QA. A Rant. - Page 4

QA. A Rant.

Discuss the source code and development of Spring Engine in general from a technical point of view. Patches go here too.

Moderator: Moderators

User avatar
Argh
Posts: 10920
Joined: 21 Feb 2005, 03:38

Re: QA. A Rant.

Post by Argh »

hmmm Id think multitexturing would be a little dubious at best, cant you just rerender the unit but with the second/third/ etc textures? It'd be more robust/flexible in terms of graphics card support
Multitexturing's cheaper per render pass, on this engine. Geometry and setup becomes a major drag, due to duplication (more on this further down). And of course, it's all in-hand with the approach I'm building, where textures for maps are part of a big tileset collection- instead of having, say, 50MB for a single SMF, I can spend 200MB on tilesets for 40 maps...

Besides which, it's what Blizzard's doing, except they're smarter than I am and they've figured out a cute way to rebuild the normalmaps from a two-channel solution (which, if I were smart enough to do, I'd have another channel of data available for all sorts of fun tricks) and I am almost entirely certain that they're using a vertex shader to displace the map grid. One of the things I've found out during all this is that map geometry load isn't a major big deal- we can go up to about 90K triangles before it really starts to hurt. 90K triangles, FYI, is a ridiculous amount of real detail, if you use an optimized mesh approach... but because the shadowmaps still don't handle large triangles correctly (distortion is a major issue) I've been forced to use next-best, which is a uniform mesh that has had some tessellation applied- I'll show a screenshot of the latest, compare it with the ones above, the difference is pretty obvious. This is an OK solution, but then I hit resolution problems, in terms of mismatch between heightmap and geometry, but I can get pretty close by forcing the heightmap downwards, I think. Haven't had time to add that last little tweak.

Anyhow, long story short, multitexture's the way Blizzard is doing it; they were doing it with the Warcraft 3 engine, they've refined it for SC2, but it's basically the same stuff but with extra fanciness (normalmaps and reflectivity, dunno whether they bothered with glow, but glow is very very cheap).



Anyhow, in terms of Spring, there are other issues that make a multi-pass concept very un-attractive; you'd need to use gl.Unit, and there isn't a normalmap shader for gl.Unit passes, and I'm not sure what I'd need to change in terms of the matrix side to make it happen. I presume that's easy, if you understand the matrix side of things, which I don't. But really, it's a geometry throughput problem- each Unit is multiple display lists, instead of being one display list whose vertexes are being manipulated by a vertex shader that handles the Piece conditions.

And there are other complications, in terms of the purely unitRendering stuff I am doing right now:

A. UnitRendering does not actually allow us to use one geometry load, texture load and OpenGL state setup for all three of the major passes- diffuse, reflection, shadowmap. This really sucks; amongst other things, it means that the tex1 / tex2 are being re-sent to the video card, and that there's a lot of other needless duplication going on.

B. More specifically, unitRendering, as currently set up, has some very big problems if you go outside some very limited scope.

For example, as in the screenshot below, I got translucency working in jK's shader framework; however, it costs a lot per pass because some genius set the alpha test threshold in Spring to GL_GREATER,0.5, when in fact it should be GL_GREATER,0.01, so that anything that isn't basically black is treated as opaque. So I have to invoke gl.AlphaTest(GL.GREATER,0.01) within the predl, which is quite expensive. I really wish that was fixed, it's annoying and it's utterly unjustified- it won't screw up anything that was made properly and it won't impact 3DO rendering at all.

Then there are textures. Basically, Spring's shadow and reflection passes expect tex1 / tex2, but they don't use consistent gl.Texture positions. This is causing some needlessly wasteful behaviors, and I would like to see them fixed.

What I think should happen, more or less:

1. For this pass, load Unit textures from PBO. Tex1, Tex2, Shadow and Reflection, in that order always for all Units. If additional textures are specified in the textures defined in a unitRendering setup, then they should be loaded now. IOW, do all setup one time only per num Units of type (unitDefID) on the screen or within the shadow frustrum.

2. Render Unit reflection, using unitRendering setup (it's pretty pathetic that it uses the ARB shader instead of consolidating it into two passes with no duplication of setup).

3. Render Unit diffuse (i.e., the main render) using the reflections from Now and the shadow from Last Time (the error will be very small, and for various reasons it will be cheaper this way).

4. Render Unit shadows to shadow Now. Use FBO to flip shadow Now with shadow Last Time.

Basically, I'd like to see less duplication of stuff we don't need duplicated- the biggest issue in this sense is re-sending textures multiple times when it's really not necessary, which is quite expensive. I know there's no way around some of the other setup costs, ofc, but the texture and geometry duplication is truly wasteful behavior.



Lastly, when are we going to be able to specify multiple lights in a Spring scene? I'd really prefer to do lighting in a less clunky way- it's preposterous that the engine still only supports one light that you can't change during runtime unless you implement the entire lighting from scratch per Unit, when supporting 1 global light and 15 point lights with distance culling would be cheap enough, let alone the fancier lighting systems used in modern engines :P


Anyhow... basically, things are a lot less of an orkish kludge, I am basically just needing to split up the shaders into smaller, smarter ones for specific tasks, and then I will be able to release it.

Screenshots in a minute... I want to run the demo on this hardware, get a feel for how things are working and make sure it's non-broken on both platforms.

Big screens, so that you all can get a better feel for the sheer detail you'd see at a typical play distance- there's some tiling, but frankly that's more due to me not having put enough time into the map, not the process. I wish I'd had more time to mess with it, but I've been busy with RL lately.

Funny thing, though- shadowmaps failed to work on the 9800, even though it's working at home on the ATi... probably a driver setup issue, but whatever :roll:

Click to enlarge.

Image

Image

Image

[EDIT]jK, to illustrate what I'm talking about, in terms of required context switches and other nasty things I have to do for each Unit using your framework, here's a sample preDL setup, using your framework (with my modifications) to implement translucency.

That glTexture (normalTex) is absolutely required, btw; setting it up in the unit's mat doesn't actually work correctly- I tried doing it like it says it's supposed to work, and found that textures were getting overwritten on a regular basis...

Code: Select all

	if (unitDef.customParams.normalmaps == "yes") then
            		Spring.Echo("Found Normalmap",unitDefID,unitDef.name)
		materialDefs[unitDefID] = {
      			shader = include("LuaRules/Configs/UnitShaders/normalmappedS3O.lua"),
      			usecamera   = false,
			culling     = GL.BACK,

predl= gl.CreateList(
	function()
	local glAlphaTest = gl.AlphaTest
	local glBlending = gl.Blending
	local GL_GREATER = GL.GREATER
	local GL_SRC_ALPHA = GL.SRC_ALPHA
	local GL_ONE_MINUS_SRC_ALPHA = GL.ONE_MINUS_SRC_ALPHA
	local glTexture = gl.Texture
	glAlphaTest(GL_GREATER,0.01)
	glBlending(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA)
	local normalTex = "unittextures/"..unitDef.customParams.normalmap_name
	glTexture(4,normalTex)
	end),


			texunits    = {
			[0] = '%' .. unitDefID .. ":0",
			[1] = '%' .. unitDefID .. ":1",
			[2] = {tex = "$reflection", enable = false},
			[3] = "$shadow",
				},
			}
		unitShaderDefs[unitDefID] = unitDefID
	end
Attachments
screen00047.png
(2.82 MiB) Downloaded 2 times
screen00046.png
(1.8 MiB) Downloaded 2 times
screen00045.png
(3.07 MiB) Downloaded 2 times
User avatar
AF
AI Developer
Posts: 20687
Joined: 14 Sep 2004, 11:32

Re: QA. A Rant.

Post by AF »

Pretty ^_^ I think its imperative we get a fresh set of eyes on it, or at least make its development viewable even if we cant participate, if only so we can test for hardware compatability outside of your own hardware collection

That Alpha threshold sounds like a 1 line fix, submit a diff?
User avatar
Argh
Posts: 10920
Joined: 21 Feb 2005, 03:38

Re: QA. A Rant.

Post by Argh »

I agree with all that, will do when I'm happier with all of it and have written some docs. It's pretty close to being what it needs to be, for my purposes. Probably Monday / Tuesday; I'm rather busy this weekend.

Whether it becomes something more generally useful largely depends on people with better 3D skills. I'll be happy to share it out to get some beta-testing results and see whether people would be willing to look at the things it would take to make it a lot more useful.

And yes, I'll be happy to hunt down the line, but frankly, the process of rendering a Unit needs a much larger overhaul- the more I see how unitRendering was done, the less happy I am about the kludges that were used.
User avatar
Argh
Posts: 10920
Joined: 21 Feb 2005, 03:38

Re: QA. A Rant.

Post by Argh »

Still busy. It's been a rather awful week IRL, but it's improving. I got LUPS happy with it, other than gl.GroundQuad (which causes Spring to crash if map rendering isn't on) including distortions on ATi (there are some issues remaining, but it's a lot closer to functional than it was). Still haven't had time to write the special POPS variant to replace GroundQuad behaviors, but I have a plan. Maybe tomorrow, maybe Thursday.
User avatar
Cheesecan
Posts: 1571
Joined: 07 Feb 2005, 21:30

Re: QA. A Rant.

Post by Cheesecan »

Megatexture-based optimizations possible..frustum culling, mipmaps, vbos. That's about it.
Sm3 optimizations..plentiful.
So go SM3.
User avatar
Argh
Posts: 10920
Joined: 21 Feb 2005, 03:38

Re: QA. A Rant.

Post by Argh »

You aren't seeing a megatexture there, and it doesn't operate anything like SM3.

It's really annoying that I was almost ready to show it and I not only ended up dealing with a bunch of RL crap but my damn mobo died last Thursday- I was hoping to present this on Monday :P
User avatar
AF
AI Developer
Posts: 20687
Joined: 14 Sep 2004, 11:32

Re: QA. A Rant.

Post by AF »

I'd still very much like to see it, near completion or completed, you've played a dangerous game doing it all behind closed doors in that one of us could have pointed out a potentially major future pitfall at the start of the project, only because you've already built it, fixing the problem is now an order of magnitude greater in cost. That and we could have pointed out simple architectural improvements that increased performance significantly that are again much harder to implement at this stage because your.

What you've implemented is of great interest to some of us, and it could be magnificent, or it could be meh. We'd prefer to know sooner rather than later if we need to help.

That and Im sure it'd help a lot if you documented extensively, something like this explained in detail with lots of technical explanations of code inline, would be very useful for people at university doing first degrees learning graphics programming, or hobbyists looking to get into the field. Its a damn site better than being intimidated by Maya and trying to model a landscape from scratch, or using tiles or giant images on a heightmap.
User avatar
Argh
Posts: 10920
Joined: 21 Feb 2005, 03:38

Re: QA. A Rant.

Post by Argh »

Don't worry, I fully intend for it to be in public asap, AF. It's like POPS- it needs testing and I think it's important that everybody can verify the accuracy of my findings and see that it is what I said it was.

Unfortunately, there is nothing practical I can do until the mobo arrives :| The code's on one HD, the last built code that's outside that HD is incomplete enough that I'd have to start over again, at least on the major part of it. I thought about shifting the HD over to another box, but with my luck, the HD would die and it would get lost until I managed to re-write it... and it's somewhere around 5K of new code and 3 weeks of blue-sky dev time, which, as most of you know, I hate doing.

The annoying thing is that I was almost done. I wanted to build a nicer demo, but the time requirements kept cropping up and my schedule didn't permit- tech-wise, other than minor crap like the refactor I did for certain POPS events and some shader variants to show people things, it's been done for almost two weeks now.

In terms of "major impact"... well, IDK about all that- it's more of a working concept worth further thought atm, and in my case, I'll probably deploy it, simply because I've already burned another month on blue-sky stuff and I need to get back to my real job, i.e., making content and gameplay. I only took it to the end because the results of Beherith's tests told me that I simply wasn't going to find a way out of the hole with SMF- IDK what's wrong with it; heck, I haven't even tested a new buildbot build in a few weeks, so maybe Beherith or Kloot found the issue and killed it (which would be wonderful, if for no other reason than I would not have to re-create so much content to be back at Square One), since whatever it is, it's relatively new.

It certainly isn't a panacea atm; I can use it, I will probably deploy it, but largely because of time considerations, not because I think it's what it should be. But, as I said before... even if I gave out detailed instructions, I don't think that it will be easy to replicate the content, because of how I had to go about it.

It is not the way that this should be done- what should be done (imo) is to build multiple display list 'sectors' at true resolution and at 2, 4, 8 and 16X LOD (i.e. 1:16 reduction in mesh resolution) or even greater (even 16X is probably excessive at tab-view levels), calculate their TBN for the facets and then use a fairly simple viewtest to determine which sectors are in view and what LOD to use (average view would probably be 2 to 4X, which is about 8X better resolution than with SMF short of cranking it to stuttering point, IIRC, but it would perform quite adequately. Shadowmaps for terrain could (optimally) be burned into map data (stored in a file) or (very much sub-optimally) be created using LOD16 for sectors outside the frustrum but within the critical distance (i.e., so that mountains cast shadows correctly under most circumstances but aren't relying on brute force), greatly changing the shadowmap geometry load (which is one of the larger costs in this prototype).

I thought about demonstrating the basics of this idea in operation, at least in a crappy, Orkish fashion, but again <sigh> it was last on the list, after getting the ground POPS working, and then kablooie...

A vertex shader could handle heightmap changes for both processes, although there will be a fairly tricky issue in terms of handling vert splits, based on my initial tests; I suspect that can only be handled by a different tristrip design (the commercial engines use tristrips that alternate and sometimes form hexagons except at the borders; I suspect that it is largely to avoid this issue by making sure that there are never any triangles with verts that fall on a uv coordinate / texel boundary).

But the technology of the shaders isn't amazing; it's just stock logic that's been extended a bit, and some tweaks I did to build a fast cheating approximation of light to get a better final result, like I did with Kloot's, but it's not like jK/Kloot won't recognize where it came from or how it works. It's just a normalmap shader with a straightforward series combiner to keep it reasonably efficient, basically, and it's probably not 100% efficient as designed, but it's reasonably close. While it's pretty, it's not all that important.

The important stuff (at least from my POV) is that the technology demonstrates a lot of things I've said here in the past were quite correct; we can have a working SM3-like design; the splat renderer I outlined at one point still looks fairly attractive, although I think the sector approach will be far easier to get working... and potentially we could go really hardcore and use the ideas behind this and the ground POPS (which is the last thing that was on my must-do punchout list, after taking Tobi's advice to heart and saving some cycles for certain types of POPS burst events, making it even more ridiculously fast) and have maps that support detail objects as well as splats, doing the details in a secondary pass to a FBO and post-processing them into the final result...

Anyhow, enough speculative stuff.

It will be released, albeit not quite as nicely-cute as I would like, as soon as I get access to it again.
User avatar
AF
AI Developer
Posts: 20687
Joined: 14 Sep 2004, 11:32

Re: QA. A Rant.

Post by AF »

Really, if you'd just don't this in a github repo then none of these issues would be present, and you'd have an absurd number of backups and saved a lot of time. I reccomend it be the first thing you do once you have access to the code again, if only for the benefits of a version control system, and the branching support you'd get, let alone the savings in time and effort it brings
User avatar
Argh
Posts: 10920
Joined: 21 Feb 2005, 03:38

Re: QA. A Rant.

Post by Argh »

Eh, this is the first time I've lost the dev box, other than taking it down for upgrades, AF. This is mainly inconvenient because of timing. And if the code was on github, I'd still be stuck with a netbook that can't even run Spring, anyhow. I'm pretty confident that the data is fine.

If I ever run another team project distributed over the world, something like that makes plenty of sense. It's a lot less of a pain than using email, forums and FTP, which is how I used to do it. Right now, it'd just be another thing to maintain, with dubious benefits for all involved.
User avatar
AF
AI Developer
Posts: 20687
Joined: 14 Sep 2004, 11:32

Re: QA. A Rant.

Post by AF »

Not at all, we can run spring fine, and there'd be nothing stopping us forking your code and finishing it ourselves, or even adding new things that never occurred to you so all you have to do then is merge and hey presto we all have it.

Version control has uses outside of team coding efforts, and they are very real benefits that are worth more than their weight in gold.
User avatar
Forboding Angel
Evolution RTS Developer
Posts: 14673
Joined: 17 Nov 2005, 02:43

Re: QA. A Rant.

Post by Forboding Angel »

Yes, anyone not using svn (or git, but git fails hard on winbloze) for pretty much anything related to work in spring, deserves to be slapped repeatedly.
User avatar
AF
AI Developer
Posts: 20687
Joined: 14 Sep 2004, 11:32

Re: QA. A Rant.

Post by AF »

Git actually works quite well, the main problems arise from a culture shock moving from svn to git and expecting a 1:1 mapping of procedures.
User avatar
FLOZi
MC: Legacy & Spring 1944 Developer
Posts: 6241
Joined: 29 Apr 2005, 01:14

Re: QA. A Rant.

Post by FLOZi »

AF wrote:Git actually works quite well, the main problems arise from a culture shock moving from svn to git and expecting a 1:1 mapping of procedures.
msysgit simply cannot be said to 'work quite well', at least as far as my experience goes. Yes git-gui is not as comfortable as tortoisesvn but that is reasonably irrelevant.
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: QA. A Rant.

Post by hoijui »

there is only this one problem with line-endings under windows. there is no general, perfect solution for it that i know of.
the work-arounds i know of (you may not want to use --global):

Code: Select all

git config --global core.autocrlf input
this personally never worked for me, but some (eg. BrainDamage) said it works for them. if it does not, use:

Code: Select all

git config --global core.autocrlf false
with this, you have to make sure to "manually" convert line-endings (in springs case, to "\n") before committing, eg. with dos2unix.exe.

i used SmartSVN, but never missed it even a tiny bit after switching to git. it is a shame that some seemingly simple problem like line-endings makes git so unpopular for such a long time already. There should be windows builds/utils that enforce unix line endings in all non-binary files or something. :D
User avatar
AF
AI Developer
Posts: 20687
Joined: 14 Sep 2004, 11:32

Re: QA. A Rant.

Post by AF »

Git GUI does me fine, I'm not exactly doing gymnastics with my repos and neither are you guys using SVN with yours. Newer versions of GitGUI have explorer context menus for more things, and tortoiseGit seems to be coming along well.
User avatar
FLOZi
MC: Legacy & Spring 1944 Developer
Posts: 6241
Joined: 29 Apr 2005, 01:14

Re: QA. A Rant.

Post by FLOZi »

It's not the line endings that I'm talking about either, though that is a nuisance.
User avatar
Argh
Posts: 10920
Joined: 21 Feb 2005, 03:38

Re: QA. A Rant.

Post by Argh »

Well you're certainly welcome to put it all in a repo if you want.

Anyways, got mobo Friday, installed it last night, will be able to post it tonight, I think.
User avatar
Argh
Posts: 10920
Joined: 21 Feb 2005, 03:38

Re: QA. A Rant.

Post by Argh »

Getting closer with the basic demo:

Image

The fake-volumetric smoke isn't perfect, but what the hell, it's that, about 64K POPS snowflakes, volume fog, etc., and as you can see, I got translucency working well again.

Still not happy with this, but I think a lot of it's just the map, which I haven't had time to mess with further. It'll probably have to do for an initial release.
User avatar
Johannes
Posts: 1265
Joined: 17 Sep 2010, 15:49

Re: QA. A Rant.

Post by Johannes »

Why haven't you made a proper thread for these, makes 0 sense to post it in here
Post Reply

Return to “Engine”