TODO stuff... very long

[Argh]

I read through the TODO list thread. My impression is that basically people are talking about mechanics too much- the main issue is that we don't seem to have targets for the next release, or the slightest clue about a release date. Release Often, Release Early seems to have wandered off and died again.

So, meh. I know I will get shouted at, but hey, I decided it was time for another one of my Walls of Doom... er, text.

Here is what I think should be prioritized:

1. A clear idea of when we're supposed to see stable, with the current featureset.

2. A roll-call about what (roughly) you Devs want to get done by then.

3. Settle on how you're going to track it.

My vote is for a public forum thread here, so that we have something like vague tracking. I have nothing against using Mantis or w/e for bug-hunts. But we have a lot of non-bug-hunt stuff that is in various states, and I am concerned about this... at least concerned enough to churn out this giant Thing which I hope will stir up some dialogue.




The engine's at a fairly important set of tipping-points here; there are a lot of important projects that are in various stages of completion. I would like to take this opportunity to talk about what I see. I heard and understand Auschwabar's point about feeling like Spring has made that transition from Game to Engine. I am sad to read that he feels like this transition left him with nothing to do.

If I'd known he felt that way earlier, I would have at least tried to brainstorm about this. So I apologize for not spending any time on this, because this may be a big problem in terms of motivation. Spring-as-engine is an abstract concept, but at the same time, that huge work has made everything up to this point possible, and I think it's fair to say that nobody's game is using all of Spring's potential, or even close atm- this has been true for well over a year now, and it's a big deal. I know that coder burnout is inevitable with this thing, but we're not picking up a lot of new people in the areas we really need (imo!), either, so I'm concerned on that front as well.


For myself, I've been pretty un-motivated lately to work on P.U.R.E., other than some bug-fixing, because so much of what I want is in limbo atm and I don't know where various balls are going to drop. So while this long rant may be upsetting to some, please try not to take it that way- I am using features that aren't officially supported yet, I am pretty excited about some of the stuff I've been doing that I haven't shown a lot of yet, and I have been following what's been going on in Git. So I am mainly wondering what I need to do here in terms of spending time, and I want to hear whether certain things are going one way or another.



A. There's jK's 'universal' shader system, which I put some work into, but I let it go because I can't port Kloot's shader entire, and it's so much better than anything else we have, it's a big deal to me. So that major thing is just sitting there, unfinished, even though it's pretty important.

B. There's Kloot's work on taking Unit rendering and abstracting the model format. Currently useless, because there is no firm tie to any existing editing environment and the model format chosen doesn't support origins, so there's a lot of stuff missing atm.

My vote is that for this release, it's cut to my main request, which is that we need a generic geometry handler that Lua can use to get a display list, very badly. Our ability to do special FX would take a quantum leap forwards if this were true. I understand the end-goal here- this is either intended as a final run to moving to deprecate the S3O format or at least a practice run at same. That said, I feel like there is far too much work remaining on this area, and I would like to see the goal set to a reasonable mark for the release.

C. There is the new map format, SSMF. This is largely functional per specification. My concerns are that: 1. I have offered to spend significant time on the shader side, if minor changes are made (sending a uniform for shadows on/off, sending the reflection cube) and the demo code I released is already easier to work with than Beherith's first code, and I am sure it can be further improved and documented. I want to help here, but ever since I released that first bit of code and Behirith said, "yeah, that's easier to control" and I said, "hey I see more possibilities" I have gotten zero feedback at all, and I am not motivated to work on something in a vacuum, because I am not sure about the outcome here- if I work on reflections, for example, and no code shows up, then I will have to ship the next version of P.U.R.E. with a non-standard shader build, when I'd rather put the work in for everybody at once and work with Behirith and other interested parties to build a good standard and some docs, rather than mystery ware nobody else uses.

If this means I have to submit the modified .cpp for this, since it's two lines of code, maybe three, fine, but I feel like this is stalled and it should not be- SSMF is the single biggest improvement on the rendering side atm.

D. There is a lot broken on ATi. The major stuff in my bug thread is getting dealt with, especially that gl.Unit bug seems to be fixed atm (I will be watching and testing that one a lot, though, it made P.U.R.E. unplayable).

One thing that I feel should be said is that I have read more things and played a few modern games lately. Most of them are using dual-pass shadowmaps- one is a static, low-rez shadowmap that is burned at runtime if not in the mapping engine, and they are not rendering shadows as part of the unit geometry pass, but are doing a second pass only for shadows- this I suspect is to facilitate deferred rendering of the shadowmap.

I realize that every time I mention this, I piss people off. I know nobody wants to mess with it. I still feel like it needs to be rebuilt from scratch, with a modern approach... or at the very least, we need to find the collective will to do something about the ATi problem. It is not the shader- I am 100% certain of that. I exchanged some PMs about this with Kloot- I can see the "crude" shadows (i.e., no alpha trans) for Features, due to one section of code that is missing for Units. And map shadows are fine.

Kloot is very frustrated with this issue, or at the very least for me pushing relentlessly on this for over a year now, and says that there's something borked with ATi's drivers that is causing the problem, and the code should run as written. So we're at an impasse, now that I am getting to see all these dreaded "ATi problems" up close.

What I can tell you guys for certain is that I can run other OpenGL games that use shadowmaps OK. And that there may be something critical going on in terms of setup missing. I have found ATi's drivers to be a lot more picky about this over here (I've had to fix a few things, like most of the LUPS shaders, some issues with P.O.P.S. and other OpenGL work, and I'm still hunting down one last LUPS issue that isn't showing up here but shows up on certain other hardware). The issue of the texture not being cleared didn't fix it (but, hmm, I was right, there was an issue there and it apparently fixed lines).

So, here's a thought. Why not just trash it in favor of source we know works, that is also GPL?

If I sit down with Quake source and look at it, can we maybe look at its shadowmap code? All shadowmaps are drawn in very similar ways- you draw geometry from the POV of the lightsource, set up frustrum culling, yada yada. Other than Quake source geometry type differences, for which we can substitute drawUnit calls like the current source, surely it's practical, and we know that code works on this hardware, and a shadowmap is a shadowmap.

All I'm asking for is that I grab the code, and maybe we can all pull together on this. I don't think I'm quite up to doing this alone and submitting a patch, but I am willing to try, if people here will be a little less resistant.

This is one of the stupid things that drives me nuts about this engine- we have all this amazingly-advanced engine stuff, we finally have figured out a pretty decent workaround for the future of maps (shaders can be pushed in various directions without needing a new map format; it may be possible to write a splat-like behavior using special-case stuff for slopes with only shaders).

Yet we have basic stuff that is badly broken. We have water3, which is totally broken on all hardware I have tested with in the last 6 months. We have water4, which I can't even get one of you to write a GLSL dump for so that I can debug for ATi and figure out why it's not compiling. And we have shadows that don't work on a significant fraction of hardware- a very basic thing in any modern game engine.

So, maybe it's time to assume the current code is unfixable and start anew.

Six years in, and the shadowmaps are still mainly using SJ code with his famous statement that maybe he'd figure out how he made it work someday, and aren't anywhere near the current state of the art. Many have tried to fix it- IDK how many collective coder-hours have gone down this rat-hole, but quite a lot, and we're still not there.

But seeing a gorgeous SSMF map with working shaders and no shadows on this hardware is like going on a date with a gorgeous blonde only to discover that she lacks front teeth. Accurate lighting models that include shadows are such a fundamental part of visuals that it's a huge issue, and I finally got to see it all working at a reasonable level on nVidia hardware only to see that it's not happening on ATi.

This may not be possible within the period described under point 1. My call then is to move that brief chunk of code so that Units cast crude shadowmaps under ATi, instead of causing some bizarre distortions (which is why I still lean towards the setup issue, now that I've done a little more looking). That's a copy-pasta, and Kloot knows exactly what code I'm talking about. It's better than nothing.

E. My understanding is that the C++ AIs are going bye-bye in the next release. That is probably wrong; what I'd like to know, since I don't follow the AI dev threads, is where everybody envisions this going now, and to what extent we're in the middle of hold-everything in terms of release. The issues with AI code screwing up the Master is very troublesome to me, but I presume this means that major things have been afoot, in a good sense, that justify the problems we've seen.

F. Where is MT at, atm? I've seen zerver commits suggesting that he's trying to get it rolling again, but I'd be totally kidding myself if I had the foggiest idea what that means, in terms of timing. One of the other, more serious questions that arises is... when is Spring going to be able to use a multi-CPU approach, and, at least under Windows, have the OS changes that seem to facilitate MT as a hidden-hand process (even on XP, a lot of newer apps are significantly faster with this quad-core than the single-core clock would suggest, especially DirectX games) has made this either redundant or into a specific Windows-only tweak that may or may not be Linux-friendly in terms of sync? This goes into all sorts of nasty philosophical waters, and I am not advocating a Windows-centric concept here, I just want to hear what people think about this.

G. What is jK working on these days, besides getting through schoolwork? If you can just scrape up the time to give me that debug info about water4, jK... let alone the 'universal' shader system... which mainly just needs somebody who understands matrixes to fix what is probably a couple of lines of GLSL... since I pretty much got it ready otherwise, Lua-side, and wanted to document / deploy it nearly three months ago.

H. Anything else not on this list? What happened to SPliFF's project? Did it sink, or is it dog-paddling in shark-free waters atm? Are we ever going to see the issues with Mission Builder resolved (namely, the links to an old unitsync that make it broken with newer Spring, when frankly we could just use a text file... please?), Licho, or is that totally dead because nobody will maintain it? I feel like having a Spring Engine without a standard Mission structural tool is a major problem, especially after I got to play with Mount and Blade's level editor and saw that they have had the same issues I have- they have a high-level scripting language, of sorts, and a level-editor that does the visual shit, like I do... and have had to hand-code reams of Quests, etc., thousands of lines of code... and their codebase has become very difficult for them to debug (seriously, there were a lot of scripting bugs in Warband involving their weird pseudo-Python-thing that escaped QA, some of them serious, it's a little scary how much better our Lua callin / callout system is).

[Beherith]

c, SSMF: I requested DIFF format from you, to see what you changed.

I only noticed 1 formatting change, which you called "making it easier to work with" (which I am doubtful if the compiler will optimize properly, but i have to get some Cg compiling and assembly instruction counting done for that)

1 functional change, multiplication instead of addition for splatting. Im investigating that one, and working with Artturi to see if mult or add is better in this case.

1 removal of shadows. Why would I or anyone want that? I have nVidia, cant test with ATI since I dont have one.

For alot of the rest of your points:

I use visual studio 8 (2005, not express edition), but zerver says its just as good with 9 (2008)

You need to get latest source:
http://github.com/spring/spring/zipball/master
Unzip it.

Depending on vs 8 or vs9, get the corresponding vclibs package:
http://github.com/spring/spring/downloads
Unzip it so that you have the following dirs:
ai
doc
game
installer
rts
tools
vclibs

Then go to rts/build, and select the vs8 or v9 solution, then compile!

After compiling, you will need to move over base content and dll's from a recent install (for base files its imperative you get latest from buildbot ( can be found in /j #buildserv)

Etc: dont bother optimizing trees, I got them done and gave them to artturi.

Edit: Afaik MT is fine. I use it and it still rocks.

Edit2: If you own an ATI, I can build you a windows exe with bumpwater shader info dumping.

[AF]

Bug trackers

In my mantis install I have issues flagged as research, feature requests, suggestions, discussions, or even plain notes. Theyre not just for bugs and crashes.

Features that need finishing blocking P.U.R.E

I think your priorities arent quite right. Specular on the map or a fancy shader isn't going to make me buy a game. I'd like it but most would not appreciate it beyond an 'oooo this version looks prettier'.

Sometimes you have to cut noncritical things in order to ship, and fancy shaders are not the number one problem you face by far. Cut them and add them back in when its feasible.

Readability

Argh, readability is a major issue, I would seriously consider the following:

• Proof reading
• PHPBB formatting. The tools are there, and your ignoring them, lists alone would be of enormous help
• If your post takes longer than a minute to read, a writting plan would help.

As the length of your post rises, the less data you get across. People start to skim read because of the huge cost of reading, versus the percieved benefits, and all these details your verbosity is attempting to preserve, are being deliberatly skipped in the name of time and effort.

[Argh]

c, SSMF: I requested DIFF format from you, to see what you changed.

I'm not using SVN atm; is there another tool I can use that will do a DIFF in the appropriate format? What I changed was about 3 lines, IIRC, but it was really meant to be a, y'know, discussion about final format, specific goals (like, adding glow and reflec and maybe an optional animated offset for simplistic "streams","lava", etc.) and making it EZ to get the right results.

1 removal of shadows. Why would I or anyone want that? I have nVidia, cant test with ATI since I dont have one.

I removed it because right now, if I don't set shadows 0, rendering errors show up related to Units and Features, and the GLSL totally failed otherwise- try your shader and set shadows 0.

There is no shadows 3, that just renders map shadows to the shadowmap. So, on my hardware... it's all or nothing, and all I am asking for is that a boolean be passed to the shader via a uniform saying whether shadows are on/off, like every other major unit / water / everything shader in the engine. Please?

Edit: Afaik MT is fine. I use it and it still rocks.

Current Master? If so, IR Confuzed, as zerver led me to believe that it is not functional atm, and is catching up with the various things that have been done.

Edit2: If you own an ATI, I can build you a windows exe with bumpwater shader info dumping.

I do, and this about the zillionth time I've said it... I bought it very specifically so that I could tackle the GL issues with this engine and ATi, in fact, and I would appreciate some supportiveness.

[Beherith]

MT: Oh, I meant current release. But I have full faith in zerver that he will be able to continue to provide those of us with less fortunate cpus the awesome MT builds that he usually makes. He releases a while after full, and that's even better, because then I can stress test MT overnight with compatible replays generated while playing non mt.

SSMF reverts to ARB if shadows=0.

Please try to compile spring in the few easy steps I outlined above.

[Argh]

SSMF reverts to ARB if shadows=0.

The last time I tested, this did not happen when SSMF was enabled (or ARB is borked), so I will test again.

Who knows, that may have been other issues. There have been other things broken left and right.

I will set up compiling again.