QA. A Rant.

[Argh]

[RANT MODE: ON]

Look, I'm going to be very blunt. I got done looking at the latest portable, and I'm pretty upset.

QA for this engine needs to use a better process. Since you guys have abandoned Release Early, Release Often, the quality has actually fallen.

For example, here are 4 major issues with the current public build:

1. Shadowmaps. They're still broken on more than half the hardware that can run Spring.

Shadows are a basic feature on any engine worth giving the name to, these days, and yet the engine can't even do simple shadows from a single light source.

And shadows are such an important part of the visual experience of a game that lacking them makes the engine look downright quaint.

Yes, I know the issue's frustrating and annoying, and I know we've looked into it. We need to look again. I would be happy to root around source from other projects and discuss this- most things I've read indicate that the best solution is a scenegraph.

This has been a vague object of discussion, and the issue seemed to get nowhere on account of developer worries about a non-issue, that of synced behavior no longer matching up with unsynced animation quite as perfectly. What do I need to show people to convince them that this is not a problem at all? Go play any FPS, and if you pay close attention, you will see that characters don't match their action states perfectly. It's not a big deal; the discontinuities are too small to notice unless you're not actually playing.

2. Pathfinding. Pathfinding changes should probably be reverted to 9 months ago, when it was working about as well as it ever has. At that point, Units could use reverse movement, two turning styles, and everything worked pretty well. The only thing that was obviously awful was local steering code, and I think most players probably wish it was the only issue atm.

I don't have a better idea than to revert it; if somebody has a plan, this would be a good time to do that 'communication with game developers' stuff that you guys can't even put on your agendas any more.

3. There is something wrong with SSMF. Right now, in tests over here, SM3 is performing better, at least on this hardware, and turning off map rendering entirely results in huge performance gains. I'm not even going to guess what is wrong, but I'm fairly certain about causes now.

4. Sound. I would like the changes to sound reverted. It's ugly and it's also fairly CPU intensive. What we had before, with Auswaschbar's additions, was stable, useful, controllable and nice. What we have now is muddy sounds.




None of this should have never passed basic QA. All of these things are important issues that make the engine substantially worse. And there haven't been major features that can distract us from this- for the vast majority of players and game developers, hardly anything new has been done.

But apparently nobody is actually doing QA since I quit, largely because I got tired of listening to people whining at me for bringing things up that needed to be said. I've decided that since I'm going to Unity after this, though, that I'll just say what needs saying, and forget about worrying about whether you like what you hear.

There needs to be a QA process. It needs to involve testing with multiple games. It should be performed by somebody who will look at the engine like a player, instead of just checking stuff off on a list.

The fact that the "release manager" can't be bothered to do that job means that we probably need a new one, or that somebody else needs veto powers over official releases so that we don't have embarrassing results like this. It's very unsettling that the engine has degraded so badly, and yet the person we've put at the helm seems to be calmly oblivious.

If I wasn't planning on leaving as soon as I can, I'd probably ask the saner developers if they would like to fork this engine, with a focus on getting all this stuff fixed instead of blathering endlessly about MT changes which should not be done until the engine is where it was 9 months ago.

And if I were Licho, I would be doing it right now, instead of hoping that things get straightened out in time for his commercial project's debut.

There is no better way to convince me and others to jump ship than to have a release like 82.6 and then make bland statements that suggest it's all fine when anybody who played games on this engine a year ago would disagree.

The fact that this stuff isn't even on your agendas is very disturbing. It's all vital, it's stuff that's been known to be broken for months or even years.

Please get focused on the right problems, or we'll eventually get an MT Spring that is at least partially broken on > 60% of all hardware, performs poorly, plays poorly, and sounds like it's been dipped in a well of deep tar, largely because the people signing off apparently are too lazy to do the most rudimentary review. And then it will be "too hard to fix", and that will become the excuse forevermore, and the engine will become even less attractive than it is right now.

[/RANT]

[bobthedinosaur]

http://www.youtube.com/watch?v=UjLO_CrZRmM#t=0m33s

[Das Bruce]

I think we should use the GPU to make spring faster.

[Argh]

<shrugs>

This is feedback, from somebody who tested the engine to death for four years while dealing with people who are resistant to hearing about things they don't want to.

When you can say the same, then you can judge me.

None of it's going to effect me personally; I've already made decisions that will make most of it moot for my game. I'm not going to discuss that beyond the demonstration shots I sent a few people, because it's not something most people will be willing to try.

But it's quite clear that things aren't right ATM. For example, go check the defaults for SpringSettings- instead of presenting the game with widescreen specs, decent resolution, and the visuals people seeing screenshots will be expecting, it's running in a fairly minimalistic mode, other than a truly insane number of nano-particles.

Stuff like this tells me that whoever gave the go-ahead and adjusted the installer probably is using those settings themselves, and has no clue how badly the engine is broken when you want it to do more than that.

In short, instead of bitching, go investigate, report, be active and useful. I'm already going to be gone; this messenger's already shot. So you're just doing yourselves a disservice, if you're not asking to be heard.

[Gota]

Pathing is a ridicules issue..basically broke almost all spring games.

[Tobi]

QA for this engine needs to use a better process. Since you guys have abandoned Release Early, Release Often, the quality has actually fallen.

Would you mind actually looking at the facts?

The last 10 releases have been in the last 4 months. In the past it used to be 1 release at best every 2-3 months.

With regards to Release Early Release Often we're probably better than ever before now

There needs to be a QA process. It needs to involve testing with multiple games. It should be performed by somebody who will look at the engine like a player, instead of just checking stuff off on a list.

This happens and it does not cover any and all combinations of unit tags, hardware, Lua APIs, etc. (and it never will)

Heck, there are already too many games to even get a single decently sized game of every game going for a single release.

We never have used a checklist for testing. (I wish we had, quality would probably have been a little bit better then, contrary to what you believe.)


Anyway I'll make it short as you leave and I don't have much time myself to improve things anyway.

What needs to happen is not that developers stop developing and only play test games (as would be required to have any significant coverage), but that the lobby server finally supports multiple engine versions so that game developers can play endless of test games, to ensure their game works fine in some new version of the engine. This reduces 99.99% of the combinations that need to be tested, making decent testing actually feasible.

Even better would be to actually rip all useless features and options out of the engine to make the remaining useful 10% of a higher quality, but I bet the community (and modders) wouldn't like this at all.

The other thing we would need is automated tests - that is the only way to guarantee testing happens in a human driven project like this (no one sane is going to do repetitive testing work for free). It's hard and boring work to create such tests though.

[Johannes]

What is the reason to not reverting pathing changes?

[quantum]

Even better would be to actually rip all useless features and options out of the engine to make the remaining useful 10% of a higher quality, but I bet the community (and modders) wouldn't like this at all.

Yes please! Workarounds and hax for quirky hard coded stuff waste more dev time than implementing them from scratch.

What needs to happen is not that developers stop developing and only play test games (as would be required to have any significant coverage), but that the lobby server finally supports multiple engine versions so that game developers can play endless of test games, to ensure their game works fine in some new version of the engine. This reduces 99.99% of the combinations that need to be tested, making decent testing actually feasible.

Truth!

[zerver]

What is the reason to not reverting pathing changes?

Interface reasons, and maybe some pride as well.

Personally I think it is OK to release some partly broken stuff, as long as the overall progress is pointing in the right direction and there is someone actively working to resolve the remaining issues.

Support for multiple engine versions would be #1 priority since this makes it easy to just ignore the new release and keep using some old version.

[smoth]

Even better would be to actually rip all useless features and options out of the engine to make the remaining useful 10% of a higher quality, but I bet the community (and modders) wouldn't like this at all.

If there is a viable and just as fast lua replacement, I am A-ok with it.

[Argh]

You don't need to run endless test games to QA.

You just need to download a few of them and run a SP scenario for a bit- fire up a map, give each side a few units, give them some orders.

I could probably write some Lua that would automatically do that part.

As for the rest of what you've said, Tobi, you've totally skipped around the entire argument.

You ship a broken engine out that doesn't do a lot of things it claims it does, people will be disappointed.

You guys have already broken backwards compatibility twice this year, which means that that vast majority of games downloads on SpringFiles are literally unplayable.

And the amount of stuff broken right now is obvious to anybody who actually pays attention to the state of the engine between releases.

You can't tell me that pathfinding doesn't suck, that shadows work on ATi, that sounds aren't obeying their parameters or that turning off map drawing doesn't make a giant difference in FPS.

So, yeah, I think you guys do need a procedure for this. Because I'd honestly prefer that work on new features is halted until all of the broken features are fixed.

[hoijui]

please make a spring branch with all the pathfinder changes reverted that you think were bad. then make a test release with this branch, and we get some mod devs and possibly users to test it. if they like it, overall, we can use that until the new path finder is ready, or any other better possibility comes around. of course, you would then have to fix all path finder issues until an other solution will get used.
of course you would have to stick to the current interface, and really only change path-finder internals.

deal?

[Gota]

players already agree unit movement was better before the recent updates.There is no question about it.

[Forboding Angel]

Ok hold on a second. First up, climb off of Tobi's nuts. It's not his fault. Furthermore, hoijui sent me a pm a couple of days ago asking me to test pathing in evo. I did, screwed around for 10 minutes with various units and it all seemed ok so I told him so. It wasn't until I actually played a MP game that I realized that something was awry.

Regardless, it isn't Tobi's fault, so stop singling him out. Furthermore, yes, in my opinion, before a release, the devs should get together with various gamedevs and have a day of testing. Generally if you can get 4 people together for a game, during the course of that game should tell you more or less what you need (3 devs + gamedev).

Evo does at least 1/4th of the weird shit that the engine is capable of, and Gundam and S44 take care of the rest, so those three right there should enable some pretty accurate testing. 1 game of eash would take, what, 2 hours or so? Evo games are designed to max out at around 30 and most games are 10 - 20 mins long so whatever. I don't really know about s44 and gundam normal game lengths but... yeah.

That's where this Dev - Gamedev communication comes in. If we had that going on, then the devs would have known that something was awry.*hint hint*

And Argh, you're doing a lot of the pot calling the kettle black. Back off.

[FireStorm_]

I don't see any big personal attacks,
but it seems to me Arch did got the attention of spring devs.

Perhaps i'm being naive, but the desired place or system to ask spring devs important things, can't it be right here right now?
Why not turn off rand-mode and turn on polite-question mode?

Pathing seems to be an an issue with more than one mod-dev. Maybe this is a chance to put your annoyances in the form of well considered question.

[Peet]

Ok hold on a second. First up, climb off of Tobi's nuts. It's not his fault.

This. Your authoritarian tone is inappropriate here, Argh. Novell does not get to boss around the linux kernel devs; the same applies to you, being in the same situation of having formed a commercial product from an open source project.

[Argh]

please make a spring branch with all the pathfinder changes reverted that you think were bad. then make a test release with this branch, and we get some mod devs and possibly users to test it. if they like it, overall, we can use that until the new path finder is ready, or any other better possibility comes around. of course, you would then have to fix all path finder issues until an other solution will get used.
of course you would have to stick to the current interface, and really only change path-finder internals.

deal?

Right now, I'm frying another fish.

I'm working on duplicating the splatmap system used by Starcraft II- i.e., a one-pass splatmap format with normalmap support for each averaged splat that can be edited in realtime.

Maps are one of the biggest remaining technical issues that I've decided I need to fix before I release P.U.R.E.- the current formats either are broken on a fair percentage of hardware, or are huge. P.U.R.E. runs to half a gig, compressed, and most of that is maps.

I've decided that instead of just talking about it and getting nowhere, I'll just do it. Same dealio as POPS, which I wrote because it's obvious that CEG is a fossil and should have been replaced some time ago. The source for POPS is already public; it was in that profile test you ignored. Feel free to read it and use it; it could be made backwards-compatible with most CEGs using dirt, CSphereParticleSpawner, etc., with a few caveats which I'd be happy to talk about and discuss workarounds for if I wasn't convinced it wasn't a waste of my time.

I have some of it working already. If I get it working as planned, I'll be happy to contribute the source.

It won't be engine source, though- I'm building it with Lua, like all of the other things I've done to work around the issues you guys refuse to fix. I just wish the four things on my list weren't completely immune to that.

As for pathfinding... we all know it's broken, this isn't even something worth having a serious discussion about. Go run a Spring build from a year ago, and it will be obvious why I'm yelling at you.

This. Your authoritarian tone is inappropriate here, Argh. Novell does not get to boss around the linux kernel devs; the same applies to you, being in the same situation of having formed a commercial product from an open source project.

1. Novell has lots of resources it can throw into the mix. I'm just one dude, and I don't know C++ backwards and forwards. This is just reality, man. Every time people yell at me to swoop in like an angel and fix it all myself, I review whether it's realistic to do that, finish my project, and keep to my schedule, and the answer is "no".

If I had a budget, and could hire people to fix things, I'd surely do that. If I had a team that could keep building the game while I got my hands deep into engine development, I'd do that. I don't get the luxury of either choice- I can either spread myself too thin to do anybody any good, or I can solve the problems that I'm capable of solving. If you think that I like that, you're wrong- I really wish I was that superhuman, but I'm not.

2. Even if I could do it, there'd be the obvious issue of whether I'd be wasting my time, because it wouldn't get used. Why contribute to a project without a roadmap that people stick to and an emphasis on quality, and where the issues of who gets to do things is determined largely arbitrarily by people who may be tempted to nix things politically? I've stuck my neck out around here many times, and put lots of time into various stuff, and the chief lesson I've learned from it is that every time I do it, my time gets wasted. There is nothing I hate more.

[AF]

please make a spring branch with all the pathfinder changes reverted that you think were bad. then make a test release with this branch, and we get some mod devs and possibly users to test it. if they like it, overall, we can use that until the new path finder is ready, or any other better possibility comes around. of course, you would then have to fix all path finder issues until an other solution will get used.
of course you would have to stick to the current interface, and really only change path-finder internals.

deal?

Right now, I'm frying another fish.

I'm working on duplicating the splatmap system used by Starcraft II- i.e., a one-pass splatmap format with normalmap support for each averaged splat that can be edited in realtime.

Maps are one of the biggest remaining technical issues that I've decided I need to fix before I release P.U.R.E.- the current formats either are broken on a fair percentage of hardware, or are huge. P.U.R.E. runs to half a gig, compressed, and most of that is maps.

They do actually work, SMF, works on all machines. Sure performance could be improved, but thats not so much a problem with the format but the code in the engine rendering it. This has been said numerous times.

I've decided that instead of just talking about it and getting nowhere, I'll just do it. Same dealio as POPS, which I wrote because it's obvious that CEG is a fossil and should have been replaced some time ago. The source for POPS is already public; it was in that profile test you ignored. Feel free to read it and use it; it could be made backwards-compatible with most CEGs using dirt, CSphereParticleSpawner, etc., with a few caveats which I'd be happy to talk about and discuss workarounds for if I wasn't convinced it wasn't a waste of my time.

Most people are now using various forks or branches of L.U.P.S, often to great effect.

I have some of it working already. If I get it working as planned, I'll be happy to contribute the source.

Better yet develop it out in the open so we can help

It won't be engine source, though- I'm building it with Lua, like all of the other things I've done to work around the issues you guys refuse to fix. I just wish the four things on my list weren't completely immune to that.

Nobody has refused to fix anything. There are a number of thigns the devs have been chasing that promise significant performance boosts, such as the numerous refactors to the pathfinder interface ( note that this is not the set of changes people are complaining about in other threads, those are different ), to the multithreading situation.

As for pathfinding... we all know it's broken, this isn't even something worth having a serious discussion about. Go run a Spring build from a year ago, and it will be obvious why I'm yelling at you.

Rose tinted glasses. Im sure you'll glorify the builds from a year ago because the problems we have now are NOW, not then. All we remember from a year ago are the times when it actually worked. It would be far more productive to look at the git commit log and say this this and this has changed, why has it changed and how does it contribute to our current mess?


There are only so many developers here, and remember, they dont do this because its their job, they do it because they enjoy it. They do what they want to do either because its necessary or because they want to, and they don't have to do any of it. Complaining that they haven't fixed long standing bug X or Y when they're busy with other things is unhelpful and unnappreciative. You would do a lot better to go and do it yourself or find someone else who will rather than complain and protest against problems you yourself are guilty of in the past, and to this day have yet to correct.

[Argh]

Better yet develop it out in the open so we can help

I'll do that when it's less WIP.

Rose tinted glasses. Im sure you'll glorify the builds from a year ago because the problems we have now are NOW, not then. All we remember from a year ago are the times when it actually worked. It would be far more productive to look at the git commit log and say this this and this has changed, why has it changed and how does it contribute to our current mess?

Again, it's obvious when things went off the rails in this regard.

9 months ago, the primary issues were local steering. The pathfinder was well-tested and very mature; the few new features (reverse movement, for example) were working, if a little imperfectly, and frankly it could have been left alone, short of something that had been tested to death.

Approximately 2 months later, changes were made to the way heatmapping was deployed, and things have gotten worse since then.

I don't know if there is any way to fix the current code without it becoming enormously slow. One thing I thought about was that each Unit could re-simulate it's previous movements and remove those results from a local heatmap copy before making steering decisions, but I rejected that because it's going to be too damn slow. I think that we need to remove heatmapping entirely and rethink the strategy, frankly.

But it's totally obvious to anybody testing their games that things are badly broken right now; Units move in jerky motions, often turn arbitrarily, and generally act like stupid zombies, instead of smoothly attempting to reach the goal. I'd rather see them do the ol' bump-and-stop clusterfuck behaviors than what they're doing now, and I really don't think I'm a lone voice on that.

[AF]

heatmapping is on/offable