What is to stop engine devs ignoring the output of the tests?

how to stop game devs ignoring the changelog and not testing release candidates?

with regression tests, you can see when something broke. this will help the game AND engine dev to fix stuff. currently most things break unnoticed, most bugs are detected after a release. regression testing allows to detect breakages before a release, close to the time a commit is made... sorry i don't fully understand your question, i hope this answered it.

we could also skip all tests and just release?!

A man to my hearts desire! Lets just see how flameable, this little world on the edget to entropy really is... i get the gasoline.

I think tests should be on single map which has all sort of terrain, there should be some LUA way to make test like record button, you record and also ensure that everything is like it should be then issued comands are in some file which headless server will run. E.g give this here, give that here and do barrel roll?: passed failed warning.

On recording part there should be time flag for example reclaim some trees and metal field, if time mismatches it issues warning if it timeouts after preset reasonable timeout it marks it as broken.

Also there should be recorded each game frame where and when unit is. If not in same place at same time as it was with last test of game or engine it gives warning only.
PS. I dont know how random here works(if it does anything here :) ) and will unit move same way as in test when same orders to same units are given at different ingame clock.

For ultimate satisfaction those tests should be run and rendered(this could be done only on first test and only if test gives warning or error) and uploaded to youtube, but maybe demos are enough.


For engine test you run it with same mod version all the time.

that can be added later if needed. such a tool won't allow to test, if some unitdef tag exists and contains some specific value. also, it seems like you would have to "record" for each single unit, reusing it for other units seems to be not possible (when it should record frame + position).

imo it would make sense to store the output of test and show the diffs to the previous run.




in general, the idea isn't wrong, but each frame check the unit position wouldn't allow to change anything in the path finder. You can add such a test, if the pathfinder is perfect, but i guess a pathfinder + unitmovement can't be perfect.





buildslaves have no decent gpu, that won't work currently, also i don't see the benefits, if you watch youtube videos, you can also run the test yourself.

Benefit of those videos is that you dont have to compile(even have) each spring version(assuming some development versions), it is fast and easy way to compare, but yeah thats linux only problem I guess.

hm, for screen comparison i would personally prefer screenshots then, but uhm, as we can't do that currently... i don't care :)

*bump*


@game-devs:

can you at least use Spring.Log('filename.lua', LOG.ERROR, "There happened an error") in your gadget/widget handler to print fatal errors?

this way http://buildbot.springrts.com/builders/ ... logs/stdio :




would be in red and much more easily to find...

see also http://code.google.com/p/zero-k/issues/detail?id=1469


there is still not a single automated run test... ai vs ai is mostly crash testing... :-/

I take it the base content handlers use Log? S44 uses base content handlers (as does BTL, fwiw)

oops. sorry for the delay. yep base-handlers uses Log.