Automated Release Test

[Google_Frog]

What is to stop engine devs ignoring the output of the tests?

[abma]

What is to stop engine devs ignoring the output of the tests?

how to stop game devs ignoring the changelog and not testing release candidates?

with regression tests, you can see when something broke. this will help the game AND engine dev to fix stuff. currently most things break unnoticed, most bugs are detected after a release. regression testing allows to detect breakages before a release, close to the time a commit is made... sorry i don't fully understand your question, i hope this answered it.

we could also skip all tests and just release?!

[PicassoCT]

we could also skip all tests and just release?!

A man to my hearts desire! Lets just see how flameable, this little world on the edget to entropy really is... i get the gasoline.

[wolas]

I think tests should be on single map which has all sort of terrain, there should be some LUA way to make test like record button, you record and also ensure that everything is like it should be then issued comands are in some file which headless server will run. E.g give this here, give that here and do barrel roll?: passed failed warning.

On recording part there should be time flag for example reclaim some trees and metal field, if time mismatches it issues warning if it timeouts after preset reasonable timeout it marks it as broken.

Also there should be recorded each game frame where and when unit is. If not in same place at same time as it was with last test of game or engine it gives warning only.
PS. I dont know how random here works(if it does anything here :) ) and will unit move same way as in test when same orders to same units are given at different ingame clock.

For ultimate satisfaction those tests should be run and rendered(this could be done only on first test and only if test gives warning or error) and uploaded to youtube, but maybe demos are enough.


For engine test you run it with same mod version all the time.

[abma]

I think tests should be on single map which has all sort of terrain, there should be some LUA way to make test like record button, you record and also ensure that everything is like it should be then issued comands are in some file which headless server will run. E.g give this here, give that here and do barrel roll?: passed failed warning.

On recording part there should be time flag for example reclaim some trees and metal field, if time mismatches it issues warning if it timeouts after preset reasonable timeout it marks it as broken.

that can be added later if needed. such a tool won't allow to test, if some unitdef tag exists and contains some specific value. also, it seems like you would have to "record" for each single unit, reusing it for other units seems to be not possible (when it should record frame + position).

imo it would make sense to store the output of test and show the diffs to the previous run.

Also there should be recorded each game frame where and when unit is. If not in same place at same time as it was with last test of game or engine it gives warning only.
PS. I dont know how random here works(if it does anything here :) ) and will unit move same way as in test when same orders to same units are given at different ingame clock.

in general, the idea isn't wrong, but each frame check the unit position wouldn't allow to change anything in the path finder. You can add such a test, if the pathfinder is perfect, but i guess a pathfinder + unitmovement can't be perfect.

For ultimate satisfaction those tests should be run and rendered(this could be done only on first test and only if test gives warning or error) and uploaded to youtube, but maybe demos are enough.

buildslaves have no decent gpu, that won't work currently, also i don't see the benefits, if you watch youtube videos, you can also run the test yourself.

[wolas]

Benefit of those videos is that you dont have to compile(even have) each spring version(assuming some development versions), it is fast and easy way to compare, but yeah thats linux only problem I guess.

[abma]

hm, for screen comparison i would personally prefer screenshots then, but uhm, as we can't do that currently... i don't care :)

[abma]

*bump*


@game-devs:

can you at least use Spring.Log('filename.lua', LOG.ERROR, "There happened an error") in your gadget/widget handler to print fatal errors?

this way http://buildbot.springrts.com/builders/ ... logs/stdio :

Code: Select all

[f=0000000] WAAA, stack traceback:
	[string "LuaUI/Widgets/chili/Handlers/debughandler.l..."]:30: in function <[string "LuaUI/Widgets/chili/Handlers/debughandler.l..."]:26>
	[string "LuaUI/Widgets/chili/Handlers/debughandler.l..."]:363: in function 'OriginalRemoveWidget'

would be in red and much more easily to find...

see also http://code.google.com/p/zero-k/issues/detail?id=1469


there is still not a single automated run test... ai vs ai is mostly crash testing... :-/

[FLOZi]

I take it the base content handlers use Log? S44 uses base content handlers (as does BTL, fwiw)

[abma]

I take it the base content handlers use Log? S44 uses base content handlers (as does BTL, fwiw)

oops. sorry for the delay. yep base-handlers uses Log.

[abma]

wow, took a while, but i've created a small game which runs some tests:

https://github.com/spring/devgame.sdd

to create a test, just create a gadget and put it here:

https://github.com/spring/devgame.sdd/tree/master/tests

when the test run is over, just call gadget:TestDone(true, "ok")

example:
https://github.com/spring/devgame.sdd/b ... ittest.lua

(regression test for #4942)


the game is on rapid (devgame:test) and the most recent commit is used in the validation tests. there is a lot of space for improvements but its hopefully a good start. atm it desyncs as it uses Spring.SetConfig* / Spring.GetConfig*... but that doesn't really matter atm.


imo this test is meant for testing the simulation: stuff like Set... and then Get... to check if values are ok.

[gajop]

imo this test is meant for testing the simulation: stuff like Set... and then Get... to check if values are ok.

I have this little serialization library thing in WIP state: https://github.com/gajop/s11n, it could be used to easily test Set and Get functions on units and features.

It's currently configured to only save values that are different from default (default being the ones that are present when the unit is created), but that can be changed.

[PicassoCT]

What is to stop engine devs ignoring the output of the tests?

how to stop game devs ignoring the changelog and not testing release candidates?

Reduce the hazzle?

Download RC
Install RC
Crash
Report Bug
Reinstall old Working Version

alternative:
Silent dl RC
Offer Test
If Test fails, silently rollback and restart with the saved game settings.
Autoreport to mantis.

To long, did not load