Basically a bunch of virtual maschines, testing the installation and networking of the release candidate. No actuall gametesting there besides the - does it get ingame. Every lobby gets 24 hours before release a "does-run-does not run" flag.

We got automated tests for the builds (which is actually most impressive) so i assume this can be done (not for all configurations of course, but the most comon ones)

most of it is already done here:
http://buildbot.springrts.com/waterfall

(see the validation test)

i already asked guys of zero-k, if they were interested in implementing in-game tests, but they had no interest to do so. maybe i asked the wrong guy, don't know.

see http://springrts.com/wiki/Regression_Testing for my plan.



what does that mean? current lobbies don't allow to do run automatic tests, because they have no automatiation switches. afaik koshi works on that / his lobby-lib will allow to do such tests.

this is also how our current releases works: make release post, wait some time, if nobody complains: set it live.

I'll be able to automatically test unitsync usage, yes. I did already add tests in the python unitsync wrapper btw. Those just aren't plugged into anything.

We are interested in making tests, we just dont know how.

I already made an automated pathfinder test:

- Get Kernel Panic.
- Get this map
- Get this file or extract from Kernel_Panic_4.4.sd7 \Missions\KP\Script Exemple 3 - Pathfinder Test.txt
- Feed the file to Spring (as a startscript)
- Read results in infolog.

Here's what a successfully passed test should read like

Code:

[f=0000090] ÿ`ÿÿTrying to climb ramp...
[f=0000090] ÿ`ÿÿTrying to send nine atop...
[f=0000090] ÿ`ÿÿTrying to avoid wall...
[f=0000090] ÿ`ÿÿTrying to climb tower...
[f=0000633] ÿ`ÿÿSuccesfully managed to avoid wall!
[f=0001037] ÿ`ÿÿSuccesfully managed to climb ramp!
[f=0001522] ÿ`ÿÿSuccesfully managed to climb tower!
[f=0002740] ÿ`ÿÿSuccesfully managed to send nine atop!
[f=0003618] ÿ`ÿÿTime-Out!

The ÿ are just because for some reasons I wanted color in the ingame chat.

Many versions of Spring, such as 83, 84, 85, fails the last test about sending nine bits atop the tower through the twisted ramp.

Oh take a Spring version either higher or lower but don't use Spring 87, it crashes KP gadgets.

But how do you interfaace tests with testing suite? what should the test output?

Use Spring.Log() with errorlevel >= ERROR...

this will printf to stderror, all printed to stderror is seen as error / test failed.

so... some useful message would be nice.

@zwzsg:

nice! thats close to that what i wanted to do. i will soon try to run that test at the buildslave. i'm not sure if changes are needed, but its a nice start-off i think...

I don't really understand what you mean with the /runtest 1 option.

I disagree about how you said "tests shouldn't rely on a specific map":
Spawning units, make them move, build, attack, reclaim, etc... all depends on the map.

I don't use Spring.Log (is that C ?) but Lua's Spring.Echo.

I suppose it would be cleaner if my test printed a failure message instead of having to detect a failure by not having a success message before the time out.

and how do you run the test? can it be lua widgets? gadgets? how does it work?
does it update to laest mod?

Spring.* is Lua (mostly).

@zwzsg:
i should have noted that all points on the wiki are debatable, its currently just a suggestion how it could work. Yeah, it seems like it depends on the map... i just wanted to not have this dependency if possible somehow.

for Spring.Log see changelog.txt or https://github.com/spring/spring/commit ... 390db3691c

and yes, Spring.Log is lua...

@Licho:


my suggestion is the modoption "testsrun=1". it can be a widget and/or gadget that checks for this modoption. (thats also debatable...)

ok that sounds good

Ok. I was confused because you format your strings the C way. I guess I can make it use Spring.Log instead of Spring.Echo.

However I think the test should not just signal errors, but should also indicate when the test has run successfully. That way, when a bug is so severe the test doesn't run at all, you can detect the absence of a "test successful!" message, instead of believing all is perfect when in reality all is crashed.



Does spring-headless run widget?



Instead of a single modoption I use a whole startscript so that:

• More conditions are controlled, like the teams, the startpos, the other modoptions, making the test more repeatable.
• What the test scenario do is described in almost readable text form in that script instead of being buried in the mod. You can read and change the test without remaking the mod.
• You'd have to create a startscript anyway to run Spring.
• But in reality mostly because I already had this mission system thingy coded.

spring-headless runs everything it just doesn't render

how far can a beheaded spring run? If every step saves a doomed pirate from the ropemakers daughter?

You can report those via Spring.Log too just use INFO level

i'm still thinking about how to start a test:

using a startscript for a test currently has the huge drawback, that, for each test, the engine has to be (re-)started.

also, somehow, the map has to be downloaded to run the test.

this is why i prefered the single modoption runtests=1 to run a test.

would it make sense, to create the folder "tests" in a game-archive, that contains script.txt's? fetching the map out of script.txt should be easy, it should be a grep for the line Map ...

only bad thing is, the game name has to be updated... maybe use some placeholder for that?

Test could also be a mission - specialized mutator ... you would download it from fixed URL and run it ..

imo a mission is fine.

a fixed url is bad, because:
- it needs a (remote) service thats running to work
- bad for smaller projects, because they will have not control about the service
- bad to reproduce tests, if the service is down

also, this could be changed later. a solution, that works "offline" is more what we want here.

this is why i would prefer the data/script/mission (whatever) to be included in the game itself. this way its also easier to reproduce a failing test.

Well there is spring based general mission service
http://zero-k.info/Missions?featured=false

All missions there are always available through standard plasma service download.