Automated Release Test
Moderator: Moderators
Automated Release Test
Basically a bunch of virtual maschines, testing the installation and networking of the release candidate. No actuall gametesting there besides the - does it get ingame. Every lobby gets 24 hours before release a "does-run-does not run" flag.
We got automated tests for the builds (which is actually most impressive) so i assume this can be done (not for all configurations of course, but the most comon ones)
We got automated tests for the builds (which is actually most impressive) so i assume this can be done (not for all configurations of course, but the most comon ones)
Re: Automated Release Test
most of it is already done here:
http://buildbot.springrts.com/waterfall
(see the validation test)
i already asked guys of zero-k, if they were interested in implementing in-game tests, but they had no interest to do so. maybe i asked the wrong guy, don't know.
see http://springrts.com/wiki/Regression_Testing for my plan.
this is also how our current releases works: make release post, wait some time, if nobody complains: set it live.
http://buildbot.springrts.com/waterfall
(see the validation test)
i already asked guys of zero-k, if they were interested in implementing in-game tests, but they had no interest to do so. maybe i asked the wrong guy, don't know.
see http://springrts.com/wiki/Regression_Testing for my plan.
what does that mean? current lobbies don't allow to do run automatic tests, because they have no automatiation switches. afaik koshi works on that / his lobby-lib will allow to do such tests.Every lobby gets 24 hours before release a "does-run-does not run" flag.
this is also how our current releases works: make release post, wait some time, if nobody complains: set it live.
Re: Automated Release Test
I'll be able to automatically test unitsync usage, yes. I did already add tests in the python unitsync wrapper btw. Those just aren't plugged into anything.
Re: Automated Release Test
We are interested in making tests, we just dont know how.
Re: Automated Release Test
I already made an automated pathfinder test:
- Get Kernel Panic.
- Get this map
- Get this file or extract from Kernel_Panic_4.4.sd7 \Missions\KP\Script Exemple 3 - Pathfinder Test.txt
- Feed the file to Spring (as a startscript)
- Read results in infolog.
Here's what a successfully passed test should read like[code][f=0000090] ÿ`ÿÿTrying to climb ramp...
[f=0000090] ÿ`ÿÿTrying to send nine atop...
[f=0000090] ÿ`ÿÿTrying to avoid wall...
[f=0000090] ÿ`ÿÿTrying to climb tower...
[f=0000633] ÿ`ÿÿSuccesfully managed to avoid wall!
[f=0001037] ÿ`ÿÿSuccesfully managed to climb ramp!
[f=0001522] ÿ`ÿÿSuccesfully managed to climb tower!
[f=0002740] ÿ`ÿÿSuccesfully managed to send nine atop!
[f=0003618] ÿ`ÿÿTime-Out![/code]The ÿ are just because for some reasons I wanted color in the ingame chat.
Many versions of Spring, such as 83, 84, 85, fails the last test about sending nine bits atop the tower through the twisted ramp.
Oh take a Spring version either higher or lower but don't use Spring 87, it crashes KP gadgets.
- Get Kernel Panic.
- Get this map
- Get this file or extract from Kernel_Panic_4.4.sd7 \Missions\KP\Script Exemple 3 - Pathfinder Test.txt
- Feed the file to Spring (as a startscript)
- Read results in infolog.
Here's what a successfully passed test should read like[code][f=0000090] ÿ`ÿÿTrying to climb ramp...
[f=0000090] ÿ`ÿÿTrying to send nine atop...
[f=0000090] ÿ`ÿÿTrying to avoid wall...
[f=0000090] ÿ`ÿÿTrying to climb tower...
[f=0000633] ÿ`ÿÿSuccesfully managed to avoid wall!
[f=0001037] ÿ`ÿÿSuccesfully managed to climb ramp!
[f=0001522] ÿ`ÿÿSuccesfully managed to climb tower!
[f=0002740] ÿ`ÿÿSuccesfully managed to send nine atop!
[f=0003618] ÿ`ÿÿTime-Out![/code]The ÿ are just because for some reasons I wanted color in the ingame chat.
Many versions of Spring, such as 83, 84, 85, fails the last test about sending nine bits atop the tower through the twisted ramp.
Oh take a Spring version either higher or lower but don't use Spring 87, it crashes KP gadgets.
Re: Automated Release Test
But how do you interfaace tests with testing suite? what should the test output?
Re: Automated Release Test
Use Spring.Log() with errorlevel >= ERROR...Licho wrote:But how do you interfaace tests with testing suite? what should the test output?
this will printf to stderror, all printed to stderror is seen as error / test failed.
so... some useful message would be nice.
@zwzsg:
nice! thats close to that what i wanted to do. i will soon try to run that test at the buildslave. i'm not sure if changes are needed, but its a nice start-off i think...
Re: Automated Release Test
I don't really understand what you mean with the /runtest 1 option.
I disagree about how you said "tests shouldn't rely on a specific map":
Spawning units, make them move, build, attack, reclaim, etc... all depends on the map.
I don't use Spring.Log (is that C ?) but Lua's Spring.Echo.
I suppose it would be cleaner if my test printed a failure message instead of having to detect a failure by not having a success message before the time out.
I disagree about how you said "tests shouldn't rely on a specific map":
Spawning units, make them move, build, attack, reclaim, etc... all depends on the map.
I don't use Spring.Log (is that C ?) but Lua's Spring.Echo.
I suppose it would be cleaner if my test printed a failure message instead of having to detect a failure by not having a success message before the time out.
Re: Automated Release Test
and how do you run the test? can it be lua widgets? gadgets? how does it work?
does it update to laest mod?
does it update to laest mod?
-
- Posts: 834
- Joined: 19 May 2009, 21:10
Re: Automated Release Test
Spring.* is Lua (mostly).zwzsg wrote:I don't use Spring.Log (is that C ?) but Lua's Spring.Echo.
Re: Automated Release Test
@zwzsg:
i should have noted that all points on the wiki are debatable, its currently just a suggestion how it could work. Yeah, it seems like it depends on the map... i just wanted to not have this dependency if possible somehow.
for Spring.Log see changelog.txt or https://github.com/spring/spring/commit ... 390db3691c
and yes, Spring.Log is lua...
@Licho:
i should have noted that all points on the wiki are debatable, its currently just a suggestion how it could work. Yeah, it seems like it depends on the map... i just wanted to not have this dependency if possible somehow.
for Spring.Log see changelog.txt or https://github.com/spring/spring/commit ... 390db3691c
and yes, Spring.Log is lua...
@Licho:
my suggestion is the modoption "testsrun=1". it can be a widget and/or gadget that checks for this modoption. (thats also debatable...)and how do you run the test? can it be lua widgets? gadgets? how does it work?
does it update to laest mod?
Re: Automated Release Test
ok that sounds good
Re: Automated Release Test
Ok. I was confused because you format your strings the C way. I guess I can make it use Spring.Log instead of Spring.Echo.abma wrote:and yes, Spring.Log is lua...
However I think the test should not just signal errors, but should also indicate when the test has run successfully. That way, when a bug is so severe the test doesn't run at all, you can detect the absence of a "test successful!" message, instead of believing all is perfect when in reality all is crashed.
Does spring-headless run widget?abma wrote:it can be a widget and/or gadget
Instead of a single modoption I use a whole startscript so that:abma wrote:checks for this modoption. (thats also debatable...)
- More conditions are controlled, like the teams, the startpos, the other modoptions, making the test more repeatable.
- What the test scenario do is described in almost readable text form in that script instead of being buried in the mod. You can read and change the test without remaking the mod.
- You'd have to create a startscript anyway to run Spring.
- But in reality mostly because I already had this mission system thingy coded.
Re: Automated Release Test
spring-headless runs everything it just doesn't renderzwzsg wrote:Does spring-headless run widget?abma wrote:it can be a widget and/or gadget
Re: Automated Release Test
how far can a beheaded spring run? If every step saves a doomed pirate from the ropemakers daughter?
Re: Automated Release Test
You can report those via Spring.Log too just use INFO levelzwzsg wrote:Ok. I was confused because you format your strings the C way. I guess I can make it use Spring.Log instead of Spring.Echo.abma wrote:and yes, Spring.Log is lua...
However I think the test should not just signal errors, but should also indicate when the test has run successfully. That way, when a bug is so severe the test doesn't run at all, you can detect the absence of a "test successful!" message, instead of believing all is perfect when in reality all is crashed.
Re: Automated Release Test
i'm still thinking about how to start a test:
using a startscript for a test currently has the huge drawback, that, for each test, the engine has to be (re-)started.
also, somehow, the map has to be downloaded to run the test.
this is why i prefered the single modoption runtests=1 to run a test.
would it make sense, to create the folder "tests" in a game-archive, that contains script.txt's? fetching the map out of script.txt should be easy, it should be a grep for the line Map ...
only bad thing is, the game name has to be updated... maybe use some placeholder for that?
using a startscript for a test currently has the huge drawback, that, for each test, the engine has to be (re-)started.
also, somehow, the map has to be downloaded to run the test.
this is why i prefered the single modoption runtests=1 to run a test.
would it make sense, to create the folder "tests" in a game-archive, that contains script.txt's? fetching the map out of script.txt should be easy, it should be a grep for the line Map ...
only bad thing is, the game name has to be updated... maybe use some placeholder for that?
Re: Automated Release Test
Test could also be a mission - specialized mutator ... you would download it from fixed URL and run it ..
Re: Automated Release Test
imo a mission is fine.Licho wrote:Test could also be a mission - specialized mutator ... you would download it from fixed URL and run it ..
a fixed url is bad, because:
- it needs a (remote) service thats running to work
- bad for smaller projects, because they will have not control about the service
- bad to reproduce tests, if the service is down
also, this could be changed later. a solution, that works "offline" is more what we want here.
this is why i would prefer the data/script/mission (whatever) to be included in the game itself. this way its also easier to reproduce a failing test.
Re: Automated Release Test
Well there is spring based general mission service
http://zero-k.info/Missions?featured=false
All missions there are always available through standard plasma service download.
http://zero-k.info/Missions?featured=false
All missions there are always available through standard plasma service download.