Automated Release Test

Automated Release Test

Requests for features in the spring code.

Moderator: Moderators

User avatar
PicassoCT
Journeywar Developer & Mapper
Posts: 10450
Joined: 24 Jan 2006, 21:12

Automated Release Test

Post by PicassoCT »

Basically a bunch of virtual maschines, testing the installation and networking of the release candidate. No actuall gametesting there besides the - does it get ingame. Every lobby gets 24 hours before release a "does-run-does not run" flag.

We got automated tests for the builds (which is actually most impressive) so i assume this can be done (not for all configurations of course, but the most comon ones)
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Automated Release Test

Post by abma »

most of it is already done here:
http://buildbot.springrts.com/waterfall

(see the validation test)

i already asked guys of zero-k, if they were interested in implementing in-game tests, but they had no interest to do so. maybe i asked the wrong guy, don't know.

see http://springrts.com/wiki/Regression_Testing for my plan.
Every lobby gets 24 hours before release a "does-run-does not run" flag.
what does that mean? current lobbies don't allow to do run automatic tests, because they have no automatiation switches. afaik koshi works on that / his lobby-lib will allow to do such tests.

this is also how our current releases works: make release post, wait some time, if nobody complains: set it live.
User avatar
koshi
Lobby Developer
Posts: 1059
Joined: 14 Aug 2007, 16:15

Re: Automated Release Test

Post by koshi »

I'll be able to automatically test unitsync usage, yes. I did already add tests in the python unitsync wrapper btw. Those just aren't plugged into anything.
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Automated Release Test

Post by Licho »

We are interested in making tests, we just dont know how.
User avatar
zwzsg
Kernel Panic Co-Developer
Posts: 7049
Joined: 16 Nov 2004, 13:08

Re: Automated Release Test

Post by zwzsg »

I already made an automated pathfinder test:

- Get Kernel Panic.
- Get this map
- Get this file or extract from Kernel_Panic_4.4.sd7 \Missions\KP\Script Exemple 3 - Pathfinder Test.txt
- Feed the file to Spring (as a startscript)
- Read results in infolog.

Here's what a successfully passed test should read like[code][f=0000090] ÿ`ÿÿTrying to climb ramp...
[f=0000090] ÿ`ÿÿTrying to send nine atop...
[f=0000090] ÿ`ÿÿTrying to avoid wall...
[f=0000090] ÿ`ÿÿTrying to climb tower...
[f=0000633] ÿ`ÿÿSuccesfully managed to avoid wall!
[f=0001037] ÿ`ÿÿSuccesfully managed to climb ramp!
[f=0001522] ÿ`ÿÿSuccesfully managed to climb tower!
[f=0002740] ÿ`ÿÿSuccesfully managed to send nine atop!
[f=0003618] ÿ`ÿÿTime-Out![/code]The ÿ are just because for some reasons I wanted color in the ingame chat.

Many versions of Spring, such as 83, 84, 85, fails the last test about sending nine bits atop the tower through the twisted ramp.

Oh take a Spring version either higher or lower but don't use Spring 87, it crashes KP gadgets.
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Automated Release Test

Post by Licho »

But how do you interfaace tests with testing suite? what should the test output?
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Automated Release Test

Post by abma »

Licho wrote:But how do you interfaace tests with testing suite? what should the test output?
Use Spring.Log() with errorlevel >= ERROR...

this will printf to stderror, all printed to stderror is seen as error / test failed.

so... some useful message would be nice.

@zwzsg:

nice! thats close to that what i wanted to do. i will soon try to run that test at the buildslave. i'm not sure if changes are needed, but its a nice start-off i think...
User avatar
zwzsg
Kernel Panic Co-Developer
Posts: 7049
Joined: 16 Nov 2004, 13:08

Re: Automated Release Test

Post by zwzsg »

I don't really understand what you mean with the /runtest 1 option.

I disagree about how you said "tests shouldn't rely on a specific map":
Spawning units, make them move, build, attack, reclaim, etc... all depends on the map.

I don't use Spring.Log (is that C ?) but Lua's Spring.Echo.

I suppose it would be cleaner if my test printed a failure message instead of having to detect a failure by not having a success message before the time out.
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Automated Release Test

Post by Licho »

and how do you run the test? can it be lua widgets? gadgets? how does it work?
does it update to laest mod?
SirMaverick
Posts: 834
Joined: 19 May 2009, 21:10

Re: Automated Release Test

Post by SirMaverick »

zwzsg wrote:I don't use Spring.Log (is that C ?) but Lua's Spring.Echo.
Spring.* is Lua (mostly).
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Automated Release Test

Post by abma »

@zwzsg:
i should have noted that all points on the wiki are debatable, its currently just a suggestion how it could work. Yeah, it seems like it depends on the map... i just wanted to not have this dependency if possible somehow.

for Spring.Log see changelog.txt or https://github.com/spring/spring/commit ... 390db3691c

and yes, Spring.Log is lua...

@Licho:
and how do you run the test? can it be lua widgets? gadgets? how does it work?
does it update to laest mod?
my suggestion is the modoption "testsrun=1". it can be a widget and/or gadget that checks for this modoption. (thats also debatable...)
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Automated Release Test

Post by Licho »

ok that sounds good
User avatar
zwzsg
Kernel Panic Co-Developer
Posts: 7049
Joined: 16 Nov 2004, 13:08

Re: Automated Release Test

Post by zwzsg »

abma wrote:and yes, Spring.Log is lua...
Ok. I was confused because you format your strings the C way. I guess I can make it use Spring.Log instead of Spring.Echo.

However I think the test should not just signal errors, but should also indicate when the test has run successfully. That way, when a bug is so severe the test doesn't run at all, you can detect the absence of a "test successful!" message, instead of believing all is perfect when in reality all is crashed.


abma wrote:it can be a widget and/or gadget
Does spring-headless run widget?


abma wrote:checks for this modoption. (thats also debatable...)
Instead of a single modoption I use a whole startscript so that:
  • More conditions are controlled, like the teams, the startpos, the other modoptions, making the test more repeatable.
  • What the test scenario do is described in almost readable text form in that script instead of being buried in the mod. You can read and change the test without remaking the mod.
  • You'd have to create a startscript anyway to run Spring.
  • But in reality mostly because I already had this mission system thingy coded.
User avatar
jK
Spring Developer
Posts: 2299
Joined: 28 Jun 2007, 07:30

Re: Automated Release Test

Post by jK »

zwzsg wrote:
abma wrote:it can be a widget and/or gadget
Does spring-headless run widget?
spring-headless runs everything it just doesn't render
User avatar
PicassoCT
Journeywar Developer & Mapper
Posts: 10450
Joined: 24 Jan 2006, 21:12

Re: Automated Release Test

Post by PicassoCT »

how far can a beheaded spring run? If every step saves a doomed pirate from the ropemakers daughter?
User avatar
jK
Spring Developer
Posts: 2299
Joined: 28 Jun 2007, 07:30

Re: Automated Release Test

Post by jK »

zwzsg wrote:
abma wrote:and yes, Spring.Log is lua...
Ok. I was confused because you format your strings the C way. I guess I can make it use Spring.Log instead of Spring.Echo.

However I think the test should not just signal errors, but should also indicate when the test has run successfully. That way, when a bug is so severe the test doesn't run at all, you can detect the absence of a "test successful!" message, instead of believing all is perfect when in reality all is crashed.
You can report those via Spring.Log too just use INFO level
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Automated Release Test

Post by abma »

i'm still thinking about how to start a test:

using a startscript for a test currently has the huge drawback, that, for each test, the engine has to be (re-)started.

also, somehow, the map has to be downloaded to run the test.

this is why i prefered the single modoption runtests=1 to run a test.

would it make sense, to create the folder "tests" in a game-archive, that contains script.txt's? fetching the map out of script.txt should be easy, it should be a grep for the line Map ...

only bad thing is, the game name has to be updated... maybe use some placeholder for that?
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Automated Release Test

Post by Licho »

Test could also be a mission - specialized mutator ... you would download it from fixed URL and run it ..
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Automated Release Test

Post by abma »

Licho wrote:Test could also be a mission - specialized mutator ... you would download it from fixed URL and run it ..
imo a mission is fine.

a fixed url is bad, because:
- it needs a (remote) service thats running to work
- bad for smaller projects, because they will have not control about the service
- bad to reproduce tests, if the service is down

also, this could be changed later. a solution, that works "offline" is more what we want here.

this is why i would prefer the data/script/mission (whatever) to be included in the game itself. this way its also easier to reproduce a failing test.
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Automated Release Test

Post by Licho »

Well there is spring based general mission service
http://zero-k.info/Missions?featured=false

All missions there are always available through standard plasma service download.
Post Reply

Return to “Feature Requests”