Release Candidate prerequisites/ladder

[hoijui]

this would be for the future, not this release

We could define prerequisites for a release candidate of the engine, which had to be fulfilled before it would go live.

example prerequisites:
min games: 30
min different users involved in games: 10
min OS: win32,linux32,linux64
...

maybe anything more then min games is not really needed.
Also, how would this be done best?

i can think of two ways:
1. The lobby server/a lobby bot monitors all games, and forwards info to the dev channel or a website or whatever.
2. A Release (Candidate) ladder/tournament

any thoughts?

[imbaczek]

+1, but sounds like work. also, lobbies need to lower the barrier of entry - springlobby e.g. requires you to set DisableVersionCheck=1 in the config file to allow joining games. (/me looks at aegis and other lobby server devs)

[Auswaschbar]

Putting something like this in fixed numbers is predetermined to fail imho.

Such numbers mean next to nothing. Which mods are tested, how big the games were, how long they lasted etc etc. Also it doesn't take into account the amount of testing before release candidate, the amount of changes since last release, time since last release etc. Also, have fun organising 30 testgames, how many did you do for current release? When there were 10 testgames, that would be much.

[Gota]

What would be cool is if this whole thing could work through spring downloader.
ATM you can put mods to autoupdate.
If it was possible to auto update test releases so you'd have the latest automatically and could join games that would be very good..
I suppose this also needs other stuff to support such a systemlike the lobbies.
This would make it much easier for people to test pre releases.

[Tobi]

Putting something like this in fixed numbers is predetermined to fail imho.

Such numbers mean next to nothing. Which mods are tested, how big the games were, how long they lasted etc etc. Also it doesn't take into account the amount of testing before release candidate, the amount of changes since last release, time since last release etc. Also, have fun organising 30 testgames, how many did you do for current release? When there were 10 testgames, that would be much.

If you want proper numbers we should calculate combined branch coverage of all test games, and only release when that is high enough

More useful could be to make a checklist of stuff to test, but that would be quite boring to actually test. (i.e. test /kick, test playing a demo, test all startpos types, etc.)

[Agon]

Unit tests like JUnit/NUnit?

[Tobi]

That would by far be the best solution yeah.

Unfortunately unit testing in C++ is kinda painful in my experience, and also lots of refactoring would already be needed to even be able to test the separate components.

Also some kind of automated regression tests would be cool, but I haven't been able to come up with something useable yet... (Best I got at some point was script that played lots of AI vs AI games on random maps, IIRC, but this was effectively more testing the AIs then the engine, and, most importantly, network code wasn't tested at all.)

[Umbra]

Id play more in-development games if:

• The main lobby server could allow multiple versions of the engine alongside each other.
• A list of svn/git commit comments could be shown somewhere convinient so people would know what has changed, and what to test.
• A quick bug reporting system.
• Maybe track the time spent playing each release(or all dev releases combined maybe) and provide a function similar to .ingame, but for the dev/test games instead?
• Show a medal next to someones name based on how much dev test games they've played in at least the battle lobby for games using a dev engine.
• Even go as far as to announce the dev tester(s) of the month in the #main topic for any significant amount of testing done by one or more people?

[hoijui]

Id play more in-development games if:

• A list of svn/git commit comments could be shown somewhere convinient so people would know what has changed, and what to test.

in addition to the changelog:
http://github.com/spring/spring/commits/master