Testing (Split from Graphical Performance & Spring)
Moderator: Moderators
Holy grail of all software developmentTobi wrote:TBH any attempt at optimization is wasted time until we have a development process that doesn't allow major bugs to slip in so easily as it is now.
impossible to achieve, though. Distributed development + feature freezes on candidate/release branches are the best you can get.What we really need is testers. Testing is like sex - if you have lots, great. If you have little, still better than none. Currently, we have sometimes little Wednesdays and none rest of the time
Therefore I hereby request ladder players to help us make spring better and open a Future Spring (Your Favourite Mod) ladder. Please always use the newest automatic build whenever possible (if unitsync doesn't work, copy it over from 75b2 release) and post your observations and bugs. We (as the devs) really need your input.
No, not impossible. Of course you will always have bugs but with Spring it are really major bugs that slip through, not small minor things.imbaczek wrote:Holy grail of all software developmentTobi wrote:TBH any attempt at optimization is wasted time until we have a development process that doesn't allow major bugs to slip in so easily as it is now.
Maybe it is impossible for Spring though due to it's very nature, but in general, with decent unit testing (with good coverage), good functional testing, frequent deployments (AKA agile development) you can get a steady rate of improvement instead of the three step forward two step back model Spring currently has (or is it two step forward three step back, I doubt you can really tell this after a release, which is the dangerous thing...).
+1What we really need is testers.
And we also need lead testers since it seems now that if I'm not advertizing it and actively organizing the test then it won't happen, or maybe it does happen but no results are posted anywhere? lavishpcar2 was an excellent lead tester.
Sounds like a great idea.Therefore I hereby request ladder players to help us make spring better and open a Future Spring (Your Favourite Mod) ladder. Please always use the newest automatic build whenever possible (if unitsync doesn't work, copy it over from 75b2 release) and post your observations and bugs. We (as the devs) really need your input.
Another idea I had in mind is writing a functional test, basically a checklist of very small points so anyone being bored can test a bit and fill in the checklist and sent it to some e-mail or something, then we can keep track of what has been tested and what hasn't, and the tester himself has some hints and guidelines for where to look for.
And yet another idea: auto-feedback, make Spring by default phone home after any game and transfer stacktrace of previous game if that crashed and some stats like # of players, # of spectators, mod, map, duration, peak unit count, etc. (or maybe just the infolog.txt, since that contains almost all that kind of stuff)
The first idea takes much less work though, second has more perspective. With the second idea it would be trivial to plot for example a graph of percentage of stacktraces over time, or percentage of desyncs. If you see a jump upward on release, you know stuff's wrong. If you see a jump downward on release, you know you did something right
Last edited by Tobi on 21 Sep 2007, 00:40, edited 1 time in total.
You can read this about testing: http://spring.clan-sy.com/phpbb/viewtopic.php?t=10555.
About the bugs/testing: would it be possible to make some automatic testing also ? Something that would send 'clicks' to the spring window, and in some time check let's say the statistics.
The main question is : would that be worth ? If there are many bugs that would be caught like that, maybe it is worth studying.
About the bugs/testing: would it be possible to make some automatic testing also ? Something that would send 'clicks' to the spring window, and in some time check let's say the statistics.
The main question is : would that be worth ? If there are many bugs that would be caught like that, maybe it is worth studying.
-
mr sharpoblunto
- Posts: 24
- Joined: 12 Dec 2005, 03:47
Sounds like getting some automated unit testing going for the core classes in the engine would be very beneficial. Having predictable and repeatable unit tests will provide much more confidence that a given build will free of *major* bugs. The added benefit of this is that it reduced the reliance on human testers to run through test scripts again and again which can be very long and boring, especially if they are sufficiently thorough.
While there is overhead in writing up the initial test cases it more than pays for itself every time you refactor or add a new peice of code as you know straight that away your changes have not broken anything else in the system.
The other thing is is that once you get used to writing code in a test driven fashion there's not much effort required to keep the unit tests up to date - it just takes a bit of a change in your coding habits, even if you don't write tests first, you can write a bit of code, then write a quick test to ensure its all going according to plan. Also when you do discover a suspected bug, isolate the cause by writing a test case this means you now have a repeatable check to ensure the bug never reappears. If a piece of code is proving hard to write a test for it often indicates that the code is not well designed (refactoring to use design patterns such as inversion of control can usually help in this case), in general code that is testable is modular and well structured so unit testing can help isolate areas of poorly written code aswell as buggy code.
The only major problem I have with testing in c++ is the lack of good testing frameworks available, I've used cppUnit before but wasn't particularly impressed, perhaps there's no better way to do it given that c++ doesn't support reflection. Anyone have any recommendations in this area?
While there is overhead in writing up the initial test cases it more than pays for itself every time you refactor or add a new peice of code as you know straight that away your changes have not broken anything else in the system.
The other thing is is that once you get used to writing code in a test driven fashion there's not much effort required to keep the unit tests up to date - it just takes a bit of a change in your coding habits, even if you don't write tests first, you can write a bit of code, then write a quick test to ensure its all going according to plan. Also when you do discover a suspected bug, isolate the cause by writing a test case this means you now have a repeatable check to ensure the bug never reappears. If a piece of code is proving hard to write a test for it often indicates that the code is not well designed (refactoring to use design patterns such as inversion of control can usually help in this case), in general code that is testable is modular and well structured so unit testing can help isolate areas of poorly written code aswell as buggy code.
The only major problem I have with testing in c++ is the lack of good testing frameworks available, I've used cppUnit before but wasn't particularly impressed, perhaps there's no better way to do it given that c++ doesn't support reflection. Anyone have any recommendations in this area?
I've asked several times for spring to write out stacktraces to seperate files in the past so I can use the lobby to auto report them, however because they're always in infolog.txt theyre very quickly lost forever.
Also Spring crashes numerous times for me and every time it crashes I see either the usual windows prompt asking if I want to ask for a solution, close the program, or run through a debugger.
Other times its a gpu crash and Im presented with a message box from the c++ runtime dll saying the program terminated unexpectedly. No stack traces in either case. In the latter case spring continues to run as a blank screen that hides the mouse which is extremely annoying.
Also Spring crashes numerous times for me and every time it crashes I see either the usual windows prompt asking if I want to ask for a solution, close the program, or run through a debugger.
Other times its a gpu crash and Im presented with a message box from the c++ runtime dll saying the program terminated unexpectedly. No stack traces in either case. In the latter case spring continues to run as a blank screen that hides the mouse which is extremely annoying.
Automatic tests are great, but I am afraid that spring will prove to have a lot of that what you said below:mr sharpoblunto wrote:Sounds like getting some automated unit testing going for the core classes in the engine would be very beneficial.
mr sharpoblunto wrote:If a piece of code is proving hard to write a test for it often indicates that the code is not well designed (refactoring to use design patterns such as inversion of control can usually help in this case), in general code that is testable is modular and well structured so unit testing can help isolate areas of poorly written code aswell as buggy code.
Yes there is
but you need to strip out the graphics engine first
then analyze desired quantities in "random motion" (very dumb) AIs' battles @ 10x speed, that .give all their shit to themselves to build etc.
dumb to logfile, analyze more. it is entirely possible to autmoate a lot of stuff, but its not easy. especially with the engine in its current state (a state that wont become better to the extent we all want. ever. i think.)
heck we could set up a distributed auto-test for spring that runs as a background process on users' and send in updates and downloads "do this thing next for testing".
and stuf
a
sfdasfd
but you need to strip out the graphics engine first
then analyze desired quantities in "random motion" (very dumb) AIs' battles @ 10x speed, that .give all their shit to themselves to build etc.
dumb to logfile, analyze more. it is entirely possible to autmoate a lot of stuff, but its not easy. especially with the engine in its current state (a state that wont become better to the extent we all want. ever. i think.)
heck we could set up a distributed auto-test for spring that runs as a background process on users' and send in updates and downloads "do this thing next for testing".
and stuf
a
sfdasfd