automated path regression testing

[BrainDamage]

the system is a bit rough, but does work, here's what it does:

spawns units on the map
issue move order if requested
monitor distance to target and elapsed time
if distance < wanted distance or timeouts to target, end test
save info on unit position, distance to target, target reached, etc

regressions can be verified by units never reaching the target ( timeout ) or taking considerably longer due to collisions, etc

config test files belong to luaui/pathTests/config/mapname-modname.lua
the widget writes back infos to: luaui/pathTests/result/mapname-modname-springversion-date.lua

things it needs: actual test cases! (attached is a tiny example script)

devs/modders/mappers should create config files for the stuff they want tested (push resistant vs not push resistant, all terrain units scaling a cliff, etc ), obviously they won't cover all the cases but with time there should be a decent lib of test cases for most problems

use the attached pathtest_script.txt to launch spring, just have care to edit map & mod first, you can also change minspeed to a very high number to zip trough the simulation ( headless spring can also help there )

abma will probably setup a buildbot slave to automatically execute the tests for submitted scripts
I guess to keep the script simple, at first it should just check for timeouts ( by simply grepping the result file for timeout = true ), in the future parse for arrival time and see if there's substantial changes ( for the better or worse ) between revisions

[abma]

if you replace echo with Spring.Log + fix params it could be directly used in validation tests. everything >= LOG.WARNING is seen as error...

thats exactly what i wanted to see as start point for regression tests...

[BrainDamage]

updated to use Log and did some small bugfixes

[BrainDamage]

updated the test widget and example file, now it handles distance in 3d instead of 2d, and supports multiple move waypoints per unit, and sets all units in hold fire

I created a small widget that lets you easily create or remove path tests, here's how it works:

launch a game and enable the widget ( it'll enable cheats, godmode and globallos )

spawn or build some units around the map where you want them to be tested ( can be ally or enemy ), the widget will automatically set them on hold fire
pause the game and issue one or more move orders ( any other orders will be ignored by the widget)

select all units that you want to be included by the test

issue /addpathtest [targetDistanceTollerance] [delayToNextTest] [arrivalTimeout]

if params are omitted the widget will try to estimate ok values for them

if you want to remove a test, use /delpathtest number

[zerver]

Although I did not check it in detail yet, an excellent idea. You could randomly spawn units across the map and issue random move orders. Some units will (and should) get stuck but the interesting part (provided that you make a very large number of such tests) is the percentage that succeed, and could be used to construct a pathfinder fitness index. Any commits that bring the index score down will be auto-reverted :)

[Silentwings]

I think these tests are a good idea, but we should use it as a way of catching bugs and not as a way of comparing *good* path finders.

Basically I think its easy to spot if something big is wrong, but seperating out which of two good pathfinders work best is something you can't automate and only feel when you're playing games. Since pathfinder code is hard to write I think we can't expect a constant stream of improvements and sometimes things have to get worse before better.

[KingRaptor]

Integrated with some tweaks into Benchmarker; source here. Currently only includes one test case (for ZK, but should work with any *A except XTA out of the box, and can be easily modified for any game) on the map Crossing_4_final though.

Made some cool charts using the data: