Sync regression testing?

[hughperkins]

At the risk of stating the obvious, because it's pretty far from my field of expertise, and lots of people have been looking at this for a long time, but, just on the off-chance there's something useful:

Sync is really hard to test rigorously.

I couldn't help thinking however that as a game unfolds, the actions that should affect sync state are most likely well defined, and thus easy to record? I imagine something like a list of commands per frame?

So, idea:
- during a game - could be a human game, bot game, whatever - we record exactly all the actions that affected state, along with the frame number
- during the game, we also snapshot the full game state once every 1 to 5 game minutes or so (so every 60 * 30 frames or 5 * 60 * 30 frames), ie we record the exact position and orientation of every unit, and maybe a couple of other measures, such as health, player metal, and player energy
- after the game, we rerun the game, by feeding into the sync state exactly the same commands as before
- during the rerun, we create the same full game state snapshots at the exact same interval as before
- we compare the snapshots created during the game with those afterwards
- we can also compare snapshots created on different machines of course

Example of regressions this could detect:
- play a single game on one computer with an ai in
-- if the ai somehow makes calls that manipulate sync state directly (eg pathfinder), then the snapshots created during the game will not match those created when the sync commands are rerun later

In addition:
- once we have the list of sync commands, we can rerun the sync commands both for a current version of spring and a proposed new release
-- if there are any regressions in sync between the current version and the new version of spring, then the snapshots will not match

[Auswaschbar]

Ever heard of syncdebug?

[Kloot]

We already have lists of actions that affect sync state, they're called demos. (Albeit not indexed per frame, but they don't need to be.) Even so, automated testing only goes so far, and sometimes sends you completely in the wrong direction (cf. the most recent case, which boiled down to an uninitialized memory read in a section of code the sync debugger never even hinted at). Making and comparing shapshots of the entire state does not help then, and in practice desyncs are detected quickly anyway.

[hughperkins]

Ok great I'll go back to stuff I actually know about

[imbaczek]

(cf. the most recent case, which boiled down to an uninitialized memory read in a section of code the sync debugger never even hinted at).

that's a huge problem - synced state is not marked as such. I introduced ASSERT_SYNCED way back then (free time, I miss you...), but never got around to using it much. It would be IMHO very good if any sync fixes were guarded against regressions by ASSERT_SYNCED somewhere.

[Tobi]

Could change more types to SyncedTypes, but this does have some performance impact (mostly on compile time tho, I think )