Development < Debugging Sync Errors

Desync happens when simulation gives different results for different clients. Currently it is detected by means of checksumming crucial data on every write; this checksum is then transferred to the host, who compares them to his own checksum.

For purposes of debugging, the sync debugger stores every such write with some debugging information. When a desync happens, the write history is compared on every client, who then sends locations where computation has given different results. The location is then resolved to line number of the engine source, which should provide information about the nature of the desync.

To enable sync debugging, you have use a special build of Spring. To obtain it, you must:

1. Go to #buildserv and request a syncdebug build with !rebuild profile=syncdebug. You can request a specific version by adding tag=<svn tag> or rev=<svn rev number>. Type !help for help.
2. If the build works, an executable will be uploaded to the usual place; at the time of this writing, it's http://buildbot.eat-peet.net/spring/executable/. The generated archive will have [syncdebug] in its name.
3. To compile it with linux, first get the same revision from git, and then use either of the two:
   • scons configure syncdebug=yes synctrace=yes optimize=yes debug=1 gml=false
   • cmake -DCMAKE_BUILD_TYPE=DEBUG2 -DTRACE_SYNC=true -DSYNCDEBUG=true
4. Host a game. Make sure every player who joins has this exe; it's not needed to have it at the time of joining the battle, but when the game starts, everyone must have it. Expect random undefined behaviour otherwise.
5. If you want engine source lines, the host has to download two more files: debug symbols and addr2line. Debug symbols are generated and uploaded by buildserv; as of now, they can be found at http://buildbot.eat-peet.net/spring/debug/. addr2line is a part of MinGW32 binutils package.
   1. Download debug symbols
   2. Download addr2line
   3. Extract both to where your syncdebug-enabled spring.exe lives
   4. Rename debug symbols to spring.dbg

Only the host needs to download addr2line and debug symbols.

When everybody is in game, test your setup by enabling cheats (/cheat) and having one of the players do a test desync (/desync). The game should pause and syncdebugger should do its work. If the game pauses, but you won't see SD messages, it's likely that one of the clients doesn't have the correct exe.

In addition to syncdebugger logs, syncdebug engine builds also have synctrace enabled. This option makes spring.exe write a file called trace<playernum>.log which logs important simulation events in a diff-friendly format. After a desync, get this file from the desynced player(s) and do a diff -u yourtrace histrace to find the first logged sync mismatch. The real desync probably happened somewhere in between the two log lines.

If you're a mod developer, please be aware that Lua may be a source of desyncs. E.g. table iteration using pairs when you have tables, coroutines, or functions as keys is not a sync-safe operation, see mantis #1050 for example of such a desync.