Task: modularizing the replay file format

[Tobi]

I post this here as an idea for someone to implement.

The features this adds to Spring have been requested quite often already for use in replay/league/ladder sites, so you can be sure it will be useful.

If you are interested, post your intention to implement this (to prevent double work) and questions/ideas here.

The goal is to make the replay file format better designed and defined, in a way smooth transitions are possible if the format changes, and in a way which separates raw demo data (network stream) from clearly defined chunks of (meta)data for use by third party applications.

This includes, but is not limited to:

• A proper DemoFileHeader with a version, offsets to other chunks of data and an easy way to add new chunks of data without having to modify any application reading the format. (look at the map format for an example)
• The header should contain a synced 128 bit GUID (generated by the server and sent over the network to all players), so there is an easy and accurate way to identify demo files of the same game.
• Other stuff which would be useful in the header: demo file format version (in case the modular structure ever needs a redesign), Spring version, Unix timestamp, etc.
• A chunk with player stats (MC/m, KP/m etc.).
• A chunk with team stats (metal/energy used/spent, damage dealt etc.)
• A chunk with the network stream (the stream itself can and should be undefined format as it is now, as I don't want to limit our freedom to change it however we want).
• A chunk with the startscript, if it is a game with gameSetup. (as it sits now at the start of the demo file)
• Maybe a chunk with the gameSetup (startscript) data available in binary format (so a replay-reader does not need a TDF parser to parse the startscript).

But in the end the most important feature should be that the demo format is extensible, without breaking the applications that use it.

[Tim Blokdijk]

Looks like Tobi read the site development process..

And as feedback on the task itself, I understand this won't allow people to rewind a replay as that would be part of the network stream? Or would such a function be implemented outside the network stream (and inside the replay file format)?

[Tobi]

Rewind functionality falls outside the scope of this task, though of course it would be possible to extend the format later with something like keyframes in a new chunk in the format. (without breaking the format for other applications due to it's extensability)

This task is only ment to solve the problem of third party application developers (league/replay/ladder sites, mostly) having to hack their way through a totally unconventional and not extendable format, and to add often requested metadata to it (e.g. stats, which allyteam won the game, etc).

[Tobi]

FYI I am implementing this (read: it's almost finished)

[rattle]

Keyframes as well?

I could imagine a progress slider in demo view mode which allows you to step through the keyframes... man that would be awesome.

[Tobi]

No, that would require support in other parts of the engine (to properly serialize/deserialize game state).

[lurker]

No, that would require support in other parts of the engine (to properly serialize/deserialize game state).

What do you need for keyframes that save/load doesn't do?

[Tobi]

Nothing, but just try save/load a few times and you'll see why I can not reasonably include that in the replay. Heck, some people have even called F8 (save game) an instacrash key

That, and it may bloat the replay file too much.

[lurker]

Nothing, but just try save/load a few times and you'll see why I can not reasonably include that in the replay. Heck, some people have even called F8 (save game) an instacrash key

That, and it may bloat the replay file too much.

What I was thinking was along the lines of only creating keyframes as an option upon playback of the replay, possibly saving them in a different file. That way you can have rewind if you want it, and don't have to worry about space otherwise.

[jcnossen]

Problem with that is that if even saving can crash spring, then your game can crash at any keyframe saving point...

On the other hand, if this is implemented, it might help a lot to determine which unit types, actions or game states can crash the saving. Just enable saving every 10 frame or so and play around a bit.

[Tobi]

Committed the changes. Everything mentioned in my OP is implemented.

I included an example/test program DemoDumper in which you can see how to read the format, in case you are interested. (Thanks trepan/zizu for the ArchiveMover source btw, I ripped half of it for this program )

Some ideas I can think off:

• A lobby program could start spring with --quit=HUGENUMBER so Spring quits automatically at game end instead of showing the endgame graphs. It could then show the graphs in it's own user interface.
• Player/team statistics on replay/ladder sites.
• Matching replays of the same game together on replay/ladder sites (that work with replays) using the GameID.
• Extracting the winning allyteam of the game from the replay (on a ladder site).
• A tool to analyze your own statistics (whether you got better or not) based on your replay collection
• Adding a XML switch to DemoDumper so other apps can just filter a demo through DemoDumper and parse the XML instead of bothering with the custom binary format.

If you have good ideas that are trivial to add to the format or other changes/comments/questions, feel free to post.