Sync errors

DemO · Post by **DemO** » 30 Dec 2006, 22:39

No I just mean in general, theres people I know that host games that repeatedly desync on specific maps, and others that can host those maps with no problems the majority of the time, and vice versa.

Like licho saying 100% of his games on spring desync - for most of us that's simply not the case, and surely it can't just be a run of bad luck, or recurring patterns in games.

imbaczek · Post by **imbaczek** » 30 Dec 2006, 23:17

And most likely differences in AMD and Intel FPUs. Someone must have run into this problem before spring...

Peet · Post by **Peet** » 30 Dec 2006, 23:24

Most games either force the client to the server's calculated data (most FPSs), or simply drop the desynced client (C&C Generals does this iirc).

LordMatt · Post by **LordMatt** » 30 Dec 2006, 23:56

What about stuff with CPUs that have different instruction sets, like SSE2 and SSE3. Does spring use those?

SwiftSpear · Post by **SwiftSpear** » 31 Dec 2006, 00:02

P3374H wrote:Most games either force the client to the server's calculated data (most FPSs), or simply drop the desynced client (C&C Generals does this iirc).

FPS games use a totally different networking system, they don't run simultaneous calculations, the server does most of the calculations and relays that info to the clients, which run only the calculations necessary to communicate sensibly with the server and create a seamless visual play experiance. However the system is very hackable.

AFAIK most commercial RTS games use the same system. They don't use the system spring uses but "force the player to the server's data set" or "drop the desynced player". The server does all the viable calculation and transfers the data to the clients. However, the clients can easily be hacked so that they see more gamestate data then they are intended to. But spring clients will desync if they are modified as such.

The problem with spring is that the gamestates are HUGE, they can't just instantly be streamed over, everyone would have to pause the game for like 10 minutes to stream over the current gamestate on a desync. We either need a method of correcting only the specific states that have desynced (alot of work), or we need to force the systems to evaluate numbers the same way. At some point in the code there is some math that different PC's are emulating to different results. This is the root of our desync problem.

CautionToTheWind · Post by **CautionToTheWind** » 31 Dec 2006, 00:57

Maybe both the client and the server could create a gamestate file designed with spatial consisntency in mind and then rsync it instead of a full download.

SwiftSpear · Post by **SwiftSpear** » 31 Dec 2006, 09:53

CautionToTheWind wrote:Maybe both the client and the server could create a gamestate file designed with spatial consisntency in mind and then rsync it instead of a full download.

If they could do that likely they wouldn't desync in the first place.

Tobi · Post by **Tobi** » 31 Dec 2006, 11:57

jcnossen wrote:Maybe its an idea to send hardware specs once people are desyncing... I have the feeling some people have a lot of desyncs and others have none (like me).
At least people could know which CPU not to buy if they want to play spring

Yeah it's a good step to find a possible HW cause of sync errors. I've thought about it, just didn't have the time to implement it

Ideally I'd set up a database with player + hardware specs and desyncs vs which other players. That way it would be easily visible if e.g. Intel vs AMD has a bigger chance to desync then Intel vs Intel or AMD vs AMD (just an example).

LordMatt wrote:Multiple versions of tangerine were released without changing something to make the two version appear different, is that what you are talking about?

With 0.74b2 it should be impossible to start games if map/mod aren't binary equal. User overriden files can still cause desync tho, but that's a less common bug I think.

LordMatt wrote:What about stuff with CPUs that have different instruction sets, like SSE2 and SSE3. Does spring use those?

No. Currently we compile with X87 math, not with SSE math. X87 math desyncs vs SSE math, so switching to SSE math would mean people on e.g. VIA C3 CPUs and other reasonable recent CPUs without SSE can't play spring anymore.

SwiftSpear wrote:AFAIK most commercial RTS games use the same system. They don't use the system spring uses but "force the player to the server's data set" or "drop the desynced player". The server does all the viable calculation and transfers the data to the clients. However, the clients can easily be hacked so that they see more gamestate data then they are intended to. But spring clients will desync if they are modified as such.

Spring can be hacked to see more gamestate too without desync. It's active cheats (.give'ing units, removing enemy units) that are impossible with our design.

SwiftSpear wrote:The problem with spring is that the gamestates are HUGE, they can't just instantly be streamed over, everyone would have to pause the game for like 10 minutes to stream over the current gamestate on a desync. We either need a method of correcting only the specific states that have desynced (alot of work), or we need to force the systems to evaluate numbers the same way. At some point in the code there is some math that different PC's are emulating to different results. This is the root of our desync problem.

That is if you'd download everything regardless of what is broken. I'm sure I can make up an algorithm that only needs to transfer maybe 5-30 kilobytes to resync the game (by only resyncing the desynced parts...). Heck, I wont even attempt to make something that streams everything over because it wouldn't be really useable anyway.

You hit the nail on the head regarding the actual desync problem

(apart from spring bugs like the uninitialized commonSonarJammerMap of course)

SwiftSpear wrote:If they could do that likely they wouldn't desync in the first place.

Why? It's very well possible for client/server to write the gamestate to a file, and on desync problably a few dwords differ (by only a few bits).

It gets harder when the desync caused one object to exist only at server and not at the client (or the other way round) of course.

SwiftSpear · Post by **SwiftSpear** » 31 Dec 2006, 12:22

Meh, tobi, I don't know, I was just making guesses based on info I had gleaned from various dev forum readings and stuff. I'm no expert. I trust that you know more about that kind of thing then I do.

Tobi · Post by **Tobi** » 31 Dec 2006, 14:28

That's why I correct you, even tho it was with a question

Licho · Post by **Licho** » 31 Dec 2006, 15:00

Btw active cheats are also impossible in server based model. (You can't hax to suddenly have 300HP and such). I think that main benefit of spring's synced model is minimial data transfer and low apparent lag even in high ping/delayed sync scenarios.

CautionToTheWind · Post by **CautionToTheWind** » 31 Dec 2006, 15:24

Tobi wrote: Why? It's very well possible for client/server to write the gamestate to a file, and on desync problably a few dwords differ (by only a few bits).

Yes, if you stop the game at the first notice of desync, and multithread the sync negotation, you could be looking at resycs as fast as 10 secs or so, which would bring much joy.

Also, this could lay the groundwork for game saving/loading, joining games in progress, saving games on player disconnects so that they can be resumed straight away without restarting, etc... Its so easy to give other ppl work. :D

Don't take offense devs, just giving ideas.

imbaczek · Post by **imbaczek** » 31 Dec 2006, 15:25

An idea: if Licho says he can 100% reproduce desyncs, maybe make a version that keeps a cyclic buffer of state (lots of data, but not THAT much I hope...) and right after first desync happens, kill both games and find the first difference. Not easy, but IMHO possible.

Or a less data-intesive trace method - after a desync is detected, calculate several checksums of different parts of state and try to find desynced data by some kind of bisection.

MadRat · Post by **MadRat** » 31 Dec 2006, 16:47

I get desyncs in skirmish mode using strictly 4 ai's duking it out. Its always at first sign of an army of units marching across a map and suddenly a few cannot make it to their planned end points for whatever reasons. The funny thing is that the exact numbers of frames cited ebbs and flows until the offending units get destroyed. Because you only see messages from one team at a time in skirmish mode its tough to catch it happening until the whole game bogs down.

I have a feeling its all the small units in the mods I playtest. Mostly it happens with the tiny soldier units in mods like 1944 and world domination.

Tobi · Post by **Tobi** » 31 Dec 2006, 18:03

CautionToTheWind wrote:Yes, if you stop the game at the first notice of desync, and multithread the sync negotation, you could be looking at resycs as fast as 10 secs or so, which would bring much joy.

Also, this could lay the groundwork for game saving/loading, joining games in progress, saving games on player disconnects so that they can be resumed straight away without restarting, etc... Its so easy to give other ppl work. :D

I actually started working on (preparations for) resync already :D
Indeed I'm targetting resync in seconds, not minutes.

imbaczek wrote:An idea: if Licho says he can 100% reproduce desyncs, maybe make a version that keeps a cyclic buffer of state (lots of data, but not THAT much I hope...) and right after first desync happens, kill both games and find the first difference. Not easy, but IMHO possible.

I've already implemented that (the sync debugger), it's just not compiled in the releases because the buffer I use is about 200 megs. It is an idea tho to actually build a version with it compiled in and give it to Licho... I might do that sometime...

imbaczek wrote:Or a less data-intesive trace method - after a desync is detected, calculate several checksums of different parts of state and try to find desynced data by some kind of bisection.

That's what I plan for resync. It's actually the reason I switched target from fixing sync errors to writing resync, because the latter also gives me a second powerful sync debugging tool basically for free.

MadRat wrote:I have a feeling its all the small units in the mods I playtest. Mostly it happens with the tiny soldier units in mods like 1944 and world domination.

Hmm, interesting.

DemO · Post by **DemO** » 31 Dec 2006, 18:17

Good luck Tobi, this would really help spring.

LordMatt · Post by **LordMatt** » 31 Dec 2006, 18:56

Tobi wrote: I've already implemented that (the sync debugger), it's just not compiled in the releases because the buffer I use is about 200 megs. It is an idea tho to actually build a version with it compiled in and give it to Licho... I might do that sometime...

I would also be happy to run such a build on my main machine. I have 2GB of RAM there.

AF · Post by AF » 31 Dec 2006, 19:07

madrat that sounds like Delayed Sync message and not Desync messages, afterall how cna you desync with yourself if only 1 simulation is running?

el_matarife · Post by **el_matarife** » 31 Dec 2006, 23:57

Does anyone even use VIA C3s for gaming? Their FPU was TERRIBLY slow last time I checked, like 1/2 the clock speed which is only like 1-1.5ghz. Any post 500 mhz CPU besides the Via and Transmeta chips should work with SSE. Look at Valve's Steam hardware survey results, 99% of people have it now. http://www.steampowered.com/status/survey.html

Anyway, if the sync fixer isn't done next patch, at least make the game pause itself instantly on desync so we can kick the broken players, or decide to restart without the sync errors getting bigger. Not to mention a little HUD display of desync would be better so it wouldn't flood the chat box when we try to make plans about what to do.

CautionToTheWind · Post by **CautionToTheWind** » 01 Jan 2007, 03:23

Just for clarity, when i said rsync i meant usage of the rsync algorithm that can be found here: http://www.samba.org/rsync/tech_report/

The algorithm identifies parts of the source file which are identical to some part of the destination file, and only sends those parts which cannot be matched in this way. Effectively, the algorithm computes a set of differences without having both files on the same machine. The algorithm works best when the files are similar, but will also function correctly and reasonably efficiently when the files are quite different.

An important feature of rsync not found in most similar programs/protocols is that the mirroring takes place with only one transmission in each direction.

And of which mature GPL implementation exist for the taking.

Spring RTS Engine

Sync errors