View Issue Details

IDProjectCategoryView StatusLast Update
0001019Spring engineGeneralpublic2008-09-04 23:15
Reporterimbaczek Assigned ToAuswaschbar  
PrioritynormalSeverityblockReproducibilityalways
Status resolvedResolutionfixed 
Product Version0.76b1+svn 
Summary0001019: major desyncs in all mods
Descriptionright from the start.
TagsNo tags attached.
Attached Files
20080822_224112_SmallDivide_0.76b1+.sdf (Attachment missing)
20080822_224133_SmallDivide_0.76b1+.sdf (Attachment missing)
desu-smalldivide1.sdf (Attachment missing)
fatality-20080822_224132_SmallDivide_0.76b1+.sdf (Attachment missing)
bibim-20080822_225057_SmallDivide_0.76b1+.sdf (Attachment missing)
host_20080824_210838_SmallDivide_0.76b1+.sdf (Attachment missing)
client_20080824_210843_SmallDivide_0.76b1+.sdf (Attachment missing)
Checked infolog.txt for Errors

Relationships

related to 0001017 closed some commands don't get transmitted in CA r2697 

Activities

jK

2008-08-22 23:13

developer   ~0002486

r6303 + ca 2697

imbaczek

2008-08-22 23:27

reporter   ~0002488

the replays are kinda broken, too; only one player on each team works, and there's error spam about invalid startpos messages right from the start.

bibim_

2008-08-23 12:54

reporter   ~0002502

I tried replaying my demo with r6308. Now all players are here, but the demo doesn't match what I saw and did in game when playing (lots of commands skipped :/)

imbaczek

2008-08-23 13:11

reporter   ~0002504

I can't reproduce this reliably on lan with 4 players and 2 machines, with or without 6243+6262. It only happened once, when not needed...

Maybe latency is an issue here?

Auswaschbar

2008-08-24 18:49

reporter   ~0002512

I did some testing:
Not reproducable with two instances of the same build. Even if I add much latency and packet loss (networking works fine).

I can 100% reproduce desyncs with:
Run a self-compiled and a windows version from buildserv with BA
-> build an core anti-air-tower
-> build some fighters (every plane works)
-> let them flew over aa gun
-> desync

imbaczek

2008-08-24 21:11

reporter   ~0002513

reproduced with two exactly same builds, with much less elaborate setup; sometimes, you don't need to do _anything_, it desyncs on frame 1 (CA r2697). Not sure if there's any useful information in replays, but posting them anyway...

Kloot

2008-08-24 21:36

developer   ~0002514

I've also had it desync with identical builds on frame 1 (before any user commands are ever transmitted), so we're probably / hopefully just looking at uninitialized data reads.

Kloot

2008-08-25 13:53

developer   ~0002515

Last edited: 2008-08-25 13:57

I traced the source of the desync in your (imbaczek) latest demos to a crucial difference in the setup scripts:


Client:
    [TEAM0]
    {
        TeamLeader=0;
        AllyTeam=0;
        RGBColor=0.07843 0.07843 0.96078;
        Side=ARM;
        Handicap=0;
    }
    [TEAM1]
    {
        TeamLeader=1;
        AllyTeam=1;
        RGBColor=0.78431 0.00000 0.00000;
        Side=ARM; <======================== !
        Handicap=0;
    }


Server:
    [TEAM0]
    {
        TeamLeader=0;
        AllyTeam=0;
        RGBColor=0.07843 0.07843 0.96078;
        Side=ARM;
        Handicap=0;
    }
    [TEAM1]
    {
        TeamLeader=1;
        AllyTeam=1;
        RGBColor=0.78431 0.00000 0.00000;
        Side=CORE; <======================== !!
        Handicap=0;
    }


Ironically I didn't notice this by just watching them, I had to hack the sync checker to find that the point of divergence was actually somewhere in CCommanderScript::GameStart() (which didn't leave much room for interpretation). Bottom line: always compare the scripts when faced with a
very early sync error like this!

imbaczek

2008-08-25 14:00

reporter   ~0002517

Last edited: 2008-08-25 14:00

well duh! /me fails. sorry for your wasted time. (commit synccheker hacks, tho ;p)

maybe implement script.txt (or just sycned team/player data) transfer and/or checksumming? something to consider after release.

Kloot

2008-08-25 19:07

developer   ~0002521

Last edited: 2008-08-25 23:14

Yeah, good idea. Only problem with checksumming the scripts directly is that they can differ wrt. case (ex. "Arm" vs "ARM" vs "arm") and formatting (ex. "[GAME] {" vs "[GAME]\n{"), it would need to be done over the parsed key/value pairs.

I've also just examined the demos of that 3v3 game on SmallDivide with a self-compiled build of r6316, all of them play out exactly the same way as far as I can tell (locally calculated simframe checksums are equal across each demo for the first 5 minutes at least). I don't think we're going to extract any useful information out of them, it's impossible to know why fi. Crayfish desynced as early as frame 32 from just the original warning message (maybe it is build-specfic like Auswaschbar's post suggests).

Auswaschbar

2008-08-29 01:05

reporter   ~0002531

I can now 100% reproduce a desync with two different builds, every time in frame 32. No orders are given, it jsut desyncs while running.

I have the suspicion that in frame 32 (31) something reads an unsynced variable and writs it to synced. Anyone have an idea what runs in frame 32?

imbaczek

2008-08-29 09:29

reporter   ~0002539

Isn't that the first SlowUpdate?

Kloot

2008-08-29 13:43

developer   ~0002541

Not for units; they follow a staggered update pattern in which the first (activeUnits.size() / 16) units are SlowUpdate()'d when !(gs->frameNum & 15) == true (so every 16th frame, starting at frame 0), then the second (activeUnits.size() / 16) units on the next frame, etc. CUnit::SlowUpdate() in turn SlowUpdate()'s the unit's commandAI, movetype, and weapons.

Teams are SlowUpdate()'d every 32 frames though (the only class in Spring that is).

Auswaschbar

2008-09-01 16:52

reporter   ~0002549

It's gone away. I tried it with buildserv-build from wine and my local build, but no desync any more. Anyone here still getting sync errors?

Kloot

2008-09-01 17:17

developer   ~0002550

I've unfortunately never been able to trigger it with any trunk revision >= r6320 (r6320 was the earliest I tested with, my own gcc 4.1.3 Linux build vs. bibim's crosscomp buildserv-build run in wine), so I can't confirm that the desync is really gone (just that I haven't seen it). It would be interesting to know which exact rev fixed things though, especially because of the 100% reproducability (there were very few synced code changes recently, so should be doable).

imbaczek

2008-09-03 00:09

reporter   ~0002553

i cannot reproduce a desync between two boxes, one compiled with gcc 4.3.2, another with bibim's build. needs further testing in 4+ player matches.

Auswaschbar

2008-09-04 23:15

reporter   ~0002558

After some testing, we could finally track this down and resolve this one

Issue History

Date Modified Username Field Change
2008-08-22 23:12 imbaczek New Issue
2008-08-22 23:13 jK Note Added: 0002486
2008-08-22 23:14 imbaczek File Added: 20080822_224112_SmallDivide_0.76b1+.sdf
2008-08-22 23:14 jK File Added: 20080822_224133_SmallDivide_0.76b1+.sdf
2008-08-22 23:18 imbaczek File Added: desu-smalldivide1.sdf
2008-08-22 23:19 imbaczek File Added: fatality-20080822_224132_SmallDivide_0.76b1+.sdf
2008-08-22 23:27 imbaczek Note Added: 0002488
2008-08-22 23:51 imbaczek Summary major desyncs in CA => major desyncs in all mods
2008-08-22 23:52 imbaczek Severity minor => block
2008-08-23 12:49 bibim_ File Added: bibim-20080822_225057_SmallDivide_0.76b1+.sdf
2008-08-23 12:54 bibim_ Note Added: 0002502
2008-08-23 13:11 imbaczek Note Added: 0002504
2008-08-24 16:02 imbaczek Relationship added related to 0001017
2008-08-24 18:49 Auswaschbar Note Added: 0002512
2008-08-24 21:11 imbaczek Note Added: 0002513
2008-08-24 21:12 imbaczek File Added: host_20080824_210838_SmallDivide_0.76b1+.sdf
2008-08-24 21:12 imbaczek File Added: client_20080824_210843_SmallDivide_0.76b1+.sdf
2008-08-24 21:36 Kloot Note Added: 0002514
2008-08-25 13:53 Kloot Note Added: 0002515
2008-08-25 13:53 Kloot Status new => feedback
2008-08-25 13:55 Kloot Note Edited: 0002515
2008-08-25 13:57 Kloot Note Edited: 0002515
2008-08-25 14:00 imbaczek Note Added: 0002517
2008-08-25 14:00 imbaczek Note Edited: 0002517
2008-08-25 19:07 Kloot Note Added: 0002521
2008-08-25 19:11 Kloot Note Edited: 0002521
2008-08-25 22:18 Kloot Note Edited: 0002521
2008-08-25 22:31 Kloot Note Edited: 0002521
2008-08-25 23:14 Kloot Note Edited: 0002521
2008-08-25 23:14 Kloot Note Edited: 0002521
2008-08-29 01:05 Auswaschbar Note Added: 0002531
2008-08-29 09:29 imbaczek Note Added: 0002539
2008-08-29 13:43 Kloot Note Added: 0002541
2008-09-01 16:52 Auswaschbar Note Added: 0002549
2008-09-01 17:17 Kloot Note Added: 0002550
2008-09-03 00:09 imbaczek Note Added: 0002553
2008-09-04 23:15 Auswaschbar Status feedback => resolved
2008-09-04 23:15 Auswaschbar Resolution open => fixed
2008-09-04 23:15 Auswaschbar Assigned To => Auswaschbar
2008-09-04 23:15 Auswaschbar Note Added: 0002558