Patch delivery, Sync Problems

[lurker]

Go code it and it'll be added in a few minutes. PM not necessary?

[Argh]

I'll look into that when I get a chance.

[YokoZar]

Syncing is done based on only filenames (full paths including directories) and contents of files. Timestamps, empty directories, order of files in archive or even archive format itself (sd7, sdz, sdd, ufo all sync with each other) are all irrelevant.

Out of curiosity does this include the case of the filename? I'm wondering if some of the "I zipped the exact same thing and got a different checksum" is coming from FOO.lua being autoconverted to foo.lua and the linux clients actually caring.

[det]

Syncing is done based on only filenames (full paths including directories) and contents of files. Timestamps, empty directories, order of files in archive or even archive format itself (sd7, sdz, sdd, ufo all sync with each other) are all irrelevant.

Out of curiosity does this include the case of the filename? I'm wondering if some of the "I zipped the exact same thing and got a different checksum" is coming from FOO.lua being autoconverted to foo.lua and the linux clients actually caring.

It's not case-sensitive.

[Argh]

We ran into a major problem on Thursday.

Seems that if you remove a file from a build that was previously syncing, then it will still sync with other builds. Ripped a Gadget out of a build on one machine, and it still synced with the other build.

So, what triggers UnitSync to rebuild ArchiveCache, exactly? This is a non-trivial issue, obviously.

[lurker]

When the modification date on the file changes, it gets scanned again. The complication comes in about *when* it changes on an sdd. I think windows will change the mod date on a folder when you change something in the folder, but not if you change something in a subdirectory.

This is a non-trivial issue, obviously.

It is? The end user shouldn't have any mod files get edited.

[Argh]

The sync system's pretty worthless, if it can't even catch that a file is present in one build, but is not present in another, imo.

So... let's review. I'm getting a little frustrated here. Telling me that I have a non-problem isn't cutting it, guys. Thus far, I see that either I do complete builds, or I go to SDD, in which case I can expect all sorts of "impossible" stuff to desync players.



1. It appears that version numbers don't trigger rebuild of the hash for a given archive. IOW, changing that internally doesn't seem to cause Archivecache to be rebuilt as expected, when that should trigger a complete rebuild every time. Just making that happen would take care of 90% of the problem, unless a maintainer forgets to update the version strings, in which case it's their fault anyhow.

2. You can delete a file from one build, and it won't desync with another build, because this process is made dependent on the file-creation date of the root folder.

3. There are various processes in Windows that can cause the file-creation date to differ from one install to the next. I can't prove that this is causing desync between builds that should otherwise be binary-identical, but that's a possibility, and would explain a few things.

4. When building the hash, it's probably taking into account junk files such as thumbs.db, which can be updated by Windows when an install occurs under certain circumstances. This may also cause desync, and if these junk files aren't being excluded, they should be.

5. Because /mods and /maps are cached together, the rebuild of the hashes from start takes bloody forever. Hence the understandable desire to make it infrequent.

[lurker]

The sdd sync system is worthless.

either I do complete builds, or I go to SDD

Do you mean single-zip builds? What errors have you had with multiple zips?


1. It doesn't read the version unless it's rescanning.
2. Yes.
3. No.
4. Yes. Tell me Argh, how is spring supposed to detect 'junk' files?
5. Yeah, it's slow if you delete the cache, but you should never have to delete the entire cache file. But worse than that was in the past, when it would do a full rescan of every sdd every update, taking forever, and even that would fail on automatic OS-made files.

Imo the best way to handle things is to completely remove sync checking on sdds.

[Argh]

If I change one tree, the entire forest should not become invalid. SDZ sync creates this problem, in spades.

One change to any file in a content module, and players need to re-download a file that's anywhere from 50MB up, or they will not be able to sync. Whereas I could just send them that one file in an installer, and voila, we're in sync again, if the system worked like it should.

The problem here is not that SDD isn't a good way to build easily-maintained archives. The problem here is that the sync system is checking the wrong stuff, is not checking when it needs to, and has mysterious failures to correctly determine that files are binary-identical or not.

[lurker]

just send them that one file

And do what with it?

The problem here is not that SDD isn't a good way to build easily-maintained archives. The problem here is that the sync system is checking the wrong stuff, is not checking when it needs to, and has mysterious failures to correctly determine that files are binary-identical or not.

When sdd isn't involved, what does it check wrong, or fail to check?

And iirc you never showed any evidence of errors with identical files, or said how to reproduce.

[Argh]

And do what with it?

And install it to the correct sub-directory, of course! Which should trigger re-sync. If sync isn't checking #files and creationdates of the files, it's not really doing anything except causing problems.

Meh, I could do something like wipe out archivecache when I send out builds, I suppose... but that's like using a Buick to swat a fruitfly.

[lurker]

It takes ages to scan an sdd for modified files. When you said that the problem wasn't "that SDD isn't a good way" I thought you weren't using sdd in this scenario.

sdd is not a viable option, because of metafiles created merely by viewing the content, and because of how slow it is. sdd is for LOCAL DEV WORK. So, ignoring sdd, what does spring do wrong?

[Pxtl]

I'm curious: what's wrong with using a tonne of SDZ files? Like, one SDZ per-file for any large objects, and then use another SDZ for all the text data, with a really long modinfo.tdf referencing all the dependancies?

[lurker]

Probably not much. But put them in a subfolder if there are a lot. However, going all the way to per-file strikes me as a waste.

[Argh]

Maintenance of such a thing, without a tool to auto-build that list of dependencies, would be improbable.

Meh, most of this is not terribly important, in a commercial release. Most distributors will produce binary patches for your games and distribute them to customers to keep them up to date, and they maintain a CRC check over everything.

I could do that with Windows users already, although I'm not going that way atm, there's little point when 90% of the downloads are for full installers anyhow. The "bare bones" thing is going the way of the dodo fairly soon, it was mainly done for convenience of a small number of people around here anyhow.

I'm just hoping there's a solution that combines easy maintenance with reliability and lack of issues during QA or player testing, and the answer seems to be no, at least for me. That makes release early, release often an impractical goal for anybody maintaining large games unless they're really willing to put huge time into it, which is a fairly terrifying barrier to entry, though. One thing that might save the day at some later point is for the current CA updater tool to support binary patchers from any game, and a common point to upload them to.

[lurker]

You only need a simple script to build update packages, and SD already supports what you want just fine.

[det]

You claim the sync system is flawed, yet CA has been doing incremental updates for more than a year and every possible scenario involving adding/removing/modifying files has happened. There has never been any issues, whether it be things syncing when they shouldn't, or things not syncing when they should. If your issue is only with sdd, then you need to be more specific. Sdd will never be a viable choice for end users. If you are claiming there are syncing issues with sdz or sd7 then I think you are wrong.

[Argh]

Well, CA's delivered as one solid archive, which you're patching with a binary patcher delivered via your torrent system. Keeps things simple. I don't have those options, the game's a lot more invasive and involves files in more places. So I have seen a lot more issues than you guys have.

I really don't have any more time for this. Moving on.

[det]

Well, CA's delivered as one solid archive, which you're patching with a binary patcher delivered via your torrent system. Keeps things simple. I don't have those options, the game's a lot more invasive and involves files in more places. So I have seen a lot more issues than you guys have.

I really don't have any more time for this. Moving on.

CA doesn't use a binary patcher and it doesn't use the torrent system. I don't see any reason at all that CA's system wouldn't work for you.

[Tobi]

When the modification date on the file changes, it gets scanned again. The complication comes in about *when* it changes on an sdd. I think windows will change the mod date on a folder when you change something in the folder, but not if you change something in a subdirectory.

On Linux, the directory timestamp only changes when a file in the directory (not a subdirectory) is renamed, moved or deleted.

1. It appears that version numbers don't trigger rebuild of the hash for a given archive. IOW, changing that internally doesn't seem to cause Archivecache to be rebuilt as expected, when that should trigger a complete rebuild every time. Just making that happen would take care of 90% of the problem, unless a maintainer forgets to update the version strings, in which case it's their fault anyhow.

Hmm, always reading modinfo.tdf / modinfo.lua for .sdd's might be a way to get reasonably proper checksum calculation without an unbearable performance overhead.

Probably needs quite a lot of internal rework though to actually make it as well performing as it is now. (Just opening an .sdd archive currently makes Spring recursively go through all files and stat() them.)

3. There are various processes in Windows that can cause the file-creation date to differ from one install to the next. I can't prove that this is causing desync between builds that should otherwise be binary-identical, but that's a possibility, and would explain a few things.

I can probably prove this can't cause desync

(For example: file creation dates on springcontent.sdz, bitmaps.sdz, cursors.sdz and maphelper.sdz archives (and their content!) is different on every manually built Linux version of Spring, and yet they all do sync.)

4. When building the hash, it's probably taking into account junk files such as thumbs.db, which can be updated by Windows when an install occurs under certain circumstances. This may also cause desync, and if these junk files aren't being excluded, they should be.

There's springignore.txt for that. Just put names like thumbs.db in that.

5. Because /mods and /maps are cached together, the rebuild of the hashes from start takes bloody forever. Hence the understandable desire to make it infrequent.

That's why .sdd's aren't cached everytime. Full rebuild of the archive cache takes at most a minute here (probably closer to 30 sec anyway) for about 7 gigs worth of content...

And do what with it?

And install it to the correct sub-directory, of course! Which should trigger re-sync. If sync isn't checking #files and creationdates of the files, it's not really doing anything except causing problems.

Performance, see above. I think it were modders in the first place who complained that starting Spring would take ages when they had many .sdd archives in their mods folder. Ironic it's now a modder again who wants the uberslow behaviour back

Meh, I could do something like wipe out archivecache when I send out builds, I suppose... but that's like using a Buick to swat a fruitfly.

Just update the timestamp on the .sdd directory. That triggers rescan of the archive on next unitsync/Spring load and hence an update to the checksum.