Patch delivery, Sync Problems

[Argh]

Ok, as most of you know, I've released P.U.R.E. RC4. Due to my time-crunch and inadequate QA, a problem I'm sure you're all familiar with, I need to deliver patches to players over the next week or so.

Here's the problem. In tests with the "bare bones" installer, it wouldn't sync with the full installer. Unitsync's screwing up the hashes, because when the files are extracted from the zip archive, they get different file-creation dates, which is what Unitsync's using to make the hashes.

I would like, in an ideal world, to put up a fixed version of the full installer (call it, 4.1) with all bugfixes pre-applied, and I'd like to also release separate SDZs in a small zip archive, for people who have already downloaded the full installer. I'd also like to resolve the issues with the bare-bones, as that's the installer I'd like to put up on Jobjol and make available via torrent.

Can anybody help me resolve these issues? I hate the idea of having to make people re-download a 500MB installer, I am annoyed that I'm effectively leaving Linux players in the lurch, and I'd like solutions, but I'm not sure how to proceed here.


Also! If I need to selectively replace a file that's in another archive- how do I make sure that the patch version overwrites the non-patch version? I.E., if I make a change to a .S3O, but I don't want people to have to download the entire 250MB PURE_CONTENT.sdz, can this be done?

[koshi]

I've recently built an ubuntu package for S44 that depends on the avail. spring ubuntu packages. It's kinda simplistic, just putting modarchive and some maps where they'll be found by unitsync. It's not modular, meaning you can't update just the maps for example. Phasing components out into seperate packages and combining them into a meta package should be easy enough tho.
I could provide you with the necessary files to get a package for PURE started, in fact i should have the whole shebang (including s44 windows installer stuff) somewhere in a git repo.

[Argh]

Meh. Right now, the immediate "hump" is how to deal with the file-creation date issue, which is what's preventing sync. Is there an archive pack / unpack that I should be using, that preserves the creation dates when unpacked?

[lurker]

Argh, start over from the top, what exactly did you do to make the two different test environments, because I can tell you that your guess about file dates is wrong. Did you rebuild a giant 4.1 sdz on one end, and have multiple archives on the other? Because that method makes the checksum math work differently.

[Auswaschbar]

Here's the problem. In tests with the "bare bones" installer, it wouldn't sync with the full installer. Unitsync's screwing up the hashes, because when the files are extracted from the zip archive, they get different file-creation dates, which is what Unitsync's using to make the hashes.

U sure with that? Afaik ca-installer generates the archives on the fly, so it definitely has different creation times, but they still sync.

[Argh]

Afaik ca-installer generates the archives on the fly, so it definitely has different creation times, but they still sync.

All I know is what I observed. And I know that I used the exact same SDZs in both installers. My guess is that ca-installer won't sync between two versions made at different times, even if they're otherwise identical.



1. I have a huge installer, which seems to preserve the file-creation dates when it unpacks, and syncs correctly.

2. I have "barebones", which is just a zip archive with a bunch of SDZs (the exact same SDZs in the installer).

When unzipped, at least under Vista they had different creation dates, and would not sync.

3. I need to deliver patches, which will need to overwrite certain files, so that I don't have to make people re-download giant archives because one small thing was changed.

I need to know what order I need to do things- whether that's the order that the master loads all of the dependencies (see PURE_COMPLETE.sdz for what I'm talking about, I've done what I said I'd do about building an example of modularizing our deliveries to clearly deal with license issues) or whatever, so that all players have exactly the same stuff, and bugfixes work.

All three problems are major obstacles to delivering patches, which is a primary need going forward.

I've gtg for now, I'll bbl, see what can be done after I'm done writing the initial patch.

[lurker]

The only thing modification dates are used for is whether unitsync rescans a file. Do they still differ if you delete archivecache?

[Argh]

They seemed to, yes, but maybe we weren't thorough enough. Meh, I'd been up until 7AM that morning, got maybe 5 hours of sleep, and DRB wanted to commence testing right away, and was grouchy that it didn't work right out of the box, which is understandable.

At any rate... let's assume, for the sake of argument, that I screwed up somewhere, and the two sets of SDZs are not identical. Let's assume that I fix that issue, and that takes care of sync.

Now... assuming all of that... how do I deliver a patch that will overwrite previous material? Just load the patch file first, or second, when going down the list of dependencies? Or is it potluck, and I'm stuck?

[lurker]

Each file overwrites the last; for example, to turn a unit red you only need an sdz with two files, a modinfo depending on the original archive, and the new texture.

[Argh]

Ok, so if patch files are last in the line of dependencies, it should work. Cool, that makes that part a lot more... relaxed.

[yuritch]

We made S'44 testing releases like that - there were some 'base' sd7 files (textures, sounds, models even) that weren't likely to change, and then there was an sd7 with fbi/scripts/etc that changed a lot, but it was small (those text files compress rather well). So a next testing build just depended on the previous (a-la mod mutator), and files in the new build overridden similar files in the previous. It synced ok in my experience (the only sync problems were while using sdd). Side effect was that players had a list of S'44 versions available at once (because 'old' testing archives counted as mods, too).

[SpliFF]

Now that SpringLobby looks set to become the primary lobby it would be interesting to build an incremental update mechanism right into it. It already has a bittorrent system so I could see that being extended to downloading patches which contain only the files that changed between versions. The system would then add/replace or patch the previously downloaded data as required.

The advantage over yuritch's method is that the patched sdz would have the same checksum as a fresh jobjol download so desyncing wouldn't be an issue and old versions wouldn't hang around in the mod list.

[Argh]

According to Eman, the ModDB full installer of P.U.R.E. RC5 and the various smaller installers will not sync.

This should be impossible. They're literally the same collection of SDZs. The install of Spring over here should be identical to everybody else's, and people downloading the ModDB version are using standard Spring.

[imbaczek]

add

Code: Select all

LogSubsystems=ArchiveScanner

to springsettings.cfg, compare checksums of both installs.

[Argh]

Okie doke. IDK what could be causing this, though, unless people have been trying to upgrade RC3 to RC5, which I told them not to do.

I literally just zipped up the maps, mods, and music directories from the exact same environment of the install, and my Spring install has synced with more than one player online with RC4+.

[det]

Syncing is done based on only filenames (full paths including directories) and contents of files. Timestamps, empty directories, order of files in archive or even archive format itself (sd7, sdz, sdd, ufo all sync with each other) are all irrelevant.

[Argh]

Then it should work. But it doesn't. I guess I'll try a totally clean install and see.

[KDR_11k]

Here's the problem. In tests with the "bare bones" installer, it wouldn't sync with the full installer. Unitsync's screwing up the hashes, because when the files are extracted from the zip archive, they get different file-creation dates, which is what Unitsync's using to make the hashes.

U sure with that? Afaik ca-installer generates the archives on the fly, so it definitely has different creation times, but they still sync.

My browser sets the creation date of anything downloaded to the current time and downloading sd*s works fine.

[det]

Here's the problem. In tests with the "bare bones" installer, it wouldn't sync with the full installer. Unitsync's screwing up the hashes, because when the files are extracted from the zip archive, they get different file-creation dates, which is what Unitsync's using to make the hashes.

U sure with that? Afaik ca-installer generates the archives on the fly, so it definitely has different creation times, but they still sync.

My browser sets the creation date of anything downloaded to the current time and downloading sd*s works fine.

I think they are talking about timestamps in the archive, of individual files. I don't think they are talking about the timestamps in your filesystem of the archive itself.

[Tobi]

Timestamps inside archives (all types) are not used at all by Spring.