Improvements for rapid/http-downloads

[abma]

I'm still thinking about how files could be transfered more efficient.

rapid is nice, but it's hard to distribute/mirror, as it contains thousands of files and it produces a high server load.

Best thing i found is zsync: http://zsync.moria.org.uk/
its something like rsync over http. Requirements on server side is only the file <originalfile>.zsync that is created by zsyncmake.

its not so efficient like rapid, but it produces zero load on the server side and can be mirrored easily.

I tested it with zk-v0.8.0 and zk-v0.8.4.4.

Zsync detected 84.9% as already downloaded, so download size was only about 21MB for an update.

To test yourself:

zsync http://springfiles.abma.de/zsync/zk-v0. ... .sdz.zsync

and the update run:

zsync http://springfiles.abma.de/zsync/zk-v0. ... .sdz.zsync -i zk-v0.8.0_0.sdz

I know, this requires a working versioning detection of the files.

and suggestions/thoughts about that?

[SinbadEV]

that's kinda awesome... it would make a good additional option at the very least.

[Licho]

What are you talking about, high server load? Have you seen streamer? It does not send "thousand of files" it sends one stream and yes you can distribute it easilly around multiple mirrors if mirroring software is smart!
When its mirrored, just put streamer there.

Furthermore, rapid reuses over 99% of content for ZK versions you mention and allows shared content to also be stored in this shared form (no duplicated content stored, even parts of different games are shared).
Thats a MAJOR bonus if you deal with hundreds of thousands versions of the game.

Its tailored for spring and works extremely well, so im saying - waste of work to try to force some other system with obviously lower performance.

Its not "awesome" its kind of "poor" compared to what we have atm.

[SinbadEV]

Its not "awesome" its kind of "poor" compared to what we have atm.

Sorry, I don't know anything about rapid/streamer so I can't comment on "relative awesomeness".

I meant that the technology/idea was awesome... especially the part where there is no additional server load beyond the initial metafile generation. I also like how it works for updates of compressed data while rapid only works on decompressed files (though I seem to recall that it compresses them before being transmitted or something).

[Licho]

Rapid is essentially the same, it can also work only with "metadata" and ordinary http hosting. But as well as this, it needs preprocessing (in rapid's case basically unpacking files from sdz archive and calculating their hash).

This ensures good reusability because only modified files are transmitted andonly if they dont match say historic version or file from other mod already on client.

For improved performance there is simple streamer program which sends all requested files in one http request to the client.

[FLOZi]

If rapid could unpack sd7 it would be even better. Nudgey winkey.

[abma]

@licho:
yes, you're right, this solution isn't worth to implement it.

but this should be possible and solve most problems with rapid:
make a cache-mirror:

client asks for files on cache-mirror, if cache mirror has not this file, download it from master + send to client. the script could be also in php, so requirements are pretty low for the cache-mirror.

should be easy to implement + should work with small modifications on client side (or without, if client uses only one mirror).

DNS with round-robin would be possible, too, but if one of the mirrors is offline.... that would be bad, too.

[Licho]

You dont need any such complicated thigns imo..
simply make streamer to support HTTP range header..

Rapid client can then use several mirrors at once in multipart http download - pretty ordinary thing used to download normal files already.

Streamer should support some caching in that case though (as its likely to be rerequested for same stream soon).

But I dont think we need that .. rapid was made to minimize transferred data, assuming most downloads are from people who just upgrade their existing stuff it should be fine (=> not worth using mirrors if the download is 100kb).

[Forboding Angel]

Not to mention the ginormous amount of things that rapid can do, e.g. make sdz, make sdd, upgrade, clean upgrade (which btw... seriously, zk needs to start using this), etc etc etc.

[hoijui]

forb, that are mostly client features (of tobis rapid client), while this discussion is about rapid protocol (not that this matters much).

what we would need for rapid is an easier way for non-ZK devs to have a say in the protocol ... thats what i remember SL devs wanted, but i also remember there was a discussion about this already...

[koshi]

Not speaking for BrainDamage, but I personally won't care one bit about rapid protocol (or any other downloader) details once we've switched to using abma's lib.

[dansan]

DNS with round-robin would be possible, too, but if one of the mirrors is offline.... that would be bad, too.

_If_ you set up DNS with RR, then you _should_ have a prog that detects offline mirrors, and takes them out of the rotation.

[Licho]

The system is done and runs without changes what 3 years?

So why would you discuss anything? Its pretty easy to implement too (easier than say torrent download) and as koshi said abma made it into library for lobby.
So SL can start using it too.

Nobody from ZK team touched it for years and I certainly dont plan to change it unless there is good reason.
As the saying goes: don't fix what isnt broken.

Basic procol and idea is even older, dating to old caupdater, perhaps more than 4 years old.

[SinbadEV]

could zsync be used to improve the functionality of the existing rapid protocol? for example if a map was updates to fix the locations of some features this would only affect about %10 of the smf file, with rapid it would transfer the entire changed file while zsync would theoretically only transfer the differences... would only really improve performance in a very small number of situations (like changes to SMF or textures) but might be useful in the long run for total server load.

[hoijui]

i make enlightenment: changes != fixes

[Licho]

Indeed it would improve map download, but rapid is not used for maps atm, just fo mods. Im not sure if its worth versioning maps.

[SirMaverick]

Indeed it would improve map download, but rapid is not used for maps atm, just fo mods. Im not sure if its worth versioning maps.

Less worth than for mods as there are less versions. But if done right it can reduce network load as it does for game updates.

[zwzsg]

Mappers don't care about filesizes anyway.

[AF]

tbh Rapid appears to do its job, any system that adds features should be built on a layer above Rapid.

As such, I don't see what exactly there is to improve and nobody has quite definitively stated an improvement that can be stated nice and neatly without prior knowledge

[abma]

@licho:

rapid runs on only one server, i don't want to see what happens, when some game gets slashdoted and all users download full files through rapid.

would it make sense to add maps to rapid? updates of maps often only changes features...

and there are still some feature-requests, for example this one:

http://code.google.com/p/zero-k/issues/detail?id=304

but this should be done by plasma or the xml-rpc api.