Improvements for rapid/http-downloads

Improvements for rapid/http-downloads

For the discussion of infrastructure improvements and changes.

Moderator: Moderators

abma
Spring Developer
Posts: 3540
Joined: 01 Jun 2009, 00:08

Improvements for rapid/http-downloads

Post by abma » 15 Apr 2011, 10:35

I'm still thinking about how files could be transfered more efficient.

rapid is nice, but it's hard to distribute/mirror, as it contains thousands of files and it produces a high server load.

Best thing i found is zsync: http://zsync.moria.org.uk/
its something like rsync over http. Requirements on server side is only the file <originalfile>.zsync that is created by zsyncmake.

its not so efficient like rapid, but it produces zero load on the server side and can be mirrored easily.

I tested it with zk-v0.8.0 and zk-v0.8.4.4.

Zsync detected 84.9% as already downloaded, so download size was only about 21MB for an update.

To test yourself:

zsync http://springfiles.abma.de/zsync/zk-v0. ... .sdz.zsync

and the update run:

zsync http://springfiles.abma.de/zsync/zk-v0. ... .sdz.zsync -i zk-v0.8.0_0.sdz

I know, this requires a working versioning detection of the files.

and suggestions/thoughts about that?
0 x

User avatar
SinbadEV
Posts: 6475
Joined: 02 May 2005, 03:56

Re: Improvements for rapid/http-downloads

Post by SinbadEV » 15 Apr 2011, 15:22

that's kinda awesome... it would make a good additional option at the very least.
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Improvements for rapid/http-downloads

Post by Licho » 15 Apr 2011, 16:35

What are you talking about, high server load? Have you seen streamer? It does not send "thousand of files" it sends one stream and yes you can distribute it easilly around multiple mirrors if mirroring software is smart!
When its mirrored, just put streamer there.

Furthermore, rapid reuses over 99% of content for ZK versions you mention and allows shared content to also be stored in this shared form (no duplicated content stored, even parts of different games are shared).
Thats a MAJOR bonus if you deal with hundreds of thousands versions of the game.

Its tailored for spring and works extremely well, so im saying - waste of work to try to force some other system with obviously lower performance.

Its not "awesome" its kind of "poor" compared to what we have atm.
0 x

User avatar
SinbadEV
Posts: 6475
Joined: 02 May 2005, 03:56

Re: Improvements for rapid/http-downloads

Post by SinbadEV » 15 Apr 2011, 16:43

Licho wrote:Its not "awesome" its kind of "poor" compared to what we have atm.
Sorry, I don't know anything about rapid/streamer so I can't comment on "relative awesomeness".

I meant that the technology/idea was awesome... especially the part where there is no additional server load beyond the initial metafile generation. I also like how it works for updates of compressed data while rapid only works on decompressed files (though I seem to recall that it compresses them before being transmitted or something).
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Improvements for rapid/http-downloads

Post by Licho » 15 Apr 2011, 16:52

Rapid is essentially the same, it can also work only with "metadata" and ordinary http hosting. But as well as this, it needs preprocessing (in rapid's case basically unpacking files from sdz archive and calculating their hash).

This ensures good reusability because only modified files are transmitted andonly if they dont match say historic version or file from other mod already on client.

For improved performance there is simple streamer program which sends all requested files in one http request to the client.
0 x

User avatar
FLOZi
MC: Legacy & Spring 1944 Developer
Posts: 6100
Joined: 29 Apr 2005, 01:14

Re: Improvements for rapid/http-downloads

Post by FLOZi » 15 Apr 2011, 17:12

If rapid could unpack sd7 it would be even better. Nudgey winkey.
0 x

abma
Spring Developer
Posts: 3540
Joined: 01 Jun 2009, 00:08

Re: Improvements for rapid/http-downloads

Post by abma » 15 Apr 2011, 19:13

@licho:
yes, you're right, this solution isn't worth to implement it.

but this should be possible and solve most problems with rapid:
make a cache-mirror:

client asks for files on cache-mirror, if cache mirror has not this file, download it from master + send to client. the script could be also in php, so requirements are pretty low for the cache-mirror.

should be easy to implement + should work with small modifications on client side (or without, if client uses only one mirror).

DNS with round-robin would be possible, too, but if one of the mirrors is offline.... that would be bad, too.
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Improvements for rapid/http-downloads

Post by Licho » 15 Apr 2011, 19:55

You dont need any such complicated thigns imo..
simply make streamer to support HTTP range header..

Rapid client can then use several mirrors at once in multipart http download - pretty ordinary thing used to download normal files already.

Streamer should support some caching in that case though (as its likely to be rerequested for same stream soon).

But I dont think we need that .. rapid was made to minimize transferred data, assuming most downloads are from people who just upgrade their existing stuff it should be fine (=> not worth using mirrors if the download is 100kb).
0 x

User avatar
Forboding Angel
Evolution RTS Developer
Posts: 14572
Joined: 17 Nov 2005, 02:43

Re: Improvements for rapid/http-downloads

Post by Forboding Angel » 16 Apr 2011, 04:26

Not to mention the ginormous amount of things that rapid can do, e.g. make sdz, make sdd, upgrade, clean upgrade (which btw... seriously, zk needs to start using this), etc etc etc.
0 x

User avatar
hoijui
Former Engine Dev
Posts: 4342
Joined: 22 Sep 2007, 09:51

Re: Improvements for rapid/http-downloads

Post by hoijui » 19 Apr 2011, 17:07

forb, that are mostly client features (of tobis rapid client), while this discussion is about rapid protocol (not that this matters much).

what we would need for rapid is an easier way for non-ZK devs to have a say in the protocol ... thats what i remember SL devs wanted, but i also remember there was a discussion about this already...
0 x

User avatar
koshi
Lobby Developer
Posts: 1058
Joined: 14 Aug 2007, 16:15

Re: Improvements for rapid/http-downloads

Post by koshi » 19 Apr 2011, 17:10

Not speaking for BrainDamage, but I personally won't care one bit about rapid protocol (or any other downloader) details once we've switched to using abma's lib.
0 x

dansan
Server Owner & Developer
Posts: 1190
Joined: 29 May 2010, 23:40

Re: Improvements for rapid/http-downloads

Post by dansan » 19 Apr 2011, 17:17

abma wrote:DNS with round-robin would be possible, too, but if one of the mirrors is offline.... that would be bad, too.
_If_ you set up DNS with RR, then you _should_ have a prog that detects offline mirrors, and takes them out of the rotation.
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Improvements for rapid/http-downloads

Post by Licho » 19 Apr 2011, 17:17

The system is done and runs without changes what 3 years?

So why would you discuss anything? Its pretty easy to implement too (easier than say torrent download) and as koshi said abma made it into library for lobby.
So SL can start using it too.

Nobody from ZK team touched it for years and I certainly dont plan to change it unless there is good reason.
As the saying goes: don't fix what isnt broken.

Basic procol and idea is even older, dating to old caupdater, perhaps more than 4 years old.
0 x

User avatar
SinbadEV
Posts: 6475
Joined: 02 May 2005, 03:56

Re: Improvements for rapid/http-downloads

Post by SinbadEV » 19 Apr 2011, 17:32

could zsync be used to improve the functionality of the existing rapid protocol? for example if a map was updates to fix the locations of some features this would only affect about %10 of the smf file, with rapid it would transfer the entire changed file while zsync would theoretically only transfer the differences... would only really improve performance in a very small number of situations (like changes to SMF or textures) but might be useful in the long run for total server load.
0 x

User avatar
hoijui
Former Engine Dev
Posts: 4342
Joined: 22 Sep 2007, 09:51

Re: Improvements for rapid/http-downloads

Post by hoijui » 19 Apr 2011, 17:42

i make enlightenment: changes != fixes
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Improvements for rapid/http-downloads

Post by Licho » 19 Apr 2011, 20:40

Indeed it would improve map download, but rapid is not used for maps atm, just fo mods. Im not sure if its worth versioning maps.
0 x

SirMaverick
Posts: 834
Joined: 19 May 2009, 21:10

Re: Improvements for rapid/http-downloads

Post by SirMaverick » 19 Apr 2011, 22:37

Licho wrote:Indeed it would improve map download, but rapid is not used for maps atm, just fo mods. Im not sure if its worth versioning maps.
Less worth than for mods as there are less versions. But if done right it can reduce network load as it does for game updates.
0 x

User avatar
zwzsg
Kernel Panic Co-Developer
Posts: 7014
Joined: 16 Nov 2004, 13:08

Re: Improvements for rapid/http-downloads

Post by zwzsg » 19 Apr 2011, 22:55

Mappers don't care about filesizes anyway.
0 x

User avatar
AF
AI Developer
Posts: 20667
Joined: 14 Sep 2004, 11:32

Re: Improvements for rapid/http-downloads

Post by AF » 19 Apr 2011, 23:26

tbh Rapid appears to do its job, any system that adds features should be built on a layer above Rapid.

As such, I don't see what exactly there is to improve and nobody has quite definitively stated an improvement that can be stated nice and neatly without prior knowledge
0 x

abma
Spring Developer
Posts: 3540
Joined: 01 Jun 2009, 00:08

Re: Improvements for rapid/http-downloads

Post by abma » 21 Apr 2011, 16:06

@licho:

rapid runs on only one server, i don't want to see what happens, when some game gets slashdoted and all users download full files through rapid.

would it make sense to add maps to rapid? updates of maps often only changes features...

and there are still some feature-requests, for example this one:

http://code.google.com/p/zero-k/issues/detail?id=304

but this should be done by plasma or the xml-rpc api.
0 x

Post Reply

Return to “Infrastructure Development”

cron