Spring RTS Engine

Currently, whenever a change in a mod has been made and a new version has been published, you have to download the entire file, sometimes as big as 20MBs. Often however, there weren't many changes, and even sometimes none were done to unit models, which probably (guesswork here) makes most of the mod size.

I propose that we have hosting sites, as well as the internal spring file sharing system done by torrents, use a binary difference model. Whenever someone is interested in downloading a new mod (through a spring client), a local check is done to see what previous versions of the mod he has. In case he has none, the regular full download is done.

In case the user has an existing version, he sends the hash of the file to the system. The system then responds by giving the URL of the patch, which can be a torrent of the patch, or the regular http/ftp file download link.

This can be applied to initial file uploads as well, where the mod uploader would do a local diff and send only the file difference to the site.

Now as far as tools are concerned, doing a quick search on linux packages gives me bsdiff (http://www.daemonology.net/bsdiff/), a generic binary diff/patch program, which seems to have a Windows port available as well.
A couple of examples on it's efficiency and usage are shown below: As can be seen, all patches are smaller than full downloads, with some having drastic decrease in size.
An example creation and application of a patch This could be trivially used by any of the sites/clients.

PS: All the above should be done "behind the scene", which means user should be doing the download using a spring client (springlobby f.e) which does the process for him - he just has to specify what he wants to download. In case when a user goes to a site to download mods, it can be done by downloading a (small) file which will complete the download process in one of the spring clients (similar to how .torrent files work).

Spring already has such a system, it's called 'rapid' and is included in recent installers (and may be integrated in some of the lobbies). Someone else will have to fill you in on the more technical details, but at the least it uses existing content to avoid downloading.

What Niobium said.
Sadly this rapid downloader has no manual afaik.
Though there is some technical blabla no player understands:
http://springrts.com/wiki/Dev:LobbyDownloadSystems

On windows, start the rapid\rapid-gui.exe (comes with spring install) and click things on the left site until you get what you want.
Such automatic download is also built into some lobbies already.

knorke wrote:What Niobium said.
Sadly this rapid downloader has no manual afaik.
Though there is some technical blabla no player understands:
http://springrts.com/wiki/Dev:LobbyDownloadSystems

On windows, start the rapid\rapid-gui.exe (comes with spring install) and click things on the left site until you get what you want.
Such automatic download is also built into some lobbies already.

No kidding, it's not easy to understand that. But basically, it seems that spring uses archive files - .sdz seem to be archives with a bunch of files, and rapid can see if certain files exist/have been changed, and it will download those files.

Still not clear whether or not it downloads files partially, or how exactly it works across different mods, but I guess it's something.

However, it's still not used in springlobby (at least not when I click "Download this Mod"), not sure if there's any plan to implement it or not (unless it's easier to use rapid than it is to read that "manual", I can understand why no one is implementing it).

yes, .sdz file = renamed .zip archive
.sd7 = renamed 7zip archive
For rapid only .sdz works though.

However, it's still not used in springlobby (at least not when I click "Download this Mod")

sl might simply always downloads the complete file.

Anyway, I think you want those subforums:
http://springrts.com/phpbb/viewforum.php?f=71
http://springrts.com/phpbb/viewforum.php?f=64

It all depends on the game creators if they release new version as a 'patch' (small size, only changed files included) or 'full release' (large, all files inside). Rapid isn't even needed to get patches, it's perfectly possible to distribute them as .sdz which springlobby can then download (but it won't get all the requirements, so the 'base' game version for the patch has to be present anyway).

Also lol at 'as big as 20 MBs' for the whole game/mod. BA maybe, but then take a look at S1944 - more like 100 MB. Gundam is even larger afaik.

What yuritch means by "patch" is http://springrts.com/wiki/Modinfo.lua#Mutator

Like Spring1944 patches or XTA betas: http://springrts.com/phpbb/viewtopic.php?f=48&t=21739

Though I dont know if that is a good solution, the download is a bit smaller but you need multiple files (=confused players) and you have the work of later merging the files anyway.

I am moving to rapid instead of doing full archive releases

Rapid does not work with .sdz files.
For rapid to work game has to use special rapid format. Stuff it uses is stored in pool and packages folders.
It then

1) shares content between games (like models, sounds etc)
2) shares content between different versions of same game
3) downloads only changes
4) handles dependencies
5) supports tagging (tag a version as "ba:test" for example)
6) supports notification about new versions

This thing exists for ages, sadly SL didnt implement it.
Use Zero-K lobby (original implementation of rapid), Rapid GUI (packed with spring) or Tasclient lobby.

Licho wrote:Rapid does not work with .sdz files.
For rapid to work game has to use special rapid format. Stuff it uses is stored in pool and packages folders.
It then

1) shares content between games (like models, sounds etc)
2) shares content between different versions of same game
3) downloads only changes
4) handles dependencies
5) supports tagging (tag a version as "ba:test" for example)
6) supports notification about new versions

This thing exists for ages, sadly SL didnt implement it.
Use Zero-K lobby (original implementation of rapid), Rapid GUI (packed with spring) or Tasclient lobby.

Which of these work with linux? (I fear none)

Rapid commandline, described in many places, including springrts.com wiki and zero-k download page.

gajop wrote:Which of these work with linux? (I fear none)

did you try? or are you just "shooting from the hip?"

smoth wrote:

gajop wrote:Which of these work with linux? (I fear none)

did you try? or are you just "shooting from the hip?"

Yes, in fact, when I last checked the only working lobbies for spring on linux are springlobby and qtlobby (not sure what happened with this one). Rapid is a command line tool I have been using for a while now, but it's no GUI that can help me download stuff by choosing on a game and right clicking "Download this Mod".

gajop wrote: Rapid is a command line tool I have been using for a while now, but it's no GUI that can help me download stuff by choosing on a game and right clicking "Download this Mod".

you are a 'nux user who cannot do commandline.... wow, can someone turn off the internet? I have seen the end..

In case you are interested, here is how rapid works, it is very simple. There are files called .sdp, these list all files in the mod and their md5 hash. These are stored in the packages folder. Individual files are stored in the pool folder, named by their md5 hash. Downloading a rapid mod is done by downloading the .sdp, then determining which files you don't have and then downloading those. It does not do any partial downloading of files, but that wouldn't be very useful in the context of spring mods. Partial .lua wouldn't save you much and its unlikely to get a useful patch of a sound/model/image due to compression. Also note that a patch tool like bsdiff would be incapable of downloading partial files too (even of lua), because the way .zip compresses files individually. The spring engine has native support for the .sdp format. Also, there is special cgi program called streamer which accepts a compressed bitarray representing which files to download and then sends them all in a single stream. Even on Linux, there exists the rapid-gui program.

Rather than technical details some text for players would be nice.

Even on Linux, there exists the rapid-gui program.

where to get it? (appearently it is not trivial to find, eg see varikonniemi 's post: http://springrts.com/phpbb/viewtopic.php?f=12&t=27188 )

Rapid commandline, described in many places, including springrts.com wiki and zero-k download page.

rapid-gui is described nowhere (or cant find)
This is the only site on wiki I found: http://springrts.com/wiki/Dev:LobbyDown ... tems#Rapid
It has no information for players.
Somebody with the knowledge please make a manual for players.

A readme.txt or "about"-button would be nice too, so you know where to report bugs and all that.

This was a technical topic, my response was entirely appropriate. rapid-gui is packaged in debian/ubuntu, I have no idea about other distros. I would expect to find it anywhere rapid is found. A manual means its too complicated, double click in lobby should be sufficient.

Pretty much every distro should have python-pip in their repositories.

If you have rapid installed you also have the gui installed. Simply type "rapid-gui" in the terminal instead.

That was not the case for me, ubuntu had only rapid-spring, not rapid-gui.

I needed to install pip and python-dev to get rapid-gui.