Updating mods [PROPOSAL]

Updating mods [PROPOSAL]

Discuss game development here, from a distinct game project to an accessible third-party mutator, down to the interaction and design of individual units if you like.

Moderators: Moderators, Moderators

gajop
Moderator
Posts: 3027
Joined: 05 Aug 2009, 20:42

Updating mods [PROPOSAL]

Post by gajop »

Currently, whenever a change in a mod has been made and a new version has been published, you have to download the entire file, sometimes as big as 20MBs. Often however, there weren't many changes, and even sometimes none were done to unit models, which probably (guesswork here) makes most of the mod size.

I propose that we have hosting sites, as well as the internal spring file sharing system done by torrents, use a binary difference model. Whenever someone is interested in downloading a new mod (through a spring client), a local check is done to see what previous versions of the mod he has. In case he has none, the regular full download is done.

In case the user has an existing version, he sends the hash of the file to the system. The system then responds by giving the URL of the patch, which can be a torrent of the patch, or the regular http/ftp file download link.

This can be applied to initial file uploads as well, where the mod uploader would do a local diff and send only the file difference to the site.

Now as far as tools are concerned, doing a quick search on linux packages gives me bsdiff (http://www.daemonology.net/bsdiff/), a generic binary diff/patch program, which seems to have a Windows port available as well.
A couple of examples on it's efficiency and usage are shown below:

Code: Select all

Patches:
19M       731_750
485K      750_760
17M       760_762
148K      762_763

Full files:
21M       ba731.sdz
21M       ba750.sdz
21M       ba760.sdz
21M       ba762.sdz
21M       ba763.sdz
As can be seen, all patches are smaller than full downloads, with some having drastic decrease in size.
An example creation and application of a patch

Code: Select all

#patch creation
$ bsdiff ba762.sdz ba763.sdz 762_763 

#patch application
$ bspatch ba762.sdz ba763_patched.sdz 762_763_diff 

#produces nothing, they're the same
$ diff ba763_patched.sdz ba763.sdz 
This could be trivially used by any of the sites/clients.

PS: All the above should be done "behind the scene", which means user should be doing the download using a spring client (springlobby f.e) which does the process for him - he just has to specify what he wants to download. In case when a user goes to a site to download mods, it can be done by downloading a (small) file which will complete the download process in one of the spring clients (similar to how .torrent files work).
0 x

User avatar
Niobium
Posts: 456
Joined: 07 Dec 2008, 02:35

Re: Updating mods [PROPOSAL]

Post by Niobium »

Spring already has such a system, it's called 'rapid' and is included in recent installers (and may be integrated in some of the lobbies). Someone else will have to fill you in on the more technical details, but at the least it uses existing content to avoid downloading.
0 x

User avatar
knorke
Posts: 7971
Joined: 22 Feb 2006, 01:02

Re: Updating mods [PROPOSAL]

Post by knorke »

What Niobium said.
Sadly this rapid downloader has no manual afaik.
Though there is some technical blabla no player understands: ;)
http://springrts.com/wiki/Dev:LobbyDownloadSystems

On windows, start the rapid\rapid-gui.exe (comes with spring install) and click things on the left site until you get what you want.
Such automatic download is also built into some lobbies already.
0 x

gajop
Moderator
Posts: 3027
Joined: 05 Aug 2009, 20:42

Re: Updating mods [PROPOSAL]

Post by gajop »

knorke wrote:What Niobium said.
Sadly this rapid downloader has no manual afaik.
Though there is some technical blabla no player understands: ;)
http://springrts.com/wiki/Dev:LobbyDownloadSystems

On windows, start the rapid\rapid-gui.exe (comes with spring install) and click things on the left site until you get what you want.
Such automatic download is also built into some lobbies already.
No kidding, it's not easy to understand that. But basically, it seems that spring uses archive files - .sdz seem to be archives with a bunch of files, and rapid can see if certain files exist/have been changed, and it will download those files.

Still not clear whether or not it downloads files partially, or how exactly it works across different mods, but I guess it's something.

However, it's still not used in springlobby (at least not when I click "Download this Mod"), not sure if there's any plan to implement it or not (unless it's easier to use rapid than it is to read that "manual", I can understand why no one is implementing it).
0 x

User avatar
knorke
Posts: 7971
Joined: 22 Feb 2006, 01:02

Re: Updating mods [PROPOSAL]

Post by knorke »

yes, .sdz file = renamed .zip archive
.sd7 = renamed 7zip archive
For rapid only .sdz works though.
However, it's still not used in springlobby (at least not when I click "Download this Mod")
sl might simply always downloads the complete file.

Anyway, I think you want those subforums:
http://springrts.com/phpbb/viewforum.php?f=71
http://springrts.com/phpbb/viewforum.php?f=64
0 x

User avatar
yuritch
Spring 1944 Developer
Posts: 1018
Joined: 11 Oct 2005, 07:18

Re: Updating mods [PROPOSAL]

Post by yuritch »

It all depends on the game creators if they release new version as a 'patch' (small size, only changed files included) or 'full release' (large, all files inside). Rapid isn't even needed to get patches, it's perfectly possible to distribute them as .sdz which springlobby can then download (but it won't get all the requirements, so the 'base' game version for the patch has to be present anyway).

Also lol at 'as big as 20 MBs' for the whole game/mod. BA maybe, but then take a look at S1944 - more like 100 MB. Gundam is even larger afaik.
0 x

User avatar
knorke
Posts: 7971
Joined: 22 Feb 2006, 01:02

Re: Updating mods [PROPOSAL]

Post by knorke »

What yuritch means by "patch" is http://springrts.com/wiki/Modinfo.lua#Mutator

Like Spring1944 patches or XTA betas: http://springrts.com/phpbb/viewtopic.php?f=48&t=21739

Though I dont know if that is a good solution, the download is a bit smaller but you need multiple files (=confused players) and you have the work of later merging the files anyway.
0 x

User avatar
smoth
Posts: 22300
Joined: 13 Jan 2005, 00:46

Re: Updating mods [PROPOSAL]

Post by smoth »

I am moving to rapid instead of doing full archive releases
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Updating mods [PROPOSAL]

Post by Licho »

Rapid does not work with .sdz files.
For rapid to work game has to use special rapid format. Stuff it uses is stored in pool and packages folders.
It then

1) shares content between games (like models, sounds etc)
2) shares content between different versions of same game
3) downloads only changes
4) handles dependencies
5) supports tagging (tag a version as "ba:test" for example)
6) supports notification about new versions

This thing exists for ages, sadly SL didnt implement it.
Use Zero-K lobby (original implementation of rapid), Rapid GUI (packed with spring) or Tasclient lobby.
0 x

gajop
Moderator
Posts: 3027
Joined: 05 Aug 2009, 20:42

Re: Updating mods [PROPOSAL]

Post by gajop »

Licho wrote:Rapid does not work with .sdz files.
For rapid to work game has to use special rapid format. Stuff it uses is stored in pool and packages folders.
It then

1) shares content between games (like models, sounds etc)
2) shares content between different versions of same game
3) downloads only changes
4) handles dependencies
5) supports tagging (tag a version as "ba:test" for example)
6) supports notification about new versions

This thing exists for ages, sadly SL didnt implement it.
Use Zero-K lobby (original implementation of rapid), Rapid GUI (packed with spring) or Tasclient lobby.
Which of these work with linux? (I fear none)
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Updating mods [PROPOSAL]

Post by Licho »

Rapid commandline, described in many places, including springrts.com wiki and zero-k download page.
0 x

User avatar
smoth
Posts: 22300
Joined: 13 Jan 2005, 00:46

Re: Updating mods [PROPOSAL]

Post by smoth »

gajop wrote:Which of these work with linux? (I fear none)
did you try? or are you just "shooting from the hip?"
0 x

gajop
Moderator
Posts: 3027
Joined: 05 Aug 2009, 20:42

Re: Updating mods [PROPOSAL]

Post by gajop »

smoth wrote:
gajop wrote:Which of these work with linux? (I fear none)
did you try? or are you just "shooting from the hip?"
Yes, in fact, when I last checked the only working lobbies for spring on linux are springlobby and qtlobby (not sure what happened with this one). Rapid is a command line tool I have been using for a while now, but it's no GUI that can help me download stuff by choosing on a game and right clicking "Download this Mod".
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Updating mods [PROPOSAL]

Post by Licho »

0 x

User avatar
smoth
Posts: 22300
Joined: 13 Jan 2005, 00:46

Re: Updating mods [PROPOSAL]

Post by smoth »

gajop wrote: Rapid is a command line tool I have been using for a while now, but it's no GUI that can help me download stuff by choosing on a game and right clicking "Download this Mod".
you are a 'nux user who cannot do commandline.... wow, can someone turn off the internet? I have seen the end..
0 x

User avatar
det
Moderator
Posts: 737
Joined: 26 Nov 2005, 11:22

Re: Updating mods [PROPOSAL]

Post by det »

In case you are interested, here is how rapid works, it is very simple. There are files called .sdp, these list all files in the mod and their md5 hash. These are stored in the packages folder. Individual files are stored in the pool folder, named by their md5 hash. Downloading a rapid mod is done by downloading the .sdp, then determining which files you don't have and then downloading those. It does not do any partial downloading of files, but that wouldn't be very useful in the context of spring mods. Partial .lua wouldn't save you much and its unlikely to get a useful patch of a sound/model/image due to compression. Also note that a patch tool like bsdiff would be incapable of downloading partial files too (even of lua), because the way .zip compresses files individually. The spring engine has native support for the .sdp format. Also, there is special cgi program called streamer which accepts a compressed bitarray representing which files to download and then sends them all in a single stream. Even on Linux, there exists the rapid-gui program.
0 x

User avatar
knorke
Posts: 7971
Joined: 22 Feb 2006, 01:02

Re: Updating mods [PROPOSAL]

Post by knorke »

Rather than technical details some text for players would be nice.
Even on Linux, there exists the rapid-gui program.
where to get it? (appearently it is not trivial to find, eg see varikonniemi 's post: http://springrts.com/phpbb/viewtopic.php?f=12&t=27188 )
Rapid commandline, described in many places, including springrts.com wiki and zero-k download page.
rapid-gui is described nowhere (or cant find)
This is the only site on wiki I found: http://springrts.com/wiki/Dev:LobbyDown ... tems#Rapid
It has no information for players.
Somebody with the knowledge please make a manual for players.

A readme.txt or "about"-button would be nice too, so you know where to report bugs and all that.
0 x

User avatar
det
Moderator
Posts: 737
Joined: 26 Nov 2005, 11:22

Re: Updating mods [PROPOSAL]

Post by det »

This was a technical topic, my response was entirely appropriate. rapid-gui is packaged in debian/ubuntu, I have no idea about other distros. I would expect to find it anywhere rapid is found. A manual means its too complicated, double click in lobby should be sufficient.
0 x

luckywaldo7
Posts: 1397
Joined: 17 Sep 2008, 04:36

Re: Updating mods [PROPOSAL]

Post by luckywaldo7 »

Pretty much every distro should have python-pip in their repositories.

If you have rapid installed you also have the gui installed. Simply type "rapid-gui" in the terminal instead.
0 x

varikonniemi
Posts: 447
Joined: 03 Jul 2011, 11:54

Re: Updating mods [PROPOSAL]

Post by varikonniemi »

That was not the case for me, ubuntu had only rapid-spring, not rapid-gui.

I needed to install pip and python-dev to get rapid-gui.

Code: Select all

$ sudo apt-get install pip
$ sudo apt-get install python-dev
$ pip install -U rapid-spring
$ rapid-gui
Last edited by varikonniemi on 02 Jan 2012, 20:01, edited 4 times in total.
0 x

Post Reply

Return to “Game Development”

cron