SpringDownloader - maps/mods/widgets/lobby

[Licho]

0.99.1.0

* proper mission downloading integreation with tasclient
* button to start mission and download stuff you need to run mission. Doubleclicking on row start missions.
* comments and download count.
* proper rating averaging - each player has 1 vote

[Tobi]

Where is the spec for the mission stuff so it can be integrated into other software (e.g. SpringLobby) too?

[Licho]

PM sent

[Licho]

1.0.0.0

* very bad soldier add new feature to widget list - ability to select which widgets are active in spring

* fixed random bugs when mission fail to starts

[Licho]

1.0.2.0
* added support for automated online scoring of missions
* disabled automatic widget list refresh - this will prevent bad memory leak

[Licho]

1.10

* changed to new system - higher speeds and reliability and handles dependencies (mods with multiple files etc)
* uses pool system for some mods (allows incremental updates and shared content among different mods and their versions)
* upgraded widget manager - shows profiles for each mod
* TS2 integration removed
* automatic download of map/mod for current game removed (unused?) - if used together with tasclient, tasclient requests download anyway
* removed seeding
* missiondownloader fixed
* modstats data submission fixed
* simplifications and optimizations


Note, current mod and maps list is incomplete. Will add remaining of those on springfiles in near future.

[Tobi]

How to add a game to the pool system? Is there any documentation for this?

[Licho]

Aww 1.1 has self-updater disabled internally..

Please download 1.1.2 manually from here:

http://files.caspring.org/caupdater/Spr ... loader.exe


Tobi: I will ask det to write specs down.
Atm there are tools that auto process SVN commits to pool. But its simple to make commandline tool for people who have ordinary archives without source control.

[Licho]

Re pool system.

All files in pool system are gzipped.

Master file is here:
http://repos.caspring.org/

It's content is something like this:
ca,http://packages.caspring.org,^Complete Annihilation (stable|test)-(\d+),revision.{1}

This tells the system that the mod that matches the regex is to be found on http://packages.caspring.org

Each mod can have completely different site.

Master file is manually edited by det or me.

Each repository (like http://packages.caspring.org/ ) can contain any number of mods.

Main file there is versions.gz which lists what versions are present in the repository.
(There is also versions.digest used to quickly check whether repository changed).

Downloader determines version name using regular expression from master - for example it gets "revision.6666" for some CA version and looks up this version name in versions.gz.

It gets name of package from "packages" folder.

Packages are in native sdp format supported by engine.

Sdp is basically list of files in the mod, each entry contains relative path in lowercase, size and md5 hash of the content.

Md5 hash is name of the pool file with content. Pool files are in pool folder, first byte is folder name.


Atm there is ocaml program, which gets mod from SVN and constructs/maintains this (pool, packages and versions.gz/digest).

Det said its trivial to change it to commandline utility.

There is also old C# which does same thing.

So it all depends on what you want to achieve.

Pool system is meant mainly for mods that update often or are very big.

If you want it for say s44 steps should be simple, setup web folder, post commit hook for that ocaml generator and tell us URL to enter into master file.

If you want something else, tell me what, i can make necessary tools. Its all demand based atm.

[hoijui]

would it possibly make sense to use this for AIs too?
my brainmap is currently empty for this, as in.. i dont know what additional problems there would be.
i am thinking about AI packages, where the AI dev supplies a zip that would be extracted to AI/Skirmish/<shortName>/.

[Pxtl]

would it possibly make sense to use this for AIs too?
my brainmap is currently empty for this, as in.. i dont know what additional problems there would be.
i am thinking about AI packages, where the AI dev supplies a zip that would be extracted to AI/Skirmish/<shortName>/.

Safety issues. AIs are DLLs, which means that malicious code could be hidden in them. That sort of thing should not be given to the user through a P2P system where nobody is accountable for hosting the file.

[hoijui]

true that.

[Licho]

Technically yes. It's no problem to store data about AI in the system.
Some resources can also be server-side checked before being published, like widgets already are.


Only question is, what will use it? SpringLobby is lagging behind atm in features and SD isn't very useful without rarely used tasclient.

I offered making sure SD runs smoothly on linux and interfacing with springlobby, but it was declined (mono hate).

So atm we have to wait until SL implements existing SD features (torrent download, pool downloader, lua manager, single player missions, mod statistics) and then add new..

[AF]

Im thinking that an AI would be fine until you realise that certain files dont work on every single machine, what about linux where theres skirmishai.so, 32 bit or 64 bit? which distribution? etc etc etc

[hoijui]

thanks for feedback.
but i think the safety argument rules AIs out anyway, no?
for so/dll/dylib and 32/64 bit ... not that i know, but the system that Licho describes sounds relatively powerfull. this wouldneed passing parameters from the client, or maybe that info could even be put into the download name, eg: "E323AI_version_SO_32"

[AF]

I thought 32 bit SO libs could be incompatible between different linux machines based on distribution?

[hoijui]

they should only be so if they rely on native libs, where the different distors have different versions of (eg: boost), and they are not supplied with the AI. otherwise it shoudl be no problem.. i did some AI packages with linux 32bit SOs in the past, and just recently did the same for E323AI (error does the same with 64bit), and i never got a complain about it not working. error even uses boost.

[Tobi]

(g)libc version would be the major troublemaker in that case, I suppose

[Licho]

SpringDownloader 1.3
=================
* automatic download of map/mod for joined game re-enabled
* linux support
* "rapid" module - package browser - browses mods/maps which are using engine's pool system and keeps selected packages updated. (This will replace now outdated CA module)
* upgraded/fixed torrent lib for faster and more reliable downloads (fixes bug where it sometimes refused to finish at 95% or so)
* dependency downgraded from .NET 3.5 back down to .NET 2.0
* added -verbose parameter for extra spam to console
* removed "auto start on windows start"


Details:
-------
Package system:
* packages are stored in pool and packages in spring folder (windows) or writable datadir (linux)
* packages may auto-select their dependencies too and some normal downloads might select packages
* some package can actually be a tag - like "ca.stable" and can change over time. Such packages are automatically updated when selected.
* selected packages are listed in packages/selected.list

Linux support:
* widget module is not yet ported, will be soon
* installation:
wget http://files.caspring.org/caupdater/Spr ... loader.exe
sudo aptitude install mono libmono-winforms2.0-cil
mono SpringDownloader.exe

Lobby interface:
* added secondary interface for simpler "tasking"/monitoring by lobby - especially on Linux
* file <writable datadir>/SDCache/transfers.txt contains updated list of transfers in the format:

internal name|percentage|eta|status

eta is string, percentage is 0-100, status is downloading, done, failed, aborting,aborted.
Ended transfers are removed from the list on next download request.

* if you create file <writable datadir>/SDCache/requests.txt and put your requests there line by line, SD will "consume" it and delete file.
Requests should be internal names of resources wanted.
Request can also be "cancel" request in the format

internal name|cancel

[Licho]