Mod/map downloader and lobby integration

[Licho]

It runs under mono but needs some platform specific debugging/changes because of spring registry etc..

I dont have linux but feel free to help fixing it for linux.

[Jonanin]

I know for sure that SpringLobby won't be using it if it isn't cross platform. I think it would be better if the lobbies handled it themselves with an http server (With UF - I know it's down, but either that will be fixed or there will be a replacement eventually.)

[aegis]

RSS feed

[Jonanin]

Exactly, http! :)

[Licho]

Regarding integration I think its best to use commandline and standard output for starters.

That way lobby will easilly detect crash/unexpected exit and I think its also easier to implement for lobby than some extra tcp link.

so:

input:
<executable> <list of comma separated resource hash>

possible response:
ERROR <reason> - typically "CaDownloader not running" or "hash <hash> not found"

input:
GETSTATE <hash>

response:
STATE <hash> state,size,progress,speed,peers,seeds

response:
DONE <hash> - file was succesfully installed in target spring folder

input:
ABORT - tells downloader to abort requested downloads and exit talking process


Is that enough for initial integration?

[koshi]

use a bittorrent library compatible with your lobby ^_^

probably the way we will go with springlobby, if one of us finds the time to do it

[Licho]

use a bittorrent library compatible with your lobby ^_^

probably the way we will go with springlobby, if one of us finds the time to do it

Yeah you can use same central repository of torrents - sharing maps with all other users while having independent client implementation.

Since "server" for this system is modified tracker I made which runs on both linux and windows and torrents take just few kb of space, it will be easy to run backups of cental server.

That way we can create ultra-stable system.

[aegis]

we could sync torrents with my uber seed and use it as a tracker/alternate torrent downloader

[LordMatt]

Anyone know if this runs under wine?

[Licho]

It does not. It should be native application (one binary running under both windows and mono on linux), but someone has to fix it for mono..
Same binary of caTracker (server part) runs well on both my windows and debian server though.

[AF]

I'd support that spec. But I'd also like a way of retrieving version numbers in case that internal protocol changes.

[Licho]

I upgraded central tracker to new version, this is basic description how it works (for those who want to implement new client using the same system):

I tried to design whole system to be very easy to implement and debug.
When data are transfered or stored, "|" is used to divide records, if original record contains "|" its replaced with "&divider&"

Central server:
===========
torrent tracker: http://tracker.caspring.org:8201/announce
p2p coordinator: tracker.caspring.org:8202 (tcp stream)
site: http://tracker.caspring.org/caupdater (will change to root of tracker in the future)

Site
----
site provides torrent browsing in /torrents subfolder, each torrent named by unitsync hash.
Site also provides torrent uploading (using upload_torrent.php <- put file to fname file variable).
Torrent must contains special data in its comment: name|type
type being MOD or MAP now and name name of mod or map from unitsync.

Site also contains and maintains several legacy files, like torrent.txt, requests.txt and battles.txt

P2P coordinator
---------------
service runs at tracker.caspring.org:8202 - its purpose is to notify clients about new torrents and to distribute information that seeding is needed for some torrent. It can also provide "tracking" - it can take lobby user name and direct client to start downloading maps needed for his game or notify client that his spring started.

Protocol used by coordinator is similar to TasServer, each command is one line and arguments are divided by "|" symbol.

Here are commands sent by server:

T+|hash|name|type
informs client that new torrent was added to server (type is either MOD or MAP)

T-|hash
informs client that torrent was removed from server

S+|hash|seeders|leechers
tells client that seed is needed for this torrent

S-|hash
tells client that seed is no longer neede for this torrent

L+|hash
tells client that it should leech this torrent

L-|hash
tells client that he no longer needs this torrent

INGAME|mode
tells client that his ingame status changed - mode is 1 or 0

PING
every minute - client must respond with its own "PING"

M+|hash|url
tells client that http mirror exists for given hash

Here are commands sent by client:

TRACK_MAP|mode|username
this tells server that client requests tracking. Mode is either "NONE", "ALL", "PLAYER"
NONE means that server only informs about ingame status changes.

PING
every minute

N+|hash
requests torrent seeding and mirrors from server

N-|hash
indicates that seeding is no longer needed

PLAYER means that server only sends leech commands for maps needed by this player (if he joins some lobby game or lobby map changes server sends leech commands).

ALL means that leech commands will be sent for all currently hosted maps.


Basic client operations
===========================
- Upon start or when map folder changes, torrent files are created for all maps and mods and stored in special folder. Torrents have announce address set to tracker and comment set by special rules notified earlier.

- Client connects to p2p coordinator and coordinator and coordinator sends its list of known torrents.

- When seed command is recieved, client starts seeding requested file (if it has it)

- When client wants to download new file it gets torrent for it from http://tracker.caspring.org/caupdater/t ... sh.torrent and starts leeching it. All other clients are notified instantly that seeding is needed using p2p coordinator.

- New files are added to system using torrent upload to http://tracker.caspring.org/caupdater/t ... orrent.php (again all clients are notified instantly through p2p coordinator that file has been added).

General remarks
----------------
- client should not download more than 5 torrents at once, server enforces this internally by not sending further seed requests if more than 5 files are requested

- client should reannounce every 15s during first minute of leeching/seeding to ensure optimal performance and after that every minute.

Sources svn://svn.caspring.org/
(to be added)

[Licho]

Note that I plan to add M+ command from server.

It will indicate that http mirror exists for given hash.

M+|hash|url

p2p coordinator will scavenge some http file hosts and discover some mirrors :)

That way we can combine all power of torrents and http file hosts together!

[Licho]

Added MIRROS command -> server responds with list of http mirrors.
Right now it either responds with M+|hash|url or M+|hash| <- signaling no known mirror exists.

[Licho]

Jonani yes if you make it crossplatform.
I have no idea what is spring layout on linux so you have to modify code..

But springlobby is implementing own client for this system so map downloads will be availiable directly in it for linux users.

[Licho]

I added N+ command and N- (sent by client).
It tells server that this client needs given hash.
Requests are no longer generated form tracker announcements but from direct requests.

After N+ possible S+ request is sent to other clients (if it was not requested before, if its not more than 5th request of the user etc) and http mirrors are sent back to client who needs it.

[Ifein]

Hi, I have two remarks:

The server sends "T+|hash|..." and "S-|hash|..." for each torrent in the beginning.
To get rid of those "S-" messages you could introduce a "begin torrent list" and "end torrent list" (like for some commands in the lobby protocol) to save some messages. Or just a "T|hash|" to announce that this torrent exists. (That would save about 10kB for each client connecting)

Each time the server sends a "S+" it sends later a "S-" (caused by client requests). We have some good leechers it seems (or the files are small), because often I see a "S+" followed by a "S-" for the same hash (<5) seconds later. Additionally sometime there is a "S+ S- S+ S-" chain for the same hash (new mod - many want it?, new map - several people join host who do not have it yet?).
From my point of view S- (as N-) should be avoided. It the client wants something it should indicate it (N+..). Server sends request (S+) to all others. When this client has finished downloading, all other clients already see it, because there are no leechers for the torrent anymore.
They even can keep the torrent for some time, that does not hurt. And would help with the request chains: this would stop clients from starting and stopping torrents too often, causing overhead for the torrent tracker.

These remarks would reduce some traffic and make the whole thing more scalable.
Btw: how many clients are connected in the "rush hour"?

[AF]

Did you implement the lobby interface <-> client at all or was that just a proposal you haven't followed up yet? Or are we all definitely going to have to implement our own clients?

[Licho]

Ifein:
the S- on startup are sent, because if client loses connection to coordinator, it keeps seeding (to not break existing downloads, p2p works well even if server randomly crashes).

So on startup server tries to "synchronize" clients by sending them full lis of S-.

The S+, S- in quick succession are caused by some clients using web download. They request a file, and coordinator tells other to seed but also provides http links. Client then decides to use http only instead of torrent so unrequests the file.

Also because many people have it set to "auto download all hosted maps", it can repeat S+, S- for several people in quick succcession.

Both of these later problems wont happen when I implement full http seeding.

[Licho]

I can implement lobby <-> downloader its not a problem. But so far I dont know of anyone wanting to integrate it with lobby.

SpringLobby is developing their own client for this system and I thought you are going same way.