remove reliance on map/mod hosts (Aka Unknown files)

[Drugar]

With Unknown files being down for several days now. It seems like a very bad idea on relying on them or any commercial host to provide us with our maps and mods. The installer doesn't do much good if it can't download mods/maps from UF

I am guessing that UF isn't very economically feasible. If the site was hosted for profit, having the game lobby directly link to maps/mods might hurt his Ad revenue.

A Peer 2 Peer based file distribution makes the most sense for an open source application to flourish and expand when it requires so many large data files.

Has anybody in the 'inner circle' considered moving towards something like this? Imagine a solution where as soon as you connect to the lobby, all your maps and mods are 'shared'. New players could connect for the first time and pick and choose maps/mods to download.

I know that this won't magically happen but maybe a gradual migration to this would be a good thing to start. Maybe temporarily promote people to use a stable torrent site that doesn't have to much negative press.

[zwzsg]

MTR

(woot, I feel like a aged troll vet, replying cryptic three letters acronyms to a newcomer's first post. Beside it isn't even listed in the MTR list as no one bother to update it.)

[LordMatt]

I have to remember to ask fnordia to give me write access to the php file that controls redirects from the installer. A p2p system would be great, please write one for us.

[Drugar]

I have to remember to ask fnordia to give me write access to the php file that controls redirects from the installer. A p2p system would be great, please write one for us.

My sarcasm detector is faulty but I am guessing thats mostly a joke, otherwise I might bite (aka run with it, do it, write code etc)

[LordMatt]

If you have the ability to make such a thing it would get serious consideration here.

[Argh]

Yeah, we're not at all joking. A BitTorrent solution built into Spring would be very welcome.

[aegis]

licho said he tried bittorrent and it was too resource-intensive for the amount of files we have

[sombriks]

torrent library?

http://deluge-torrent.org/
http://ktorrent.org/

those two have GPL'ed code, it's a nice start even for me (since i'm still accepting C++ in my life, )

[aegis]

torrent library?

http://deluge-torrent.org/
http://ktorrent.org/

those two have GPL'ed code, it's a nice start even for me (since i'm still accepting C++ in my life, )

licho said he tried bittorrent and it was too resource-intensive for the amount of files we have

we can do bittorrent... we are just looking at alternatives

[zwARREN69]

Bit torrent is the way to go. Unknown files was slowing to a crawl, this past weekend. I was installed Spring on 3 different pcs and the UF addons were going as low as 2-3kb/s. It was freaking ridiculous. I'm interested to know why UF went down, actually.

BTW, any Azureus fans? uTorrent is awesome, but I like my plugins, so I stuck with Azureus.

[sombriks]

torrent library?

http://deluge-torrent.org/
http://ktorrent.org/

those two have GPL'ed code, it's a nice start even for me (since i'm still accepting C++ in my life, )

licho said he tried bittorrent and it was too resource-intensive for the amount of files we have

we can do bittorrent... we are just looking at alternatives

i mean that some day there will be a dedicated C/C++ module to get things via bittorrent for, example, the guys in the lobby which aren't playing.

for now map torrents could help. mappers could start imediately

[SwiftSpear]

I don't think this is really feasible at this point in time... UF was in large part an attempt to decentralize our file hosting, UF has the capability of offing hosting to other file hosts, as opposed to file universe which did all the file hosting itself. There's just too many files in spring to not have a central database of them somewhere.

Torrents just seem completely unfeasible. Torrent clients take up signifigant system resources running 20-30 torrents simultaneously. There would just be no way to torrent every single map and mod in your folder without grouping them together in torrent packs, and if it's packed downloaders can't quickly access the files they need to start up and play a game.

[smoth]

do we know what is going on with uf?

[NightfallGemini]

anyone have a way of getting in touch with the current UF webmaster?

[Forboding Angel]

It may come down to us just needing to buckle down and pay for servers ourselves.

TBH I'm not real sure what we should do. It's pretty much impossible to use p2p for this stuff. If we all lived in the United states it might be a different story as bandwidth here is dirt cheap, however upstream is still a problem. I currently have 10 mbit down and 1 mbit up which costs me about 55 bucks per month. The difference tho is the fact that everyone in the spring community is spread butter thin across the world.

At this particular moment I am lost. If there was a server that I could find and afford I would pay for it, I've even considered setting up a website on my local fileserver here where ppl ingame with me could leech off my connection to get maps, but that solution sucks balls too, cause then my connection gets 110% lolraeped in hte process.


Guys, I think the time to force a switch to sm3 type of mapping may be necessary to keep thing alive. We keep hopping and hopping from server to server, raping bandwidth wherever we go, and nothing is being done to alleviate it. I used to think that we would always just manage to get by, but now I have my doubts.



Is there any Dev here willing to put balls to the wall on sm3? Hell I'll even pay for it to be done (Tho I'm not rich by any means).

If we could consolidate the bulk of our maps into 5 mb files, then we would really be getting somewhere. ATM sm3 has some huge downfalls. If we can alleviate those, then switching from smf>sm3 will not be so much of a kick in the balls.

[SwiftSpear]

http://www.springplayersclub.com/

Not UF but it will work for now. (thanks Sefidel)

I haven't been in contact with cuppy recently, but I know school is pretty much kicking his ass right now. I'm not sure what the root of the problem is here, hopefully we can get in contact with cuppy sometime soon.

[aegis]

seems to be something trivial like restarting the server from the web interface...

[very_bad_soldier]

Maybe we wont need a P2P-system if enough users could donate some bandwith. I uploaded some maps by myself to this URL.

http://spring.vsync.de/maps

Isnt there an open source website system which allows to manage different files by categories and so on just like UF?

[BrainDamage]

there are few downsides with every solution, the most notables i can think are:

bittorent:

-huge client overhead on torrent creation/joining
-uses tcp so STUN(a sort of advanced hole punching) is impossible, all the peers must open their incoming ports
-torrent creation: a server would still be needed to keep track of the files and the peers which is not just just the tracker and reset the torrent and all the peers if a new file gets added (creating multiple torrents would be even more penalizing due to the aforementioned overhead)
-requires servers(trackers) to keep track of peers(although this can be solved by having the KAD network to pass peer and files infos)

- ed2k/emule:

- no any-leeching system, and the emule credit system doesn't work well because it creates long queues, especially for the rare files or new players (the long queues might frustrate new players and have them abandon the game)
-not very efficient
-uses both tcp and udp (again, no STUN)
-requires servers to keep track of peers (although this can be solved by having the KAD network to pass peer and files infos)
-the more rare files doesn't get priority

the ideal p2p network for spring players has:

- arbitrary metadata fields, so files can have listed the hash of their dependencies so solving dependency chains becomes a breeze
- uses udp so STUN protocol can be used (open battles which are not started can be used as bootstrap nodes, since they'll have open spring udp port), so every user which is in the lobby server but not in a game will automatically share his spring files
- capable of handling large files efficiently
- serverless
- doesn't have any queue system (people expects to start the download in relatively short time, especially if trying to join a battle)

some p2p general problems:

-the majority if not totality of people have asymmetrical bandwith, so like all the p2p networks, the demand for the files is higher than the offer
-the less common a file is, the lower is the possibility to find/achive a decent download speed
-even with the aid of external libraries, it's very hard to code and get running efficiently
-no control over distributed files

some centralized systems problems:

- a single point of failure, if the the main node fails, the whole network goes down
- load can only increase with the more users spring will get
- the server has to supply the demand of most(if not all) the users without glitches
- it's much harder (compared to p2p) to add more server nodes to increase capacity


off course some of the problems could be solved such as still rely unknown files (or similar external hosting) for the more rare files (in the p2p case) or have lists of multiple mirror servers ( server case ) but some of them are intrinsecally present in the type of solution

as a possibility for tracking files metadata, the kad protocol could be used to keep track of the files (it's used both by ed2k/emule and trackerless bittorent networks), but that is not capable of sending bulk data like the files themself, another network which supports distributed hash table would have to be used but i don't know which yet, suggestions would be welcome ( again bittorent and emule would not be ideal for the aforementioned reasons)

[imbaczek]

bittorrent would be cool if it was made to work like the WoW updater: seeds are e.g. on UF, but every client that wants to download something joins the cloud and eases the server's life.