What you mean - if you can list custom mirrors in the downloads you can sure remove this one and add fixed one instead..

Jj assured me there are no php downloads and now you are saying me there are and cannot be turned off?

The /system url is made by drupal and can't be really switched of.

Disallow /system/files/downloads/spring/ is a bit ugly + may break other things and will make trouble after updates of drupal.

Even after removing the /system links from displaying, it will be still valid. As it isn't returned by the search-api, the /system links should be deleted from the plasma-service.

Best would be, if plasma detects the HTTP/500 error + mark the link as invalid / damaged.

Valid links are not a problem if they are not linked. I can remove it from plasma no problem.

But atm its also primary download link on the site and this should be fixed.

I also think that disabling it in apache should be pretty safe.

So please

1) do not link the php based drupal file downlaods

2) remove springfiles.planet-wars.eu from mirror list
Its identical server so there is no point mirroring it

1) They won't be linked for the RPC service. They are linked in the site itself because its a drupal thing and lots of modules depends on it. It takes a lot of time to alter it.

2) its not used anywhere.

Cant you hide them or something? If not we should use apache rewrite rule to direct that request away from php.

the old links system/... are 301-redirected to real http-files by an apache rewrite-rule, manual-downloads are through a 302 redirect php script (for stats), so no more php-downloads.

when watching the xml-rpc logs, i saw that most of the file-searches won't return anything, only searching for the springname of a file will return a result... so... removing the "or filename" in a searchrequest in plasma would return the same files.

Shall i normalize the filenames in the results? that would mean generating the filename from springname + filetype? Afaik the filenames on mirrors arent always the same filename on springfiles.

something like:

getfilename(springname){
springname.tolower()
springname.replace specialchars with _
springname.addsuffix(springname.filetype)
}


@licho: did you see any problems with the searchapi?

Or i dont understand much :)

I dont think there are problems, but after recent change mirror server count seems to go down.

with normalizing i mean:

in modinfo.lua: name="This is a awesome Game v1"
file will renamed to = This_is_a_awesome_Game_v1.sd7

(even it is uploaded with filename awesomegame.sd7)


there are currently only 3 active mirrors, so for new files you'll get at max 4 urls. i asked other sites for an ftp-access, if they give springfiles.com one, mirrors will increase.

i'm still thinking about how to make the system a bit more distributed. I dislike, that only the main server has the xml-rpc interface, but the mirrors shouldn't require a database. any idea about that, how lobby-clients can find the required files without relying on a single host?

In rapid each repository has simple zipped text file with list of content published on web..

You could simply generate list.

Or, if the files are named exactly by internal name, you dont need any index.

Server would simply ask each mirror site for given file name ...

Regarding normalization, please do not do it this way..

Either do it for all files at once and ommit random .sdz/.sd7 extension, or dont do it at all..

Reason is plasma server mirror checking. In fact it does what i suggested, in case RPC is offline, it queries each known mirror server for filename.

Filename is from the list of known filenames users had locally.

If you rename it on the server, this part would fail.

ok, i won't rename files. only stuff on todo is, to move files from old file-structure into new, for example:
2/*.sd7 -> downloads/spring/games/*.sd7

on some mirrors there are currently both file-structures :-/


i (re-)added www.tasdownloads.com and files.springrts.de as mirrors! thanks to dave and very_bad_soldier! :)

mirrors for files should grow soon.

It shouldnt be a problem if there are multiple folder structures @ mirrors.

SpringFiles will index existing files when a mirror is added. For new uploads the new folder structure will be used.

Jj, please run rsync and other synchronization tools (ftp copies) at low priority, use ionice -c 3 to prevent overloading server disk.

the upload-script can be easily "identified", as it runs as user jj and not as the apache-user.it runs already with nice -19 ionice -c3... so, lower priority isn't possible. it runs currently so long, because 3 mirrors were recently added, about ~90GB have to be uploaded, this takes some time, as only 10 files are uploaded at once and some mirrors allow uploading only at 200KB/s.
when mirrors are synced, it should only run a few secs.

the php-script should do few io, because the php script does a sql-query + reads only needed files for uploading (and doesn't a stat() on all files).

uploading by ftp would mean, to re-upload huge-parts of the files, as folder structure changed.

rsync/lftp have drawbacks, too: either they are invoked by cron, and don't know which files to upload and need to rescan all files or they know it (but this has to be written somewhere in a script too...) so why not upload it directly by the script?

hm, one thing is bad with plasma: if i disable a mirror, plasma still lists this site as mirror :-/

can you make plasma work like this:
-always request xml-rpc api (or cache it only for a few minutes), return this mirrors.
-if it doesn't return an answer, return all mirrors

currently evolutionrts.info seems to be overloaded, i already disabled it as file-mirror (isn't returned as result in xml-rpc interface), but plasma still uses it as mirror

edit: reenabled it, as it had no effect :-/

i wrote a small script that moved all files from the old file structure in the new.

it looks like some files where deleted on springfiles but still exists on some mirrors:

http://files.springrts.de/2/

what to do with this files? re-add to springfiles.com?

imo deleting is a bad idea, think about engine devs try to debug a replay with a old map...

I still see rsync running at full CPU and IO priorities under jj..
(nice 0, best effort 4)

sorry, forgot to set nice on the remote rsync.

(parameter for rsync is --rsync-path="nice -19 ionice -c3 rsync" if anyone else needs this)

hmm, BA 7.31 was uploaded at 18:39, plasma still doesn't list any mirrors :-/

searching should return results for "Balanced Annihilation v7.31" since 18:59... (how) can we improve that?