XML-RPC searchapi on springfiles.com - Page 4

XML-RPC searchapi on springfiles.com

For the discussion of infrastructure improvements and changes.

Moderator: Moderators

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles.com

Post by Licho »

If you changed interface or renamed files randomly in repositiories it might cause it, i will try to debug that when i have time. Do you get requests for the file?
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma »

yes, now it works... i don't know where the ~3 hours of delay come from.

i improved partly the cron job (for uploading / indexing new files) so it is run soon when new files are uploaded or at the given schedule (currently every two hours) so we hopefully have only a delay of a few minutes.
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma »

plasma lists this url as mirror:

"http://www.darkstars.co.uk/downloads/vi ... nculta.sd7", this is also a duplicate url of http://jj.darkstars.co.uk/c2/Inculta.sd7

but this file is damaged :-/ (md5sum incorrect)

this currently breaks downloading for springlobby


please do something against listing urls not returned by xml-rpc interface :-/
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles.com

Post by Licho »

Plasma server returns torrent file to verify hashes for each piece .. all downloaders should always verify downloaded piece.

I recommend deleting broken file.

There are more mirrors than those tracked by springfiles, including old web repositories which are simply remotely checked for presence of correctly sized file.

I cannot turn off this ability because its needed for some things which are not trackedby SF, like missions or builds of rapid mods for springlobby.

I also dont want to be 100% dependent on springfiles because it had to be shutdown many times in the past.

Please return "banlist" instead - mirrors which should be avoided.
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma »

i have no access to this file, i can't delete it :-/

where to return the banlist?

the banlist would be all links that are not listed in the result... the xml-rpc doesn't know links outside of the mirror-system.

so: i had to query plasma-service to get all urls + compare with own list -> make banlist :-/

imo you can delete all urls with ? or & in its url.
User avatar
AF
AI Developer
Posts: 20687
Joined: 14 Sep 2004, 11:32

Re: XML-RPC searchapi on springfiles.com

Post by AF »

http://jj.darkstars.co.uk was managed by JJ. He did not comply with requests and thus I had to do it myself, requiring I change the password as I didn't have the original email containing it. JJ never asked for the new password despite it being offered. All I have done since is remove htaccess files and change file permissions to make things more open

The details for the account are stored on my main machine ( which recently broke and is being replaced ). It would be a few days minimum before I can regain access to that particular subdomain. I would recommend you amend your system so that host involvement is not necessary ( if only because I can now launch a makeshift attack by modifying the files on my servers, and you would require the cooperation of the attacker to fix it )
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles.com

Post by Licho »

abma wrote:i have no access to this file, i can't delete it :-/

where to return the banlist?

the banlist would be all links that are not listed in the result... the xml-rpc doesn't know links outside of the mirror-system.

so: i had to query plasma-service to get all urls + compare with own list -> make banlist :-/

imo you can delete all urls with ? or & in its url.

But in the case of evolution you know that right? You could ban those..

These are mirrors it uses:

http://www.tasdownloads.com/uploads/%t/
http://www.evolutionrts.info/%t/%f

http://spring.vsync.de/%t/%f

http://www.jobjol.com/spring/%t/%f

http://www.ta-spring.com/%t/%f

http://www.fileuniverse.com/spring/%t/%f

http://packages.caspring.org/builds/%f

http://packages.springrts.com/builds/%f


It also used in the past:
http://www.springfiles.com/system/files ... g/games/%f
http://www.springfiles.com/system/files ... ng-maps/%f
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles.com

Post by Licho »

I could give you edit access to this list but deleting old mirror links means deleting it from database.

Even if I remove mirror site from the list, if the file is still present it will keep usingit.
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma »

i think deleting by hand makes no sense, a blacklist is bad, too.

as springlobby + zero-k (i hope so it does) checks for valid files for now, that shouldn't be a problem.

i made a small update/fix on the searchapi:

the comparison of springname is now binary + underscore is now threated as underscore and not as wildcard. I found that because when searching for "Lost_v2" two results where returned. "Lost-v2", exists, too.

results should be now more accurate.
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles.com

Post by Licho »

They should both check that file content is valid, however tasclient probably does not do that, so it could download faulty file.

Sometimes all mirrors are "corrupted" meaning torrent is for another file than in the mirror system. In such cases i delete the file from the system and it gets back with correct torrent usually.
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma »

now "name" and "version" are additionally returned, torrent data will be returned when "torrent" is set in request. (see doc/exampleclient.py)
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma »

added "tag" to search + added "sdp" to the result.

./exampleclient.py "zk:stable"

[...]
'tags': ['zk:revision:3767', 'zk:stable'],
'sdp': 'http://packages.springrts.com/packages/ ... 087aa0.sdp',
[...]

note:

tags are returned when they are on rapid _and_ springfiles.com.

the info is added by a script that is currently run daily by cron:
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma »

major update:

http://api.springfiles.com/, the page allows testing of input to the xmlrpc interface and shows results.

please use http://api.springfiles.com/xmlrpc.php as soon as possible, the old url will be switched off soon.

its still a work in progress, but should work nicely.

changes:

- result returns links to splash / map images
- added metadata in results
- sdp hash for all files is calculated
- its much more faster than the old one!
- search for tags works (ba:latest for example), currently only for files that are also known on springfiles.com
- stuff i forgot...
- all source can be found at http://github.com/springfiles/upq (moved many stuff from the drupal module into upq)
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma »

small update (maybe major...):

i normalized all filenames, new files filenames will be normalized, too:

the name is now always:

<lowercase springname>[-version].<suffix>

also: duplicate filenames and version-numbers aren't possible any more on the springfiles.com mirror-system. currently only files on api.springfiles.com are normalized, soon i'll normalize all filenames on the mirrors, too.

broken files (files that cant't be loaded by unitsync) are moved to http://api.springfiles.com/broken/ , maybe this way we get a higher quality of files that are downloadable...


note: i hopefully have soon the courage to activate auto-creating of pages for files on springfiles.com. the code is ready and partly tested, but i fear a bit, that it does stupid things ;)
User avatar
hoijui
Former Engine Dev
Posts: 4344
Joined: 22 Sep 2007, 09:51

Re: XML-RPC searchapi on springfiles.com

Post by hoijui »

ehhh that sound good! especially the broken files thing. :-)
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles.com

Post by Licho »

As soon as unitsync SPRING_DATADIR is fixed (it ignores it), i will do that tool to auto regiser all new content so they will be instantly available on autohosts.
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma »

Licho wrote:As soon as unitsync SPRING_DATADIR is fixed (it ignores it), i will do that tool to auto regiser all new content so they will be instantly available on autohosts.
i tapped into the same issue on linux and but made a work-around by setting HOME... don't know if that works on win32...

can't we bundle manpower and don't write the same stuff twice? it feels stupid, to code it in python and .net, one language should be enough?!
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles.com

Post by Licho »

Sure!
Hmm setting home.. i can try it but i doubt it wil use it on win.
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma »

added the parameter "limit" which allows to restrict amount of results returned. defaults to 10 when not set (as before)

now for example a list of all engine versions can be shown:

http://api.springfiles.com/?category=en ... ws&limit=0

(the parameter works for the xml-interface, too)

please don't abuse this parameter, as in 99% of cases you don't need a whole list: cache results as they shouldn't change very often.

https://github.com/springfiles/upq/comm ... b7d3bda669
Post Reply

Return to “Infrastructure Development”