View topic - XML-RPC searchapi on springfiles.com



All times are UTC + 1 hour


Post new topic Reply to topic  [ 78 posts ]  Go to page Previous  1, 2, 3, 4
Author Message
PostPosted: 28 Mar 2011, 01:53 
Server Owner & Developer
User avatar

Joined: 19 May 2006, 18:13
Location: Brno, Czech rep., EU, Terra, Sol, Orion arm, Milky way, Virgo supercluster
If you changed interface or renamed files randomly in repositiories it might cause it, i will try to debug that when i have time. Do you get requests for the file?


Top
 Offline Profile  
 
PostPosted: 28 Mar 2011, 02:21 
Spring Developer

Joined: 31 May 2009, 23:08
yes, now it works... i don't know where the ~3 hours of delay come from.

i improved partly the cron job (for uploading / indexing new files) so it is run soon when new files are uploaded or at the given schedule (currently every two hours) so we hopefully have only a delay of a few minutes.


Top
 Offline Profile  
 
PostPosted: 30 Mar 2011, 01:25 
Spring Developer

Joined: 31 May 2009, 23:08
plasma lists this url as mirror:

"http://www.darkstars.co.uk/downloads/view.php?dl=1&file=spring/maps/Inculta.sd7", this is also a duplicate url of http://jj.darkstars.co.uk/c2/Inculta.sd7

but this file is damaged :-/ (md5sum incorrect)

this currently breaks downloading for springlobby


please do something against listing urls not returned by xml-rpc interface :-/


Top
 Offline Profile  
 
PostPosted: 30 Mar 2011, 10:40 
Server Owner & Developer
User avatar

Joined: 19 May 2006, 18:13
Location: Brno, Czech rep., EU, Terra, Sol, Orion arm, Milky way, Virgo supercluster
Plasma server returns torrent file to verify hashes for each piece .. all downloaders should always verify downloaded piece.

I recommend deleting broken file.

There are more mirrors than those tracked by springfiles, including old web repositories which are simply remotely checked for presence of correctly sized file.

I cannot turn off this ability because its needed for some things which are not trackedby SF, like missions or builds of rapid mods for springlobby.

I also dont want to be 100% dependent on springfiles because it had to be shutdown many times in the past.

Please return "banlist" instead - mirrors which should be avoided.


Top
 Offline Profile  
 
PostPosted: 30 Mar 2011, 12:27 
Spring Developer

Joined: 31 May 2009, 23:08
i have no access to this file, i can't delete it :-/

where to return the banlist?

the banlist would be all links that are not listed in the result... the xml-rpc doesn't know links outside of the mirror-system.

so: i had to query plasma-service to get all urls + compare with own list -> make banlist :-/

imo you can delete all urls with ? or & in its url.


Top
 Offline Profile  
 
PostPosted: 30 Mar 2011, 12:35 
AI Coder
User avatar

Joined: 14 Sep 2004, 10:32
Location: Cookieland
http://jj.darkstars.co.uk was managed by JJ. He did not comply with requests and thus I had to do it myself, requiring I change the password as I didn't have the original email containing it. JJ never asked for the new password despite it being offered. All I have done since is remove htaccess files and change file permissions to make things more open

The details for the account are stored on my main machine ( which recently broke and is being replaced ). It would be a few days minimum before I can regain access to that particular subdomain. I would recommend you amend your system so that host involvement is not necessary ( if only because I can now launch a makeshift attack by modifying the files on my servers, and you would require the cooperation of the attacker to fix it )


Top
 Offline Profile  
 
PostPosted: 30 Mar 2011, 14:42 
Server Owner & Developer
User avatar

Joined: 19 May 2006, 18:13
Location: Brno, Czech rep., EU, Terra, Sol, Orion arm, Milky way, Virgo supercluster
abma wrote:
i have no access to this file, i can't delete it :-/

where to return the banlist?

the banlist would be all links that are not listed in the result... the xml-rpc doesn't know links outside of the mirror-system.

so: i had to query plasma-service to get all urls + compare with own list -> make banlist :-/

imo you can delete all urls with ? or & in its url.



But in the case of evolution you know that right? You could ban those..

These are mirrors it uses:

http://www.tasdownloads.com/uploads/%t/
http://www.evolutionrts.info/%t/%f

http://spring.vsync.de/%t/%f

http://www.jobjol.com/spring/%t/%f

http://www.ta-spring.com/%t/%f

http://www.fileuniverse.com/spring/%t/%f

http://packages.caspring.org/builds/%f

http://packages.springrts.com/builds/%f


It also used in the past:
http://www.springfiles.com/system/files ... ing/games/%f
http://www.springfiles.com/system/files ... ring-maps/%f


Top
 Offline Profile  
 
PostPosted: 30 Mar 2011, 14:47 
Server Owner & Developer
User avatar

Joined: 19 May 2006, 18:13
Location: Brno, Czech rep., EU, Terra, Sol, Orion arm, Milky way, Virgo supercluster
I could give you edit access to this list but deleting old mirror links means deleting it from database.

Even if I remove mirror site from the list, if the file is still present it will keep usingit.


Top
 Offline Profile  
 
PostPosted: 09 Apr 2011, 17:50 
Spring Developer

Joined: 31 May 2009, 23:08
i think deleting by hand makes no sense, a blacklist is bad, too.

as springlobby + zero-k (i hope so it does) checks for valid files for now, that shouldn't be a problem.

i made a small update/fix on the searchapi:

the comparison of springname is now binary + underscore is now threated as underscore and not as wildcard. I found that because when searching for "Lost_v2" two results where returned. "Lost-v2", exists, too.

results should be now more accurate.


Top
 Offline Profile  
 
PostPosted: 09 Apr 2011, 19:01 
Server Owner & Developer
User avatar

Joined: 19 May 2006, 18:13
Location: Brno, Czech rep., EU, Terra, Sol, Orion arm, Milky way, Virgo supercluster
They should both check that file content is valid, however tasclient probably does not do that, so it could download faulty file.

Sometimes all mirrors are "corrupted" meaning torrent is for another file than in the mirror system. In such cases i delete the file from the system and it gets back with correct torrent usually.


Top
 Offline Profile  
 
PostPosted: 30 May 2011, 23:37 
Spring Developer

Joined: 31 May 2009, 23:08
now "name" and "version" are additionally returned, torrent data will be returned when "torrent" is set in request. (see doc/exampleclient.py)


Top
 Offline Profile  
 
PostPosted: 12 Sep 2011, 18:08 
Spring Developer

Joined: 31 May 2009, 23:08
added "tag" to search + added "sdp" to the result.

./exampleclient.py "zk:stable"

[...]
'tags': ['zk:revision:3767', 'zk:stable'],
'sdp': 'http://packages.springrts.com/packages/91950372c85cb9c6b965c03061087aa0.sdp',
[...]

note:

tags are returned when they are on rapid _and_ springfiles.com.

the info is added by a script that is currently run daily by cron:


Top
 Offline Profile  
 
PostPosted: 17 Dec 2011, 11:43 
Spring Developer

Joined: 31 May 2009, 23:08
major update:

http://api.springfiles.com/, the page allows testing of input to the xmlrpc interface and shows results.

please use http://api.springfiles.com/xmlrpc.php as soon as possible, the old url will be switched off soon.

its still a work in progress, but should work nicely.

changes:

- result returns links to splash / map images
- added metadata in results
- sdp hash for all files is calculated
- its much more faster than the old one!
- search for tags works (ba:latest for example), currently only for files that are also known on springfiles.com
- stuff i forgot...
- all source can be found at http://github.com/springfiles/upq (moved many stuff from the drupal module into upq)


Top
 Offline Profile  
 
PostPosted: 23 Dec 2011, 21:04 
Spring Developer

Joined: 31 May 2009, 23:08
small update (maybe major...):

i normalized all filenames, new files filenames will be normalized, too:

the name is now always:

<lowercase springname>[-version].<suffix>

also: duplicate filenames and version-numbers aren't possible any more on the springfiles.com mirror-system. currently only files on api.springfiles.com are normalized, soon i'll normalize all filenames on the mirrors, too.

broken files (files that cant't be loaded by unitsync) are moved to http://api.springfiles.com/broken/ , maybe this way we get a higher quality of files that are downloadable...


note: i hopefully have soon the courage to activate auto-creating of pages for files on springfiles.com. the code is ready and partly tested, but i fear a bit, that it does stupid things ;)


Top
 Offline Profile  
 
PostPosted: 23 Dec 2011, 21:44 
Spring Developer
User avatar

Joined: 22 Sep 2007, 08:51
ehhh that sound good! especially the broken files thing. :-)


Top
 Offline Profile  
 
PostPosted: 23 Dec 2011, 23:02 
Server Owner & Developer
User avatar

Joined: 19 May 2006, 18:13
Location: Brno, Czech rep., EU, Terra, Sol, Orion arm, Milky way, Virgo supercluster
As soon as unitsync SPRING_DATADIR is fixed (it ignores it), i will do that tool to auto regiser all new content so they will be instantly available on autohosts.


Top
 Offline Profile  
 
PostPosted: 23 Dec 2011, 23:53 
Spring Developer

Joined: 31 May 2009, 23:08
Licho wrote:
As soon as unitsync SPRING_DATADIR is fixed (it ignores it), i will do that tool to auto regiser all new content so they will be instantly available on autohosts.


i tapped into the same issue on linux and but made a work-around by setting HOME... don't know if that works on win32...

can't we bundle manpower and don't write the same stuff twice? it feels stupid, to code it in python and .net, one language should be enough?!


Top
 Offline Profile  
 
PostPosted: 24 Dec 2011, 01:52 
Server Owner & Developer
User avatar

Joined: 19 May 2006, 18:13
Location: Brno, Czech rep., EU, Terra, Sol, Orion arm, Milky way, Virgo supercluster
Sure!
Hmm setting home.. i can try it but i doubt it wil use it on win.


Top
 Offline Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 78 posts ]  Go to page Previous  1, 2, 3, 4

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 0 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group

Site layout created by Roflcopter et al.