View topic - XML-RPC searchapi on springfiles.com

View unanswered posts | View active topics



Board index » Development » Development » Infrastructure Development

All times are UTC + 1 hour


Post new topic Reply to topic  Page 1 of 4
  [ 78 posts ]  Go to page 1, 2, 3, 4  Next
  Previous topic | Next topic 
Author Message
abma
PostPosted: 18 Jan 2011, 10:09 
Spring Developer

Joined: 31 May 2009, 23:08
i've written a module for http://springfiles.com, that allows searching trough XML-RPC for files:

https://github.com/springfiles/upq

python implementation of a client:

https://github.com/springfiles/upq/blob ... eclient.py

example-query + response:
https://github.com/springfiles/upq/blob ... amplequery

implementation in c++:

https://github.com/abma/pr-downloader/b ... loader.cpp

any hints, suggestions, critism or questions? please fill in an issue-request: https://github.com/springfiles/upq/issues

for testing use http://api.springfiles.com/ it uses the same parameters as the xml-rpc interface.


Last edited by abma on 17 Dec 2011, 11:53, edited 18 times in total.

Top
 Offline Profile  
 
koshi
PostPosted: 18 Jan 2011, 14:36 
Lobby Developer
User avatar

Joined: 14 Aug 2007, 15:15
- search by hash and dependencies and whatever I've mentioned in pm a couple of days back ;-)

Is it that currently only complete matches for springname are returned or am I just having the worst of luck trying different inputs to that python script?
Top
 Offline Profile  
 
abma
PostPosted: 18 Jan 2011, 18:37 
Spring Developer

Joined: 31 May 2009, 23:08
that's still on the todo :)

is search by hash a must-have? isn't the internal name enough?

afaik can the hash change at each spring release, the springname shouldn't change...

yep, only exact matches. placeholders are currently possible, just search for a "%". but i'm unsure if that is really needed.

currently i try to get unitsync extracting that data, that i want to have.




(updated: now an array is returned with structs inside)
Top
 Offline Profile  
 
koshi
PostPosted: 18 Jan 2011, 18:46 
Lobby Developer
User avatar

Joined: 14 Aug 2007, 15:15
For an integrated auto-downloader partial matches aren't really needed, no. A more elaborate GUI that might offer choices to the user could use it though.

Hashes are kindof needed to foil attempts of poisoned downloads replacing the 'real' file with something that just has the same thing in modinfo imo.

Are requests cached server-side already?
Top
 Offline Profile  
 
abma
PostPosted: 18 Jan 2011, 19:43 
Spring Developer

Joined: 31 May 2009, 23:08
no, there is no caching currently, the mysql-caching could be enough. as searching is only one long sql-query with some joins, it should be fast.

how to get the checksums? GetMapChecksum + GetPrimaryModChecksum?

(Update: depends added)
Top
 Offline Profile  
 
abma
PostPosted: 19 Jan 2011, 09:06 
Spring Developer

Joined: 31 May 2009, 23:08
Update: added param "logical", allowed values are "or" / "and", defaults to "and". (Thanks Licho for this suggestion)
Top
 Offline Profile  
 
abma
PostPosted: 25 Jan 2011, 00:07 
Spring Developer

Joined: 31 May 2009, 23:08
Update: will be set live, soon + i added github url to drupal-module + scripts
Top
 Offline Profile  
 
abma
PostPosted: 26 Jan 2011, 13:24 
Spring Developer

Joined: 31 May 2009, 23:08
is now avaiable on springfiles.com! :-)
Top
 Offline Profile  
 
Licho
PostPosted: 26 Jan 2011, 13:24 
Server Owner & Developer
User avatar

Joined: 19 May 2006, 18:13
Location: Brno, Czech rep., EU, Terra, Sol, Orion arm, Milky way, Virgo supercluster
What url?

For others - this will be used as part of "plasmaserver" to find more mirrors on springfiles for downloaders in tasclient, springlobby and zero-k. Past method was to use normal search on SF and read resulting html.


Last edited by Licho on 26 Jan 2011, 13:26, edited 1 time in total.

Top
 Offline Profile  
 
abma
PostPosted: 26 Jan 2011, 13:25 
Spring Developer

Joined: 31 May 2009, 23:08
http://springfiles.com/xmlrpc.php
Top
 Offline Profile  
 
Licho
PostPosted: 27 Jan 2011, 02:58 
Server Owner & Developer
User avatar

Joined: 19 May 2006, 18:13
Location: Brno, Czech rep., EU, Terra, Sol, Orion arm, Milky way, Virgo supercluster
This does not generate WSDL with ?wsdl as normal.
How to get WSDL ?

Without WSDL we cannot generate proxy classes for client.
Top
 Offline Profile  
 
Licho
PostPosted: 27 Jan 2011, 03:50 
Server Owner & Developer
User avatar

Joined: 19 May 2006, 18:13
Location: Brno, Czech rep., EU, Terra, Sol, Orion arm, Milky way, Virgo supercluster
I analysed result and generated proxy classes manualy - it works now.

However you should provide standard description format for future consumers.

Plasma server now use this service to discover more mirrors for the benefit of lobbies.
Top
 Offline Profile  
 
abma
PostPosted: 27 Jan 2011, 13:57 
Spring Developer

Joined: 31 May 2009, 23:08
nice!

about wsdl: xmlrpc can't generate such file automaticly, this introduced with soap, because soap is complicated :)

i added an example request + response to the docs, maybe this helps, too.
Top
 Offline Profile  
 
jj
PostPosted: 27 Jan 2011, 18:10 
User avatar

Joined: 01 Apr 2006, 18:03
Location: The Netherlands
Does the auto downloader downloads a file multiple times or from different mirrors at the same time?

If so i have to adapt the download statistics module
Top
 Offline Profile  
 
Licho
PostPosted: 27 Jan 2011, 19:41 
Server Owner & Developer
User avatar

Joined: 19 May 2006, 18:13
Location: Brno, Czech rep., EU, Terra, Sol, Orion arm, Milky way, Virgo supercluster
It depends which one. For example tasclient picks one mirror and uses it.

Zero-K uses multiple pieces and often opens new connection meaning it could produce easilly 20 "download stats".
Springlobby use multiple pieces but keeps connection open.

Thats why plasma server itself counts downloads.

I already said it would be best to provide unmeasured URL and simply add plasmaserver download count to yours.
Top
 Offline Profile  
 
Licho
PostPosted: 28 Jan 2011, 02:59 
Server Owner & Developer
User avatar

Joined: 19 May 2006, 18:13
Location: Brno, Czech rep., EU, Terra, Sol, Orion arm, Milky way, Virgo supercluster
Critical problem! All URL you return do not respond to http range requests!

PLEASE DO NOT SEND ME URL REDIRECT WITHOUT RANGE SUPPORT, I NEED DIRECT URL..

As a result downloads are broken in lobbies. Im disabling use of this system until its fixed.

Multipart downloaders need URL with range header support for partial downloads.
Top
 Offline Profile  
 
Licho
PostPosted: 28 Jan 2011, 03:32 
Server Owner & Developer
User avatar

Joined: 19 May 2006, 18:13
Location: Brno, Czech rep., EU, Terra, Sol, Orion arm, Milky way, Virgo supercluster
Also they don't appear to be redirects which means you are wasting valuable resources on running php to provide download. Please make it return full url, main server php count is limited, dont waste it on long running downloads!
Top
 Offline Profile  
 
hoijui
PostPosted: 28 Jan 2011, 12:03 
Spring Developer
User avatar

Joined: 22 Sep 2007, 08:51
could you probably call/ping an additional URL, when the download is finished? i assume that would be easiest for jj/abma. at least it appears easier to me then fetching finished downloads from plasma server though some other system, even though it may be less efficient.
maybe that only makes sense form my very remote point of view though.
Top
 Offline Profile  
 
abma
PostPosted: 28 Jan 2011, 12:06 
Spring Developer

Joined: 31 May 2009, 23:08
@licho:
afaik only on the main site it isn't redirected to a plain file, for mirrors this is wrong:
Code:
wget -v http://www.springfiles.com/download/3668/5
--2011-01-28 12:06:55--  http://www.springfiles.com/download/3668/5
Resolving www.springfiles.com... 94.23.170.70
Connecting to www.springfiles.com|94.23.170.70|:80... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: http://springfiles.abma.de/downloads/spring/games/zk-v0.6.1.sdz [following]
--2011-01-28 12:06:56--  http://springfiles.abma.de/downloads/spring/games/zk-v0.6.1.sdz
Resolving springfiles.abma.de... 109.230.210.75
Connecting to springfiles.abma.de|109.230.210.75|:80... connected.
HTTP request sent, awaiting response... 200 OK


the problem seems here, that the clients ignore the 301 redirect and try to get a part from the php script which will fail...
Top
 Offline Profile  
 
Licho
PostPosted: 28 Jan 2011, 13:18 
Server Owner & Developer
User avatar

Joined: 19 May 2006, 18:13
Location: Brno, Czech rep., EU, Terra, Sol, Orion arm, Milky way, Virgo supercluster
No, it's not redirected for many URL and does not listen to range.

I'm sending the request manually on TCP connection and i see it returns full result when i request range.

This is the worst outcome.
It confuses all clients and overloads server because apparently its being served by php!

All the URL in the format
http://www.springfiles.com/download/3668/5

respond directly with file and ignore range -> they are served by php despite the fact that I was being assured by jj he wont waste server resources on that.

Any single URL with no range compatibility easilly screws torrent download or makes it very slow.
Top
 Offline Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  Page 1 of 4
  [ 78 posts ]  Go to page 1, 2, 3, 4  Next

Board index » Development » Development » Infrastructure Development

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 1 guest

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group

Site layout created by Roflcopter et al.