XML-RPC searchapi on springfiles.com

XML-RPC searchapi on springfiles.com

For the discussion of infrastructure improvements and changes.

Moderator: Moderators

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

XML-RPC searchapi on springfiles.com

Post by abma » 18 Jan 2011, 10:09

i've written a module for http://springfiles.com, that allows searching trough XML-RPC for files:

https://github.com/springfiles/upq

python implementation of a client:

https://github.com/springfiles/upq/blob ... eclient.py

example-query + response:
https://github.com/springfiles/upq/blob ... amplequery

implementation in c++:

https://github.com/abma/pr-downloader/b ... loader.cpp

any hints, suggestions, critism or questions? please fill in an issue-request: https://github.com/springfiles/upq/issues

for testing use http://api.springfiles.com/ it uses the same parameters as the xml-rpc interface.
Last edited by abma on 17 Dec 2011, 11:53, edited 18 times in total.
0 x

User avatar
koshi
Lobby Developer
Posts: 1058
Joined: 14 Aug 2007, 16:15

Re: feedback for XML-RPC searchapi on springfiles wanted

Post by koshi » 18 Jan 2011, 14:36

- search by hash and dependencies and whatever I've mentioned in pm a couple of days back ;-)

Is it that currently only complete matches for springname are returned or am I just having the worst of luck trying different inputs to that python script?
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: feedback for XML-RPC searchapi on springfiles wanted

Post by abma » 18 Jan 2011, 18:37

that's still on the todo :)

is search by hash a must-have? isn't the internal name enough?

afaik can the hash change at each spring release, the springname shouldn't change...

yep, only exact matches. placeholders are currently possible, just search for a "%". but i'm unsure if that is really needed.

currently i try to get unitsync extracting that data, that i want to have.




(updated: now an array is returned with structs inside)
0 x

User avatar
koshi
Lobby Developer
Posts: 1058
Joined: 14 Aug 2007, 16:15

Re: feedback for XML-RPC searchapi on springfiles wanted

Post by koshi » 18 Jan 2011, 18:46

For an integrated auto-downloader partial matches aren't really needed, no. A more elaborate GUI that might offer choices to the user could use it though.

Hashes are kindof needed to foil attempts of poisoned downloads replacing the 'real' file with something that just has the same thing in modinfo imo.

Are requests cached server-side already?
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: feedback for XML-RPC searchapi on springfiles wanted

Post by abma » 18 Jan 2011, 19:43

no, there is no caching currently, the mysql-caching could be enough. as searching is only one long sql-query with some joins, it should be fast.

how to get the checksums? GetMapChecksum + GetPrimaryModChecksum?

(Update: depends added)
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: feedback for XML-RPC searchapi on springfiles wanted

Post by abma » 19 Jan 2011, 09:06

Update: added param "logical", allowed values are "or" / "and", defaults to "and". (Thanks Licho for this suggestion)
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: feedback for XML-RPC searchapi on springfiles wanted

Post by abma » 25 Jan 2011, 00:07

Update: will be set live, soon + i added github url to drupal-module + scripts
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles

Post by abma » 26 Jan 2011, 13:24

is now avaiable on springfiles.com! :-)
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles

Post by Licho » 26 Jan 2011, 13:24

What url?

For others - this will be used as part of "plasmaserver" to find more mirrors on springfiles for downloaders in tasclient, springlobby and zero-k. Past method was to use normal search on SF and read resulting html.
Last edited by Licho on 26 Jan 2011, 13:26, edited 1 time in total.
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles

Post by abma » 26 Jan 2011, 13:25

0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles

Post by Licho » 27 Jan 2011, 02:58

This does not generate WSDL with ?wsdl as normal.
How to get WSDL ?

Without WSDL we cannot generate proxy classes for client.
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles

Post by Licho » 27 Jan 2011, 03:50

I analysed result and generated proxy classes manualy - it works now.

However you should provide standard description format for future consumers.

Plasma server now use this service to discover more mirrors for the benefit of lobbies.
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma » 27 Jan 2011, 13:57

nice!

about wsdl: xmlrpc can't generate such file automaticly, this introduced with soap, because soap is complicated :)

i added an example request + response to the docs, maybe this helps, too.
0 x

User avatar
jj
Posts: 273
Joined: 01 Apr 2006, 19:03

Re: XML-RPC searchapi on springfiles.com

Post by jj » 27 Jan 2011, 18:10

Does the auto downloader downloads a file multiple times or from different mirrors at the same time?

If so i have to adapt the download statistics module
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles.com

Post by Licho » 27 Jan 2011, 19:41

It depends which one. For example tasclient picks one mirror and uses it.

Zero-K uses multiple pieces and often opens new connection meaning it could produce easilly 20 "download stats".
Springlobby use multiple pieces but keeps connection open.

Thats why plasma server itself counts downloads.

I already said it would be best to provide unmeasured URL and simply add plasmaserver download count to yours.
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles.com

Post by Licho » 28 Jan 2011, 02:59

Critical problem! All URL you return do not respond to http range requests!

PLEASE DO NOT SEND ME URL REDIRECT WITHOUT RANGE SUPPORT, I NEED DIRECT URL..

As a result downloads are broken in lobbies. Im disabling use of this system until its fixed.

Multipart downloaders need URL with range header support for partial downloads.
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles.com

Post by Licho » 28 Jan 2011, 03:32

Also they don't appear to be redirects which means you are wasting valuable resources on running php to provide download. Please make it return full url, main server php count is limited, dont waste it on long running downloads!
0 x

User avatar
hoijui
Former Engine Dev
Posts: 4342
Joined: 22 Sep 2007, 09:51

Re: XML-RPC searchapi on springfiles.com

Post by hoijui » 28 Jan 2011, 12:03

could you probably call/ping an additional URL, when the download is finished? i assume that would be easiest for jj/abma. at least it appears easier to me then fetching finished downloads from plasma server though some other system, even though it may be less efficient.
maybe that only makes sense form my very remote point of view though.
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma » 28 Jan 2011, 12:06

@licho:
afaik only on the main site it isn't redirected to a plain file, for mirrors this is wrong:

Code: Select all

wget -v http://www.springfiles.com/download/3668/5
--2011-01-28 12:06:55--  http://www.springfiles.com/download/3668/5
Resolving www.springfiles.com... 94.23.170.70
Connecting to www.springfiles.com|94.23.170.70|:80... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: http://springfiles.abma.de/downloads/spring/games/zk-v0.6.1.sdz [following]
--2011-01-28 12:06:56--  http://springfiles.abma.de/downloads/spring/games/zk-v0.6.1.sdz
Resolving springfiles.abma.de... 109.230.210.75
Connecting to springfiles.abma.de|109.230.210.75|:80... connected.
HTTP request sent, awaiting response... 200 OK
the problem seems here, that the clients ignore the 301 redirect and try to get a part from the php script which will fail...
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles.com

Post by Licho » 28 Jan 2011, 13:18

No, it's not redirected for many URL and does not listen to range.

I'm sending the request manually on TCP connection and i see it returns full result when i request range.

This is the worst outcome.
It confuses all clients and overloads server because apparently its being served by php!

All the URL in the format
http://www.springfiles.com/download/3668/5

respond directly with file and ignore range -> they are served by php despite the fact that I was being assured by jj he wont waste server resources on that.

Any single URL with no range compatibility easilly screws torrent download or makes it very slow.
0 x

Post Reply

Return to “Infrastructure Development”