XML-RPC searchapi on springfiles.com
Moderator: Moderators
XML-RPC searchapi on springfiles.com
i've written a module for http://springfiles.com, that allows searching trough XML-RPC for files:
https://github.com/springfiles/upq
python implementation of a client:
https://github.com/springfiles/upq/blob ... eclient.py
example-query + response:
https://github.com/springfiles/upq/blob ... amplequery
implementation in c++:
https://github.com/abma/pr-downloader/b ... loader.cpp
any hints, suggestions, critism or questions? please fill in an issue-request: https://github.com/springfiles/upq/issues
for testing use http://api.springfiles.com/ it uses the same parameters as the xml-rpc interface.
https://github.com/springfiles/upq
python implementation of a client:
https://github.com/springfiles/upq/blob ... eclient.py
example-query + response:
https://github.com/springfiles/upq/blob ... amplequery
implementation in c++:
https://github.com/abma/pr-downloader/b ... loader.cpp
any hints, suggestions, critism or questions? please fill in an issue-request: https://github.com/springfiles/upq/issues
for testing use http://api.springfiles.com/ it uses the same parameters as the xml-rpc interface.
Last edited by abma on 17 Dec 2011, 11:53, edited 18 times in total.
Re: feedback for XML-RPC searchapi on springfiles wanted
- search by hash and dependencies and whatever I've mentioned in pm a couple of days back
Is it that currently only complete matches for springname are returned or am I just having the worst of luck trying different inputs to that python script?
Is it that currently only complete matches for springname are returned or am I just having the worst of luck trying different inputs to that python script?
Re: feedback for XML-RPC searchapi on springfiles wanted
that's still on the todo :)
is search by hash a must-have? isn't the internal name enough?
afaik can the hash change at each spring release, the springname shouldn't change...
yep, only exact matches. placeholders are currently possible, just search for a "%". but i'm unsure if that is really needed.
currently i try to get unitsync extracting that data, that i want to have.
(updated: now an array is returned with structs inside)
is search by hash a must-have? isn't the internal name enough?
afaik can the hash change at each spring release, the springname shouldn't change...
yep, only exact matches. placeholders are currently possible, just search for a "%". but i'm unsure if that is really needed.
currently i try to get unitsync extracting that data, that i want to have.
(updated: now an array is returned with structs inside)
Re: feedback for XML-RPC searchapi on springfiles wanted
For an integrated auto-downloader partial matches aren't really needed, no. A more elaborate GUI that might offer choices to the user could use it though.
Hashes are kindof needed to foil attempts of poisoned downloads replacing the 'real' file with something that just has the same thing in modinfo imo.
Are requests cached server-side already?
Hashes are kindof needed to foil attempts of poisoned downloads replacing the 'real' file with something that just has the same thing in modinfo imo.
Are requests cached server-side already?
Re: feedback for XML-RPC searchapi on springfiles wanted
no, there is no caching currently, the mysql-caching could be enough. as searching is only one long sql-query with some joins, it should be fast.
how to get the checksums? GetMapChecksum + GetPrimaryModChecksum?
(Update: depends added)
how to get the checksums? GetMapChecksum + GetPrimaryModChecksum?
(Update: depends added)
Re: feedback for XML-RPC searchapi on springfiles wanted
Update: added param "logical", allowed values are "or" / "and", defaults to "and". (Thanks Licho for this suggestion)
Re: feedback for XML-RPC searchapi on springfiles wanted
Update: will be set live, soon + i added github url to drupal-module + scripts
Re: XML-RPC searchapi on springfiles
is now avaiable on springfiles.com!
Re: XML-RPC searchapi on springfiles
What url?
For others - this will be used as part of "plasmaserver" to find more mirrors on springfiles for downloaders in tasclient, springlobby and zero-k. Past method was to use normal search on SF and read resulting html.
For others - this will be used as part of "plasmaserver" to find more mirrors on springfiles for downloaders in tasclient, springlobby and zero-k. Past method was to use normal search on SF and read resulting html.
Last edited by Licho on 26 Jan 2011, 13:26, edited 1 time in total.
Re: XML-RPC searchapi on springfiles
This does not generate WSDL with ?wsdl as normal.
How to get WSDL ?
Without WSDL we cannot generate proxy classes for client.
How to get WSDL ?
Without WSDL we cannot generate proxy classes for client.
Re: XML-RPC searchapi on springfiles
I analysed result and generated proxy classes manualy - it works now.
However you should provide standard description format for future consumers.
Plasma server now use this service to discover more mirrors for the benefit of lobbies.
However you should provide standard description format for future consumers.
Plasma server now use this service to discover more mirrors for the benefit of lobbies.
Re: XML-RPC searchapi on springfiles.com
nice!
about wsdl: xmlrpc can't generate such file automaticly, this introduced with soap, because soap is complicated :)
i added an example request + response to the docs, maybe this helps, too.
about wsdl: xmlrpc can't generate such file automaticly, this introduced with soap, because soap is complicated :)
i added an example request + response to the docs, maybe this helps, too.
Re: XML-RPC searchapi on springfiles.com
Does the auto downloader downloads a file multiple times or from different mirrors at the same time?
If so i have to adapt the download statistics module
If so i have to adapt the download statistics module
Re: XML-RPC searchapi on springfiles.com
It depends which one. For example tasclient picks one mirror and uses it.
Zero-K uses multiple pieces and often opens new connection meaning it could produce easilly 20 "download stats".
Springlobby use multiple pieces but keeps connection open.
Thats why plasma server itself counts downloads.
I already said it would be best to provide unmeasured URL and simply add plasmaserver download count to yours.
Zero-K uses multiple pieces and often opens new connection meaning it could produce easilly 20 "download stats".
Springlobby use multiple pieces but keeps connection open.
Thats why plasma server itself counts downloads.
I already said it would be best to provide unmeasured URL and simply add plasmaserver download count to yours.
Re: XML-RPC searchapi on springfiles.com
Critical problem! All URL you return do not respond to http range requests!
PLEASE DO NOT SEND ME URL REDIRECT WITHOUT RANGE SUPPORT, I NEED DIRECT URL..
As a result downloads are broken in lobbies. Im disabling use of this system until its fixed.
Multipart downloaders need URL with range header support for partial downloads.
PLEASE DO NOT SEND ME URL REDIRECT WITHOUT RANGE SUPPORT, I NEED DIRECT URL..
As a result downloads are broken in lobbies. Im disabling use of this system until its fixed.
Multipart downloaders need URL with range header support for partial downloads.
Re: XML-RPC searchapi on springfiles.com
Also they don't appear to be redirects which means you are wasting valuable resources on running php to provide download. Please make it return full url, main server php count is limited, dont waste it on long running downloads!
Re: XML-RPC searchapi on springfiles.com
could you probably call/ping an additional URL, when the download is finished? i assume that would be easiest for jj/abma. at least it appears easier to me then fetching finished downloads from plasma server though some other system, even though it may be less efficient.
maybe that only makes sense form my very remote point of view though.
maybe that only makes sense form my very remote point of view though.
Re: XML-RPC searchapi on springfiles.com
@licho:
afaik only on the main site it isn't redirected to a plain file, for mirrors this is wrong:
the problem seems here, that the clients ignore the 301 redirect and try to get a part from the php script which will fail...
afaik only on the main site it isn't redirected to a plain file, for mirrors this is wrong:
Code: Select all
wget -v http://www.springfiles.com/download/3668/5
--2011-01-28 12:06:55-- http://www.springfiles.com/download/3668/5
Resolving www.springfiles.com... 94.23.170.70
Connecting to www.springfiles.com|94.23.170.70|:80... connected.
HTTP request sent, awaiting response... 302 Moved Temporarily
Location: http://springfiles.abma.de/downloads/spring/games/zk-v0.6.1.sdz [following]
--2011-01-28 12:06:56-- http://springfiles.abma.de/downloads/spring/games/zk-v0.6.1.sdz
Resolving springfiles.abma.de... 109.230.210.75
Connecting to springfiles.abma.de|109.230.210.75|:80... connected.
HTTP request sent, awaiting response... 200 OK
Re: XML-RPC searchapi on springfiles.com
No, it's not redirected for many URL and does not listen to range.
I'm sending the request manually on TCP connection and i see it returns full result when i request range.
This is the worst outcome.
It confuses all clients and overloads server because apparently its being served by php!
All the URL in the format
http://www.springfiles.com/download/3668/5
respond directly with file and ignore range -> they are served by php despite the fact that I was being assured by jj he wont waste server resources on that.
Any single URL with no range compatibility easilly screws torrent download or makes it very slow.
I'm sending the request manually on TCP connection and i see it returns full result when i request range.
This is the worst outcome.
It confuses all clients and overloads server because apparently its being served by php!
All the URL in the format
http://www.springfiles.com/download/3668/5
respond directly with file and ignore range -> they are served by php despite the fact that I was being assured by jj he wont waste server resources on that.
Any single URL with no range compatibility easilly screws torrent download or makes it very slow.