XML-RPC searchapi on springfiles.com - Page 2

XML-RPC searchapi on springfiles.com

For the discussion of infrastructure improvements and changes.

Moderator: Moderators

abma
Spring Developer
Posts: 3556
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma » 28 Jan 2011, 15:16

you didn't understand: http://www.springfiles.com/download/3668/5 does send a HTTP 301 as response, if you want to get that file, you've to follow to that new location and request the file there with a range-request. So resources aren't wasted...

currently client + server implementation don't understand each other... one side has to be fixed, imo the server side. but i can't do anything, because i don't have access to springfiles, only to a dev-server.

jj wants accurate stats, but currently this is a bit broken because of this problem... let's wait what he says about this.

edit: i've fixed it on the dev-server it returns now the real url. jj please upload it to springfiles.com (https://github.com/springfiles/searchap ... api.module)
0 x

User avatar
jj
Posts: 273
Joined: 01 Apr 2006, 19:03

Re: XML-RPC searchapi on springfiles.com

Post by jj » 28 Jan 2011, 16:04

Licho, the files are not served by php, it's just a redirect, not a php- download script.

I dont like the approach to serve the direct url.
How can we check the used bandwidth for mirror owners?

But i will put abma's new code online, as soon as i am home (tonight)
That will solve all the problems.

ps. nice work abma!
0 x

abma
Spring Developer
Posts: 3556
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma » 28 Jan 2011, 18:48

uh... forgot to say, size and timestamp of the file is now returned, too.
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles.com

Post by Licho » 29 Jan 2011, 00:59

Why the hell you care about used bandwith, i pay for the server and it has unlimited one. You cannot measure it this way anyway unless you process range headers correctly or measure it lower level.

I care about stable and reliable downloads for users..

And no Abma, you are mistaken, it does not return 302 for all URLs. Example:

Request

Code: Select all

GET /system/files/downloads/spring/spring-maps/DeltaSiegeDry.sd7 HTTP/1.1
Host: www.springfiles.com
Range: bytes=262144-524288

Response

Code: Select all

HTTP/1.1 200 OK
Date: Fri, 28 Jan 2011 23:54:43 GMT
Server: Apache/2.2.12 (Ubuntu)
X-Powered-By: PHP/5.2.10-2ubuntu6.5
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Cache-Control: post-check=0, pre-check=0, private
Content-Disposition: attachment; filename="DeltaSiegeDry.sd7"
Set-Cookie: SESS91c96e8a48be43d5eb7b4b5d71a52400=cff809990db8c91808f0ad7af7f11b14; expires=Mon, 21-Feb-2011 03:28:03 GMT; path=/; domain=.springfiles.com
Last-Modified: Fri, 28 Jan 2011 23:54:43 GMT
Content-Length: 17640007
Content-Type: application/octet-stream; name="DeltaSiegeDry.sd7"
If this isnt php served file then what is it jj and why it ignores range?
0 x

User avatar
AF
AI Developer
Posts: 20669
Joined: 14 Sep 2004, 11:32

Re: XML-RPC searchapi on springfiles.com

Post by AF » 29 Jan 2011, 02:36

If we want bandwidth measured isn't that something that can be done via apache or some other method rather than php?
0 x

User avatar
jj
Posts: 273
Joined: 01 Apr 2006, 19:03

Re: XML-RPC searchapi on springfiles.com

Post by jj » 29 Jan 2011, 20:28

Abma has updated the code, so the problem with range headers will be gone.

Licho, i wanted to measure the dl stats because some mirrors have bandwidth limitations. Now lots of the traffic will be unmanaged which can be bad for particular mirrors. It has nothing to do with your server.
0 x

abma
Spring Developer
Posts: 3556
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma » 30 Jan 2011, 12:34

@licho: fixed, now only plain urls are returned
0 x

abma
Spring Developer
Posts: 3556
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma » 30 Jan 2011, 13:28

hm.. one issue i found is:

when watching the logs of springfiles, i saw some accesses like this:

GET /system/files/downloads/spring/games/Tech Annihilation.sd7 HTTP/1.1" 404 10643

this looks like some client's don't url-encode the request. maybe i should do this server-side in the xml-rpc result? (for example replace space with %20)
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles.com

Post by Licho » 30 Jan 2011, 16:33

You could, but server has even more mirror sources so i should encode it there too.

And ideally clients should do that themselves but fastest fix is for me to url encode mirrors provided to lobbies.
0 x

User avatar
Agon
Posts: 527
Joined: 16 May 2007, 18:33

Re: XML-RPC searchapi on springfiles.com

Post by Agon » 30 Jan 2011, 19:58

Hey, nice work.

I don't like xmlrpc, but it is fine.
I implemented it in my Go package: https://github.com/Agon/go-slc/blob/mas ... archapi.go
0 x

User avatar
Agon
Posts: 527
Joined: 16 May 2007, 18:33

Re: XML-RPC searchapi on springfiles.com

Post by Agon » 30 Jan 2011, 20:16

Got a bot running which listens to following command:
"map mapname"
Prints out all mirrors which could be found.
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles.com

Post by Licho » 30 Jan 2011, 21:29

Every springie autohost already does that but lists more than just those mirrors, because it has its own list as well.
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles.com

Post by Licho » 30 Jan 2011, 22:12

jj, are you 100% sure nor

http://springfiles.com/system/files/dow ... egeDry.sd7

nor

http://springfiles.com/sites/default/fi ... egeDry.sd7

run using php?

I dont want ANY big downloads running through web technology. Make sure there are none please.

Yesterday your cron script killed server again and I don't want to waste extra resources :(
0 x

abma
Spring Developer
Posts: 3556
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma » 30 Jan 2011, 22:34

system/files/... is using php
sites/default/files ... isn't!
0 x

abma
Spring Developer
Posts: 3556
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma » 30 Jan 2011, 23:51

next addition: "description" field added,
this is a url to the download page + descriptions of a map/game, please use it!

the direct link is for autodownloaders, end-users should use the webpage.
0 x

abma
Spring Developer
Posts: 3556
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma » 31 Jan 2011, 13:28

@licho:
can you delete all urls from plasma starting with

http://springfiles.com/system
http://springfiles.com/download
http://www.springfiles.com/

i hope these are all invalid/duplicate urls...
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles.com

Post by Licho » 31 Jan 2011, 13:41

They will be auto deleted when those becoe inalid
0 x

abma
Spring Developer
Posts: 3556
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma » 31 Jan 2011, 13:59

http://springfiles.com/system urls aren't invalid, they don't support the-range stuff.

the others are duplicates and should all return a 301 to http://springfiles.com but are valid, too.
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: XML-RPC searchapi on springfiles.com

Post by Licho » 31 Jan 2011, 14:00

Hmm then http://springfiles.com/system
should be removed from springfiles (if they are linked from somewhere).
I still see them used by the site itself.
0 x

abma
Spring Developer
Posts: 3556
Joined: 01 Jun 2009, 00:08

Re: XML-RPC searchapi on springfiles.com

Post by abma » 31 Jan 2011, 14:15

drupal is using it, that can't be removed.
0 x

Post Reply

Return to “Infrastructure Development”

cron