refactoring repos.springrts.com (rapid repositories)

refactoring repos.springrts.com (rapid repositories)

For the discussion of infrastructure improvements and changes.

Moderator: Moderators

abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

refactoring repos.springrts.com (rapid repositories)

Post by abma »

Ive tried to refactor repos.springrts.com:

old:

http://ba.repo.springrts.com/
http://zk.repo.springrts.com/
...

new:

http://repos.springrts.com/ba/
http://repos.springrts.com/zk/
http://repos.springrts.com/tchobby/
...


also packages.springrts.com and repos.springrts.com was split, before it pointed to the same stuff.


atm only tchobby inside http://repos.springrts.com/repos.gz points to the new url schema to test if clients can handle it.

this change doesn't hopefully break anything: if so, please report here!

thanks!


you may ask why:

each subdomain requires a new dns request + http connection to be made. this can be painfully slow. when all repos are at the same subdomain, only one request has to be made and this should be much faster + less error-prone. also this allows to add / remove repos without reconfiguring apache and easily allows to use https.

also WTF: 42 git repos are hosted on springrts.com!
User avatar
PicassoCT
Journeywar Developer & Mapper
Posts: 10450
Joined: 24 Jan 2006, 21:12

Re: Experimenting with refactoring repos.springrts.com

Post by PicassoCT »

But what was the question again - you know, the one question, the question - to the meaning of life, the answer to everything- you know the question
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Experimenting with refactoring repos.springrts.com

Post by abma »

meh, found a bug: streamer users DOCUMENT_ROOT: https://github.com/spring/pr-downloader ... er.cpp#L98

but it should use DOCUMENT_ROOT + SCRIPT_NAME i guess...
User avatar
Jools
XTA Developer
Posts: 2816
Joined: 23 Feb 2009, 16:29

Re: Experimenting with refactoring repos.springrts.com

Post by Jools »

Why are we using apache? Why not nginx?
User avatar
Jools
XTA Developer
Posts: 2816
Joined: 23 Feb 2009, 16:29

Re: Experimenting with refactoring repos.springrts.com

Post by Jools »

abma wrote: 06 Mar 2019, 20:13
new:

http://repos.springrts.com/ba/
http://repos.springrts.com/zk/
http://repos.springrts.com/tchobby/
...
also WTF: 42 git repos are hosted on springrts.com!
Why can't we use url:s like https://repos.springrts.com/zk/ instead?
User avatar
Forboding Angel
Evolution RTS Developer
Posts: 14673
Joined: 17 Nov 2005, 02:43

Re: Experimenting with refactoring repos.springrts.com

Post by Forboding Angel »

Yeah, personally I don't see any benefit to using subdomains for this. Subfolders make a lot more sense.
User avatar
Silentwings
Posts: 3720
Joined: 25 Oct 2008, 00:23

Re: Experimenting with refactoring repos.springrts.com

Post by Silentwings »

Why can't we use url:s like https://repos.springrts.com/zk/ instead?
I think you will be able to use https, just not yet.
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Experimenting with refactoring repos.springrts.com

Post by abma »

Silentwings wrote: 07 Mar 2019, 16:45 I think you will be able to use https, just not yet.
yes, thats the plan! i don't want to break all at once, this is why https isn't enabled yet.
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Experimenting with refactoring repos.springrts.com

Post by abma »

ok, fixed a blocking bug. https://github.com/spring/pr-downloader/issues/113

downloading should now work via http://repos.springrts.com/tchobby at least. can you please test it if you can download "tchobby:test", too?

when i hear no problem/bug reports, i'll change all repos in repos.gz to the new urls.
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Experimenting with refactoring repos.springrts.com

Post by abma »

abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Experimenting with refactoring repos.springrts.com

Post by abma »

User avatar
ThinkSome
Posts: 387
Joined: 14 Jun 2015, 13:36

Re: Experimenting with refactoring repos.springrts.com

Post by ThinkSome »

Does rapid work for anyone?

Code: Select all

/opt/springrts.com/spring/104.0/bin/pr-downloader  --filesystem-writepath /data/users/s44/spads/game_data --rapid-download s44:test 
pr-downloader tarball                                                                                                                                        
[Info] Using filesystem-writepath: /data/users/s44/spads/game_data                                                                                   
[Info] Free disk space: 10902 MB                                                                                                                                                  
[Info] Found 42 repos in /data/users/s44/spads/game_data/rapid/repos.springrts.com/repos.gz                                                                                     
                                                                                                                                                      
[Download] Spring: 1944 test-5448-ae935de                                                                                                                                         
[Info] Using rapid                                                                                                                               
[Info] http://repos.springrts.com/s44/streamer.cgi?debae699faac80db2422860cefacee09                                                                        
[Progress]   2% [                              ] 14271/925208 ^M[Progress] 100% [==============================] 925208/925208 ^M[Info] Missing file: /data/users/s44/spads/game_data/pool/9a/0b8dce4cf805ddcaedb0a6a9341736.gz                                                                          
[Info] compare failed(): 96c51cd2ee6c2a7dc2c0717afc336974 569168b394351c7eda9442de43f7af4f                                                                                        
[Info] Removing invalid file: /data/users/s44/spads/game_data/pool/56/9168b394351c7eda9442de43f7af4f.gz                                                                           
[Info] Missing file: /data/users/s44/spads/game_data/pool/38/2f9e03e4de37425f4ab5a4a18b8a2e.gz            
[Info] Missing file: /data/users/s44/spads/game_data/pool/38/2f9e03e4de37425f4ab5a4a18b8a2e.gz



wget -O /tmp/test  'http://repos.springrts.com/s44/streamer.cgi?debae699faac80db2422860cefacee09'
file /tmp/test

/tmp/test: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=568ee4e3ef19f67c55f9d32b3e7f8165bf4bbed2, with debug_info, not stripped

SHA1 b147f1862236bf696a1eda0bcf4fdb0cec496e2c  /tmp/test

Is the streamer streaming itself?
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Experimenting with refactoring repos.springrts.com

Post by abma »

ugh, weird, when testing it worked for me :?
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Experimenting with refactoring repos.springrts.com

Post by abma »

ThinkSome wrote: 27 Mar 2019, 00:07 Does rapid work for anyone?

Is the streamer streaming itself?

should be fixed now! apache config was invalid, sorry :-|

the regex was wrong: https://github.com/spring/pr-downloader ... e.conf#L18
DeinFreund
Posts: 12
Joined: 14 Aug 2014, 00:12

Re: Experimenting with refactoring repos.springrts.com

Post by DeinFreund »

Edit: ZK has its own downloader that failed due to this update.
Google_Frog
Moderator
Posts: 2464
Joined: 12 Oct 2007, 09:24

Re: Experimenting with refactoring repos.springrts.com

Post by Google_Frog »

It looks like singleplayer was broken for 6 hours, during which we received about 60 crash reports. It would be nice if repository names didn't change around.
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Experimenting with refactoring repos.springrts.com

Post by abma »

Google_Frog wrote: 28 Mar 2019, 13:06 It looks like singleplayer was broken for 6 hours, during which we received about 60 crash reports. It would be nice if repository names didn't change around.
could you provide more details please? the names did not change, the url of the individual repos did change. When the rapid client is implemented probably, it should have worked without any changes. It was announced and a test repo was added also the old urls still work, too.

the entry url did not change: http://repos.springrts.com/repos.gz which contains the (new) prefixes to the individual repos.
Google_Frog
Moderator
Posts: 2464
Joined: 12 Oct 2007, 09:24

Re: Experimenting with refactoring repos.springrts.com

Post by Google_Frog »

I don't know details. Apparently this change fixes it: https://github.com/ZeroK-RTS/Zero-K-Inf ... d5a512be17
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Experimenting with refactoring repos.springrts.com

Post by abma »

Google_Frog wrote: 28 Mar 2019, 13:06 It would be nice if repository names didn't change around.

improving / fixing these:
abma wrote: 06 Mar 2019, 20:13 each subdomain requires a new dns request + http connection to be made. this can be painfully slow. when all repos are at the same subdomain, only one request has to be made and this should be much faster + less error-prone. also this allows to add / remove repos without reconfiguring apache and easily allows to use https.

was not possible without changing the repos urls.
User avatar
Silentwings
Posts: 3720
Joined: 25 Oct 2008, 00:23

Re: refactoring repos.springrts.com (rapid repositories)

Post by Silentwings »

Which urls should we now use to get hold of the sdz built by rapid for major versions?

I don't see any (or even a blank list) on http://repos.springrts.com/ba/, all I get there is a link to the rapid wiki page
Post Reply

Return to “Infrastructure Development”