refactoring repos.springrts.com (rapid repositories)

refactoring repos.springrts.com (rapid repositories)

For the discussion of infrastructure improvements and changes.

Moderator: Moderators

abma
Spring Developer
Posts: 3544
Joined: 01 Jun 2009, 00:08

refactoring repos.springrts.com (rapid repositories)

Post by abma » 06 Mar 2019, 20:13

Ive tried to refactor repos.springrts.com:

old:

http://ba.repo.springrts.com/
http://zk.repo.springrts.com/
...

new:

http://repos.springrts.com/ba/
http://repos.springrts.com/zk/
http://repos.springrts.com/tchobby/
...


also packages.springrts.com and repos.springrts.com was split, before it pointed to the same stuff.


atm only tchobby inside http://repos.springrts.com/repos.gz points to the new url schema to test if clients can handle it.

this change doesn't hopefully break anything: if so, please report here!

thanks!


you may ask why:

each subdomain requires a new dns request + http connection to be made. this can be painfully slow. when all repos are at the same subdomain, only one request has to be made and this should be much faster + less error-prone. also this allows to add / remove repos without reconfiguring apache and easily allows to use https.

also WTF: 42 git repos are hosted on springrts.com!
4 x

User avatar
PicassoCT
Journeywar Developer & Mapper
Posts: 10213
Joined: 24 Jan 2006, 21:12

Re: Experimenting with refactoring repos.springrts.com

Post by PicassoCT » 06 Mar 2019, 21:10

But what was the question again - you know, the one question, the question - to the meaning of life, the answer to everything- you know the question
1 x

abma
Spring Developer
Posts: 3544
Joined: 01 Jun 2009, 00:08

Re: Experimenting with refactoring repos.springrts.com

Post by abma » 07 Mar 2019, 10:22

meh, found a bug: streamer users DOCUMENT_ROOT: https://github.com/spring/pr-downloader ... er.cpp#L98

but it should use DOCUMENT_ROOT + SCRIPT_NAME i guess...
1 x

User avatar
Jools
XTA Developer
Posts: 2804
Joined: 23 Feb 2009, 16:29

Re: Experimenting with refactoring repos.springrts.com

Post by Jools » 07 Mar 2019, 14:42

Why are we using apache? Why not nginx?
1 x

User avatar
Jools
XTA Developer
Posts: 2804
Joined: 23 Feb 2009, 16:29

Re: Experimenting with refactoring repos.springrts.com

Post by Jools » 07 Mar 2019, 14:44

abma wrote:
06 Mar 2019, 20:13

new:

http://repos.springrts.com/ba/
http://repos.springrts.com/zk/
http://repos.springrts.com/tchobby/
...
also WTF: 42 git repos are hosted on springrts.com!
Why can't we use url:s like https://repos.springrts.com/zk/ instead?
1 x

User avatar
Forboding Angel
Evolution RTS Developer
Posts: 14575
Joined: 17 Nov 2005, 02:43

Re: Experimenting with refactoring repos.springrts.com

Post by Forboding Angel » 07 Mar 2019, 15:39

Yeah, personally I don't see any benefit to using subdomains for this. Subfolders make a lot more sense.
1 x

User avatar
Silentwings
Moderator
Posts: 3582
Joined: 25 Oct 2008, 00:23

Re: Experimenting with refactoring repos.springrts.com

Post by Silentwings » 07 Mar 2019, 16:45

Why can't we use url:s like https://repos.springrts.com/zk/ instead?
I think you will be able to use https, just not yet.
1 x

abma
Spring Developer
Posts: 3544
Joined: 01 Jun 2009, 00:08

Re: Experimenting with refactoring repos.springrts.com

Post by abma » 07 Mar 2019, 17:18

Silentwings wrote:
07 Mar 2019, 16:45
I think you will be able to use https, just not yet.
yes, thats the plan! i don't want to break all at once, this is why https isn't enabled yet.
1 x

abma
Spring Developer
Posts: 3544
Joined: 01 Jun 2009, 00:08

Re: Experimenting with refactoring repos.springrts.com

Post by abma » 07 Mar 2019, 23:26

ok, fixed a blocking bug. https://github.com/spring/pr-downloader/issues/113

downloading should now work via http://repos.springrts.com/tchobby at least. can you please test it if you can download "tchobby:test", too?

when i hear no problem/bug reports, i'll change all repos in repos.gz to the new urls.
2 x

abma
Spring Developer
Posts: 3544
Joined: 01 Jun 2009, 00:08

Re: Experimenting with refactoring repos.springrts.com

Post by abma » 25 Mar 2019, 22:03

1 x

abma
Spring Developer
Posts: 3544
Joined: 01 Jun 2009, 00:08

Re: Experimenting with refactoring repos.springrts.com

Post by abma » 25 Mar 2019, 22:09

2 x

User avatar
ThinkSome
Posts: 287
Joined: 14 Jun 2015, 13:36

Re: Experimenting with refactoring repos.springrts.com

Post by ThinkSome » 27 Mar 2019, 00:07

Does rapid work for anyone?

Code: Select all

/opt/springrts.com/spring/104.0/bin/pr-downloader  --filesystem-writepath /data/users/s44/spads/game_data --rapid-download s44:test 
pr-downloader tarball                                                                                                                                        
[Info] Using filesystem-writepath: /data/users/s44/spads/game_data                                                                                   
[Info] Free disk space: 10902 MB                                                                                                                                                  
[Info] Found 42 repos in /data/users/s44/spads/game_data/rapid/repos.springrts.com/repos.gz                                                                                     
                                                                                                                                                      
[Download] Spring: 1944 test-5448-ae935de                                                                                                                                         
[Info] Using rapid                                                                                                                               
[Info] http://repos.springrts.com/s44/streamer.cgi?debae699faac80db2422860cefacee09                                                                        
[Progress]   2% [                              ] 14271/925208 ^M[Progress] 100% [==============================] 925208/925208 ^M[Info] Missing file: /data/users/s44/spads/game_data/pool/9a/0b8dce4cf805ddcaedb0a6a9341736.gz                                                                          
[Info] compare failed(): 96c51cd2ee6c2a7dc2c0717afc336974 569168b394351c7eda9442de43f7af4f                                                                                        
[Info] Removing invalid file: /data/users/s44/spads/game_data/pool/56/9168b394351c7eda9442de43f7af4f.gz                                                                           
[Info] Missing file: /data/users/s44/spads/game_data/pool/38/2f9e03e4de37425f4ab5a4a18b8a2e.gz            
[Info] Missing file: /data/users/s44/spads/game_data/pool/38/2f9e03e4de37425f4ab5a4a18b8a2e.gz



wget -O /tmp/test  'http://repos.springrts.com/s44/streamer.cgi?debae699faac80db2422860cefacee09'
file /tmp/test

/tmp/test: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=568ee4e3ef19f67c55f9d32b3e7f8165bf4bbed2, with debug_info, not stripped

SHA1 b147f1862236bf696a1eda0bcf4fdb0cec496e2c  /tmp/test

Is the streamer streaming itself?
1 x

abma
Spring Developer
Posts: 3544
Joined: 01 Jun 2009, 00:08

Re: Experimenting with refactoring repos.springrts.com

Post by abma » 27 Mar 2019, 11:51

ugh, weird, when testing it worked for me :?
0 x

abma
Spring Developer
Posts: 3544
Joined: 01 Jun 2009, 00:08

Re: Experimenting with refactoring repos.springrts.com

Post by abma » 27 Mar 2019, 21:36

ThinkSome wrote:
27 Mar 2019, 00:07
Does rapid work for anyone?

Is the streamer streaming itself?

should be fixed now! apache config was invalid, sorry :-|

the regex was wrong: https://github.com/spring/pr-downloader ... e.conf#L18
1 x

DeinFreund
Posts: 12
Joined: 14 Aug 2014, 00:12

Re: Experimenting with refactoring repos.springrts.com

Post by DeinFreund » 28 Mar 2019, 11:24

Edit: ZK has its own downloader that failed due to this update.
0 x

Google_Frog
Moderator
Posts: 2434
Joined: 12 Oct 2007, 09:24

Re: Experimenting with refactoring repos.springrts.com

Post by Google_Frog » 28 Mar 2019, 13:06

It looks like singleplayer was broken for 6 hours, during which we received about 60 crash reports. It would be nice if repository names didn't change around.
0 x

abma
Spring Developer
Posts: 3544
Joined: 01 Jun 2009, 00:08

Re: Experimenting with refactoring repos.springrts.com

Post by abma » 28 Mar 2019, 13:38

Google_Frog wrote:
28 Mar 2019, 13:06
It looks like singleplayer was broken for 6 hours, during which we received about 60 crash reports. It would be nice if repository names didn't change around.
could you provide more details please? the names did not change, the url of the individual repos did change. When the rapid client is implemented probably, it should have worked without any changes. It was announced and a test repo was added also the old urls still work, too.

the entry url did not change: http://repos.springrts.com/repos.gz which contains the (new) prefixes to the individual repos.
0 x

Google_Frog
Moderator
Posts: 2434
Joined: 12 Oct 2007, 09:24

Re: Experimenting with refactoring repos.springrts.com

Post by Google_Frog » 28 Mar 2019, 15:55

I don't know details. Apparently this change fixes it: https://github.com/ZeroK-RTS/Zero-K-Inf ... d5a512be17
0 x

abma
Spring Developer
Posts: 3544
Joined: 01 Jun 2009, 00:08

Re: Experimenting with refactoring repos.springrts.com

Post by abma » 28 Mar 2019, 19:21

Google_Frog wrote:
28 Mar 2019, 13:06
It would be nice if repository names didn't change around.

improving / fixing these:
abma wrote:
06 Mar 2019, 20:13
each subdomain requires a new dns request + http connection to be made. this can be painfully slow. when all repos are at the same subdomain, only one request has to be made and this should be much faster + less error-prone. also this allows to add / remove repos without reconfiguring apache and easily allows to use https.

was not possible without changing the repos urls.
0 x

User avatar
Silentwings
Moderator
Posts: 3582
Joined: 25 Oct 2008, 00:23

Re: refactoring repos.springrts.com (rapid repositories)

Post by Silentwings » 29 Mar 2019, 23:27

Which urls should we now use to get hold of the sdz built by rapid for major versions?

I don't see any (or even a blank list) on http://repos.springrts.com/ba/, all I get there is a link to the rapid wiki page
0 x

Post Reply

Return to “Infrastructure Development”

cron