Lobby server mess

Lobby server mess

For the discussion of infrastructure improvements and changes.

Moderator: Moderators

Post Reply
abma
Spring Developer
Posts: 3544
Joined: 01 Jun 2009, 00:08

Lobby server mess

Post by abma » 17 May 2015, 05:38

Most of you noticed that this weekend the lobby server produced a lot of errors in lobby clients, like:

- users were in multiple battles at the same time
- users which are offline were in some channel or battle
- other inconsitency...

i've tried to clean the code of the lobby server up, but i failed in detecting its complexity:

there is(was) duplicate code which tries to make data consistent but it still failed in some cases (which were mostly not noticed at client side, but showed up as error messages on server side):
https://github.com/spring/uberserver/bl ... nt.py#L397

i've tried to fix this by remove/rewrite that code but introduced some new bugs which caused A LOT of trouble this weekend. i thought i could fix it within time / fixed it but new errors showed up faster than i could fix it.

i've reverted the current running lobby server to a known mostly well working version.

atm i'm not sure how the lobby server can be fixed without fully rewriting it, so it is 100% consistent.

any help is welcome!

i'm very sorry about the trouble it caused :-|

imo the problems are:
- no (good) automatic testing happens as its difficult to emulate ~200 clients
- i'm mostly working alone on this
- the code of uberserver was already a mess when i started trying to improve it
0 x

User avatar
Silentwings
Moderator
Posts: 3582
Joined: 25 Oct 2008, 00:23

Re: Lobby server mess

Post by Silentwings » 17 May 2015, 08:54

Thanks for all your efforts! I don't think you should worry about things like this happening sometimes, I guess there is often no way to test big changes to this without going live.

For non-protocol changes, maybe it would work to record the incoming messages stream for a few mins after a restart (during a busy period, so lots happens) and then play them back at the test server with the changes. Obviously would need a bit of extra code to record snd spoof the message passing.
0 x

gajop
Moderator
Posts: 3022
Joined: 05 Aug 2009, 20:42

Re: Lobby server mess

Post by gajop » 17 May 2015, 09:07

I think we should write tests. This will help not only with the regression issues such as these, but also when developing new features.
0 x

User avatar
FabriceFABS
Posts: 345
Joined: 28 Jul 2010, 16:20

Re: Lobby server mess

Post by FabriceFABS » 17 May 2015, 23:23

abma wrote:...
i'm very sorry about the trouble it caused :-|
...
Don't be sorry... Thank you for all the job you do on UberSERV and others.
All the best,
FABS
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Lobby server mess

Post by Licho » 18 May 2015, 10:59

I have some simple tests that just connect couple thousand people, create battles and make them join those battles. Dunno if it's useful, I used that to stress test lobby server in semi manual way (add 2k people, join lobby to see if the state is consistent, all people are in battles etc).
0 x

abma
Spring Developer
Posts: 3544
Joined: 01 Jun 2009, 00:08

Re: Lobby server mess

Post by abma » 18 May 2015, 11:22

Licho wrote:I have some simple tests that just connect couple thousand people
code? that would be very useful!
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Lobby server mess

Post by Licho » 18 May 2015, 13:15

It's rather trivial https://github.com/ZeroK-RTS/Zero-K-Inf ... mulator.cs
you would just need to run it with pre ZK lobby server version of ZK infrastructure and it should work.
Also uberserver needs to be modified for test (anti registration spam disabled).
0 x

gajop
Moderator
Posts: 3022
Joined: 05 Aug 2009, 20:42

Re: Lobby server mess

Post by gajop » 18 May 2015, 16:12

I've seen that. I think it's overly simplistic and shouldn't be used as it adds a C# dependency.
Rewritting that in Python isn't that hard, a few hours of work at the most, but we need to create a lot of tests so a proper testing framework needs to be designed.
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Lobby server mess

Post by Licho » 18 May 2015, 16:51

It's not trivial because for proper automated test, you need to verify state as seen both from server and client.
I agree it should go to python.
0 x

abma
Spring Developer
Posts: 3544
Joined: 01 Jun 2009, 00:08

Re: Lobby server mess

Post by abma » 23 May 2015, 03:05

ok, i've changed code of uberserver so it uses twisted which replaces most of the network code. with this change it seems it is faster, more stable and more reliable. needs a lot more testing, but i hope thats the right direction. also this should make it work on windows again.

also i've added a small stresstest script which does similar stuff like the .net testing program.
0 x

dansan
Server Owner & Developer
Posts: 1190
Joined: 29 May 2010, 23:40

Re: Lobby server mess

Post by dansan » 24 May 2015, 12:01

Such a small notice for such a big step for springkind.
0 x

User avatar
Nemo
Spring 1944 Developer
Posts: 1376
Joined: 30 Jan 2005, 19:44

Re: Lobby server mess

Post by Nemo » 26 May 2015, 11:03

Woohoo! Great news, awesome work Abma :-)
0 x

abma
Spring Developer
Posts: 3544
Joined: 01 Jun 2009, 00:08

Re: Lobby server mess

Post by abma » 26 May 2015, 16:02

Such a small notice for such a big step for springkind.
not sure if this is a big step. hopefully the changes can be soon used as lobby server, still some stuff isn't fully working (i.e. "reload).
0 x

Post Reply

Return to “Infrastructure Development”