Lobby server mess
Moderator: Moderators
Lobby server mess
Most of you noticed that this weekend the lobby server produced a lot of errors in lobby clients, like:
- users were in multiple battles at the same time
- users which are offline were in some channel or battle
- other inconsitency...
i've tried to clean the code of the lobby server up, but i failed in detecting its complexity:
there is(was) duplicate code which tries to make data consistent but it still failed in some cases (which were mostly not noticed at client side, but showed up as error messages on server side):
https://github.com/spring/uberserver/bl ... nt.py#L397
i've tried to fix this by remove/rewrite that code but introduced some new bugs which caused A LOT of trouble this weekend. i thought i could fix it within time / fixed it but new errors showed up faster than i could fix it.
i've reverted the current running lobby server to a known mostly well working version.
atm i'm not sure how the lobby server can be fixed without fully rewriting it, so it is 100% consistent.
any help is welcome!
i'm very sorry about the trouble it caused
imo the problems are:
- no (good) automatic testing happens as its difficult to emulate ~200 clients
- i'm mostly working alone on this
- the code of uberserver was already a mess when i started trying to improve it
- users were in multiple battles at the same time
- users which are offline were in some channel or battle
- other inconsitency...
i've tried to clean the code of the lobby server up, but i failed in detecting its complexity:
there is(was) duplicate code which tries to make data consistent but it still failed in some cases (which were mostly not noticed at client side, but showed up as error messages on server side):
https://github.com/spring/uberserver/bl ... nt.py#L397
i've tried to fix this by remove/rewrite that code but introduced some new bugs which caused A LOT of trouble this weekend. i thought i could fix it within time / fixed it but new errors showed up faster than i could fix it.
i've reverted the current running lobby server to a known mostly well working version.
atm i'm not sure how the lobby server can be fixed without fully rewriting it, so it is 100% consistent.
any help is welcome!
i'm very sorry about the trouble it caused
imo the problems are:
- no (good) automatic testing happens as its difficult to emulate ~200 clients
- i'm mostly working alone on this
- the code of uberserver was already a mess when i started trying to improve it
- Silentwings
- Posts: 3720
- Joined: 25 Oct 2008, 00:23
Re: Lobby server mess
Thanks for all your efforts! I don't think you should worry about things like this happening sometimes, I guess there is often no way to test big changes to this without going live.
For non-protocol changes, maybe it would work to record the incoming messages stream for a few mins after a restart (during a busy period, so lots happens) and then play them back at the test server with the changes. Obviously would need a bit of extra code to record snd spoof the message passing.
For non-protocol changes, maybe it would work to record the incoming messages stream for a few mins after a restart (during a busy period, so lots happens) and then play them back at the test server with the changes. Obviously would need a bit of extra code to record snd spoof the message passing.
Re: Lobby server mess
I think we should write tests. This will help not only with the regression issues such as these, but also when developing new features.
- FabriceFABS
- Posts: 354
- Joined: 28 Jul 2010, 16:20
Re: Lobby server mess
Don't be sorry... Thank you for all the job you do on UberSERV and others.abma wrote:...
i'm very sorry about the trouble it caused
...
All the best,
FABS
Re: Lobby server mess
I have some simple tests that just connect couple thousand people, create battles and make them join those battles. Dunno if it's useful, I used that to stress test lobby server in semi manual way (add 2k people, join lobby to see if the state is consistent, all people are in battles etc).
Re: Lobby server mess
code? that would be very useful!Licho wrote:I have some simple tests that just connect couple thousand people
Re: Lobby server mess
It's rather trivial https://github.com/ZeroK-RTS/Zero-K-Inf ... mulator.cs
you would just need to run it with pre ZK lobby server version of ZK infrastructure and it should work.
Also uberserver needs to be modified for test (anti registration spam disabled).
you would just need to run it with pre ZK lobby server version of ZK infrastructure and it should work.
Also uberserver needs to be modified for test (anti registration spam disabled).
Re: Lobby server mess
I've seen that. I think it's overly simplistic and shouldn't be used as it adds a C# dependency.
Rewritting that in Python isn't that hard, a few hours of work at the most, but we need to create a lot of tests so a proper testing framework needs to be designed.
Rewritting that in Python isn't that hard, a few hours of work at the most, but we need to create a lot of tests so a proper testing framework needs to be designed.
Re: Lobby server mess
It's not trivial because for proper automated test, you need to verify state as seen both from server and client.
I agree it should go to python.
I agree it should go to python.
Re: Lobby server mess
ok, i've changed code of uberserver so it uses twisted which replaces most of the network code. with this change it seems it is faster, more stable and more reliable. needs a lot more testing, but i hope thats the right direction. also this should make it work on windows again.
also i've added a small stresstest script which does similar stuff like the .net testing program.
also i've added a small stresstest script which does similar stuff like the .net testing program.
Re: Lobby server mess
Such a small notice for such a big step for springkind.
Re: Lobby server mess
Woohoo! Great news, awesome work Abma
Re: Lobby server mess
not sure if this is a big step. hopefully the changes can be soon used as lobby server, still some stuff isn't fully working (i.e. "reload).Such a small notice for such a big step for springkind.