Lobby server mess

[abma]

Most of you noticed that this weekend the lobby server produced a lot of errors in lobby clients, like:

- users were in multiple battles at the same time
- users which are offline were in some channel or battle
- other inconsitency...

i've tried to clean the code of the lobby server up, but i failed in detecting its complexity:

there is(was) duplicate code which tries to make data consistent but it still failed in some cases (which were mostly not noticed at client side, but showed up as error messages on server side):
https://github.com/spring/uberserver/bl ... nt.py#L397

i've tried to fix this by remove/rewrite that code but introduced some new bugs which caused A LOT of trouble this weekend. i thought i could fix it within time / fixed it but new errors showed up faster than i could fix it.

i've reverted the current running lobby server to a known mostly well working version.

atm i'm not sure how the lobby server can be fixed without fully rewriting it, so it is 100% consistent.

any help is welcome!

i'm very sorry about the trouble it caused

imo the problems are:
- no (good) automatic testing happens as its difficult to emulate ~200 clients
- i'm mostly working alone on this
- the code of uberserver was already a mess when i started trying to improve it

[Silentwings]

Thanks for all your efforts! I don't think you should worry about things like this happening sometimes, I guess there is often no way to test big changes to this without going live.

For non-protocol changes, maybe it would work to record the incoming messages stream for a few mins after a restart (during a busy period, so lots happens) and then play them back at the test server with the changes. Obviously would need a bit of extra code to record snd spoof the message passing.

[gajop]

I think we should write tests. This will help not only with the regression issues such as these, but also when developing new features.

[FabriceFABS]

...
i'm very sorry about the trouble it caused
...

Don't be sorry... Thank you for all the job you do on UberSERV and others.
All the best,
FABS

[Licho]

I have some simple tests that just connect couple thousand people, create battles and make them join those battles. Dunno if it's useful, I used that to stress test lobby server in semi manual way (add 2k people, join lobby to see if the state is consistent, all people are in battles etc).

[abma]

I have some simple tests that just connect couple thousand people

code? that would be very useful!

[Licho]

It's rather trivial https://github.com/ZeroK-RTS/Zero-K-Inf ... mulator.cs
you would just need to run it with pre ZK lobby server version of ZK infrastructure and it should work.
Also uberserver needs to be modified for test (anti registration spam disabled).

[gajop]

I've seen that. I think it's overly simplistic and shouldn't be used as it adds a C# dependency.
Rewritting that in Python isn't that hard, a few hours of work at the most, but we need to create a lot of tests so a proper testing framework needs to be designed.

[Licho]

It's not trivial because for proper automated test, you need to verify state as seen both from server and client.
I agree it should go to python.

[abma]

ok, i've changed code of uberserver so it uses twisted which replaces most of the network code. with this change it seems it is faster, more stable and more reliable. needs a lot more testing, but i hope thats the right direction. also this should make it work on windows again.

also i've added a small stresstest script which does similar stuff like the .net testing program.

[dansan]

Such a small notice for such a big step for springkind.

[Nemo]

Woohoo! Great news, awesome work Abma

[abma]

Such a small notice for such a big step for springkind.

not sure if this is a big step. hopefully the changes can be soon used as lobby server, still some stuff isn't fully working (i.e. "reload).