Server lightweight mode

Server lightweight mode

For the discussion of infrastructure improvements and changes.

Moderator: Moderators

Post Reply
gajop
Moderator
Posts: 3051
Joined: 05 Aug 2009, 20:42

Server lightweight mode

Post by gajop »

Lobby server currently scales badly. A lot of commands scale as O(n^2), where n is the amount of players/users, which is not going to work if we ever get a large enough amount of users. Most of the commands just don't need to be send.

I propose to limit when some user-related commands are sent (mostly by limiting it them to friends only).

Related uberserver tickets:
https://github.com/spring/uberserver/issues/111
https://github.com/spring/uberserver/issues/122

Commands to change:
CLIENTSTATUS (should only be sent to friends)
JOINBATTLE, LEFTBATTLE (should only be sent to friends or for other people in the same battleroom)
ADDUSER, REMOVEUSER (should only be sent to friends)

CLIENTS (sent in channels that are not in lightweight mode)
JOINED, LEFT (sent in channels that are not in lightweight mode)

Channels could enter a lightweight mode, which needs one command:
CHANNELSTATUS channelName lightweightMode (parametric command that could be extended later on)
Ideally the lobby server would put channels with more than an X amount of people in the lightweight mode. X would be an arguably large number, e.g. 100 or 1000.

PS: Not sure what to do with BATTLE* commands, they're also kinda spammy and not of interest most of the time. Maybe polling is better in this case..? Or a way to turn on/off whether such information is received.
PSS: I really don't want to allow some sort of "object listeners". It shouldn't be the clients that decide what kind of information they receive. Too many options in this regard makes the protocol complicated and not clear how it would scale.
User avatar
aegis
Posts: 2456
Joined: 11 Jul 2007, 17:47

Re: Server lightweight mode

Post by aegis »

Maybe it's worth connecting to the server for a day and logging the command breakdown (e.g. 25% CLIENTSTATUS, 10% BATTLEOPENED, etc) to see what's worth optimizing for long-running connections.

I'd recommend optimizing the new user connect, as that's probably the most bandwidth-expensive thing. When a new user joins, several commands (ADDUSER, CLIENTSTATUS, JOINED #main, JOINED #help) are sent to pretty much all connected users, *and* a full list of ADDUSER, CLIENTSTATUS, CLIENTS, BATTLEOPENED, UPDATEBATTLEINFO, JOINEDBATTLE are sent.

From the days of TASClient, I remember the protocol spam on connect to be pretty huge even with the low user counts then.

It might help to make common channels blind (you can't see the channel userlist, but you can still talk to them). A cheap "friends list" is to just join a channel with a few other people. You can see when they're online because they're in the channel.

Matchmaking can also help with this, because you don't need to see anyone or any battles until the game is ready.

If CPU load is becoming an issue, it might be worth trying uberserver under pypy.
MetalSucker
Posts: 98
Joined: 22 Sep 2014, 20:29

Re: Server lightweight mode

Post by MetalSucker »

What backend should this have? SQL or some internal persistent tables/arrays native to python? I also know there is an opinion that the right way to do it is thin lobbies (and I get why) but how thick should the server be in the best case scenario?
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Server lightweight mode

Post by Licho »

ZK changed the protocol so that the server keeps track of who know what and sends only data you need. Information content is unchanged but you only get "user info" if you for example joined the same channel and can actually see him in GUI.

Server then knows that client X knows client Y and in what state, if state changes it can only send change itself and also sends left information as needed.

Benefit of this is that client didn't need big changes and can still for example show rich tooltips with all the info for all people available in GUI without roundtrips.
gajop
Moderator
Posts: 3051
Joined: 05 Aug 2009, 20:42

Re: Server lightweight mode

Post by gajop »

aegis wrote:Maybe it's worth connecting to the server for a day and logging the command breakdown (e.g. 25% CLIENTSTATUS, 10% BATTLEOPENED, etc) to see what's worth optimizing for long-running connections.
I agree, but only after matchmaking is implemented and becomes widely used.
We should compare server performance under two conditions: massive login and constant, high workload.
aegis wrote:I'd recommend optimizing the new user connect, as that's probably the most bandwidth-expensive thing. When a new user joins, several commands (ADDUSER, CLIENTSTATUS, JOINED #main, JOINED #help) are sent to pretty much all connected users, *and* a full list of ADDUSER, CLIENTSTATUS, CLIENTS, BATTLEOPENED, UPDATEBATTLEINFO, JOINEDBATTLE are sent.
Actually, most of these are to be removed or to be received by a select few users.
aegis wrote:It might help to make common channels blind (you can't see the channel userlist, but you can still talk to them).
This is what the lightweight channel mode will do essentialy, but it should be done dynamically instead of just using a predefined list of channels.
aegis wrote:A cheap "friends list" is to just join a channel with a few other people. You can see when they're online because they're in the channel.
Friendship is not a transitive relation and should be voluntary. We already have a friend list implemented so I don't see a need for this.
aegis wrote:Matchmaking can also help with this, because you don't need to see anyone or any battles until the game is ready.
Indeed, although if we want browsable spectating, we need to allow battle lists (which don't scale well imo). But yeah, MM should scale pretty well.
aegis wrote:If CPU load is becoming an issue, it might be worth trying uberserver under pypy.
That's the last thing I want to consider now. There were also some suggestions about using tornado instead. It's good to know these options are open, but a protocol change is needed first.
gajop
Moderator
Posts: 3051
Joined: 05 Aug 2009, 20:42

Re: Server lightweight mode

Post by gajop »

MetalSucker wrote:What backend should this have? SQL or some internal persistent tables/arrays native to python? I also know there is an opinion that the right way to do it is thin lobbies (and I get why) but how thick should the server be in the best case scenario?
Server already uses SQL.
Server should be "wide", it should define the needed operations to cover all the common use cases, and it should allow extension to fill others that might be required.
It shouldn't have any complicated logic like matchmaking, but rather allow an interface for it.
gajop
Moderator
Posts: 3051
Joined: 05 Aug 2009, 20:42

Re: Server lightweight mode

Post by gajop »

Licho wrote:ZK changed the protocol so that the server keeps track of who know what and sends only data you need. Information content is unchanged but you only get "user info" if you for example joined the same channel and can actually see him in GUI.

Server then knows that client X knows client Y and in what state, if state changes it can only send change itself and also sends left information as needed.

Benefit of this is that client didn't need big changes and can still for example show rich tooltips with all the info for all people available in GUI without roundtrips.
I'm not sure how this is different than how uberserver works, as almost all people in your case join #zk, so you just end up doing unnecessary checks.
Still I think that's a lot of information which a user doesn't need and that scales badly. For non-friends I'd rather have polling instead, for the few other cases a user would be interested.
User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Server lightweight mode

Post by Licho »

It comes together with two changes

1) max channel size will be limited - if more people join it they automatically go to numbered channel clone

2) there will be no auto channel joining in new ZKL, unless they go to chat section in lobby (less prominent than currently)
User avatar
PicassoCT
Journeywar Developer & Mapper
Posts: 10450
Joined: 24 Jan 2006, 21:12

Re: Server lightweight mode

Post by PicassoCT »

Maybee seems unimportant, but is there a clean way to migrate the server in the new protocoll? Meaning, that the server can transfer all its buisness to a second one, without the whole affair shuting down?

New crowd might not be so tolerant when it comes too two hours outages..
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Server lightweight mode

Post by abma »

aegis wrote:Maybe it's worth connecting to the server for a day and logging the command breakdown (e.g. 25% CLIENTSTATUS, 10% BATTLEOPENED, etc) to see what's worth optimizing for long-running connections.

thats # of commands usage between 2015-02-16T10:54:05 and 2015-02-21T23:07:56

http://paste.springfiles.com/view/raw/d71ef39c

but this is client -> server only. server -> client is missing which produces more the network load.
MetalSucker
Posts: 98
Joined: 22 Sep 2014, 20:29

Re: Server lightweight mode

Post by MetalSucker »

What the ping...
PING 2516944
HEY it was NOT a full quote it was an ORDERED list

Ah screw this why am I even trying ...
Last edited by abma on 22 Feb 2015, 01:59, edited 3 times in total.
Reason: removed full quote (sorry my initial post was already VERY unreadable)
gajop
Moderator
Posts: 3051
Joined: 05 Aug 2009, 20:42

Re: Server lightweight mode

Post by gajop »

Licho wrote: 1) max channel size will be limited - if more people join it they automatically go to numbered channel clone
This is something you can do for #zk and similar, but not for #sy, #moddev and similar.

Btw, how do you plan to send statuses of users that are just in private chat?
This is something we need to consider for uber as well.
ikinz
Posts: 5
Joined: 11 Apr 2014, 19:41

Re: Server lightweight mode

Post by ikinz »

Licho wrote: 1) max channel size will be limited - if more people join it they automatically go to numbered channel clone
What's wrong with not sending the user list and statuses once the number of users in the channel is past a certain point? Seems much cleaner than hacks with numbered channels.
User avatar
Silentwings
Posts: 3720
Joined: 25 Oct 2008, 00:23

Re: Server lightweight mode

Post by Silentwings »

I think its cleaner to decide which channels are lightweight and which aren't on a case by case basis - with the default being lightweight and lobby moderators able to set a flag for it.

There are some usergroups (lobby moderators, bots that collect stats or manage autohosts/matchmaking) which do need access to all info (except maybe who is in which channels). So a flag for if users see this, also set by lobby mods, is necessary afaics.

I haven't read enough to check if the number of users inside (but maybe not their names) would be visible to someone not inside of a battle. I think that info can't be removed from anyone, since its the most common way in which people decide what battles they want to join.
gajop
Moderator
Posts: 3051
Joined: 05 Aug 2009, 20:42

Re: Server lightweight mode

Post by gajop »

Silentwings wrote:I think its cleaner to decide which channels are lightweight and which aren't on a case by case basis - with the default being lightweight and lobby moderators able to set a flag for it.
I'm surprised to see you say this. I see no reason it can't be automated, but I guess we can allow moderators to force channels to a certain state (always lightweight or never lightweight, regardless of the number of users present). Small channels shouldn't be lightweight, and having to process it on a case-by-case basis would make the whole process much slower and less usable.
Silentwings wrote: There are some usergroups (lobby moderators, bots that collect stats or manage autohosts/matchmaking) which do need access to all info (except maybe who is in which channels). So a flag for if users see this, also set by lobby mods, is necessary afaics.
I don't see why those entities would need access to everything, but they should definitely get all user info relevant to their battleroom, queue and similar.
Although moderator clients could be made to work outside of lightweight mode, maybe it would be better to consider a polling option instead (something we'd need for non-friend private chat as well). No one needs all information, all the time.
Stats bots could be made to retrieve only statistical information, rather than on a per-user level.
Silentwings wrote: I haven't read enough to check if the number of users inside (but maybe not their names) would be visible to someone not inside of a battle. I think that info can't be removed from anyone, since its the most common way in which people decide what battles they want to join.
I'm not sure what to do with battlerooms yet. I think it might be best to have an on/off mode that triggers battleroom messages being sent (as they're not) - it should be off by default. If matchmaking replaces battlerooms (which I hope it would), it really wouldn't be a big issue as there'd be just a few custom games going on.
User avatar
Silentwings
Posts: 3720
Joined: 25 Oct 2008, 00:23

Re: Server lightweight mode

Post by Silentwings »

I don't think its realistic at all to want/expect matchmaking to be used for all battles. I think its unlikely to be wanted for types of game which feature a large social element - and consequently rely on the persistence of a battleroom and often also on semi-human management of that battleroom. E.g. large team games, "showcase" games where lots of spectators watch a handful of top players repeatedly fight, clan games between friends, special game modes (where players choose/explore specific maps or modoptions), dedicated game development, and so on.

I can see some situations where I expect matchmaking to be useful:
(1) Games between players who don't really know each other, but want a "normal" small team or 1v1 game.
(2) Games between spectators who are currently spectating some other game while they wait.

Lobby moderators trying to handle a persisently abusing or cheating player (who is not necessarily easy to identify) do need access to all data about what all players are doing. I don't think its sensible to try and predict what types of abuse might occur and only cater for lobby mods having enough info to deal with those specifically. Similarly, stat bots that try to record data about what types of battles, or how regularly/much users play, etc etc, couldn't work without access to nearly all data about all battles. I don't think its possible to predict what data people (i.e. game devs) might want to collect. Ofc there aren't many of either case.
gajop
Moderator
Posts: 3051
Joined: 05 Aug 2009, 20:42

Re: Server lightweight mode

Post by gajop »

Silentwings wrote:I don't think its realistic at all to want/expect matchmaking to be used for all battles. I think its unlikely to be wanted for types of game which feature a large social element - and consequently rely on the persistence of a battleroom and often also on semi-human management of that battleroom. E.g. large team games, "showcase" games where lots of spectators watch a handful of top players repeatedly fight, clan games between friends, special game modes (where players choose/explore specific maps or modoptions), dedicated game development, and so on.
I think those type of games (battles) you mentioned are the exception rather than the norm. I'd also say that large team games could be done well with matchmaking if we allow spectacting.
In no way do I want to completely remove custom games, but I do hope MM becomes the most commonly used option.
Silentwings wrote: Lobby moderators trying to handle a persisently abusing or cheating player (who is not necessarily easy to identify) do need access to all data about what all players are doing. I don't think its sensible to try and predict what types of abuse might occur and only cater for lobby mods having enough info to deal with those specifically. Similarly, stat bots that try to record data about what types of battles, or how regularly/much users play, etc etc, couldn't work without access to nearly all data about all battles. I don't think its possible to predict what data people (i.e. game devs) might want to collect. Ofc there aren't many of either case.
In any case, lightweight mode will initially be optional, and even later, we don't have to make it mandatory for certain types of users. Functionality comes first, and if those types of users ever cause significant performance issues we can look into designing a polling-based option for them.
abma
Spring Developer
Posts: 3798
Joined: 01 Jun 2009, 00:08

Re: Server lightweight mode

Post by abma »

spent 10 minutes in writing down complexity of current lobby commands, the performance killer commands are:

(client -> server)

Code: Select all

UPDATEBATTLEINFO
MYSTATUS
JOIN
LOGIN
JOINBATTLE
EXIT
LEAVEBATTLE
OPENBATTLE / OPENBATTLEEX
KICKFROMBATTLE
all these commands result in updates sent to all logged in clients.

so when trying to speed things up, UPDATEBATTLEINFO & MYSTATUS should be changed first!

http://paste.springfiles.com/view/raw/d71ef39c

(idk an easy solution sharing a .ods atm)


imo viewtopic.php?f=71&t=33072 is the right approach for that. if it works well the "normal" battles can be tuned to scale for more players as well.
Post Reply

Return to “Infrastructure Development”