Server lightweight mode
Moderator: Moderators
Server lightweight mode
Lobby server currently scales badly. A lot of commands scale as O(n^2), where n is the amount of players/users, which is not going to work if we ever get a large enough amount of users. Most of the commands just don't need to be send.
I propose to limit when some user-related commands are sent (mostly by limiting it them to friends only).
Related uberserver tickets:
https://github.com/spring/uberserver/issues/111
https://github.com/spring/uberserver/issues/122
Commands to change:
CLIENTSTATUS (should only be sent to friends)
JOINBATTLE, LEFTBATTLE (should only be sent to friends or for other people in the same battleroom)
ADDUSER, REMOVEUSER (should only be sent to friends)
CLIENTS (sent in channels that are not in lightweight mode)
JOINED, LEFT (sent in channels that are not in lightweight mode)
Channels could enter a lightweight mode, which needs one command:
CHANNELSTATUS channelName lightweightMode (parametric command that could be extended later on)
Ideally the lobby server would put channels with more than an X amount of people in the lightweight mode. X would be an arguably large number, e.g. 100 or 1000.
PS: Not sure what to do with BATTLE* commands, they're also kinda spammy and not of interest most of the time. Maybe polling is better in this case..? Or a way to turn on/off whether such information is received.
PSS: I really don't want to allow some sort of "object listeners". It shouldn't be the clients that decide what kind of information they receive. Too many options in this regard makes the protocol complicated and not clear how it would scale.
I propose to limit when some user-related commands are sent (mostly by limiting it them to friends only).
Related uberserver tickets:
https://github.com/spring/uberserver/issues/111
https://github.com/spring/uberserver/issues/122
Commands to change:
CLIENTSTATUS (should only be sent to friends)
JOINBATTLE, LEFTBATTLE (should only be sent to friends or for other people in the same battleroom)
ADDUSER, REMOVEUSER (should only be sent to friends)
CLIENTS (sent in channels that are not in lightweight mode)
JOINED, LEFT (sent in channels that are not in lightweight mode)
Channels could enter a lightweight mode, which needs one command:
CHANNELSTATUS channelName lightweightMode (parametric command that could be extended later on)
Ideally the lobby server would put channels with more than an X amount of people in the lightweight mode. X would be an arguably large number, e.g. 100 or 1000.
PS: Not sure what to do with BATTLE* commands, they're also kinda spammy and not of interest most of the time. Maybe polling is better in this case..? Or a way to turn on/off whether such information is received.
PSS: I really don't want to allow some sort of "object listeners". It shouldn't be the clients that decide what kind of information they receive. Too many options in this regard makes the protocol complicated and not clear how it would scale.
Re: Server lightweight mode
Maybe it's worth connecting to the server for a day and logging the command breakdown (e.g. 25% CLIENTSTATUS, 10% BATTLEOPENED, etc) to see what's worth optimizing for long-running connections.
I'd recommend optimizing the new user connect, as that's probably the most bandwidth-expensive thing. When a new user joins, several commands (ADDUSER, CLIENTSTATUS, JOINED #main, JOINED #help) are sent to pretty much all connected users, *and* a full list of ADDUSER, CLIENTSTATUS, CLIENTS, BATTLEOPENED, UPDATEBATTLEINFO, JOINEDBATTLE are sent.
From the days of TASClient, I remember the protocol spam on connect to be pretty huge even with the low user counts then.
It might help to make common channels blind (you can't see the channel userlist, but you can still talk to them). A cheap "friends list" is to just join a channel with a few other people. You can see when they're online because they're in the channel.
Matchmaking can also help with this, because you don't need to see anyone or any battles until the game is ready.
If CPU load is becoming an issue, it might be worth trying uberserver under pypy.
I'd recommend optimizing the new user connect, as that's probably the most bandwidth-expensive thing. When a new user joins, several commands (ADDUSER, CLIENTSTATUS, JOINED #main, JOINED #help) are sent to pretty much all connected users, *and* a full list of ADDUSER, CLIENTSTATUS, CLIENTS, BATTLEOPENED, UPDATEBATTLEINFO, JOINEDBATTLE are sent.
From the days of TASClient, I remember the protocol spam on connect to be pretty huge even with the low user counts then.
It might help to make common channels blind (you can't see the channel userlist, but you can still talk to them). A cheap "friends list" is to just join a channel with a few other people. You can see when they're online because they're in the channel.
Matchmaking can also help with this, because you don't need to see anyone or any battles until the game is ready.
If CPU load is becoming an issue, it might be worth trying uberserver under pypy.
-
- Posts: 98
- Joined: 22 Sep 2014, 20:29
Re: Server lightweight mode
What backend should this have? SQL or some internal persistent tables/arrays native to python? I also know there is an opinion that the right way to do it is thin lobbies (and I get why) but how thick should the server be in the best case scenario?
Re: Server lightweight mode
ZK changed the protocol so that the server keeps track of who know what and sends only data you need. Information content is unchanged but you only get "user info" if you for example joined the same channel and can actually see him in GUI.
Server then knows that client X knows client Y and in what state, if state changes it can only send change itself and also sends left information as needed.
Benefit of this is that client didn't need big changes and can still for example show rich tooltips with all the info for all people available in GUI without roundtrips.
Server then knows that client X knows client Y and in what state, if state changes it can only send change itself and also sends left information as needed.
Benefit of this is that client didn't need big changes and can still for example show rich tooltips with all the info for all people available in GUI without roundtrips.
Re: Server lightweight mode
I agree, but only after matchmaking is implemented and becomes widely used.aegis wrote:Maybe it's worth connecting to the server for a day and logging the command breakdown (e.g. 25% CLIENTSTATUS, 10% BATTLEOPENED, etc) to see what's worth optimizing for long-running connections.
We should compare server performance under two conditions: massive login and constant, high workload.
Actually, most of these are to be removed or to be received by a select few users.aegis wrote:I'd recommend optimizing the new user connect, as that's probably the most bandwidth-expensive thing. When a new user joins, several commands (ADDUSER, CLIENTSTATUS, JOINED #main, JOINED #help) are sent to pretty much all connected users, *and* a full list of ADDUSER, CLIENTSTATUS, CLIENTS, BATTLEOPENED, UPDATEBATTLEINFO, JOINEDBATTLE are sent.
This is what the lightweight channel mode will do essentialy, but it should be done dynamically instead of just using a predefined list of channels.aegis wrote:It might help to make common channels blind (you can't see the channel userlist, but you can still talk to them).
Friendship is not a transitive relation and should be voluntary. We already have a friend list implemented so I don't see a need for this.aegis wrote:A cheap "friends list" is to just join a channel with a few other people. You can see when they're online because they're in the channel.
Indeed, although if we want browsable spectating, we need to allow battle lists (which don't scale well imo). But yeah, MM should scale pretty well.aegis wrote:Matchmaking can also help with this, because you don't need to see anyone or any battles until the game is ready.
That's the last thing I want to consider now. There were also some suggestions about using tornado instead. It's good to know these options are open, but a protocol change is needed first.aegis wrote:If CPU load is becoming an issue, it might be worth trying uberserver under pypy.
Re: Server lightweight mode
Server already uses SQL.MetalSucker wrote:What backend should this have? SQL or some internal persistent tables/arrays native to python? I also know there is an opinion that the right way to do it is thin lobbies (and I get why) but how thick should the server be in the best case scenario?
Server should be "wide", it should define the needed operations to cover all the common use cases, and it should allow extension to fill others that might be required.
It shouldn't have any complicated logic like matchmaking, but rather allow an interface for it.
Re: Server lightweight mode
I'm not sure how this is different than how uberserver works, as almost all people in your case join #zk, so you just end up doing unnecessary checks.Licho wrote:ZK changed the protocol so that the server keeps track of who know what and sends only data you need. Information content is unchanged but you only get "user info" if you for example joined the same channel and can actually see him in GUI.
Server then knows that client X knows client Y and in what state, if state changes it can only send change itself and also sends left information as needed.
Benefit of this is that client didn't need big changes and can still for example show rich tooltips with all the info for all people available in GUI without roundtrips.
Still I think that's a lot of information which a user doesn't need and that scales badly. For non-friends I'd rather have polling instead, for the few other cases a user would be interested.
Re: Server lightweight mode
It comes together with two changes
1) max channel size will be limited - if more people join it they automatically go to numbered channel clone
2) there will be no auto channel joining in new ZKL, unless they go to chat section in lobby (less prominent than currently)
1) max channel size will be limited - if more people join it they automatically go to numbered channel clone
2) there will be no auto channel joining in new ZKL, unless they go to chat section in lobby (less prominent than currently)
Re: Server lightweight mode
Maybee seems unimportant, but is there a clean way to migrate the server in the new protocoll? Meaning, that the server can transfer all its buisness to a second one, without the whole affair shuting down?
New crowd might not be so tolerant when it comes too two hours outages..
New crowd might not be so tolerant when it comes too two hours outages..
Re: Server lightweight mode
aegis wrote:Maybe it's worth connecting to the server for a day and logging the command breakdown (e.g. 25% CLIENTSTATUS, 10% BATTLEOPENED, etc) to see what's worth optimizing for long-running connections.
thats # of commands usage between 2015-02-16T10:54:05 and 2015-02-21T23:07:56
http://paste.springfiles.com/view/raw/d71ef39c
but this is client -> server only. server -> client is missing which produces more the network load.
-
- Posts: 98
- Joined: 22 Sep 2014, 20:29
Re: Server lightweight mode
What the ping...
Ah screw this why am I even trying ...
HEY it was NOT a full quote it was an ORDERED listPING 2516944
Ah screw this why am I even trying ...
Last edited by abma on 22 Feb 2015, 01:59, edited 3 times in total.
Reason: removed full quote (sorry my initial post was already VERY unreadable)
Reason: removed full quote (sorry my initial post was already VERY unreadable)
Re: Server lightweight mode
This is something you can do for #zk and similar, but not for #sy, #moddev and similar.Licho wrote: 1) max channel size will be limited - if more people join it they automatically go to numbered channel clone
Btw, how do you plan to send statuses of users that are just in private chat?
This is something we need to consider for uber as well.
Re: Server lightweight mode
What's wrong with not sending the user list and statuses once the number of users in the channel is past a certain point? Seems much cleaner than hacks with numbered channels.Licho wrote: 1) max channel size will be limited - if more people join it they automatically go to numbered channel clone
- Silentwings
- Posts: 3720
- Joined: 25 Oct 2008, 00:23
Re: Server lightweight mode
I think its cleaner to decide which channels are lightweight and which aren't on a case by case basis - with the default being lightweight and lobby moderators able to set a flag for it.
There are some usergroups (lobby moderators, bots that collect stats or manage autohosts/matchmaking) which do need access to all info (except maybe who is in which channels). So a flag for if users see this, also set by lobby mods, is necessary afaics.
I haven't read enough to check if the number of users inside (but maybe not their names) would be visible to someone not inside of a battle. I think that info can't be removed from anyone, since its the most common way in which people decide what battles they want to join.
There are some usergroups (lobby moderators, bots that collect stats or manage autohosts/matchmaking) which do need access to all info (except maybe who is in which channels). So a flag for if users see this, also set by lobby mods, is necessary afaics.
I haven't read enough to check if the number of users inside (but maybe not their names) would be visible to someone not inside of a battle. I think that info can't be removed from anyone, since its the most common way in which people decide what battles they want to join.
Re: Server lightweight mode
I'm surprised to see you say this. I see no reason it can't be automated, but I guess we can allow moderators to force channels to a certain state (always lightweight or never lightweight, regardless of the number of users present). Small channels shouldn't be lightweight, and having to process it on a case-by-case basis would make the whole process much slower and less usable.Silentwings wrote:I think its cleaner to decide which channels are lightweight and which aren't on a case by case basis - with the default being lightweight and lobby moderators able to set a flag for it.
I don't see why those entities would need access to everything, but they should definitely get all user info relevant to their battleroom, queue and similar.Silentwings wrote: There are some usergroups (lobby moderators, bots that collect stats or manage autohosts/matchmaking) which do need access to all info (except maybe who is in which channels). So a flag for if users see this, also set by lobby mods, is necessary afaics.
Although moderator clients could be made to work outside of lightweight mode, maybe it would be better to consider a polling option instead (something we'd need for non-friend private chat as well). No one needs all information, all the time.
Stats bots could be made to retrieve only statistical information, rather than on a per-user level.
I'm not sure what to do with battlerooms yet. I think it might be best to have an on/off mode that triggers battleroom messages being sent (as they're not) - it should be off by default. If matchmaking replaces battlerooms (which I hope it would), it really wouldn't be a big issue as there'd be just a few custom games going on.Silentwings wrote: I haven't read enough to check if the number of users inside (but maybe not their names) would be visible to someone not inside of a battle. I think that info can't be removed from anyone, since its the most common way in which people decide what battles they want to join.
- Silentwings
- Posts: 3720
- Joined: 25 Oct 2008, 00:23
Re: Server lightweight mode
I don't think its realistic at all to want/expect matchmaking to be used for all battles. I think its unlikely to be wanted for types of game which feature a large social element - and consequently rely on the persistence of a battleroom and often also on semi-human management of that battleroom. E.g. large team games, "showcase" games where lots of spectators watch a handful of top players repeatedly fight, clan games between friends, special game modes (where players choose/explore specific maps or modoptions), dedicated game development, and so on.
I can see some situations where I expect matchmaking to be useful:
(1) Games between players who don't really know each other, but want a "normal" small team or 1v1 game.
(2) Games between spectators who are currently spectating some other game while they wait.
Lobby moderators trying to handle a persisently abusing or cheating player (who is not necessarily easy to identify) do need access to all data about what all players are doing. I don't think its sensible to try and predict what types of abuse might occur and only cater for lobby mods having enough info to deal with those specifically. Similarly, stat bots that try to record data about what types of battles, or how regularly/much users play, etc etc, couldn't work without access to nearly all data about all battles. I don't think its possible to predict what data people (i.e. game devs) might want to collect. Ofc there aren't many of either case.
I can see some situations where I expect matchmaking to be useful:
(1) Games between players who don't really know each other, but want a "normal" small team or 1v1 game.
(2) Games between spectators who are currently spectating some other game while they wait.
Lobby moderators trying to handle a persisently abusing or cheating player (who is not necessarily easy to identify) do need access to all data about what all players are doing. I don't think its sensible to try and predict what types of abuse might occur and only cater for lobby mods having enough info to deal with those specifically. Similarly, stat bots that try to record data about what types of battles, or how regularly/much users play, etc etc, couldn't work without access to nearly all data about all battles. I don't think its possible to predict what data people (i.e. game devs) might want to collect. Ofc there aren't many of either case.
Re: Server lightweight mode
I think those type of games (battles) you mentioned are the exception rather than the norm. I'd also say that large team games could be done well with matchmaking if we allow spectacting.Silentwings wrote:I don't think its realistic at all to want/expect matchmaking to be used for all battles. I think its unlikely to be wanted for types of game which feature a large social element - and consequently rely on the persistence of a battleroom and often also on semi-human management of that battleroom. E.g. large team games, "showcase" games where lots of spectators watch a handful of top players repeatedly fight, clan games between friends, special game modes (where players choose/explore specific maps or modoptions), dedicated game development, and so on.
In no way do I want to completely remove custom games, but I do hope MM becomes the most commonly used option.
In any case, lightweight mode will initially be optional, and even later, we don't have to make it mandatory for certain types of users. Functionality comes first, and if those types of users ever cause significant performance issues we can look into designing a polling-based option for them.Silentwings wrote: Lobby moderators trying to handle a persisently abusing or cheating player (who is not necessarily easy to identify) do need access to all data about what all players are doing. I don't think its sensible to try and predict what types of abuse might occur and only cater for lobby mods having enough info to deal with those specifically. Similarly, stat bots that try to record data about what types of battles, or how regularly/much users play, etc etc, couldn't work without access to nearly all data about all battles. I don't think its possible to predict what data people (i.e. game devs) might want to collect. Ofc there aren't many of either case.
Re: Server lightweight mode
spent 10 minutes in writing down complexity of current lobby commands, the performance killer commands are:
(client -> server)
all these commands result in updates sent to all logged in clients.
so when trying to speed things up, UPDATEBATTLEINFO & MYSTATUS should be changed first!
http://paste.springfiles.com/view/raw/d71ef39c
(idk an easy solution sharing a .ods atm)
imo viewtopic.php?f=71&t=33072 is the right approach for that. if it works well the "normal" battles can be tuned to scale for more players as well.
(client -> server)
Code: Select all
UPDATEBATTLEINFO
MYSTATUS
JOIN
LOGIN
JOINBATTLE
EXIT
LEAVEBATTLE
OPENBATTLE / OPENBATTLEEX
KICKFROMBATTLE
so when trying to speed things up, UPDATEBATTLEINFO & MYSTATUS should be changed first!
http://paste.springfiles.com/view/raw/d71ef39c
(idk an easy solution sharing a .ods atm)
imo viewtopic.php?f=71&t=33072 is the right approach for that. if it works well the "normal" battles can be tuned to scale for more players as well.