PING/PONG

[abma]

i've investigated current network problems and found, that PING is send by some lobby clients way to often.

ATM we have sometimes ~300 clients connected to the server, some lobbies send every 10 seconds a PING which leads to ~30 PING/PONG a second.

i know thats data-wise seen nothing, but its a first step to reduce the needed traffic for the lobby protocol. i've also made a change to the lobby server (which is not active yet) which shows a warning and doesn't respond to PING when the time between PING send is < 15 seconds:

https://github.com/spring/LobbyProtocol ... 384c5c5e3a
and
https://github.com/spring/uberserver/co ... f3a25d2f6d

my suggestion is to send a PING every 30 seconds, or is it still to low?

any thoughts about this?

[cleanrock]

I am pretty sure PING is not required atm.
flobby do not send it and is not disconnected if not sending stuff for many minutes, works fine anyway.
I dont think we need to require PING, perhaps we should not even encourage lobbies to send it.

[abma]

i see it similar, imo only a ping sent by server would make sense to detect "dead" connections. with id's every command sent by client with a response can be used to detect round-trip.

[abma]

hm, talked a bit with licho.

my proposal: add the parameter TIMEOUT (or nextping or...?) to PING, if set, when after this time no next ping is received, disconnect the user, for example:

PING 60
PONG
PING 10
<~10 seconds>
disconnect user


thoughts? this would solve the problem with dead connections not instantly detected by the server, but would also allow connections to withstand some heavy network lag, user. the lobby or user/autohost can decide, also PING/PONG would be still optional. lobby servers interval checking for dead connection is fixed, so PING 10 won't kill the connection exactly 10 seconds after no PONG is received, but it should be some "low" value.

PING without parameter clears the timeout / doesn't require an additional PING to be send.

[cleanrock]

Only use for PING i see is when lobby client is hanging without closing the socket.
I am skeptical to add this uberserver overhead just to catch bugged lobby clients.

[Silentwings]

It's def useful to give users some significant time before disconnecting them - if someone briefly loses connection while ingame and then wants to reconnect; when they time out they may lose their slot in the host & then can't reconnect. The setting as it is now is perfect imo.

[bibim]

Only use for PING i see is when lobby client is hanging without closing the socket.
I am skeptical to add this uberserver overhead just to catch bugged lobby clients.

It's not only bugged lobby clients hanging without closing the socket, but also players having system crashes, big network outages (frequent for wifi users) etc. I just tested the ghost "functionality" to see how much time someone can appear online without being actually connected, and my ghost lasted 17 minutes.

I wonder how many players hours have been lost since we use uberserver, due to ghost players still appearing online in the battleroom whereas they hardcrashed or got a network disconnection and aren't coming back. As the other players still see the ghost online in the battleroom they think he just got a spring crash but will try to reconnect to the game. So they try to communicate with him during several minutes, then they have to call a vote to spec him so they can retry to start the game etc. I've seen that happening tons of times... Lot of wasted time...

It's the basics, for realtime communications and especially to prepare games, you need to know who is online or not, and the PING/PONG scheme with decent timeout is the simplest way to implement that. We don't need systems to make connection persistent after outages of several minutes and waste players time due to side effects (at least if you implement something like that you need to add a flag on these ghosts so other players know it is a ghost that didn't send data for say 60 seconds).
Furthermore, if the ghost reconnects, other players don't know that he went offline (they see an instant leave/join) and that he didn't see the messages written during this time etc.
All this really is a regression for me...

my proposal: add the parameter TIMEOUT (or nextping or...?) to PING, if set, when after this time no next ping is received, disconnect the user, for example:

PING 60
PONG
PING 10
<~10 seconds>
disconnect user


thoughts? this would solve the problem with dead connections not instantly detected by the server, but would also allow connections to withstand some heavy network lag, user. the lobby or user/autohost can decide, also PING/PONG would be still optional. lobby servers interval checking for dead connection is fixed, so PING 10 won't kill the connection exactly 10 seconds after no PONG is received, but it should be some "low" value.

PING without parameter clears the timeout / doesn't require an additional PING to be send.

Do you really want to allow anyone to make his account(s) appear online whereas he's been offline for a long time, really?
At least if you plan to allow that, please make 60 seconds the default ("PING" = "PING 60") so we have a decent behavior by default... And 180 should be the maximum value allowed...

Still I don't see the point of all this, any lobby client implemented correctly would consider that the communication with lobby server is down if no communication has been possible for minutes, and will try to reconnect. And when you reconnect, your previous ghost account is removed by uberserver and you just spawn a normal new connection, leaving all your channels, battles... And of course as I said you don't receive the messages sent to your ghost during this offline time.

It's def useful to give users some significant time before disconnecting them - if someone briefly loses connection while ingame and then wants to reconnect; when they time out they may lose their slot in the host & then can't reconnect. The setting as it is now is perfect imo.

0.11.2: allow disconnected players who didn't lose yet to reconnect to the battle lobby even if the battle is full, so that they can rejoin the game

You just have to use !unlockSpec and it will open the battle for you even if spec slots are full, then you can rejoin normally.

[abma]

short off-topic:

maybe it should be noticed that NOBODY created a bug report about this ghosts. i had to investigate the code myself without knowing any details. i've created the bug report myself: https://github.com/spring/uberserver/issues/4
also the lobby protocol documentation was VERY unspecific about PING/PONG.

It's not only bugged lobby clients hanging without closing the socket, but also players having system crashes, big network outages (frequent for wifi users) etc. I just tested the ghost "functionality" to see how much time someone can appear online without being actually connected, and my ghost lasted 17 minutes.

my ghost only appeared ~5 minutes. did you login to some channel or something? imo the fewer data the client receives, the longer he can "survive".

i'll try with with tcp-keepalive first as kicking out users sending no pings would break some clients.

[dansan]

+1 bibim

Ghosted players are very annoying! Unfortunately you have to wait a lot in spring games already before the match starts... 1min should be enough for users to come back online if they had a disconnect. If waiting longer, then I think the the tolerance-barrier of the other players is hit.

Regarding lobbies: IMO implement a defined and documented behavior and notify lobby-devs that in x weeks the change will go online. Then it's their job to fix their clients.

Less waiting -> more fun

[bibim]

maybe it should be noticed that NOBODY created a bug report about this ghosts.

That's because few persons are really aware of this problem. When you don't know how it works you just think the players went afk.

also the lobby protocol documentation was VERY unspecific about PING/PONG.

Protocol rules

A client must maintain a constant connection with the server. That is if no data is to be transfered, the client must send PING commands to the server on regular intervals. This ensures that the server can properly detect any network timeouts/disconnects. The server will automatically disconnect a client if no data has been received from it within a certain time period. A client should send some data at least after half that interval passed.

The value of the interval is missing, but apart from that it clearly states that lobby clients must send data regularly to stay connected, doesn't it?

my ghost only appeared ~5 minutes. did you login to some channel or something? imo the fewer data the client receives, the longer he can "survive".

I just joined a battle. I guess it's directly related to the size of the socket sending buffer and the activity on the server yes.

i'll try with with tcp-keepalive first as kicking out users sending no pings would break some clients.

It would only break the clients which don't respect current protocol specification.

[abma]

The value of the interval is missing, but apart from that it clearly states that lobby clients must send data regularly to stay connected, doesn't it?

i've already added a suggestion how often a client should ping, not sure if the values make sense.

I just joined a battle. I guess it's directly related to the size of the socket sending buffer and the activity on the server yes.

can you please try again? my test seems to fail, i'm mostly instandly disconnected... tcp-keepalive is now used, should disconnect after ~1 minute.

[aegis]

I think it would resolve both the player and host issue with battles if the server kicked any user from their battle after a reasonable idle time - when it would've normally kicked them from the server.

"Ghost" users should also have their clientstatus set to away, which could be done by the same timer.

A test was made to re-establish channels/chat/battles when a user was ghosted, but the lobbies at the time could not handle this well. Maybe reform discussions were had but nothing ever came of it.

[bibim]

can you please try again? my test seems to fail, i'm mostly instandly disconnected... tcp-keepalive is now used, should disconnect after ~1 minute.

I just tested, my ghost lasted more than 15min again...

I think it would resolve both the player and host issue with battles if the server kicked any user from their battle after a reasonable idle time - when it would've normally kicked them from the server.

This wouldn't prevent players trying to convince the ghost player to go back through PM chat etc... Also they wouldn't understand why he would leave whereas he wanted to play and was ringing people 1 minute before for instance.

"Ghost" users should also have their clientstatus set to away, which could be done by the same timer.

This could have reduced the amount of wtf indeed. Although reusing a state which normally means "connected but not in front of computer" for a different meaning "experiencing connection difficulties" isn't the good way to implement it. This wouldn't prevent other players trying to send messages to the ghost thinking he will get them when he's back for instance.

[abma]

I just tested, my ghost lasted more than 15min again...

can you try again please? next "patch" applyed a few minutes ago. how do you test? didn't find an easy/clean way to check...

this commit imo should fix it: (first try was broken, as it enabled keepalive only on server-socket which seems to be fail)
https://github.com/spring/uberserver/co ... 42551d8b8b

if it works, it should kick in ~1 minute after disconnecting.

[bibim]

No it still doesn't work (ghost lasted > 15 min again).
I just test by disabling the network while lobby client is running.

[abma]

No it still doesn't work (ghost lasted > 15 min again).
I just test by disabling the network while lobby client is running.

meh, thanks. an other approach: now the server disconnects clients sending now data for >=1 minute. this only happens when the client send one PING. dead clients should be disconnected in at max ~90 seconds. clients not using PING won't be disconnected by this.

see https://github.com/spring/uberserver/co ... 3c5160b2f5

also this is close to what the protocol said how it should be implemented.

can you test again? i've only one computer here, so disabling network isn't a good test as i don't see the result.

[bibim]

this only happens when the client send one PING. clients not using PING won't be disconnected by this.

So this means we will still have ghost problems due to broken lobby clients...

see https://github.com/spring/uberserver/co ... 3c5160b2f5

also this is close to what the protocol said how it should be implemented.

This is basically how it should be implemented, except from the clients-not-sending-ping exception.

can you test again? i've only one computer here, so disabling network isn't a good test as i don't see the result.

Did you try to close your lobby client while your network is disabled, then re-enable your network, restart your lobby client and reconnect (with a different account of course)?

[abma]

This is basically how it should be implemented, except from the clients-not-sending-ping exception.

i dislike possible breaking current clients, we need some "migration" time / maybe a compatibility flag and a date when it is enforced. when adding the compatibility flag i would like to add a server side ping, this doesn't require the client to constantly spam pings to the server.

also afaik all "major" (Springlobby / ZK-Lobby / SpringWebLobby / Springie) clients already send PING. For IRC/Chat-Clients it shouldn't matter as they don't join battles.

but true, at some point all dead clients should be detected after a short time.

[bibim]

i dislike possible breaking current clients, we need some "migration" time / maybe a compatibility flag and a date when it is enforced.

You want to add a compatibility flag to let clients say they are compatible with the protocol which existed and was specified even before I added the compatibility flag system into it?

also afaik all "major" lobbies already send PING.

Exactly, and that's the reason why it should be enforced now as it used to be, to avoid more broken clients appearing and inducing ghost problem.

when adding the compatibility flag i would like to add a server side ping, this doesn't require the client to constantly spam pings to the server.

For a given timeout value, the amount of PING/PONG commands required to keep a connection alive doesn't change, whatever the peer sending the PING.

If client and server aren't aware of each other timeout values, there could be extra PING commands sent, whatever the peer sending the PING initially: if client is sending PING and thinks server uses a timeout of 30s whereas server uses a timeout of 60s, the client will send unneeded PING. If server is sending PING every 60s but client uses a timeout of 30s, the client will send unneeded PING (or any basic command with response such as GETINGAMETIME) to check network connection => unneeded traffic again.

So if a client spams PING commands, it's not a problem of protocol design, it's a problem of implementation at client side, not consistent with server's timeout value. It's not related to the PING command itself.

If lobby traffic is really problematic currently, you could try to improve lobby command spam detection. It's easier to force lobby clients to adjust their timeouts than force them to implement server-side ping AND adjust their timeouts...

[abma]

If client and server aren't aware of each other timeout values, there could be extra PING commands sent, whatever the peer sending the PING initially: if client is sending PING and thinks server uses a timeout of 30s whereas server uses a timeout of 60s, the client will send unneeded PING. If server is sending PING every 60s but client uses a timeout of 30s, the client will send unneeded PING (or any basic command with response such as GETINGAMETIME) to check network connection => unneeded traffic again.

thats why i want a server initiated ping:

the idea for server side ping is:
- if no data received from client longer than x, server sends ping
- if x+y time no data/PONG received from client: disconnect "dead" client

client doesn't have to care about values, it just have to respond with PONG to a PING from server. also, ping is only send when needed.

benefit: client doesn't need to care about timeout values & timeout values can be changed server-side when needed, no useless PING's send.