PING/PONG

PING/PONG

Discuss development of lobby clients, server, autohosts and auto-download software.

Moderators: Moderators, Lobby Developers

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

PING/PONG

Post by abma » 26 Sep 2013, 04:21

i've investigated current network problems and found, that PING is send by some lobby clients way to often.

ATM we have sometimes ~300 clients connected to the server, some lobbies send every 10 seconds a PING which leads to ~30 PING/PONG a second.

i know thats data-wise seen nothing, but its a first step to reduce the needed traffic for the lobby protocol. i've also made a change to the lobby server (which is not active yet) which shows a warning and doesn't respond to PING when the time between PING send is < 15 seconds:

https://github.com/spring/LobbyProtocol ... 384c5c5e3a
and
https://github.com/spring/uberserver/co ... f3a25d2f6d

my suggestion is to send a PING every 30 seconds, or is it still to low?

any thoughts about this?
0 x

cleanrock
Former Engine Dev
Posts: 115
Joined: 21 Feb 2009, 07:42

Re: PING/PONG

Post by cleanrock » 26 Sep 2013, 16:10

I am pretty sure PING is not required atm.
flobby do not send it and is not disconnected if not sending stuff for many minutes, works fine anyway.
I dont think we need to require PING, perhaps we should not even encourage lobbies to send it.
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: PING/PONG

Post by abma » 26 Sep 2013, 16:41

i see it similar, imo only a ping sent by server would make sense to detect "dead" connections. with id's every command sent by client with a response can be used to detect round-trip.
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: PING/PONG

Post by abma » 27 Sep 2013, 01:23

hm, talked a bit with licho.

my proposal: add the parameter TIMEOUT (or nextping or...?) to PING, if set, when after this time no next ping is received, disconnect the user, for example:

PING 60
PONG
PING 10
<~10 seconds>
disconnect user


thoughts? this would solve the problem with dead connections not instantly detected by the server, but would also allow connections to withstand some heavy network lag, user. the lobby or user/autohost can decide, also PING/PONG would be still optional. lobby servers interval checking for dead connection is fixed, so PING 10 won't kill the connection exactly 10 seconds after no PONG is received, but it should be some "low" value.

PING without parameter clears the timeout / doesn't require an additional PING to be send.
0 x

cleanrock
Former Engine Dev
Posts: 115
Joined: 21 Feb 2009, 07:42

Re: PING/PONG

Post by cleanrock » 27 Sep 2013, 16:22

Only use for PING i see is when lobby client is hanging without closing the socket.
I am skeptical to add this uberserver overhead just to catch bugged lobby clients.
0 x

User avatar
Silentwings
Moderator
Posts: 3595
Joined: 25 Oct 2008, 00:23

Re: PING/PONG

Post by Silentwings » 27 Sep 2013, 21:08

It's def useful to give users some significant time before disconnecting them - if someone briefly loses connection while ingame and then wants to reconnect; when they time out they may lose their slot in the host & then can't reconnect. The setting as it is now is perfect imo.
0 x

User avatar
bibim
Lobby Developer
Posts: 901
Joined: 06 Dec 2007, 11:12

Re: PING/PONG

Post by bibim » 30 Sep 2013, 21:12

cleanrock wrote:Only use for PING i see is when lobby client is hanging without closing the socket.
I am skeptical to add this uberserver overhead just to catch bugged lobby clients.
It's not only bugged lobby clients hanging without closing the socket, but also players having system crashes, big network outages (frequent for wifi users) etc. I just tested the ghost "functionality" to see how much time someone can appear online without being actually connected, and my ghost lasted 17 minutes.

I wonder how many players hours have been lost since we use uberserver, due to ghost players still appearing online in the battleroom whereas they hardcrashed or got a network disconnection and aren't coming back. As the other players still see the ghost online in the battleroom they think he just got a spring crash but will try to reconnect to the game. So they try to communicate with him during several minutes, then they have to call a vote to spec him so they can retry to start the game etc. I've seen that happening tons of times... Lot of wasted time...

It's the basics, for realtime communications and especially to prepare games, you need to know who is online or not, and the PING/PONG scheme with decent timeout is the simplest way to implement that. We don't need systems to make connection persistent after outages of several minutes and waste players time due to side effects (at least if you implement something like that you need to add a flag on these ghosts so other players know it is a ghost that didn't send data for say 60 seconds).
Furthermore, if the ghost reconnects, other players don't know that he went offline (they see an instant leave/join) and that he didn't see the messages written during this time etc.
All this really is a regression for me...
abma wrote:my proposal: add the parameter TIMEOUT (or nextping or...?) to PING, if set, when after this time no next ping is received, disconnect the user, for example:

PING 60
PONG
PING 10
<~10 seconds>
disconnect user


thoughts? this would solve the problem with dead connections not instantly detected by the server, but would also allow connections to withstand some heavy network lag, user. the lobby or user/autohost can decide, also PING/PONG would be still optional. lobby servers interval checking for dead connection is fixed, so PING 10 won't kill the connection exactly 10 seconds after no PONG is received, but it should be some "low" value.

PING without parameter clears the timeout / doesn't require an additional PING to be send.
Do you really want to allow anyone to make his account(s) appear online whereas he's been offline for a long time, really?
At least if you plan to allow that, please make 60 seconds the default ("PING" = "PING 60") so we have a decent behavior by default... And 180 should be the maximum value allowed...

Still I don't see the point of all this, any lobby client implemented correctly would consider that the communication with lobby server is down if no communication has been possible for minutes, and will try to reconnect. And when you reconnect, your previous ghost account is removed by uberserver and you just spawn a normal new connection, leaving all your channels, battles... And of course as I said you don't receive the messages sent to your ghost during this offline time.
Silentwings wrote:It's def useful to give users some significant time before disconnecting them - if someone briefly loses connection while ingame and then wants to reconnect; when they time out they may lose their slot in the host & then can't reconnect. The setting as it is now is perfect imo.
bibim wrote:
SPADS changelog wrote:0.11.2: allow disconnected players who didn't lose yet to reconnect to the battle lobby even if the battle is full, so that they can rejoin the game
You just have to use !unlockSpec and it will open the battle for you even if spec slots are full, then you can rejoin normally.
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: PING/PONG

Post by abma » 30 Sep 2013, 21:43

short off-topic:

maybe it should be noticed that NOBODY created a bug report about this ghosts. i had to investigate the code myself without knowing any details. i've created the bug report myself: https://github.com/spring/uberserver/issues/4
also the lobby protocol documentation was VERY unspecific about PING/PONG.
bibim wrote:It's not only bugged lobby clients hanging without closing the socket, but also players having system crashes, big network outages (frequent for wifi users) etc. I just tested the ghost "functionality" to see how much time someone can appear online without being actually connected, and my ghost lasted 17 minutes.
my ghost only appeared ~5 minutes. did you login to some channel or something? imo the fewer data the client receives, the longer he can "survive".

i'll try with with tcp-keepalive first as kicking out users sending no pings would break some clients.
0 x

dansan
Server Owner & Developer
Posts: 1192
Joined: 29 May 2010, 23:40

Re: PING/PONG

Post by dansan » 30 Sep 2013, 22:28

+1 bibim

Ghosted players are very annoying! Unfortunately you have to wait a lot in spring games already before the match starts... 1min should be enough for users to come back online if they had a disconnect. If waiting longer, then I think the the tolerance-barrier of the other players is hit.

Regarding lobbies: IMO implement a defined and documented behavior and notify lobby-devs that in x weeks the change will go online. Then it's their job to fix their clients.

Less waiting -> more fun :mrgreen:
0 x

User avatar
bibim
Lobby Developer
Posts: 901
Joined: 06 Dec 2007, 11:12

Re: PING/PONG

Post by bibim » 30 Sep 2013, 22:41

abma wrote:maybe it should be noticed that NOBODY created a bug report about this ghosts.
That's because few persons are really aware of this problem. When you don't know how it works you just think the players went afk.
abma wrote:also the lobby protocol documentation was VERY unspecific about PING/PONG.
Lobby Protocol Description wrote:Protocol rules

A client must maintain a constant connection with the server. That is if no data is to be transfered, the client must send PING commands to the server on regular intervals. This ensures that the server can properly detect any network timeouts/disconnects. The server will automatically disconnect a client if no data has been received from it within a certain time period. A client should send some data at least after half that interval passed.
The value of the interval is missing, but apart from that it clearly states that lobby clients must send data regularly to stay connected, doesn't it?
abma wrote:my ghost only appeared ~5 minutes. did you login to some channel or something? imo the fewer data the client receives, the longer he can "survive".
I just joined a battle. I guess it's directly related to the size of the socket sending buffer and the activity on the server yes.
abma wrote:i'll try with with tcp-keepalive first as kicking out users sending no pings would break some clients.
It would only break the clients which don't respect current protocol specification.
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: PING/PONG

Post by abma » 01 Oct 2013, 02:51

bibim wrote:The value of the interval is missing, but apart from that it clearly states that lobby clients must send data regularly to stay connected, doesn't it?
i've already added a suggestion how often a client should ping, not sure if the values make sense.
bibim wrote:I just joined a battle. I guess it's directly related to the size of the socket sending buffer and the activity on the server yes.
can you please try again? my test seems to fail, i'm mostly instandly disconnected... tcp-keepalive is now used, should disconnect after ~1 minute.
0 x

User avatar
aegis
Posts: 2456
Joined: 11 Jul 2007, 17:47

Re: PING/PONG

Post by aegis » 01 Oct 2013, 09:02

I think it would resolve both the player and host issue with battles if the server kicked any user from their battle after a reasonable idle time - when it would've normally kicked them from the server.

"Ghost" users should also have their clientstatus set to away, which could be done by the same timer.

A test was made to re-establish channels/chat/battles when a user was ghosted, but the lobbies at the time could not handle this well. Maybe reform discussions were had but nothing ever came of it.
0 x

User avatar
bibim
Lobby Developer
Posts: 901
Joined: 06 Dec 2007, 11:12

Re: PING/PONG

Post by bibim » 01 Oct 2013, 10:49

abma wrote:can you please try again? my test seems to fail, i'm mostly instandly disconnected... tcp-keepalive is now used, should disconnect after ~1 minute.
I just tested, my ghost lasted more than 15min again...
aegis wrote:I think it would resolve both the player and host issue with battles if the server kicked any user from their battle after a reasonable idle time - when it would've normally kicked them from the server.
This wouldn't prevent players trying to convince the ghost player to go back through PM chat etc... Also they wouldn't understand why he would leave whereas he wanted to play and was ringing people 1 minute before for instance.
aegis wrote:"Ghost" users should also have their clientstatus set to away, which could be done by the same timer.
This could have reduced the amount of wtf indeed. Although reusing a state which normally means "connected but not in front of computer" for a different meaning "experiencing connection difficulties" isn't the good way to implement it. This wouldn't prevent other players trying to send messages to the ghost thinking he will get them when he's back for instance.
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: PING/PONG

Post by abma » 03 Oct 2013, 06:32

bibim wrote:I just tested, my ghost lasted more than 15min again...
can you try again please? next "patch" applyed a few minutes ago. how do you test? didn't find an easy/clean way to check...

this commit imo should fix it: (first try was broken, as it enabled keepalive only on server-socket which seems to be fail)
https://github.com/spring/uberserver/co ... 42551d8b8b

if it works, it should kick in ~1 minute after disconnecting.
0 x

User avatar
bibim
Lobby Developer
Posts: 901
Joined: 06 Dec 2007, 11:12

Re: PING/PONG

Post by bibim » 03 Oct 2013, 22:28

No it still doesn't work (ghost lasted > 15 min again).
I just test by disabling the network while lobby client is running.
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: PING/PONG

Post by abma » 04 Oct 2013, 09:11

bibim wrote:No it still doesn't work (ghost lasted > 15 min again).
I just test by disabling the network while lobby client is running.
meh, thanks. an other approach: now the server disconnects clients sending now data for >=1 minute. this only happens when the client send one PING. dead clients should be disconnected in at max ~90 seconds. clients not using PING won't be disconnected by this.

see https://github.com/spring/uberserver/co ... 3c5160b2f5

also this is close to what the protocol said how it should be implemented.

can you test again? i've only one computer here, so disabling network isn't a good test as i don't see the result.
0 x

User avatar
bibim
Lobby Developer
Posts: 901
Joined: 06 Dec 2007, 11:12

Re: PING/PONG

Post by bibim » 04 Oct 2013, 10:49

abma wrote:this only happens when the client send one PING. clients not using PING won't be disconnected by this.
So this means we will still have ghost problems due to broken lobby clients...
abma wrote:see https://github.com/spring/uberserver/co ... 3c5160b2f5

also this is close to what the protocol said how it should be implemented.
This is basically how it should be implemented, except from the clients-not-sending-ping exception.
abma wrote:can you test again? i've only one computer here, so disabling network isn't a good test as i don't see the result.
Did you try to close your lobby client while your network is disabled, then re-enable your network, restart your lobby client and reconnect (with a different account of course)?
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: PING/PONG

Post by abma » 04 Oct 2013, 11:33

bibim wrote:This is basically how it should be implemented, except from the clients-not-sending-ping exception.
i dislike possible breaking current clients, we need some "migration" time / maybe a compatibility flag and a date when it is enforced. when adding the compatibility flag i would like to add a server side ping, this doesn't require the client to constantly spam pings to the server.

also afaik all "major" (Springlobby / ZK-Lobby / SpringWebLobby / Springie) clients already send PING. For IRC/Chat-Clients it shouldn't matter as they don't join battles.

but true, at some point all dead clients should be detected after a short time.
0 x

User avatar
bibim
Lobby Developer
Posts: 901
Joined: 06 Dec 2007, 11:12

Re: PING/PONG

Post by bibim » 04 Oct 2013, 12:48

abma wrote:i dislike possible breaking current clients, we need some "migration" time / maybe a compatibility flag and a date when it is enforced.
You want to add a compatibility flag to let clients say they are compatible with the protocol which existed and was specified even before I added the compatibility flag system into it? :shock:
abma wrote:also afaik all "major" lobbies already send PING.
Exactly, and that's the reason why it should be enforced now as it used to be, to avoid more broken clients appearing and inducing ghost problem.
abma wrote:when adding the compatibility flag i would like to add a server side ping, this doesn't require the client to constantly spam pings to the server.
For a given timeout value, the amount of PING/PONG commands required to keep a connection alive doesn't change, whatever the peer sending the PING.

If client and server aren't aware of each other timeout values, there could be extra PING commands sent, whatever the peer sending the PING initially: if client is sending PING and thinks server uses a timeout of 30s whereas server uses a timeout of 60s, the client will send unneeded PING. If server is sending PING every 60s but client uses a timeout of 30s, the client will send unneeded PING (or any basic command with response such as GETINGAMETIME) to check network connection => unneeded traffic again.

So if a client spams PING commands, it's not a problem of protocol design, it's a problem of implementation at client side, not consistent with server's timeout value. It's not related to the PING command itself.

If lobby traffic is really problematic currently, you could try to improve lobby command spam detection. It's easier to force lobby clients to adjust their timeouts than force them to implement server-side ping AND adjust their timeouts...
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: PING/PONG

Post by abma » 04 Oct 2013, 13:49

bibim wrote:If client and server aren't aware of each other timeout values, there could be extra PING commands sent, whatever the peer sending the PING initially: if client is sending PING and thinks server uses a timeout of 30s whereas server uses a timeout of 60s, the client will send unneeded PING. If server is sending PING every 60s but client uses a timeout of 30s, the client will send unneeded PING (or any basic command with response such as GETINGAMETIME) to check network connection => unneeded traffic again.
thats why i want a server initiated ping:

the idea for server side ping is:
- if no data received from client longer than x, server sends ping
- if x+y time no data/PONG received from client: disconnect "dead" client

client doesn't have to care about values, it just have to respond with PONG to a PING from server. also, ping is only send when needed.

benefit: client doesn't need to care about timeout values & timeout values can be changed server-side when needed, no useless PING's send.
Last edited by abma on 04 Oct 2013, 14:06, edited 1 time in total.
0 x

Post Reply

Return to “Lobby Clients & Server”