Offline chat support

Offline chat support

For the discussion of infrastructure improvements and changes.

Moderator: Moderators

gajop
Moderator
Posts: 3023
Joined: 05 Aug 2009, 20:42

Offline chat support

Post by gajop » 16 Jan 2015, 05:07

Since Nightwatch is gone we should add support for offline chat to the uberserver.
We can also extend it to allow for offline private messages which is not supported by Nightwatch afaik.
The uberserver issue is listed here: https://github.com/spring/uberserver/issues/132

I propose adding a few new commands and extending old ones, as listed:

I'd like to add the following protocol commands:
1) >SUBSCRIBE(chanName)
2) >UNSUBSCRIBE(chanName)
3) >LISTSUBSCRIPTIONS()
4) <STARTLISTSUBSCRIPTION()
5) <LISTSUBSCRIPTION(chanName)
6) <ENDLISTSUBSCRIPTION()

Extend SAID, SAIDDATA, SAIDDATAPRIVATE, SAIDPRIVATE, SAIDEX, SAIDPRIVATEEX commands to contain a datetime field as well. Do this by using the named arguments.

Alternatively we could make new SAID commands, e.g. OFFLINESAID and so on, but I think this approach is cleaner.
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: Offline chat support

Post by abma » 16 Jan 2015, 05:30

when to send stored channel messages? only on channel join or always? or on request? (like send all available offline messages since time x)

also channel messages should be deleted after some specific time i.e. at max for 3 days or so?!

for private messages it seems clear: instantly send them when user joins battle server.

hm, timeformat should be "unix time format" ?! (as its used already in CHANNELTOPIC)
0 x

gajop
Moderator
Posts: 3023
Joined: 05 Aug 2009, 20:42

Re: Offline chat support

Post by gajop » 16 Jan 2015, 05:47

abma wrote:when to send stored channel messages? only on channel join or always? or on request? (like send all available offline messages since time x)
I'd say on channel join. While it's possible, I think it's weird to receive messages if you are not in a channel. Also "on request" just bloats the protocol.
abma wrote: also channel messages should be deleted after some specific time i.e. at max for 3 days or so?!
Agreed, but not part of the lobby protocol. Imo, make it configurable and a 3day default is OK.
abma wrote: for private messages it seems clear: instantly send them when user joins battle server.
Yes, and keep them forever (or a really really long period, i.e. 1year~) and delete only when the user comes online.
abma wrote: hm, timeformat should be "unix time format" ?! (as its used already in CHANNELTOPIC)
I guess that's fine. Any potential problems with it?
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: Offline chat support

Post by abma » 16 Jan 2015, 06:10

gajop wrote:I guess that's fine. Any potential problems with it?
hmm, timezone info maybe?
0 x

gajop
Moderator
Posts: 3023
Joined: 05 Aug 2009, 20:42

Re: Offline chat support

Post by gajop » 16 Jan 2015, 06:16

I thought it was in UTC (GMT) and one can apply the timezone after?
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: Offline chat support

Post by abma » 16 Jan 2015, 08:17

good point, so i see no problems :)
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: Offline chat support

Post by abma » 16 Jan 2015, 23:07

small addition: ammount of messages should imo be limited to 300 and 3 days / what ever limit is reached first.


(300/3 = magic internal numbers)
0 x

User avatar
Anarchid
Posts: 1380
Joined: 30 Nov 2008, 04:31

Re: Offline chat support

Post by Anarchid » 17 Jan 2015, 00:41

Yes, and keep them forever (or a really really long period, i.e. 1year~) and delete only when the user comes online.
This sounds like you want to keep a message for each subscriber (inferring from the "the").

E.g. if 200 people subscribe to #sy and sign off and then there are 200 messages, you now have 40000 rows in database?

(suggestion is to use a table for all messages, a table for subscriptions, and delete when all subscribers logged in after message's timestamp, so 200 message rows + 200 subscriber rows is all you expend)
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Offline chat support

Post by Licho » 17 Jan 2015, 02:17

I would recommend implementing these non critical features using separate bots so that main server isn't clogged by this.
If it works that way some "reservername" bot can be used for that (even if implementation optionally resides withing uberserver itself, like chanserv).

And btw, NW does support offline messages..
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: Offline chat support

Post by abma » 17 Jan 2015, 04:40

Anarchid wrote:This sounds like you want to keep a message for each subscriber (inferring from the "the").
you clearly didn't read the whole. this point was about private messages.

Licho wrote:I would recommend implementing these non critical features using separate bots so that main server isn't clogged by this.
If it works that way some "reservername" bot can be used for that (even if implementation optionally resides withing uberserver itself, like chanserv).
as nightwatch showed, its a very useful feature. but nightwatchs implementation had some drawbacks:
- all data was sent to nightwatch and replaying it required it to be send from nightwatch over the lobby-server to the user
- implementating it at the lobbyserver level is standardization
- difference between having it added to ChanServ or directly to the protocol imo only about efficiency/overhead. splitting protocol.py into several smaller files isn't high priority and can be still done later by i.e. splitting sending/receiving part or by access level, or maybe add sth. like plugins for the server with some hooks for commands (whatever is best/needed & useful)
+ things i forgot
And btw, NW does support offline messages..
nighwatch isn't available any more at the lobby server as it was. its a good time to move this feature to the core-server and remove the proxy overhead.
0 x

gajop
Moderator
Posts: 3023
Joined: 05 Aug 2009, 20:42

Re: Offline chat support

Post by gajop » 17 Jan 2015, 05:15

abma wrote:small addition: ammount of messages should imo be limited to 300 and 3 days / what ever limit is reached first.


(300/3 = magic internal numbers)
Hmm, it seems too little imo. There might well be more than 300 messages per day in a channel like #sy or #moddev, and you wouldn't be able to follow the discussion in that case.
Also, 300 messages isn't that much really, this wouldn't add to much more than a couple of megabytes for all the channels in total.
But yeh, having _some_ configurable magic number would make sense.

@Licho
It doesn't matter if it's a critical feature or not. The point is that it's needed, it's easy to agree upon the specification and using a bot would just provide more overhead for the central server.
0 x

User avatar
Licho
Zero-K Developer
Posts: 3803
Joined: 19 May 2006, 19:13

Re: Offline chat support

Post by Licho » 17 Jan 2015, 09:04

If it's in a separate bot, protocol could be same for ZK and spring instances and I would also implement it (improve to match specification) and run it.

It would mean lobbies that want to use both servers wont need duplicate implementations.

And btw imo anything that does mysql should be kept out of the lobby server...
0 x

gajop
Moderator
Posts: 3023
Joined: 05 Aug 2009, 20:42

Re: Offline chat support

Post by gajop » 17 Jan 2015, 12:24

Licho wrote:If it's in a separate bot, protocol could be same for ZK and spring instances and I would also implement it (improve to match specification) and run it.

It would mean lobbies that want to use both servers wont need duplicate implementations.
We have had this separate bot for years and the specification is not available afaik. There's also little guarantee that the bot will be compatible in the future. Lastly, the bot probably uses some third party
DB or service which makes it impossible for people to deploy on their own uberserver instances. (There should be as little reliance on third party services for the lobby protocol as possible).

Why can't ZK use an uberserver implementation and rely on lobby bots or whatever in the background?
Licho wrote: And btw imo anything that does mysql should be kept out of the lobby server...
This is protocol discussion (mostly aimed at lobby clients), and not uberserver (protocol vs implementation).
The two uberservers have already diverged with regards to the implementation. Official uber needs SQL to store user, channel and other information, but this is something that is not and should not be specified by the protocol.
0 x

User avatar
jK
Spring Developer
Posts: 2299
Joined: 28 Jun 2007, 07:30

Re: Offline chat support

Post by jK » 17 Jan 2015, 13:19

Neither is SQL itself any problem at all, MMORPGs with thousands of players and x10k of actors use it without any problems. Yes SQL is MUCH better than handcoded file db crap.

The problem is the sql lib of python and that it neither supports threading nor async requests.

So I extremely question if your uberserver rewrite solves anything or if it just does stuff differently wrong cause you dislike linux&python&sql and want to redo it in windows&c#.
0 x

gajop
Moderator
Posts: 3023
Joined: 05 Aug 2009, 20:42

Re: Offline chat support

Post by gajop » 28 Jan 2015, 11:38

I think we can ignore Licho's comments as he isn't keeping the uberserver protocol anyway and writing his own thing: http://zero-k.info/Forum/Thread/6692?page=1#116897 .

I would like confirm that people (lobby devs) are OK with this:
gajop wrote: Extend SAID, SAIDDATA, SAIDDATAPRIVATE, SAIDPRIVATE, SAIDEX, SAIDPRIVATEEX commands to contain a datetime field as well. Do this by using the named arguments.
An example of such an offline command would be:
SAID sy gajop Hello World<TAB>timestamp=1422441316

The timestamp will always be in the end of the commands. It might be abbreviated by a "ts" instead. (This is not decided yet).

The alternative would be this:
gajop wrote: Alternatively we could make new SAID commands, e.g. OFFLINESAID and so on, but I think this approach is cleaner.
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: Offline chat support

Post by abma » 28 Jan 2015, 11:53

gajop wrote: I would like confirm that people (lobby devs) are OK with this:
gajop wrote: Extend SAID, SAIDDATA, SAIDDATAPRIVATE, SAIDPRIVATE, SAIDEX, SAIDPRIVATEEX commands to contain a datetime field as well. Do this by using the named arguments.
An example of such an offline command would be:
SAID sy gajop Hello World<TAB>timestamp=1422441316

The timestamp will always be in the end of the commands. It might be abbreviated by a "ts" instead. (This is not decided yet).

The alternative would be this:
gajop wrote: Alternatively we could make new SAID commands, e.g. OFFLINESAID and so on, but I think this approach is cleaner.
imo all commands should be modified, into sth. like this:

Code: Select all

SAID chan=sy user=gajop timestamp=1422441316  Hello World
adding timestamp at the end makes it a requirement to remove all tabs from the message. this would allow to drop some commands by adding a flag like data=yes

i'm not sure how to distingish between the parameters and message. either we have fixed parameters and allow tab in messages or all tabs have to be removed in messages / replaced with \t ?!

fyi: *DATA* commands aren't implemented.
0 x

gajop
Moderator
Posts: 3023
Joined: 05 Aug 2009, 20:42

Re: Offline chat support

Post by gajop » 28 Jan 2015, 12:00

abma wrote: imo all commands should be modified, into sth. like this:

Code: Select all

SAID chan=sy user=gajop timestamp=1422441316  Hello World
Wouldn't it be better to have this instead?

Code: Select all

SAID chan=sy user=gajop timestamp=1422441316 msg=Hello World
Also wouldn't this be an issue with backwards compatibility? Should we support this by a compatibility flag or?
No comment on the "data" part. I don't think it's currently important to support data transmission.
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: Offline chat support

Post by abma » 28 Jan 2015, 12:03

gajop wrote:Also wouldn't this be an issue with backwards compatibility? Should we support this by a compatibility flag or?
should be no problem with backwards compatibility. for non-supporting lobbies having the timestamp in the message would be weird -> compatibility flag is a must
SAID chan=sy user=gajop timestamp=1422441316 msg=Hello World
yep, this makes more sense.
0 x

abma
Spring Developer
Posts: 3560
Joined: 01 Jun 2009, 00:08

Re: Offline chat support

Post by abma » 28 Jan 2015, 12:14

still: how to escape a real tab in msg? replace with space? replace with \t ?

protocol spec says this:
Make sure to replace all TABs in your sentences with spaces (2 or 8 usually)! If you do not, your command is invalid, and will not be read correctly by the server.
but afaik this will break OTR. (which atm theoretically is possible through the irc-bridge)
0 x

gajop
Moderator
Posts: 3023
Joined: 05 Aug 2009, 20:42

Re: Offline chat support

Post by gajop » 28 Jan 2015, 12:21

This is something the client should do. Sending tabs in a message is not supported.
Btw, are we sure it's best to use unixtime? Conversion seems annoying: http://www.epochconverter.com/
Is there no better date format than that, like this: http://en.wikipedia.org/wiki/ISO_8601 ?
0 x

Post Reply

Return to “Infrastructure Development”