Offline chat support

[gajop]

Since Nightwatch is gone we should add support for offline chat to the uberserver.
We can also extend it to allow for offline private messages which is not supported by Nightwatch afaik.
The uberserver issue is listed here: https://github.com/spring/uberserver/issues/132

I propose adding a few new commands and extending old ones, as listed:

I'd like to add the following protocol commands:
1) >SUBSCRIBE(chanName)
2) >UNSUBSCRIBE(chanName)
3) >LISTSUBSCRIPTIONS()
4) <STARTLISTSUBSCRIPTION()
5) <LISTSUBSCRIPTION(chanName)
6) <ENDLISTSUBSCRIPTION()

Extend SAID, SAIDDATA, SAIDDATAPRIVATE, SAIDPRIVATE, SAIDEX, SAIDPRIVATEEX commands to contain a datetime field as well. Do this by using the named arguments.

Alternatively we could make new SAID commands, e.g. OFFLINESAID and so on, but I think this approach is cleaner.

[abma]

when to send stored channel messages? only on channel join or always? or on request? (like send all available offline messages since time x)

also channel messages should be deleted after some specific time i.e. at max for 3 days or so?!

for private messages it seems clear: instantly send them when user joins battle server.

hm, timeformat should be "unix time format" ?! (as its used already in CHANNELTOPIC)

[gajop]

when to send stored channel messages? only on channel join or always? or on request? (like send all available offline messages since time x)

I'd say on channel join. While it's possible, I think it's weird to receive messages if you are not in a channel. Also "on request" just bloats the protocol.

also channel messages should be deleted after some specific time i.e. at max for 3 days or so?!

Agreed, but not part of the lobby protocol. Imo, make it configurable and a 3day default is OK.

for private messages it seems clear: instantly send them when user joins battle server.

Yes, and keep them forever (or a really really long period, i.e. 1year~) and delete only when the user comes online.

hm, timeformat should be "unix time format" ?! (as its used already in CHANNELTOPIC)

I guess that's fine. Any potential problems with it?

[abma]

I guess that's fine. Any potential problems with it?

hmm, timezone info maybe?

[gajop]

I thought it was in UTC (GMT) and one can apply the timezone after?

[abma]

good point, so i see no problems :)

[abma]

small addition: ammount of messages should imo be limited to 300 and 3 days / what ever limit is reached first.


(300/3 = magic internal numbers)

[Anarchid]

Yes, and keep them forever (or a really really long period, i.e. 1year~) and delete only when the user comes online.

This sounds like you want to keep a message for each subscriber (inferring from the "the").

E.g. if 200 people subscribe to #sy and sign off and then there are 200 messages, you now have 40000 rows in database?

(suggestion is to use a table for all messages, a table for subscriptions, and delete when all subscribers logged in after message's timestamp, so 200 message rows + 200 subscriber rows is all you expend)

[Licho]

I would recommend implementing these non critical features using separate bots so that main server isn't clogged by this.
If it works that way some "reservername" bot can be used for that (even if implementation optionally resides withing uberserver itself, like chanserv).

And btw, NW does support offline messages..

[abma]

This sounds like you want to keep a message for each subscriber (inferring from the "the").

you clearly didn't read the whole. this point was about private messages.

I would recommend implementing these non critical features using separate bots so that main server isn't clogged by this.
If it works that way some "reservername" bot can be used for that (even if implementation optionally resides withing uberserver itself, like chanserv).

as nightwatch showed, its a very useful feature. but nightwatchs implementation had some drawbacks:
- all data was sent to nightwatch and replaying it required it to be send from nightwatch over the lobby-server to the user
- implementating it at the lobbyserver level is standardization
- difference between having it added to ChanServ or directly to the protocol imo only about efficiency/overhead. splitting protocol.py into several smaller files isn't high priority and can be still done later by i.e. splitting sending/receiving part or by access level, or maybe add sth. like plugins for the server with some hooks for commands (whatever is best/needed & useful)
+ things i forgot

And btw, NW does support offline messages..

nighwatch isn't available any more at the lobby server as it was. its a good time to move this feature to the core-server and remove the proxy overhead.

[gajop]

small addition: ammount of messages should imo be limited to 300 and 3 days / what ever limit is reached first.


(300/3 = magic internal numbers)

Hmm, it seems too little imo. There might well be more than 300 messages per day in a channel like #sy or #moddev, and you wouldn't be able to follow the discussion in that case.
Also, 300 messages isn't that much really, this wouldn't add to much more than a couple of megabytes for all the channels in total.
But yeh, having _some_ configurable magic number would make sense.

@Licho
It doesn't matter if it's a critical feature or not. The point is that it's needed, it's easy to agree upon the specification and using a bot would just provide more overhead for the central server.

[Licho]

If it's in a separate bot, protocol could be same for ZK and spring instances and I would also implement it (improve to match specification) and run it.

It would mean lobbies that want to use both servers wont need duplicate implementations.

And btw imo anything that does mysql should be kept out of the lobby server...

[gajop]

If it's in a separate bot, protocol could be same for ZK and spring instances and I would also implement it (improve to match specification) and run it.

It would mean lobbies that want to use both servers wont need duplicate implementations.

We have had this separate bot for years and the specification is not available afaik. There's also little guarantee that the bot will be compatible in the future. Lastly, the bot probably uses some third party
DB or service which makes it impossible for people to deploy on their own uberserver instances. (There should be as little reliance on third party services for the lobby protocol as possible).

Why can't ZK use an uberserver implementation and rely on lobby bots or whatever in the background?

And btw imo anything that does mysql should be kept out of the lobby server...

This is protocol discussion (mostly aimed at lobby clients), and not uberserver (protocol vs implementation).
The two uberservers have already diverged with regards to the implementation. Official uber needs SQL to store user, channel and other information, but this is something that is not and should not be specified by the protocol.

[jK]

Neither is SQL itself any problem at all, MMORPGs with thousands of players and x10k of actors use it without any problems. Yes SQL is MUCH better than handcoded file db crap.

The problem is the sql lib of python and that it neither supports threading nor async requests.

So I extremely question if your uberserver rewrite solves anything or if it just does stuff differently wrong cause you dislike linux&python&sql and want to redo it in windows&c#.

[gajop]

I think we can ignore Licho's comments as he isn't keeping the uberserver protocol anyway and writing his own thing: http://zero-k.info/Forum/Thread/6692?page=1#116897 .

I would like confirm that people (lobby devs) are OK with this:

Extend SAID, SAIDDATA, SAIDDATAPRIVATE, SAIDPRIVATE, SAIDEX, SAIDPRIVATEEX commands to contain a datetime field as well. Do this by using the named arguments.

An example of such an offline command would be:
SAID sy gajop Hello World<TAB>timestamp=1422441316

The timestamp will always be in the end of the commands. It might be abbreviated by a "ts" instead. (This is not decided yet).

The alternative would be this:

Alternatively we could make new SAID commands, e.g. OFFLINESAID and so on, but I think this approach is cleaner.

[abma]

I would like confirm that people (lobby devs) are OK with this:

Extend SAID, SAIDDATA, SAIDDATAPRIVATE, SAIDPRIVATE, SAIDEX, SAIDPRIVATEEX commands to contain a datetime field as well. Do this by using the named arguments.

An example of such an offline command would be:
SAID sy gajop Hello World<TAB>timestamp=1422441316

The timestamp will always be in the end of the commands. It might be abbreviated by a "ts" instead. (This is not decided yet).

The alternative would be this:

Alternatively we could make new SAID commands, e.g. OFFLINESAID and so on, but I think this approach is cleaner.

imo all commands should be modified, into sth. like this:

Code: Select all

SAID chan=sy user=gajop timestamp=1422441316  Hello World

adding timestamp at the end makes it a requirement to remove all tabs from the message. this would allow to drop some commands by adding a flag like data=yes

i'm not sure how to distingish between the parameters and message. either we have fixed parameters and allow tab in messages or all tabs have to be removed in messages / replaced with \t ?!

fyi: *DATA* commands aren't implemented.

[gajop]

imo all commands should be modified, into sth. like this:

Code: Select all

SAID chan=sy user=gajop timestamp=1422441316  Hello World

Wouldn't it be better to have this instead?

Code: Select all

SAID chan=sy user=gajop timestamp=1422441316 msg=Hello World

Also wouldn't this be an issue with backwards compatibility? Should we support this by a compatibility flag or?
No comment on the "data" part. I don't think it's currently important to support data transmission.

[abma]

Also wouldn't this be an issue with backwards compatibility? Should we support this by a compatibility flag or?

should be no problem with backwards compatibility. for non-supporting lobbies having the timestamp in the message would be weird -> compatibility flag is a must

SAID chan=sy user=gajop timestamp=1422441316 msg=Hello World

yep, this makes more sense.

[abma]

still: how to escape a real tab in msg? replace with space? replace with \t ?

protocol spec says this:

Make sure to replace all TABs in your sentences with spaces (2 or 8 usually)! If you do not, your command is invalid, and will not be read correctly by the server.

but afaik this will break OTR. (which atm theoretically is possible through the irc-bridge)

[gajop]

This is something the client should do. Sending tabs in a message is not supported.
Btw, are we sure it's best to use unixtime? Conversion seems annoying: http://www.epochconverter.com/
Is there no better date format than that, like this: http://en.wikipedia.org/wiki/ISO_8601 ?