Uberserver. Unicode messages

[lamer]

Looks like Official server supports symbols other then ascii (cyrillic). But uberserver's git version (master branch at the moment of writing) does not.
It appears that some .decode('utf-8') / .encode('utf-8') can fix this issue.
Is unicode support on todo list or did i just miss something and its already there?

update: Seems like next code can change default encoding from ascii to utf-8

Code: Select all

>>> import sys
>>> reload(sys)  # to enable `setdefaultencoding` again
<module 'sys' (built-in)>
>>> sys.setdefaultencoding("UTF-8")
>>> sys.getdefaultencoding()
'UTF-8'

but this page says it's bad practice

[abma]

what does "support" mean?

i can write some unicode char in client one and i receive it correctly in client two in a channel...

is your client broken? uberserver doesn't/shouldn't care about encoding, it just forwards the text.

uberserver master branch = current running code at springrts.com

[lamer]

Archlinux, springlobby 0.180 from community repo (also did try latest git).
Message to send: "на дворе трава, на траве дрова"
1) within regular chat (uberserver traceback):

Code: Select all

2014-03-09T20:14:39 Handler 0: Successfully logged in user <lamer> on session 1 user.
2014-03-09T20:15:06 ------------------------------------------------------------
2014-03-09T20:15:06 Traceback (most recent call last):
2014-03-09T20:15:06   File "/home/lamer/games/spring/uberserver/Dispatcher.py", line 39, in callback
2014-03-09T20:15:06     self.socketmap[s].Handle(data)
2014-03-09T20:15:06   File "/home/lamer/games/spring/uberserver/Client.py", line 147, in Handle
2014-03-09T20:15:06     self._protocol._handle(self,cmd)
2014-03-09T20:15:06   File "/home/lamer/games/spring/uberserver/protocol/Protocol.py", line 257, in _handle
2014-03-09T20:15:06     function(*([client]+arguments))
2014-03-09T20:15:06   File "/home/lamer/games/spring/uberserver/protocol/Protocol.py", line 855, in in_SAYPRIVATE
2014-03-09T20:15:06     self._root.usernames[user].Send('SAIDPRIVATE %s %s'%(client.username, msg))
2014-03-09T20:15:06 UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)
2014-03-09T20:15:06 
2014-03-09T20:15:06 ------------------------------------------------------------

2) within battleroom chat:

Code: Select all

2014-03-09T20:15:19 ------------------------------------------------------------
2014-03-09T20:15:19 Traceback (most recent call last):
2014-03-09T20:15:19   File "/home/lamer/games/spring/uberserver/Dispatcher.py", line 39, in callback
2014-03-09T20:15:19     self.socketmap[s].Handle(data)
2014-03-09T20:15:19   File "/home/lamer/games/spring/uberserver/Client.py", line 147, in Handle
2014-03-09T20:15:19     self._protocol._handle(self,cmd)
2014-03-09T20:15:19   File "/home/lamer/games/spring/uberserver/protocol/Protocol.py", line 257, in _handle
2014-03-09T20:15:19     function(*([client]+arguments))
2014-03-09T20:15:19   File "/home/lamer/games/spring/uberserver/protocol/Protocol.py", line 1247, in in_SAYBATTLE
2014-03-09T20:15:19     self.broadcast_SendBattle(battle, 'SAIDBATTLE %s %s' % (user, msg))
2014-03-09T20:15:19 UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)
2014-03-09T20:15:19 
2014-03-09T20:15:19 ------------------------------------------------------------

Maybe it's python's environment issue?

Code: Select all

Python 2.7.6 (default, Feb 26 2014, 12:07:17) 
[GCC 4.8.2 20140206 (prerelease)] on linux2

Will try to test it with ubuntu+python2.7.3

[abma]

no clue if this is related, but it could: http://stackoverflow.com/questions/1446 ... 2-or-ucs-4

on server:

Code: Select all

$ python
>>> import sys
>>> print sys.maxunicode
1114111

maybe my springlobby sends utf8 and yours unicode? does it work on the "official server"?


edit:

Code: Select all

>>> sys.getdefaultencoding()
'ascii'
>>> sys.getfilesystemencoding()
'UTF-8'

[lamer]

same error with ubuntu12.04+python2.7.3
And default encoding is

Code: Select all

>>> import sys
>>> sys.getdefaultencoding()
'ascii'
>>> sys.maxunicode
1114111
>>> sys.getfilesystemencoding()
'UTF-8'

Is there any additional steps i have to do (except git clone <uberserver>)? Change of LANG=en_US.UTF-8, LC_ALL=en_US.UTF-8 environment variables doesn't have any effect

But everything works after next ugly hack:

Code: Select all

import sys
reload(sys)  # to enable `setdefaultencoding` again
sys.setdefaultencoding("UTF-8")

[abma]

on lobby.springrts.com:

Code: Select all

$ locale 
LANG=en_US.UTF-8
LANGUAGE=en_US
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

its ubuntu 12.04.4 & python 2.7.3, too.

how do you start the server? also it seems your traceback doesn't match current uberservers code. is it up to date? it doesn't look so.

line 855 isn't "in_SAYPRIVATE": https://github.com/spring/uberserver/bl ... ol.py#L855

[abma]

weird, locally i can reproduce the problem, too. no clue why it works on the server.

[lamer]

maybe my springlobby sends utf8 and yours unicode? does it work on the "official server"?

Strange but yes, my version of springlobby works with official server (unicode i mean)

line 855 isn't "in_SAYPRIVATE": https://github.com/spring/uberserver/bl ... ol.py#L855

Actually it is. Look at https://github.com/spring/uberserver/bl ... ol.py#L843
Just did "git clone https://github.com/spring/uberserver.git" a few minutes ago before testing things (for ubuntu).

how do you start the server?

from console and uberserver's directory
$ ./server.py
or
$ python server.py
Will test locale (i have differences with your settings)

[abma]

do you use sqlite for storing as db? i guess thats the problem for some reason, but no clue atm, need time to investigate.

[lamer]

do you use sqlite for storing as db? i guess thats the problem for some reason, but no clue atm, need time to investigate.

yes, sqlite. (i mean i didn't install any mysql or other, just use default settings and thats sqlite)
And thank you for your help!

[abma]

ok, then its very likely sqlite. i guess it returns asscii while mysql returns unicode/utf8. username is taken from db and concated in this line with the message.

[lamer]

And for the moment i'm out of ideas. Copying locale settings didn't help (but maybe it's important when db created).
Changing
self.engine = sqlalchemy.create_engine(self.sqlurl)
into
self.engine = sqlalchemy.create_engine(self.sqlurl, encoding='utf-8')
inside DataHandler.py and recreating db didn't help either.
Will test MySql :) ... later.
Thanks again

edit (a few thoughts to cofirm your idea about MySql):
With sqlite
print type(user), type(msg)
<type 'unicode'> <type 'str'>
And i guess MySql result from official server configuration
type(user) = <type 'str'> and thats why it doesnt need to decode/encode anything. If it's true then it's basically a lack of "unicode support" :) Sqlalchemy uses only unicode for sqlite (AFAIK from sqlalchemy docs)

[abma]

this should fix it:

https://github.com/spring/uberserver/co ... 5408774282

just run "git pull"

[lamer]

Yes, it fixed the issue.
Thanks!