Uberserver. Unicode messages

Uberserver. Unicode messages

Discuss development of lobby clients, server, autohosts and auto-download software.

Moderators: Moderators, Lobby Developers

Post Reply
lamer
Posts: 146
Joined: 08 Mar 2014, 23:13

Uberserver. Unicode messages

Post by lamer » 08 Mar 2014, 23:57

Looks like Official server supports symbols other then ascii (cyrillic). But uberserver's git version (master branch at the moment of writing) does not.
It appears that some .decode('utf-8') / .encode('utf-8') can fix this issue.
Is unicode support on todo list or did i just miss something and its already there?

update: Seems like next code can change default encoding from ascii to utf-8

Code: Select all

>>> import sys
>>> reload(sys)  # to enable `setdefaultencoding` again
<module 'sys' (built-in)>
>>> sys.setdefaultencoding("UTF-8")
>>> sys.getdefaultencoding()
'UTF-8'
but this page says it's bad practice
0 x

abma
Spring Developer
Posts: 3507
Joined: 01 Jun 2009, 00:08

Re: Uberserver. Unicode messages

Post by abma » 09 Mar 2014, 14:13

what does "support" mean?

i can write some unicode char in client one and i receive it correctly in client two in a channel...

is your client broken? uberserver doesn't/shouldn't care about encoding, it just forwards the text.

uberserver master branch = current running code at springrts.com
0 x

lamer
Posts: 146
Joined: 08 Mar 2014, 23:13

Re: Uberserver. Unicode messages

Post by lamer » 09 Mar 2014, 19:33

Archlinux, springlobby 0.180 from community repo (also did try latest git).
Message to send: "на дворе трава, на траве дрова"
1) within regular chat (uberserver traceback):

Code: Select all

2014-03-09T20:14:39 Handler 0: Successfully logged in user <lamer> on session 1 user.
2014-03-09T20:15:06 ------------------------------------------------------------
2014-03-09T20:15:06 Traceback (most recent call last):
2014-03-09T20:15:06   File "/home/lamer/games/spring/uberserver/Dispatcher.py", line 39, in callback
2014-03-09T20:15:06     self.socketmap[s].Handle(data)
2014-03-09T20:15:06   File "/home/lamer/games/spring/uberserver/Client.py", line 147, in Handle
2014-03-09T20:15:06     self._protocol._handle(self,cmd)
2014-03-09T20:15:06   File "/home/lamer/games/spring/uberserver/protocol/Protocol.py", line 257, in _handle
2014-03-09T20:15:06     function(*([client]+arguments))
2014-03-09T20:15:06   File "/home/lamer/games/spring/uberserver/protocol/Protocol.py", line 855, in in_SAYPRIVATE
2014-03-09T20:15:06     self._root.usernames[user].Send('SAIDPRIVATE %s %s'%(client.username, msg))
2014-03-09T20:15:06 UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)
2014-03-09T20:15:06 
2014-03-09T20:15:06 ------------------------------------------------------------
2) within battleroom chat:

Code: Select all

2014-03-09T20:15:19 ------------------------------------------------------------
2014-03-09T20:15:19 Traceback (most recent call last):
2014-03-09T20:15:19   File "/home/lamer/games/spring/uberserver/Dispatcher.py", line 39, in callback
2014-03-09T20:15:19     self.socketmap[s].Handle(data)
2014-03-09T20:15:19   File "/home/lamer/games/spring/uberserver/Client.py", line 147, in Handle
2014-03-09T20:15:19     self._protocol._handle(self,cmd)
2014-03-09T20:15:19   File "/home/lamer/games/spring/uberserver/protocol/Protocol.py", line 257, in _handle
2014-03-09T20:15:19     function(*([client]+arguments))
2014-03-09T20:15:19   File "/home/lamer/games/spring/uberserver/protocol/Protocol.py", line 1247, in in_SAYBATTLE
2014-03-09T20:15:19     self.broadcast_SendBattle(battle, 'SAIDBATTLE %s %s' % (user, msg))
2014-03-09T20:15:19 UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)
2014-03-09T20:15:19 
2014-03-09T20:15:19 ------------------------------------------------------------
Maybe it's python's environment issue?

Code: Select all

Python 2.7.6 (default, Feb 26 2014, 12:07:17) 
[GCC 4.8.2 20140206 (prerelease)] on linux2
Will try to test it with ubuntu+python2.7.3
0 x

abma
Spring Developer
Posts: 3507
Joined: 01 Jun 2009, 00:08

Re: Uberserver. Unicode messages

Post by abma » 09 Mar 2014, 20:02

no clue if this is related, but it could: http://stackoverflow.com/questions/1446 ... 2-or-ucs-4

on server:

Code: Select all

$ python
>>> import sys
>>> print sys.maxunicode
1114111
maybe my springlobby sends utf8 and yours unicode? does it work on the "official server"?


edit:

Code: Select all

>>> sys.getdefaultencoding()
'ascii'
>>> sys.getfilesystemencoding()
'UTF-8'
0 x

lamer
Posts: 146
Joined: 08 Mar 2014, 23:13

Re: Uberserver. Unicode messages

Post by lamer » 09 Mar 2014, 20:12

same error with ubuntu12.04+python2.7.3
And default encoding is

Code: Select all

>>> import sys
>>> sys.getdefaultencoding()
'ascii'
>>> sys.maxunicode
1114111
>>> sys.getfilesystemencoding()
'UTF-8'
Is there any additional steps i have to do (except git clone <uberserver>)? Change of LANG=en_US.UTF-8, LC_ALL=en_US.UTF-8 environment variables doesn't have any effect

But everything works after next ugly hack:

Code: Select all

import sys
reload(sys)  # to enable `setdefaultencoding` again
sys.setdefaultencoding("UTF-8")
0 x

abma
Spring Developer
Posts: 3507
Joined: 01 Jun 2009, 00:08

Re: Uberserver. Unicode messages

Post by abma » 09 Mar 2014, 20:19

on lobby.springrts.com:

Code: Select all

$ locale 
LANG=en_US.UTF-8
LANGUAGE=en_US
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
its ubuntu 12.04.4 & python 2.7.3, too.

how do you start the server? also it seems your traceback doesn't match current uberservers code. is it up to date? it doesn't look so.

line 855 isn't "in_SAYPRIVATE": https://github.com/spring/uberserver/bl ... ol.py#L855
0 x

abma
Spring Developer
Posts: 3507
Joined: 01 Jun 2009, 00:08

Re: Uberserver. Unicode messages

Post by abma » 09 Mar 2014, 20:25

weird, locally i can reproduce the problem, too. no clue why it works on the server.
0 x

lamer
Posts: 146
Joined: 08 Mar 2014, 23:13

Re: Uberserver. Unicode messages

Post by lamer » 09 Mar 2014, 20:33

abma wrote: maybe my springlobby sends utf8 and yours unicode? does it work on the "official server"?
Strange but yes, my version of springlobby works with official server (unicode i mean)
line 855 isn't "in_SAYPRIVATE": https://github.com/spring/uberserver/bl ... ol.py#L855
Actually it is. Look at https://github.com/spring/uberserver/bl ... ol.py#L843
Just did "git clone https://github.com/spring/uberserver.git" a few minutes ago before testing things (for ubuntu).
how do you start the server?
from console and uberserver's directory
$ ./server.py
or
$ python server.py
Will test locale (i have differences with your settings)
Last edited by lamer on 09 Mar 2014, 20:36, edited 1 time in total.
0 x

abma
Spring Developer
Posts: 3507
Joined: 01 Jun 2009, 00:08

Re: Uberserver. Unicode messages

Post by abma » 09 Mar 2014, 20:34

do you use sqlite for storing as db? i guess thats the problem for some reason, but no clue atm, need time to investigate.
0 x

lamer
Posts: 146
Joined: 08 Mar 2014, 23:13

Re: Uberserver. Unicode messages

Post by lamer » 09 Mar 2014, 20:39

abma wrote:do you use sqlite for storing as db? i guess thats the problem for some reason, but no clue atm, need time to investigate.
yes, sqlite. (i mean i didn't install any mysql or other, just use default settings and thats sqlite)
And thank you for your help!
0 x

abma
Spring Developer
Posts: 3507
Joined: 01 Jun 2009, 00:08

Re: Uberserver. Unicode messages

Post by abma » 09 Mar 2014, 20:41

ok, then its very likely sqlite. i guess it returns asscii while mysql returns unicode/utf8. username is taken from db and concated in this line with the message.
0 x

lamer
Posts: 146
Joined: 08 Mar 2014, 23:13

Re: Uberserver. Unicode messages

Post by lamer » 09 Mar 2014, 21:06

And for the moment i'm out of ideas. Copying locale settings didn't help (but maybe it's important when db created).
Changing
self.engine = sqlalchemy.create_engine(self.sqlurl)
into
self.engine = sqlalchemy.create_engine(self.sqlurl, encoding='utf-8')
inside DataHandler.py and recreating db didn't help either.
Will test MySql :) ... later.
Thanks again

edit (a few thoughts to cofirm your idea about MySql):
With sqlite
print type(user), type(msg)
<type 'unicode'> <type 'str'>
And i guess MySql result from official server configuration
type(user) = <type 'str'> and thats why it doesnt need to decode/encode anything. If it's true then it's basically a lack of "unicode support" :) Sqlalchemy uses only unicode for sqlite (AFAIK from sqlalchemy docs)
Last edited by lamer on 09 Mar 2014, 22:27, edited 1 time in total.
0 x

abma
Spring Developer
Posts: 3507
Joined: 01 Jun 2009, 00:08

Re: Uberserver. Unicode messages

Post by abma » 09 Mar 2014, 21:55

this should fix it:

https://github.com/spring/uberserver/co ... 5408774282

just run "git pull"
0 x

lamer
Posts: 146
Joined: 08 Mar 2014, 23:13

Re: Uberserver. Unicode messages

Post by lamer » 09 Mar 2014, 22:31

Yes, it fixed the issue.
Thanks!
0 x

Post Reply

Return to “Lobby Clients & Server”