Page 1 of 1
Uberserver. Unicode messages
Posted: 08 Mar 2014, 23:57
by lamer
Looks like Official server supports symbols other then ascii (cyrillic). But uberserver's git version (master branch at the moment of writing) does not.
It appears that some .decode('utf-8') / .encode('utf-8') can fix this issue.
Is unicode support on todo list or did i just miss something and its already there?
update: Seems like next code can change default encoding from ascii to utf-8
Code: Select all
>>> import sys
>>> reload(sys) # to enable `setdefaultencoding` again
<module 'sys' (built-in)>
>>> sys.setdefaultencoding("UTF-8")
>>> sys.getdefaultencoding()
'UTF-8'
but
this page says it's bad practice
Re: Uberserver. Unicode messages
Posted: 09 Mar 2014, 14:13
by abma
what does "support" mean?
i can write some unicode char in client one and i receive it correctly in client two in a channel...
is your client broken? uberserver doesn't/shouldn't care about encoding, it just forwards the text.
uberserver master branch = current running code at springrts.com
Re: Uberserver. Unicode messages
Posted: 09 Mar 2014, 19:33
by lamer
Archlinux, springlobby 0.180 from community repo (also did try latest git).
Message to send: "на дворе трава, на траве дрова"
1) within regular chat (uberserver traceback):
Code: Select all
2014-03-09T20:14:39 Handler 0: Successfully logged in user <lamer> on session 1 user.
2014-03-09T20:15:06 ------------------------------------------------------------
2014-03-09T20:15:06 Traceback (most recent call last):
2014-03-09T20:15:06 File "/home/lamer/games/spring/uberserver/Dispatcher.py", line 39, in callback
2014-03-09T20:15:06 self.socketmap[s].Handle(data)
2014-03-09T20:15:06 File "/home/lamer/games/spring/uberserver/Client.py", line 147, in Handle
2014-03-09T20:15:06 self._protocol._handle(self,cmd)
2014-03-09T20:15:06 File "/home/lamer/games/spring/uberserver/protocol/Protocol.py", line 257, in _handle
2014-03-09T20:15:06 function(*([client]+arguments))
2014-03-09T20:15:06 File "/home/lamer/games/spring/uberserver/protocol/Protocol.py", line 855, in in_SAYPRIVATE
2014-03-09T20:15:06 self._root.usernames[user].Send('SAIDPRIVATE %s %s'%(client.username, msg))
2014-03-09T20:15:06 UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)
2014-03-09T20:15:06
2014-03-09T20:15:06 ------------------------------------------------------------
2) within battleroom chat:
Code: Select all
2014-03-09T20:15:19 ------------------------------------------------------------
2014-03-09T20:15:19 Traceback (most recent call last):
2014-03-09T20:15:19 File "/home/lamer/games/spring/uberserver/Dispatcher.py", line 39, in callback
2014-03-09T20:15:19 self.socketmap[s].Handle(data)
2014-03-09T20:15:19 File "/home/lamer/games/spring/uberserver/Client.py", line 147, in Handle
2014-03-09T20:15:19 self._protocol._handle(self,cmd)
2014-03-09T20:15:19 File "/home/lamer/games/spring/uberserver/protocol/Protocol.py", line 257, in _handle
2014-03-09T20:15:19 function(*([client]+arguments))
2014-03-09T20:15:19 File "/home/lamer/games/spring/uberserver/protocol/Protocol.py", line 1247, in in_SAYBATTLE
2014-03-09T20:15:19 self.broadcast_SendBattle(battle, 'SAIDBATTLE %s %s' % (user, msg))
2014-03-09T20:15:19 UnicodeDecodeError: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)
2014-03-09T20:15:19
2014-03-09T20:15:19 ------------------------------------------------------------
Maybe it's python's environment issue?
Code: Select all
Python 2.7.6 (default, Feb 26 2014, 12:07:17)
[GCC 4.8.2 20140206 (prerelease)] on linux2
Will try to test it with ubuntu+python2.7.3
Re: Uberserver. Unicode messages
Posted: 09 Mar 2014, 20:02
by abma
no clue if this is related, but it could:
http://stackoverflow.com/questions/1446 ... 2-or-ucs-4
on server:
Code: Select all
$ python
>>> import sys
>>> print sys.maxunicode
1114111
maybe my springlobby sends utf8 and yours unicode? does it work on the "official server"?
edit:
Code: Select all
>>> sys.getdefaultencoding()
'ascii'
>>> sys.getfilesystemencoding()
'UTF-8'
Re: Uberserver. Unicode messages
Posted: 09 Mar 2014, 20:12
by lamer
same error with ubuntu12.04+python2.7.3
And default encoding is
Code: Select all
>>> import sys
>>> sys.getdefaultencoding()
'ascii'
>>> sys.maxunicode
1114111
>>> sys.getfilesystemencoding()
'UTF-8'
Is there any additional steps i have to do (except git clone <uberserver>)? Change of LANG=en_US.UTF-8, LC_ALL=en_US.UTF-8 environment variables doesn't have any effect
But everything works after next ugly hack:
Code: Select all
import sys
reload(sys) # to enable `setdefaultencoding` again
sys.setdefaultencoding("UTF-8")
Re: Uberserver. Unicode messages
Posted: 09 Mar 2014, 20:19
by abma
on lobby.springrts.com:
Code: Select all
$ locale
LANG=en_US.UTF-8
LANGUAGE=en_US
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=
its ubuntu 12.04.4 & python 2.7.3, too.
how do you start the server? also it seems your traceback doesn't match current uberservers code. is it up to date? it doesn't look so.
line 855 isn't "in_SAYPRIVATE":
https://github.com/spring/uberserver/bl ... ol.py#L855
Re: Uberserver. Unicode messages
Posted: 09 Mar 2014, 20:25
by abma
weird, locally i can reproduce the problem, too. no clue why it works on the server.
Re: Uberserver. Unicode messages
Posted: 09 Mar 2014, 20:33
by lamer
abma wrote:
maybe my springlobby sends utf8 and yours unicode? does it work on the "official server"?
Strange but yes, my version of springlobby works with official server (unicode i mean)
Actually it is. Look at
https://github.com/spring/uberserver/bl ... ol.py#L843
Just did "git clone
https://github.com/spring/uberserver.git" a few minutes ago before testing things (for ubuntu).
how do you start the server?
from console and uberserver's directory
$ ./server.py
or
$ python server.py
Will test locale (i have differences with your settings)
Re: Uberserver. Unicode messages
Posted: 09 Mar 2014, 20:34
by abma
do you use sqlite for storing as db? i guess thats the problem for some reason, but no clue atm, need time to investigate.
Re: Uberserver. Unicode messages
Posted: 09 Mar 2014, 20:39
by lamer
abma wrote:do you use sqlite for storing as db? i guess thats the problem for some reason, but no clue atm, need time to investigate.
yes, sqlite. (i mean i didn't install any mysql or other, just use default settings and thats sqlite)
And thank you for your help!
Re: Uberserver. Unicode messages
Posted: 09 Mar 2014, 20:41
by abma
ok, then its very likely sqlite. i guess it returns asscii while mysql returns unicode/utf8. username is taken from db and concated in this line with the message.
Re: Uberserver. Unicode messages
Posted: 09 Mar 2014, 21:06
by lamer
And for the moment i'm out of ideas. Copying locale settings didn't help (but maybe it's important when db created).
Changing
self.engine = sqlalchemy.create_engine(self.sqlurl)
into
self.engine = sqlalchemy.create_engine(self.sqlurl, encoding='utf-8')
inside DataHandler.py and recreating db didn't help either.
Will test MySql :) ... later.
Thanks again
edit (a few thoughts to cofirm your idea about MySql):
With sqlite
print type(user), type(msg)
<type 'unicode'> <type 'str'>
And i guess MySql result from official server configuration
type(user) = <type 'str'> and thats why it doesnt need to decode/encode anything. If it's true then it's basically a lack of "unicode support" :) Sqlalchemy uses only unicode for sqlite (AFAIK from sqlalchemy docs)
Re: Uberserver. Unicode messages
Posted: 09 Mar 2014, 21:55
by abma
Re: Uberserver. Unicode messages
Posted: 09 Mar 2014, 22:31
by lamer
Yes, it fixed the issue.
Thanks!