python UnicodeEncodeError, converting unicode to ascii
In python, we often encounter the unicode convert issue. For instance, when you try to print a unicode string, you will get the following exception:
|
1 |
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe6' in position 1535: ordinal not in range(128) |
The reason is that the str() function tries to convert the unicode string using ascii, which doesn’t support the character u’\xe6′.
The solution is to convert the string into ‘utf-8’ encoding.
|
1 2 3 4 5 6 7 |
>>> x = (u'hello', u'how are you', u'xxxxxx yy \xe6r') >>> print str(x[2]) UnicodeEncodeError: 'ascii' codec can't encode character u'\xe6' in position 10: ordinal not in range(128) >>> print str(x[2].encode('utf-8')) xxxxxx yy ær >>> str(x[2].encode('utf-8')) Out[121]: 'xxxxxx yy \xc3\xa6r' |
recommended conversion workflow: input (any cp) -> convert to unicode -> (process) -> output to utf-8
See the following two examples:
|
1 2 3 4 5 |
s = u'你好' s.encode('utf-8') Out[179]: '\xe4\xbd\xa0\xe5\xa5\xbd' s.encode('utf-8').decode('utf-8') == u'你好' Out[183]: True |
|
1 2 3 4 5 6 7 8 9 |
In[185]: z = '\xe4\xbd\xa0\xe5\xa5\xbd' In[186]: print z 你好 In[187]: z Out[186]: '\xe4\xbd\xa0\xe5\xa5\xbd' In[189]: z.decode('utf-8') Out[188]: u'\u4f60\u597d' In[190]: z.decode('utf-8') == u'你好' Out[189]: True |
Best practice:
Always encode from unicode to bytes.
In this direction, you get to choose the encoding.
|
1 2 3 4 |
>>> u"你好".encode("utf8") '\xe4\xbd\xa0\xe5\xa5\xbd' >>> print _ 你好 |
The other way is to decode from bytes to unicode.
In this direction, you have to know what the encoding is.
|
1 2 3 4 5 6 7 |
>>> bytes = '\xe4\xbd\xa0\xe5\xa5\xbd' >>> print bytes 你好 >>> bytes.decode('utf-8') u'\u4f60\u597d' >>> print _ 你好 |
Reference:
http://stackoverflow.com/questions/9644099/python-ascii-codec-cant-decode-byte
