在 Python 中将多字节字符转换为 7 位 ASCII 的 Python 库 [关闭]

Question

是否有提供将多字节非 ASCII 字符转换为某种合理形式的 7 位可显示 ASCII 的 python 库。这是为了避免将charmap 硬编码为answer 到Translating multi-byte characters into 7-bit ASCII in Python 中给出的Translating multi-byte characters into 7-bit ASCII in Python

编辑：我目前使用的是 Python 2.7.11 或更高版本，但尚未使用 Python 3，但提供 Python 3 解决方案的答案将被考虑并发现有帮助。

原因是这样的：因为我是手动翻译的，所以会漏掉一些：

我的脚本是：

#!/bin/bash
# -*- mode: python; -*-

import os
import re
import requests

url = "https://system76.com/laptops/kudu"

#
# Load the text from request as a true unicode string:
#
r = requests.get(url)
r.encoding = "UTF-8"
data = r.text  # ok, data is a true unicode string

# translate offending characters in unicode:

charmap = {
    0x2014: u'-',   # em dash
    0x201D: u'"',   # comma quotation mark, double
    # etc.
}
data = data.translate(charmap)
tdata = data.encode('ascii')

我得到的错误是：

./simple_wget
Traceback (most recent call last):
  File "./simple_wget.py", line 25, in <module>
    tdata = data.encode('ascii')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2013' in position 10166: ordinal not in range(128)

这将是一场为新发现的角色更新charmap 的永无止境的战斗。是否有提供此charmap 的python 库，因此我不必以这种方式对其进行硬编码？

Answer 1

（注意：此答案适用于 Python 2.7.11+。）

https://stackoverflow.com/a/1701378/257924 的答案是指 Unidecode 包，这正是我所寻找的。在使用该软件包时，我还发现了我的困惑的最终根源，这在https://pythonhosted.org/kitchen/unicode-frustrations.html#frustration-3-inconsistent-treatment-of-output 进行了深入阐述，特别是本节：

挫折#3：输出处理不一致

好吧，既然 python 社区正在到处使用 unicode 字符串，我们不妨将所有内容都转换为 unicode 字符串并默认使用它，对吗？大多数时候听起来不错，但 至少有一个巨大的警告需要注意。每当您将文本输出到终端或文件时，都必须将文本转换为字节 str。 Python 将尝试从 unicode 隐式转换为 byte str... 但如果字节是非 ASCII 则会抛出异常：

以下是我使用它的演示脚本。 names 变量中列出的字符是我确实需要翻译成可读的字符，而不是删除，对于我正在分析的网页类型。

#!/bin/bash
# -*- mode: python; coding: utf-8 -*-
# The above coding is needed to to avoid this error: SyntaxError: Non-ASCII character '\xe2' in file ./unicodedata_normalize_test.py on line 9, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

import os
import re
import unicodedata
from unidecode import unidecode

names = [
    'HYPHEN-MINUS',
    'EM DASH',
    'EN DASH',
    'MINUS SIGN',
    'APOSTROPHE',
    'LEFT SINGLE QUOTATION MARK',
    'RIGHT SINGLE QUOTATION MARK',
    'LATIN SMALL LETTER A WITH ACUTE',
]

for name in names:
    character = unicodedata.lookup(name)
    unidecoded = unidecode(character)
    print
    print 'name      ',name
    print 'character ',character
    print 'unidecoded',unidecoded

上述脚本的示例输出为：

censored@censored:~$ unidecode_test

name       HYPHEN-MINUS
character  -
unidecoded -

name       EM DASH
character  —
unidecoded --

name       EN DASH
character  –
unidecoded -

name       MINUS SIGN
character  −
unidecoded -

name       APOSTROPHE
character  '
unidecoded '

name       LEFT SINGLE QUOTATION MARK
character  ‘
unidecoded '

name       RIGHT SINGLE QUOTATION MARK
character  ’
unidecoded '

name       LATIN SMALL LETTER A WITH ACUTE
character  á
unidecoded a

以下更详细的脚本会加载多个包含许多 unicode 字符的网页。请参阅下面脚本中的 cmets：

#!/bin/bash
# -*- mode: python; coding: utf-8 -*-

import os
import re
import subprocess
import requests
from unidecode import unidecode

urls = [
    'https://system76.com/laptops/kudu',
    'https://stackoverflow.com/a/38249916/257924',
    'https://www.peterbe.com/plog/unicode-to-ascii',
    'https://stackoverflow.com/questions/227459/ascii-value-of-a-character-in-python?rq=1#comment35813354_227472',
    # Uncomment out the following to show that this script works without throwing exceptions, but at the expense of a huge amount of diff output:
    ###'https://en.wikipedia.org/wiki/List_of_Unicode_characters',
]

# The following variable settings represent what just works without throwing exceptions.
# Setting re_encode to False and not_encode to True results in the write function throwing an exception of
#
#    Traceback (most recent call last):
#      File "./simple_wget.py", line 52, in <module>
#        file_fp.write(data[ext])
#    UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 33511: ordinal not in range(128)
#
# This is the crux of my confusion and is explained by https://pythonhosted.org/kitchen/unicode-frustrations.html#frustration-3-inconsistent-treatment-of-output
# So this is why we set re_encode to True and not_encode to False below:
force_utf_8 = False
re_encode = True
not_encode = False
do_unidecode = True

for url in urls:
    #
    # Load the text from request as a true unicode string:
    #
    r = requests.get(url)
    print "\n\n\n"
    print "url:",url
    print "current encoding:",r.encoding

    data = {}

    if force_utf_8:
        # The next two lines do not work. They cause the write to fail:
        r.encoding = "UTF-8"
        data['old'] = r.text  # ok, data is a true unicode string

    if re_encode:
        data['old'] = r.text.encode(r.encoding)

    if not_encode:
        data['old'] = r.text

    if do_unidecode:
        # translate offending characters in unicode:
        data['new'] = unidecode(r.text)

    html_base = re.sub(r'[^a-zA-Z0-9_-]+', '__', url)
    diff_cmd = "diff "
    for ext in [ 'old', 'new' ]:
        if ext in data:
            print "ext:",ext
            html_file = "{}.{}.html".format(html_base, ext)
            with open(html_file, 'w') as file_fp:
                file_fp.write(data[ext])
                print "Wrote",html_file
            diff_cmd = diff_cmd + " " + html_file

    if 'old' in data and 'new' in data:
        print 'Executing:',diff_cmd
        subprocess.call(diff_cmd, shell=True)

上述脚本的gist showing the output。这显示了在“旧”和“新”html 文件上执行 Linux diff 命令以查看翻译。德语等语言会出现误译，但这对于我获得一些有损单引号和双引号类型的字符以及类似破折号的字符的翻译来说很好。

Answer 2

str.encode() 有一个可选的 'error' 参数，可以替换不可编码的字符而不是抛出错误。这就是你要找的吗？

https://docs.python.org/3/howto/unicode.html#converting-to-bytes

Answer 3

您可以考虑使用unicodedata python 包。我认为您可能会觉得有趣的方法之一是normalize（另请参阅peterbe.come 给出的用法示例）：

import unicodedata

foo = 'abcdéfg'
unicodedata.normalize(foo).encode('ascii','ignore')