从python中的.txt文件中读取特殊字符

Question

此代码的目标是找出书中单词的使用频率。

我想阅读一本书的文本，但以下行一直在抛出我的代码：

珍贵的门徒。不，先生们；他总是会向他们展示干净的一对

特别是 é 字符

我看过以下文档，但不是很明白：https://docs.python.org/3.4/howto/unicode.html

这是我的代码：

import string
# Create word dictionary from the comprehensive word list 
word_dict = {}
def create_word_dict ():

  # open words.txt and populate dictionary
  word_file = open ("./words.txt", "r")
  for line in word_file:
    line = line.strip()
    word_dict[line] = 1

# Removes punctuation marks from a string
def parseString (st):
  st = st.encode("ascii", "replace")
  new_line = ""
  st = st.strip()
  for ch in st:
    ch = str(ch)
    if (n for n in (1,2,3,4,5,6,7,8,9,0)) in ch or ' ' in ch or ch.isspace() or ch == u'\xe9':

      print (ch)
      new_line += ch
    else:
      new_line += ""
  # now remove all instances of 's or ' at end of line
  new_line = new_line.strip()
  print (new_line)
  if (new_line[-1] == "'"):
    new_line = new_line[:-1]
  new_line.replace("'s", "")
  # Conversion from ASCII codes back to useable text
  message = new_line
  decodedMessage = ""
  for item in message.split():
    decodedMessage += chr(int(item))
  print (decodedMessage)
  return new_line

# Returns a dictionary of words and their frequencies
def getWordFreq (file):

  # Open file for reading the book.txt
  book = open (file, "r")

  # create an empty set for all Capitalized words
  cap_words = set()

  # create a dictionary for words
  book_dict = {}
  total_words = 0

  # remove all punctuation marks other than '[not s]
  for line in book:
    line = line.strip()
    if (len(line) > 0):
      line = parseString (line)

    word_list = line.split()

    # add words to the book dictionary
    for word in word_list:
      total_words += 1
      if (word in book_dict):
        book_dict[word] = book_dict[word] + 1
      else:
        book_dict[word] = 1
  print (book_dict)

  # close the file
  book.close()

def main():
  wordFreq1 = getWordFreq ("./Tale.txt")
  print (wordFreq1)

main()

我收到的错误如下：

Traceback (most recent call last):
  File "Books.py", line 80, in <module>
    main()
  File "Books.py", line 77, in main
    wordFreq1 = getWordFreq ("./Tale.txt")
  File "Books.py", line 60, in getWordFreq
    line = parseString (line)
  File "Books.py", line 36, in parseString
    decodedMessage += chr(int(item))
OverflowError: Python int too large to convert to C long                      

Answer 1

当你在python中打开一个文本文件时，默认编码是ANSI，所以它不包含你的échartecter。试试

word_file = open ("./words.txt", "r", encoding='utf-8')

Answer 2

试试：

def parseString(st):
    st = st.encode("ascii", "replace")

    # rest of code here

您遇到的新错误是因为您在 int（即数字）上调用 isalpha

试试这个：

for ch in st:
    ch = str(ch)
    if (n for n in (1,2,3,4,5,6,7,8,9,0) if n in ch) or ' ' in ch or ch.isspace() or ch == u'\xe9':

        print (ch)

Answer 3

我能想到的最好方法是将每个字符作为 ASCII 值读取到一个数组中，然后获取 char 值。例如，97 是“a”的 ASCII，如果您执行 char(97)，它将输出“a”。查看一些提供特殊字符值的在线 ASCII 表。