UnicodeDecodeError：“ascii”编解码器无法解码位置 2141 中的字节 0xef：序数不在范围内（128）

Question

这是一个将随机生成的句子发送到不和谐聊天中的脚本。但它偶尔会遇到错误：UnicodeDecodeError: 'ascii' codec cant decode byte 0xef in position 2141: ordinal not in range(128)

我该如何解决这个错误？

代码：

import asyncio
import random
import discord.ext.commands
import markovify
import nltk
import re


with open("/root/sample.txt") as f:
 text = f.read()

class POSifiedText(markovify.Text):
    def word_split(self, sentence):
        words = re.split(self.word_split_pattern, sentence)
        words = [w for w in words if len(w) > 0]
        words = [" :: ".join(tag) for tag in nltk.pos_tag(words)]
        return words

    def word_join(self, words):
        sentence = "".join(word.split("::")[0] for word in words)
        return sentence


text_model = POSifiedText(text, state_size=1)

client = discord.Client()
async def background_loop():
    await client.wait_until_ready()
    while not client.is_closed:
        channel = client.get_channel('286342556600762369')
        messages = [(text_model.make_sentence(tries=33, max_overlap_total=10, default_max_overlap_ratio=0.5))]
        await client.send_message(channel, random.choice(messages))
        await asyncio.sleep(15)

client.loop.create_task(background_loop())
client.run("MjY2NjkwNDY4MjI4NzU5NTU4.C5jcdw.WFfBTUmAY7UcrwKTwYFJ9_bFHjI")

错误发生在第 9 行。

Answer 1

我遇到了类似的问题；原来有一个 ascii 代码，python 无法变成标准符号。 为了解决这个问题，您必须告诉 python 编码 ascii 代码并忽略它无法编码的代码。然后将其解码回 utf-8。

    def word_split(self, sentence):
        words = re.split(self.word_split_pattern, sentence)
        words = [w for w in words if len(w) > 0]
        words = [" :: ".join(tag) for tag in nltk.pos_tag(words)]
        words = words.encode('ascii', 'ignore')
        words = words.decode("utf-8"))
        return words

我在您的退货声明之前添加了额外的步骤。