Python 3 在 pdb "b main" 处失败并出现 UnicodeDecodeError？

Question

我发现的唯一类似问题是Django UnicodeDecodeError when using pdb - 不幸的是，那里的解决方案不适用于这种情况。

考虑以下代码，test.py：

#!/usr/bin/env python3
# -*- coding: utf-8 -*-
# encoding: utf-8

def subtract(ina, inb):
  myresult = ina - inb
  return myresult

def main():
  y2 = 10
  y1 = 7
  # calculate (y₂-y₁)
  print("Calculating difference between y2: {} and y1: {}".format(y2, y1))
  result = subtract(y2, y1)
  print("The result is: {}".format(result))

if __name__ == '__main__':
  main()

在 Windows 10 上使用来自 Anaconda3 的 Python3：

(base) C:\tmp>conda --version
conda 4.7.12

(base) C:\tmp>python --version
Python 3.7.3

...我可以毫无问题地运行这个程序：

(base) C:\tmp>python test.py
Calculating difference between y2: 10 and y1: 7
The result is: 3

但是，如果我想使用 pdb 调试/单步执行此程序，则在我键入 b main 以在 main 函数上设置断点时它会立即失败：

(base) C:\tmp>python -m pdb test.py
> c:\tmp\test.py(6)<module>()
-> def subtract(ina, inb):
(Pdb) b main
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\pdb.py", line 648, in do_break
    lineno = int(arg)
ValueError: invalid literal for int() with base 10: 'main'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\pdb.py", line 659, in do_break
    code = func.__code__
AttributeError: 'str' object has no attribute '__code__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\pdb.py", line 1701, in main
    pdb._runscript(mainpyfile)
  File "C:\ProgramData\Anaconda3\lib\pdb.py", line 1570, in _runscript
    self.run(statement)
  File "C:\ProgramData\Anaconda3\lib\bdb.py", line 585, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "c:\tmp\test.py", line 6, in <module>
    def subtract(ina, inb):
  File "c:\tmp\test.py", line 6, in <module>
    def subtract(ina, inb):
  File "C:\ProgramData\Anaconda3\lib\bdb.py", line 88, in trace_dispatch
    return self.dispatch_line(frame)
  File "C:\ProgramData\Anaconda3\lib\bdb.py", line 112, in dispatch_line
    self.user_line(frame)
  File "C:\ProgramData\Anaconda3\lib\pdb.py", line 261, in user_line
    self.interaction(frame, None)
  File "C:\ProgramData\Anaconda3\lib\pdb.py", line 352, in interaction
    self._cmdloop()
  File "C:\ProgramData\Anaconda3\lib\pdb.py", line 321, in _cmdloop
    self.cmdloop()
  File "C:\ProgramData\Anaconda3\lib\cmd.py", line 138, in cmdloop
    stop = self.onecmd(line)
  File "C:\ProgramData\Anaconda3\lib\pdb.py", line 418, in onecmd
    return cmd.Cmd.onecmd(self, line)
  File "C:\ProgramData\Anaconda3\lib\cmd.py", line 217, in onecmd
    return func(arg)
  File "C:\ProgramData\Anaconda3\lib\pdb.py", line 667, in do_break
    (ok, filename, ln) = self.lineinfo(arg)
  File "C:\ProgramData\Anaconda3\lib\pdb.py", line 740, in lineinfo
    answer = find_function(item, fname)
  File "C:\ProgramData\Anaconda3\lib\pdb.py", line 100, in find_function
    for lineno, line in enumerate(fp, start=1):
  File "C:\ProgramData\Anaconda3\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 199: character maps to <undefined>
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> c:\programdata\anaconda3\lib\encodings\cp1252.py(23)decode()
-> return codecs.charmap_decode(input,self.errors,decoding_table)[0]
(Pdb) q
Post mortem debugger finished. The test.py will be restarted
> c:\tmp\test.py(6)<module>()
-> def subtract(ina, inb):
(Pdb) q

(base) C:\tmp>

问题出在注释行：# calculate (y₂-y₁)；如果被删除，则pdb 开始正常：

(base) C:\tmp>python -m pdb test.py
> c:\tmp\test.py(6)<module>()
-> def subtract(ina, inb):
(Pdb) b main
Breakpoint 1 at c:\tmp\test.py:10
(Pdb) q

(base) C:\tmp>

对此我有点惊讶——Python3 不应该是“默认的 utf-8”吗？

显然，这是一个微不足道的案例，我可以轻松删除导致问题的单个注释行。但是，我有一个很大的脚本，其中到处都有 utf-8 字符，无论是在 cmets 中，还是在我实际上想要逐步完成的打印中，并且进入并手动更改所有这些都是不可行的UTF-8 字符的实例。

那么，有没有办法欺骗 Python3 的 pdb，所以它可以工作 - 即使源代码中存在 utf-8 字符（不管是在 cmets 中还是在实际命令中）？

Answer 1

Python 3 默认为 UTF-8，但它运行的环境不是 - 它的默认编码为 cp1252。

您可以将 PYTHONIOENCODING 环境变量设置为 UTF-8 以覆盖默认编码，或将 change the environment 设置为使用 UTF-8。

编辑

我分析的太仓促了。上述解决方案适用于修复从 stdin/stdout 读取或写入时引发的 unicode 错误，但这里的问题是 pdb opens a file 用于读取而不指定编码：

def find_function(funcname, filename):
    cre = re.compile(r'def\s+%s\s*[(]' % re.escape(funcname))
    try:
        fp = open(filename)
    except OSError:
        return None

如果没有指定编码，根据io docs，Python将默认使用locale.getpreferredencoding的结果——在这种情况下大概是cp1252。

一种解决方案可能是在运行调试器之前设置控制台区域设置。

也可以将PYTHONUTF8 环境变量设置为1。除其他外，这将导致

open()、io.open() 和 codecs.open() 默认使用 UTF-8 编码。