交叉编译python ModuleNotFoundError：模块信息来自哪里？答案

【问题标题】：cross-compile python ModuleNotFoundError: where the module info is from?交叉编译python ModuleNotFoundError：模块信息来自哪里？
【发布时间】：2021-03-03 04:51:00
【问题描述】：

我正在交叉构建 python 3.8。两个错误看起来很相似：

$ python3 -m venv .
Error: Command '['/home/p3t/penv386/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

$ bin/python3 -Im ensurepip --upgrade --default-pip
Traceback (most recent call last):
  File "<frozen zipimport>", line 520, in _get_decompress_func
ModuleNotFoundError: No module named 'zlib'

$ pip3 install --user pipenv
Traceback (most recent call last):
  File "/usr/bin/pip3", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python3.8/site-packages/pkg_resources/__init__.py", line 35, in <module>
  File "/usr/lib/python3.8/plistlib.py", line 65, in <module>
ModuleNotFoundError: No module named 'xml'

我在 SO 上看到过一些类似的帖子：

build python with zlib：编辑模块/设置以包含 zlib 或安装 zlib。我进行了编辑并验证了 libz 存在。它还有一个 python sn-p 来验证，但如果导入正常则应该运行。
no module named zlib：类似的解决方案。
其他几个线程基本上和上面两个很相似。

由于“/usr/lib/libz*”确实存在并且xml lib“/usr/lib/libexpat*”也确实存在，我开始认为模块信息可能丢失了。如何确保这两个模块在 python 构建中正确填充了它们的信息？

我使用 buildroot 2020.08.2 进行了交叉编译。从 buildroot 配置中，选择了 xml 和 zlib。我做了一个 strace，还没有看到可能是什么原因，这里是 venv：

$ cat -n log2-venv | egrep '(openat|newfstatat)' | grep '\.so'
     5  5835  openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
     6  5835  openat(AT_FDCWD, "/lib64/tls/aarch64/libpython3.8.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
     8  5835  openat(AT_FDCWD, "/lib64/tls/libpython3.8.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    10  5835  openat(AT_FDCWD, "/lib64/aarch64/libpython3.8.so.1.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
    12  5835  openat(AT_FDCWD, "/lib64/libpython3.8.so.1.0", O_RDONLY|O_CLOEXEC) = 3

    20  5835  openat(AT_FDCWD, "/lib64/libcrypt.so.1", O_RDONLY|O_CLOEXEC) = 3
    28  5835  openat(AT_FDCWD, "/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
    36  5835  openat(AT_FDCWD, "/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
    43  5835  openat(AT_FDCWD, "/lib64/libutil.so.1", O_RDONLY|O_CLOEXEC) = 3
    50  5835  openat(AT_FDCWD, "/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
    58  5835  openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3

   123  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/encodings/__init__.cpython-38-aarch64-linux-gnu.so", 0x7fc2ad5c10, 0) = -1 ENOENT (No such file or directory)
   124  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/encodings/__init__.abi3.so", 0x7fc2ad5c10, 0) = -1 ENOENT (No such file or directory)
   125  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/encodings/__init__.so", 0x7fc2ad5c10, 0) = -1 ENOENT (No such file or directory)

   496  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/collections/__init__.cpython-38-aarch64-linux-gnu.so", 0x7fc2ad43a0, 0) = -1 ENOENT (No such file or directory)
   497  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/collections/__init__.abi3.so", 0x7fc2ad43a0, 0) = -1 ENOENT (No such file or directory)
   498  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/collections/__init__.so", 0x7fc2ad43a0, 0) = -1 ENOENT (No such file or directory)

   544  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/_heapq.cpython-38-aarch64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=18528, ...}, 0) = 0
   546  5835  openat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/_heapq.cpython-38-aarch64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3

   578  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/site-packages/pkg_resources/__init__.cpython-38-aarch64-linux-gnu.so", 0x7fc2ad60c0, 0) = -1 ENOENT (No such file or directory)
   579  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/site-packages/pkg_resources/__init__.abi3.so", 0x7fc2ad60c0, 0) = -1 ENOENT (No such file or directory)
   580  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/site-packages/pkg_resources/__init__.so", 0x7fc2ad60c0, 0) = -1 ENOENT (No such file or directory)

   615  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/binascii.cpython-38-aarch64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=27656, ...}, 0) = 0
   617  5835  openat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/binascii.cpython-38-aarch64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3

   629  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/importlib/__init__.cpython-38-aarch64-linux-gnu.so", 0x7fc2ad3af0, 0) = -1 ENOENT (No such file or directory)
   630  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/importlib/__init__.abi3.so", 0x7fc2ad3af0, 0) = -1 ENOENT (No such file or directory)
   631  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/importlib/__init__.so", 0x7fc2ad3af0, 0) = -1 ENOENT (No such file or directory)

   766  5835  newfstatat(AT_FDCWD, "/usr/bin/lzma/__init__.cpython-38-aarch64-linux-gnu.so", 0x7fc2ad3510, 0) = -1 ENOTDIR (Not a directory)
   767  5835  newfstatat(AT_FDCWD, "/usr/bin/lzma/__init__.abi3.so", 0x7fc2ad3510, 0) = -1 ENOTDIR (Not a directory)
   768  5835  newfstatat(AT_FDCWD, "/usr/bin/lzma/__init__.so", 0x7fc2ad3510, 0) = -1 ENOTDIR (Not a directory)

   788  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/grp.cpython-38-aarch64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=14784, ...}, 0) = 0
   789  5835  openat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/grp.cpython-38-aarch64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3
   810  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/_struct.cpython-38-aarch64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=38352, ...}, 0) = 0
   811  5835  openat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/_struct.cpython-38-aarch64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3

   838  5835  newfstatat(AT_FDCWD, "/usr/bin/lzma/__init__.cpython-38-aarch64-linux-gnu.so", 0x7fc2ad43a0, 0) = -1 ENOTDIR (Not a directory)
   839  5835  newfstatat(AT_FDCWD, "/usr/bin/lzma/__init__.abi3.so", 0x7fc2ad43a0, 0) = -1 ENOTDIR (Not a directory)
   840  5835  newfstatat(AT_FDCWD, "/usr/bin/lzma/__init__.so", 0x7fc2ad43a0, 0) = -1 ENOTDIR (Not a directory)

   918  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/math.cpython-38-aarch64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=45240, ...}, 0) = 0
   919  5835  openat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/math.cpython-38-aarch64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3
   930  5835  newfstatat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/_datetime.cpython-38-aarch64-linux-gnu.so", {st_mode=S_IFREG|0755, st_size=75320, ...}, 0) = 0
   931  5835  openat(AT_FDCWD, "/usr/lib/python3.8/lib-dynload/_datetime.cpython-38-aarch64-linux-gnu.so", O_RDONLY|O_CLOEXEC) = 3

【问题讨论】：

标签： python python-3.x cross-compiling buildroot

【解决方案1】：

不确定，这可能是 buildroot 中的一个错误，无法解决此场景的依赖关系。对源代码的更深入跟踪似乎表明：在配置中选择了 python-pip 时启用了 xml 和 zlib 包。但是，我猜，模块信息由 host-python 编译成 .pyc 文件并包含在目标 python 安装中。 Buildroot 不会重建 host-python。这意味着目标 python 中缺少模块信息。

根据推理，这个解决方案看起来对我有用：

在配置中启用python-pip后，运行ls output/build/ | grep python可以找到所有python相关的包。
在每个软件包上，对它们运行一个 make *-dirclean。例如，如果只找到 python3-3.8.6 和 host-python3-3.8.6，则运行以下命令：

make host-python3-dirclean
make python3-dirclean

再次运行make 进行重建。

无需修改Module/Setup或Module/Setup.local。

【讨论】：