使用 pythonbrew 编译 Python 3.2 和 2.7 的问题

Question

我正在尝试使用 Pythonbrew 构建多个版本的 python，但我遇到了一些测试失败。这是在运行的 VM 上：Ubuntu 8.04 32bit

当我使用 --enable-shared 选项时会发生这种情况

build.log 文件错误如下所示：

Python 3.2

以下是 Python 3.2 的失败测试：

LD_LIBRARY_PATH=/opt/standingcloud/stack/pythonbrew/build/Python-3.2: ./python -E -c 'import sys ; from sysconfig import get_platform ; print(get_platform()+"-"+sys.version[0:3])' >platform
find ./Lib -name '*.py[co]' -print | xargs rm -f
LD_LIBRARY_PATH=/opt/standingcloud/stack/pythonbrew/build/Python-3.2: ./python -Wd -E -bb  ./Lib/test/regrtest.py -l 
== CPython 3.2 (r32:88445, Jun 16 2011, 15:07:10) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)]
==   Linux-2.6.18.8-linode22-i686-with-debian-lenny-sid little-endian
==   /opt/standingcloud/stack/pythonbrew/build/Python-3.2/build/test_python_18839
Testing with flags: sys.flags(debug=0, division_warning=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=1, verbose=0, bytes_warning=2, quiet=0)

....SNIP....

[197/349] test_os
test test_os failed -- Traceback (most recent call last):
  File "/opt/standingcloud/stack/pythonbrew/build/Python-3.2/Lib/test/test_os.py", line 646, in test_exist_ok_existing_directory
    os.makedirs(path, mode=mode, exist_ok=True)
  File "/opt/standingcloud/stack/pythonbrew/build/Python-3.2/Lib/os.py", line 152, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '@test_18839_tmp/dir1'

....SNIP....

321 tests OK.
1 test failed:
    test_os
27 tests skipped:
    test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp
    test_codecmaps_kr test_codecmaps_tw test_curses test_dbm_gnu
    test_dbm_ndbm test_gdb test_ioctl test_kqueue test_ossaudiodev
    test_pep277 test_smtpnet test_socketserver test_sqlite
    test_startfile test_timeout test_tk test_ttk_guionly
    test_unicode_file test_urllib2net test_urllibnet test_winreg
    test_winsound test_xmlrpc_net test_zipfile64
5 skips unexpected on linux2:
    test_dbm_gnu test_dbm_ndbm test_ioctl test_tk test_ttk_guionly
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/dev/null' mode='a' encoding='ANSI_X3.4-1968'>
make: *** [test] Error 1

Python 2.7.1

以下是 Python 2.7.1 的失败测试：

./python -Wd -3 -E -tt  ./Lib/test/regrtest.py -l 
== CPython 2.7.1 (r271:86832, Jun 16 2011, 17:59:28) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)]
==   Linux-2.6.18.8-linode22-i686-with-debian-lenny-sid little-endian
==   /opt/standingcloud/stack/pythonbrew/build/Python-2.7.1/build/test_python_19286

....SNIP....

test test_file2k failed -- Traceback (most recent call last):
  File "/opt/standingcloud/stack/pythonbrew/build/Python-2.7.1/Lib/test/test_file2k.py", line 219, in testStdin
    self.assertRaises(IOError, sys.stdin.seek, -1)
AssertionError: IOError not raised

....SNIP....

343 tests OK.
2 tests failed:
    test_file test_file2k
41 tests skipped:
    test_aepack test_al test_applesingle test_bsddb test_bsddb185
    test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk
    test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
    test_dbm test_gdb test_gdbm test_gl test_imgfile test_ioctl
    test_kqueue test_linuxaudiodev test_macos test_macostools
    test_ossaudiodev test_pep277 test_scriptpackages test_smtpnet
    test_socketserver test_sqlite test_startfile test_sunaudiodev
    test_timeout test_tk test_ttk_guionly test_unicode_file
    test_urllib2net test_urllibnet test_winreg test_winsound
    test_zipfile64
8 skips unexpected on linux2:
    test_bsddb test_bsddb3 test_dbm test_gdb test_gdbm test_ioctl
    test_tk test_ttk_guionly
make: *** [test] Error 1

Python 2.7

以下是 Python 2.7 的失败测试：

LD_LIBRARY_PATH=/opt/standingcloud/stack/pythonbrew/build/Python-2.7: ./python -E -c 'import sys ; from sysconfig import get_platform ; print get_platform()+"-"+sys.version[0:3]' >platform
find ./Lib -name '*.py[co]' -print | xargs rm -f
LD_LIBRARY_PATH=/opt/standingcloud/stack/pythonbrew/build/Python-2.7: ./python -Wd -3 -E -tt ./Lib/test/regrtest.py -l 
== CPython 2.7 (r27:82500, Jun 16 2011, 15:25:09) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu4)]
==   Linux-2.6.18.8-linode22-i686-with-debian-lenny-sid little-endian
==   /opt/standingcloud/stack/pythonbrew/build/Python-2.7/build/test_python_13541

....SNIP....

test_distutils
/usr/bin/ld: cannot find -lpython2.7
collect2: ld returned 1 exit status
test test_distutils failed -- Traceback (most recent call last):
  File "/opt/standingcloud/stack/pythonbrew/build/Python-2.7/Lib/distutils/tests/test_build_ext.py", line 269, in test_get_outputs
    cmd.run()
  File "/opt/standingcloud/stack/pythonbrew/build/Python-2.7/Lib/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/opt/standingcloud/stack/pythonbrew/build/Python-2.7/Lib/distutils/command/build_ext.py", line 449, in build_extensions
    self.build_extension(ext)
  File "/opt/standingcloud/stack/pythonbrew/build/Python-2.7/Lib/distutils/command/build_ext.py", line 531, in build_extension
    target_lang=language)
  File "/opt/standingcloud/stack/pythonbrew/build/Python-2.7/Lib/distutils/ccompiler.py", line 741, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "/opt/standingcloud/stack/pythonbrew/build/Python-2.7/Lib/distutils/unixccompiler.py", line 258, in link
    raise LinkError, msg
LinkError: command 'gcc' failed with exit status 1

....SNIP....

test test_file failed -- multiple errors occurred; run in verbose mode for details
test_file2k
test test_file2k failed -- Traceback (most recent call last):
  File "/opt/standingcloud/stack/pythonbrew/build/Python-2.7/Lib/test/test_file2k.py", line 211, in testStdin
    self.assertRaises(IOError, sys.stdin.seek, -1)
AssertionError: IOError not raised

....SNIP....

339 tests OK.
3 tests failed:
    test_distutils test_file test_file2k
1 test altered the execution environment:
    test_site
41 tests skipped:
    test_aepack test_al test_applesingle test_bsddb test_bsddb185
    test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk
    test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_curses
    test_dbm test_gdb test_gdbm test_gl test_imgfile test_ioctl
    test_kqueue test_linuxaudiodev test_macos test_macostools
    test_ossaudiodev test_pep277 test_scriptpackages test_smtpnet
    test_socketserver test_sqlite test_startfile test_sunaudiodev
    test_timeout test_tk test_ttk_guionly test_unicode_file
    test_urllib2net test_urllibnet test_winreg test_winsound
    test_zipfile64
8 skips unexpected on linux2:
    test_bsddb test_bsddb3 test_dbm test_gdb test_gdbm test_ioctl
    test_tk test_ttk_guionly
make: [test] Error 1 (ignored)

Answer 1

Ryan Thompson 的回答很好……LD_RUN_PATH 是正确的补救措施。

但是，与其硬编码绝对路径，不如使用：

LD_RUN_PATH='$ORIGIN/../lib'

...其中$ORIGIN 是正在运行的二进制文件（例如python）的PWD。

# objdump -x /home/arisinger/.pythonbrew/pythons/Python-2.7.3/bin/python|grep RPATH
  RPATH                $ORIGIN/../lib
# ldd .pythonbrew/pythons/Python-2.7.3/bin/python
    [...]
    libpython2.7.so.1.0 => /home/arisinger/.pythonbrew/pythons/Python-2.7.3/bin/../lib/libpython2.7.so.1.0 (0x00007f43994f6000)
    [...]

...上面也会正确安装pip等。最后来自man ld.so：

$ORIGIN and rpath
    ld.so  understands the string $ORIGIN (or equivalently ${ORIGIN}) in an
    rpath specification (DT_RPATH or DT_RUNPATH) to mean the directory con‐
    taining  the  application  executable.  Thus, an application located in
    somedir/app could be compiled with gcc  -Wl,-rpath,'$ORIGIN/../lib'  so
    that  it  finds  an  associated shared library in somedir/lib no matter
    where somedir is located in the directory hierarchy.  This  facilitates
    the  creation  of  "turn-key"  applications  that  do  not  need  to be
    installed into special directories, but can instead  be  unpacked  into
    any directory and still find their own shared libraries.

编辑： 很遗憾 venvs 中断了，因为 virtualenv 复制了 python 二进制文件（更改了 $ORIGIN），但 没有 复制（或符号链接）库，所以 C venv 中的 /builtin 动态模块最终会链接到系统 python ... 不好 :-( ...

... IMO 这是一个需要修复的错误（在 virtualenv 中？），但一个简单的解决方法是手动将 libpythonX.Y.so* 复制/符号链接到 venv 的 lib/ 目录中......或者做一些有点恶心的事情看起来像（注意 XYZ）：

LD_RUN_PATH='$ORIGIN/../lib:$ORIGIN/../../../../pythons/Python-X.Y.Z/lib'

...选择对你来说不那么讨厌的东西；两者都经过验证可以正常工作。

Answer 2

我想我找到了一种无需设置 LD_LIBRARY_PATH 即可完成这项工作的方法。

我发现有一个名为LD_RUN_PATH 的环境变量与LD_LIBRARY_PATH 类似，只是您在编译程序时设置了它，并且它在运行时记住该路径（所以你不需要在运行时设置LD_LIBRARY_PATH）。

因此，如果您在运行 pythonbrew 以安装 Python 时适当地设置 LD_RUN_PATH，那么无论何时运行它，它都应该找到自己的共享库。例如，以下命令对我来说是成功的，包括安装 distutils 和 pip，因此模块安装准备就绪，无需任何进一步设置。

LD_RUN_PATH=$HOME/.pythonbrew/pythons/Python-2.7.3/lib pythonbrew install -j2 -C '--enable-shared' --force -v 2.7.3

然后我可以通过pythonbrew use 2.7.3 激活它，它就可以正常工作，而无需在您的 shell init 中设置任何环境变量或任何东西。

当然，你需要提前知道pythonbrew会在哪里安装你的Python，这样你才能正确设置LD_RUN_PATH。一般来说，它会是$HOME/.pythonbrew/pythons/Python-$VERSION/lib，但如果这不起作用，您可以通过实际安装一次（没有--enable-shared）然后找到lib 目录来查找。

我相信错误的发生是因为在使用--enable-shared 安装后，您安装的 Python 会找到系统 Python 的共享库并使用它，如果两个 Python 的构建方式存在任何不匹配，则会导致问题。使用 LD_LIBRARY_PATH 或 LD_RUN_PATH 解决方案会强制您的 Python 使用自己的共享库。

Answer 3

目前，我设法让 pythonbrew 使用 2.7.2 执行以下操作：

a) 首先，安装编译 python 所需的所有依赖项。

'curl' # not for build, but for steps after
'build-essential',
'libbz2-dev',
'libsqlite3-dev',
'zlib1g-dev',
'libxml2-dev',
'libxslt1-dev',
'libreadline5', # lenny
'libreadline5-dev', # lenny
'libgdbm-dev',
'libgdb-dev',
'libxml2',
'libssl-dev',
'tk-dev',
'libgdbm-dev',
'libexpat1-dev',
'libncursesw5-dev'

b) 现在安装 2.7.2：

pythonbrew install --configure="--with-threads --enable-shared" -j2 -v 2.7.2

这会安装，但会被以下bug 击中：

/home/python-deploy/.pythonbrew/pythons/Python-2.7.2/bin/python：错误 加载共享库时：libpython2.7.so.1.0：无法打开 共享对象文件：没有这样的文件或目录

*为避免在构建后尝试安装 setuptools 时出现错误，请使用 --no-setuptools 选项，并且您可能需要在某些平台上使用 --force，因为 gdb 测试由于错误而失败在测试本身中。

pythonbrew install --configure="--with-threads--enable-shared" \
                   --force \
                   --no-setuptools \
                   --jobs=2 \
                   --verbose 2.7.2 

c) 最简单的解决方法是（使用正确的 pythonbrew 安装路径）：

pythonbrew use 2.7.2
export LD_LIBRARY_PATH=$HOME/.pythonbrew/pythons/Python-2.7.2/lib

或永久添加它，将 pythonbrew lib 路径放在 /etc/ld.so.conf.d 下的文件中，该路径将在启动时添加，或立即运行 ldconfig：

sudo echo /home/user/.pythonbrew/pythons/Python-2.7.2/lib >> /etc/ld.so.conf.d/pythonbrew.conf
sudo ldconfig

d) 安装 setuptools 或手动分发，即

curl -O http://python-distribute.org/distribute_setup.py
python distribute_setup.py && easy_install pip

你应该准备好了。仅在 vagrant debian lenny VM 上使用 2.7.2 进行了测试。