导入解释器启动时不可用的 Python 模块

Question

我的代码如下所示：

import sys, importlib

try:
    import pip
except ImportError:
    raise ImportError("Please install pip")

reqs = ["sh", "vcstools"]
for req in reqs:
    sys.stdout.write("checking for %s..." % req)
    try:
        importlib.import_module(req)
        print("found")
    except ImportError:
        print("missing!")
        print("Installing %s..." % req)
        pip.main(['install', '--user', req])
        #importlib.invalidate_caches() python3 only
        # XXX fails
        importlib.import_module(req)
        new_mods = True

    locs = locals()
    locs[req] = sys.modules[req]

print(sh, vcstools)

这旨在在运行时下载（粗略）依赖项并导入它们（是的，我知道，它不尊重版本号，我可以使用 virtualenv 等。这些是单独的问题）。

如果我们运行它（在 ~/.local 中没有安装任何东西），我们会得到以下结果：

checking for sh...missing!
Installing sh...
Downloading/unpacking sh
....
Successfully installed sh
Cleaning up...
Traceback (most recent call last):
  File "test.py", line 20, in <module>
    importlib.import_module(req)
  File "/usr/local/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
ImportError: No module named sh

所以我们看到安装sh 模块后，我们不能立即导入它。如果我们重新运行脚本，它会成功，找到上次运行时安装的sh。

我觉得奇怪的是，我们看不到 vcstools 的相同行为；它在同一次运行中安装和导入就好了。是什么赋予了？ sh 有什么特别之处吗？

这是第二次运行的完整输出。请注意，我们从上次运行中选择了sh，然后我们安装了vcstools并正确地导入它：

checking for sh...found
checking for vcstools...missing!
Installing vcstools...
Downloading/unpacking vcstools
  Downloading vcstools-0.1.33.tar.gz
  Running setup.py egg_info for package vcstools

Downloading/unpacking pyyaml (from vcstools)
  Downloading PyYAML-3.11.tar.gz (248Kb): 248Kb downloaded
  Running setup.py egg_info for package pyyaml

    skipping 'ext/_yaml.c' Cython extension (up-to-date)
Requirement already satisfied (use --upgrade to upgrade): python-dateutil in /usr/local/lib/python2.7/site-packages (from vcstools)
Installing collected packages: vcstools, pyyaml
  Running setup.py install for vcstools

  Running setup.py install for pyyaml
    checking if libyaml is compilable
    cc -pthread -fno-strict-aliasing -O2 -pipe -DNDEBUG -O2 -pipe -fPIC -fPIC -I/usr/local/include/python2.7 -c build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c -o build/temp.openbsd-5.5-amd64-2.7/check_libyaml.o
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:2:18: error: yaml.h: No such file or directory
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c: In function 'main':
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:5: error: 'yaml_parser_t' undeclared (first use in this function)
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:5: error: (Each undeclared identifier is reported only once
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:5: error: for each function it appears in.)
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:5: error: expected ';' before 'parser'
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:6: error: 'yaml_emitter_t' undeclared (first use in this function)
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:6: error: expected ';' before 'emitter'
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:8: error: 'parser' undeclared (first use in this function)
    build/temp.openbsd-5.5-amd64-2.7/check_libyaml.c:11: error: 'emitter' undeclared (first use in this function)

    libyaml is not found or a compiler error: forcing --without-libyaml
    (if libyaml is installed correctly, you may need to
     specify the option --include-dirs or uncomment and
     modify the parameter include_dirs in setup.cfg)

    skipping 'ext/_yaml.c' Cython extension (up-to-date)
Successfully installed vcstools pyyaml
Cleaning up...
(<module 'sh' (built-in)>, <module 'vcstools' from '/home/edd/.local/lib/python2.7/site-packages/vcstools/__init__.pyc'>)

这是 OpenBSD 上的 Python-2.7。

干杯

编辑：刚刚注意到new_mods 行是多余的。我会把它留在那里，这样输出中的行号就不会倾斜。

Answer 1

这可能是您尝试安装到 ~/.local/... 中的第一个包 - 这意味着 ~/.local/... 仅使用 pip.main(['install', '--user', req]) 创建

如果在python启动时 ~/.local 不存在，则不会添加到 sys.path 中，并且找不到模块（即使使用 importlib.invalidate_caches() 也找不到）。

所以你必须将该路径添加到 sys.path 中，或者按照this answer 中的说明重新加载 sys.path，然后它会起作用...

我的解决方案：

import pip
import importlib
import sys

def pip_import(module_name, pip_package=None):
    try:
        mod = importlib.import_module(module_name)
    except ImportError:
        pip.main(['install', '--user', pip_package if pip_package else module_name])

        # If the install path did not exist on start up it has 
        # to be added to sys.path
        if not pip.locations.user_site in sys.path:
            sys.path.append(pip.locations.user_site)

        mod = importlib.import_module(module_name)

    # Add the imported module to the global namespace so it
    # can be used just like `import module`
    globals()[module_name] = mod