Docker 映像构建中的 Python statsmodels 要求问题答案

【问题标题】：Issue with Python statsmodels requirement in Docker image buildDocker 映像构建中的 Python statsmodels 要求问题
【发布时间】：2020-12-08 22:05:15
【问题描述】：

我正在尝试为 AWS Batch 作业构建 Docker 映像，但一个新要求（Python 包 statsmodels）正在破坏我的构建过程。

requirements.txt 行：

pandas
numpy
statsmodels

完成Dockerfile:

FROM python:3

COPY requirements.txt /home/
COPY .env /home/
COPY src home/src
COPY process.py /home/

WORKDIR /home/
ENV TZ=America/New_York
RUN /usr/local/bin/python -m pip install --upgrade pip
RUN pip install -r ./requirements.txt

ENV PYTHONPATH='.'

错误日志摘录：

#12 271.7 INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
#12 288.4 INFO: pip is looking at multiple versions of us to determine which version is compatible with other requirements. This could take a while.
#12 288.4 Collecting us
#12 288.4   Downloading us-2.0.1.tar.gz (13 kB)
#12 288.8   Downloading us-2.0.0.tar.gz (13 kB)
#12 289.3   Downloading us-1.0.0.tar.gz (13 kB)
#12 289.8   Downloading us-0.9.1.tar.gz (13 kB)
#12 290.2   Downloading us-0.9.0.tar.gz (13 kB)
#12 290.6   Downloading us-0.8.0.tar.gz (12 kB)
#12 291.2   Downloading us-0.7.1.tar.gz (12 kB)
#12 291.5 INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
#12 291.5 INFO: pip is looking at multiple versions of us to determine which version is compatible with other requirements. This could take a while.
#12 291.6   Downloading us-0.7.tar.gz (11 kB)
#12 291.9   Downloading us-0.6.tar.gz (9.9 kB)
#12 292.3   Downloading us-0.5.tar.gz (9.3 kB)
#12 292.7   Downloading us-0.4.tar.gz (9.3 kB)
#12 293.0   Downloading us-0.3.tar.gz (7.5 kB)
#12 293.4 INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
#12 293.4 INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
#12 293.4   Downloading us-0.2.tar.gz (6.9 kB)
#12 294.0   Downloading us-0.1.tar.gz (6.3 kB)
#12 294.4 INFO: pip is looking at multiple versions of statsmodels to determine which version is compatible with other requirements. This could take a while.
#12 294.9 Collecting statsmodels
#12 294.9   Downloading statsmodels-0.12.0.tar.gz (17.5 MB)
#12 297.5   Installing build dependencies: started
#12 309.5   Installing build dependencies: finished with status 'done'
#12 309.5   Getting requirements to build wheel: started
#12 392.2   Getting requirements to build wheel: still running...
#12 400.0   Getting requirements to build wheel: finished with status 'done'
#12 400.1     Preparing wheel metadata: started
#12 401.3     Preparing wheel metadata: finished with status 'done'
#12 401.9   Downloading statsmodels-0.11.1.tar.gz (15.4 MB)
#12 404.2   Installing build dependencies: started
#12 415.7   Installing build dependencies: finished with status 'done'
#12 415.7   Getting requirements to build wheel: started
#12 480.8   Getting requirements to build wheel: still running...
#12 481.5   Getting requirements to build wheel: finished with status 'done'
#12 481.5     Preparing wheel metadata: started
#12 482.6     Preparing wheel metadata: finished with status 'done'
#12 483.1   Downloading statsmodels-0.11.0.tar.gz (15.4 MB)
#12 485.5   Installing build dependencies: started
#12 488.6   Installing build dependencies: finished with status 'error'
#12 488.6   ERROR: Command errored out with exit status 1:
#12 488.6    command: /usr/local/bin/python /usr/local/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-_0xxe4fy/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'cython>=0.29.14' 'numpy==1.14.5; python_version=='"'"'3.5'"'"'' 'numpy==1.14.5; python_version=='"'"'3.6'"'"'' 'numpy==1.14.5; python_version>='"'"'3.7'"'"'' 'numpy==1.17.5; python_version>='"'"'3.8'"'"'' 'scipy>=1.0'
#12 488.6        cwd: None
#12 488.6   Complete output (13 lines):
#12 488.6   Ignoring numpy: markers 'python_version == "3.5"' don't match your environment
#12 488.6   Ignoring numpy: markers 'python_version == "3.6"' don't match your environment
#12 488.6   ERROR: Cannot install numpy==1.14.5 and numpy==1.17.5 because these package versions have conflicting dependencies.
#12 488.6   
#12 488.6   The conflict is caused by:
#12 488.6       The user requested numpy==1.14.5
#12 488.6       The user requested numpy==1.17.5
#12 488.6   
#12 488.6   To fix this you could try to:
#12 488.6   1. loosen the range of package versions you've specified
#12 488.6   2. remove package versions to allow pip attempt to solve the dependency conflict
#12 488.6   
#12 488.6   ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
#12 488.6   ----------------------------------------
#12 488.6 ERROR: Command errored out with exit status 1: /usr/local/bin/python /usr/local/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-_0xxe4fy/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'cython>=0.29.14' 'numpy==1.14.5; python_version=='"'"'3.5'"'"'' 'numpy==1.14.5; python_version=='"'"'3.6'"'"'' 'numpy==1.14.5; python_version>='"'"'3.7'"'"'' 'numpy==1.17.5; python_version>='"'"'3.8'"'"'' 'scipy>=1.0' Check the logs for full command output.
------
failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c pip install -r ./requirements.txt]: runc did not terminate sucessfully

每 cmets 更新 Dockerfile：

FROM debian:10-slim

# install Python, pip, and misc build needs
RUN apt-get update \
    && apt-get install --no-install-recommends -y \
        build-essential \
        python3-dev \
        python3-pip \
        python3-setuptools \
    && apt-get clean

# bring in program logic
COPY .env /root/.env  # this may need to go into /root/.env (or a custom user)
COPY src /app/
COPY processer.py submit-processer.py /app/
COPY requirements.txt /app/

# install Python dependencies
RUN pip3 install --no-cache-dir -r /app/requirements.txt

# set env vars
ENV TZ=America/New_York

# running process logic
WORKDIR /app
ENTRYPOINT ["python3", "processer.py"]

更新进程的错误日志：

[+] Building 0.7s (7/12)                                                        
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 979B                                       0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [internal] load metadata for docker.io/library/debian:10-slim          0.6s
 => [internal] load build context                                          0.0s
 => => transferring context: 15.91kB                                       0.0s
 => CANCELED [1/8] FROM docker.io/library/debian:10-slim@sha256:062bbd9a1  0.0s
 => => resolve docker.io/library/debian:10-slim@sha256:062bbd9a1a58c9c5b8  0.0s
 => CACHED [2/8] RUN apt-get update     && apt-get install --no-install-r  0.0s
 => ERROR [3/8] COPY .env /root/.env  # this may need to go into /root/.e  0.0s
------
 > [3/8] COPY .env /root/.env  # this may need to go into /root/.env (or a custom user):
------
failed to solve with frontend dockerfile.v0: failed to build LLB: failed to compute cache key: "/#" not found: not found

【问题讨论】：

标签： python docker numpy dockerfile statsmodels

【解决方案1】：

尚不完全清楚为什么从您的 Dockerfile 中（我很惊讶标签 3 不足以仅选择最新的稳定版 3.9.1-buster），但 pip 正在尝试解决多个版本的依赖关系Python（意味着在基础容器中安装了几个）

#12 488.6 错误：命令出错，退出状态为 1：/usr/local/bin/python /usr/local/lib/python3.9/site-packages/pip install --ignore-installed --no-用户 --prefix /tmp/pip-build-env-_0xxe4fy/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'cython>=0.29.14' 'numpy==1.14.5; python_version=='"'"'3.5'"'"'' 'numpy==1.14.5; python_version=='"'"'3.6'"'"'' 'numpy==1.14.5; python_version>='"'"'3.7'"'"'' 'numpy==1.17.5; python_version>='"'"'3.8'"'"'' 'scipy>=1.0' 检查日志以获取完整的命令输出。

指定一个次要版本可能会解决这个问题。

如果您没有特定版本的需求，3.8-slim-buster（Debian 10 with Python 3.8 .. 实际上是 3.8.9 现在虽然这可能会随着对该版本的错误修复而改变）可能成为一个实际的选择并为您解决这个问题！

from python:3.8-slim-buster
...

如果这不能解决错误，那么很可能您的 3 标签很好，而您的 .env 很可能有问题（可能是您试图保留的 Python 虚拟环境）。

使用新的 RUN 块而不是 COPY-ing 您的主机系统所具有的内容，在您的容器上下文中重新生成 venv。

在 cmets 之后，我认为值得尝试一个不同的 Dockerfile，至少在以下更改后我会使用它

直接基于 Debian 10（您可能会发现另一个容器更高效/更小/无论如何，但它极有可能开箱即用）
自己安装 Python 3
不要使用/home/ 作为基础

FROM debian:10-slim

# install Python, pip, and misc build needs
RUN apt-get update \
    && apt-get install --no-install-recommends -y \
        build-essential \
        python3-dev \
        python3-pip \
        python3-setuptools \
    && apt-get clean

# bring in program logic
COPY .env /app/  # this may need to go into /root/.env (or a custom user)
COPY src /app/
COPY process.py /app/
COPY requirements.txt /app/

# install Python dependencies
RUN pip3 install --no-cache-dir -r /app/requirements.txt

# set env vars
ENV TZ=America/New_York

# running process logic
WORKDIR /app
ENTRYPOINT ["python3", "process.py"]

如果可行

使用/home 可能有问题（而不是/home/someuser/...）
上游 python 基本容器可能有些奇怪（我从未亲自使用过它，尽管这也会令人惊讶）

如果这不起作用

您可能不在您认为的目录中工作（即，将当前目录复制到另一个类似目录中，等等）
你的src 里面可能有一些奇怪的东西
statsmodels 可能被破坏（它的版本是 0.12，所以这并非完全不合理）

【讨论】：

您好，感谢您的回复。在尝试使用3.8-slim-buster 运行时，我遇到了类似的错误，但现在是ModuleNotFoundError: No module named 'numpy'。在这种情况下，我的 .env 只是两行指定 Airtable API 参数（AIRTABLE_BASE=xxx...），没有 venv。
神秘 - 尝试删除 PYTHONPATH 并简单地调用 python3 而不是对 /usr/local 的任何引用
嗯，删除PYTHONPATH 行并调用RUN python3 -m pip install --upgrade pip 后，我返回到最初帖子中记录的错误......还有其他想法吗？
@OJT 我用另一个 Dockerfile 和一些想法做了一个大编辑，这可能对你有用
我已经更新了问题中的 Dockerfile 内容和错误日志。一般来说，我对这些东西并不是特别有经验。这里到底有什么区别，以及它如何影响在 AWS Batch EC2 实例环境中的运行（例如，使用 /app/ 与实例 /home/ 目录，因为我们在其他不需要 @987654349 的 Dockerized 进程中成功使用@?