如何使用 tensorflow-nightly 和 tensorflow 对象检测研究模型构建 docker 镜像

Question

由于 tensorflow-nightly 的 GPU 支持是 currently broken on Google Colab，我正在尝试构建自己的 docker 映像以进行开发。但是，当我从 tensorflow/models 安装 object_detection 包时，我的夜间 tensorflow 包被作为依赖项从 object_detection setup.py 拉入的版本覆盖。

我在 Google Colab 中遵循基本相同的步骤，但我的 tensorflow nightly 没有在那里被覆盖，所以我不确定我错过了什么......

这是我的Dockerfile：

FROM tensorflow/tensorflow:nightly-gpu-jupyter

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

RUN apt-get install -y \
        curl \
        git \
        less \
        zip

RUN curl -L -O https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip && unzip protoc-3.11.4-linux-x86_64.zip

RUN cp bin/protoc /usr/local/bin

RUN git clone --depth 1 https://github.com/tensorflow/models
RUN cd models/research && \
        protoc object_detection/protos/*.proto --python_out=. && \
        cp object_detection/packages/tf2/setup.py . && \
        python -m pip install .

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

我正在构建的：

docker pull tensorflow/tensorflow:nightly-gpu-jupyter
docker build --no-cache . -f models-tf-nightly.Dockerfile -t tf-nightly-models

第一个print()显示：

Tensorflow version: 2.5.0-dev20201129

但第二个显示：

Tensorflow version: 2.3.1

---

在 Google Colab 中，我执行的步骤基本相同：

# Install the Object Detection API
%%bash
pip install tf-nightly-gpu
[[ -d models ]] || git clone --depth 1 https://github.com/tensorflow/models
cd models/research/
protoc object_detection/protos/*.proto --python_out=.
cp object_detection/packages/tf2/setup.py .
python -m pip install .

之后

import tensorflow as tf
print(tf.__version__)

打印2.5.0-dev20201201

所以不知何故，我的 Google Colab 步骤保留了我的夜间 Tensorflow 安装，而在 Docker 上它被 2.3.0 覆盖。

Answer 1

如果您在安装对象检测包之前查看pip list，您会看到tf-nightly-gpu 已安装但tensorflow 未安装。当您安装对象检测包时，tensorflow 包将作为依赖项引入。 pip 认为没有安装，所以安装了。

解决此问题的一种方法是欺骗 pip install 认为已安装 tensorflow 包。可以通过符号链接dist-packages 中的tf_nightly_gpu-VERSION.dist-info 目录来做到这一点。我在下面的 Dockerfile 中添加了执行此操作的行。在这篇文章的底部，我还包含了一个 Dockerfile，它实现了一些最佳实践来最小化图像大小。

FROM tensorflow/tensorflow:nightly-gpu-jupyter

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

RUN apt-get install -y \
        curl \
        git \
        less \
        zip

# Trick pip into thinking that the 'tensorflow' package is installed.
# Installing `object_detection` attempts to install the 'tensorflow' package.
# Name the symlink with the suffix from tf_nightly_gpu.
WORKDIR /usr/local/lib/python3.6/dist-packages
RUN ln -s tf_nightly_gpu-* tensorflow-$(ls -d1 tf_nightly_gpu* | sed 's/tf_nightly_gpu-\(.*\)/\1/')

WORKDIR /tf
RUN curl -L -O https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip && unzip protoc-3.11.4-linux-x86_64.zip

RUN cp bin/protoc /usr/local/bin

RUN git clone --depth 1 https://github.com/tensorflow/models
RUN cd models/research && \
        protoc object_detection/protos/*.proto --python_out=. && \
        cp object_detection/packages/tf2/setup.py . && \
        python -m pip install .

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

---

这是一个 Dockerfile，它会生成一个稍微小一点的镜像（0.22 GB 未压缩）。值得注意的变化是清除apt 列表并在pip install 中使用--no-cache-dir。

FROM tensorflow/tensorflow:nightly-gpu-jupyter

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"

RUN apt-get install -y --no-install-recommends \
        ca-certificates \
        curl \
        git \
        less \
        zip && \
    rm -rf /var/lib/apt/lists/*

# Trick pip into thinking that the 'tensorflow' package is installed.
# Installing `object_detection` attempts to install the 'tensorflow' package.
# Name the symlink with the suffix from tf_nightly_gpu.
WORKDIR /usr/local/lib/python3.6/dist-packages
RUN ln -s tf_nightly_gpu-* tensorflow-$(ls -d1 tf_nightly_gpu* | sed 's/tf_nightly_gpu-\(.*\)/\1/')

WORKDIR /tf
RUN curl -L -O https://github.com/protocolbuffers/protobuf/releases/download/v3.11.4/protoc-3.11.4-linux-x86_64.zip && \
    unzip protoc-3.11.4-linux-x86_64.zip && \
    cp bin/protoc /usr/local/bin && \
    rm -r protoc-3.11.4-linux-x86_64.zip bin/

# Upgrade pip.
RUN python -m pip install --no-cache-dir --upgrade pip

RUN git clone --depth 1 https://github.com/tensorflow/models
WORKDIR models/research
RUN protoc object_detection/protos/*.proto --python_out=. && \
    cp object_detection/packages/tf2/setup.py . && \
    python -m pip install  --no-cache-dir .

RUN python -c "import tensorflow as tf; print(f'Tensorflow version: {tf.__version__}')"