从源代码将 Airflow 部署到 Docker

Question

我已将 Airflow repo https://github.com/apache/airflow 克隆到本地并尝试构建以便稍后在 docker 中运行到 Azure。

我不确定这个过程。我收到一个错误

错误 [airflow-build-image 11/12] 如果 [[ apache-airflow == "." 则运行]];然后 bash /scripts/docker/compile_www_assets.sh;菲;如果 [[ 2.5s**

---------------------- 修改问题------------------ 在 windows 上安装 ubuntu (WSL2) 并克隆气流 repo 并开始构建 docker 映像。我在 ma​​in 分支上。

docker build . -f Dockerfile --tag airflow-image-20210913:0.0.1

控制台错误：

    admin@Laptop-1:~/airflow$ docker build . -f Dockerfile --tag airflow-image-20210913:0.0.1
    [+] Building 41.4s (19/26)
    => [internal] load build definition from Dockerfile                                                               0.2s
    => => transferring dockerfile: 21.40kB                                                                            0.0s
    => [internal] load .dockerignore                                                                                  0.1s
    => => transferring context: 3.14kB                                                                                0.0s
    => [internal] load metadata for docker.io/library/python:3.6-slim-buster                                         24.4s
    => [internal] load build context                                                                                  0.1s
    => => transferring context: 40.64kB                                                                               0.0s
    => [airflow-build-image  1/12] FROM docker.io/library/python:3.6-slim-buster@sha256:760b66a8a90751b711bf20243c02  0.0s
    => CACHED [main  2/11] RUN apt-get update     && apt-get install -y --no-install-recommends            curl       0.0s
    => CACHED [main  3/11] RUN mkdir -pv /usr/share/man/man1     && mkdir -pv /usr/share/man/man7     && export       0.0s
    => CACHED [main  4/11] COPY scripts/docker/install_mysql.sh scripts/docker/install_pip_version.sh /scripts/docke  0.0s
    => CACHED [main  5/11] RUN chmod a+x /scripts/docker/install_mysql.sh &&     /scripts/docker/install_mysql.sh pr  0.0s
    => CACHED [airflow-build-image  2/12] RUN apt-get update     && apt-get install -y --no-install-recommends        0.0s
    => CACHED [airflow-build-image  3/12] RUN mkdir -pv /usr/share/man/man1     && mkdir -pv /usr/share/man/man7      0.0s
    => CACHED [airflow-build-image  4/12] COPY scripts/docker/*.sh /scripts/docker/                                   0.0s
    => CACHED [airflow-build-image  5/12] RUN bash ./scripts/docker/install_mysql.sh dev                              0.0s
    => [airflow-build-image  6/12] COPY docker-context-files /docker-context-files                                    0.1s
    => [airflow-build-image  7/12] RUN if [[ -f /docker-context-files/.pypirc ]]; then         cp /docker-context-fi  0.8s
    => [airflow-build-image  8/12] RUN bash /scripts/docker/install_pip_version.sh;     if [[ false == "true" &&     12.0s
    => [airflow-build-image  9/12] COPY empty /empty                                                                  0.1s
    => [airflow-build-image 10/12] WORKDIR /opt/airflow                                                               0.2s
    => ERROR [airflow-build-image 11/12] RUN if [[ apache-airflow == "." ]]; then         bash /scripts/docker/compi  3.2s
    ------
    > [airflow-build-image 11/12] RUN if [[ apache-airflow == "." ]]; then         bash /scripts/docker/compile_www_assets.sh;     fi;     if [[  == "true" ]]; then         bash /scripts/docker/install_from_docker_context_files.sh;     elif [[ true == "true" ]]; then         bash /scripts/docker/install_airflow.sh;     fi;     if [[ -n "" ]]; then         bash /scripts/docker/install_additional_dependencies.sh;     fi;     find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r || true ;     find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r || true ;     find /root/.local -executable -print0 | xargs --null chmod g+x;     find /root/.local -print0 | xargs --null chmod g+rw:
    #19 0.669 + [[ apache-airflow == \. ]]
    #19 0.669 + [[ '' == \t\r\u\e ]]
    #19 0.669 + [[ true == \t\r\u\e ]]
    #19 0.669 + bash /scripts/docker/install_airflow.sh
    #19 0.675 + common::get_airflow_version_specification
    #19 0.675 + [[ -z '' ]]
    #19 0.675 + [[ -n 2.2.0.dev0 ]]
    #19 0.675 + [[ apache-airflow != \. ]]
    #19 0.676 + AIRFLOW_VERSION_SPECIFICATION===2.2.0.dev0
    #19 0.676 + common::override_pip_version_if_needed
    #19 0.676 + [[ -n 2.2.0.dev0 ]]
    #19 0.676 + [[ 2.2.0.dev0 =~ ^2\.0.* ]]
    #19 0.676 + [[ 2.2.0.dev0 =~ ^1\.* ]]
    #19 0.676 + common::get_constraints_location
    #19 0.676 + [[ -z '' ]]
    #19 0.676 + [[ 2.2.0.dev0 =~ v?2.* ]]
    #19 0.676 + AIRFLOW_CONSTRAINTS_REFERENCE=constraints-2.2.0.dev0
    #19 0.676 + [[ -z '' ]]
    #19 0.676 + local constraints_base=https://raw.githubusercontent.com/apache/airflow/constraints-2.2.0.dev0
    #19 0.676 + local python_version
    #19 0.678 ++ cut -d ' ' -f 2
    #19 0.678 ++ python --version
    #19 0.682 ++ cut -d . -f 1-2
    #19 0.686 + python_version=3.6
    #19 0.689
    #19 0.689 Installing all packages with constraints and upgrade if needed
    #19 0.689
    #19 0.690 + AIRFLOW_CONSTRAINTS_LOCATION=https://raw.githubusercontent.com/apache/airflow/constraints-2.2.0.dev0/constraints-3.6.txt
    #19 0.690 + install_airflow
    #19 0.690 + [[ apache-airflow != \. ]]
    #19 0.690 + [[ '' == \-\-\e\d\i\t\a\b\l\e ]]
    #19 0.690 + [[ true != \t\r\u\e ]]
    #19 0.690 + [[ false != \f\a\l\s\e ]]
    #19 0.690 + echo
    #19 0.690 + echo Installing all packages with constraints and upgrade if needed
    #19 0.690 + echo
    #19 0.690 + pip install --user 'apache-airflow[async,amazon,celery,cncf.kubernetes,docker,dask,elasticsearch,ftp,grpc,hashicorp,http,ldap,google,google_auth,microsoft.azure,mysql,pandas,postgres,redis,sendgrid,sftp,slack,ssh,statsd,virtualenv]==2.2.0.dev0' --constraint https://raw.githubusercontent.com/apache/airflow/constraints-2.2.0.dev0/constraints-3.6.txt
    #19 2.227 ERROR: 404 Client Error: Not Found for url: https://raw.githubusercontent.com/apache/airflow/constraints-2.2.0.dev0/constraints-3.6.txt
    ------
    executor failed running [/bin/bash -o pipefail -e -u -x -c if [[ ${AIRFLOW_INSTALLATION_METHOD} == "." ]]; then         bash /scripts/docker/compile_www_assets.sh;     fi;     if [[ ${INSTALL_FROM_DOCKER_CONTEXT_FILES} == "true" ]]; then         bash /scripts/docker/install_from_docker_context_files.sh;     elif [[ ${INSTALL_FROM_PYPI} == "true" ]]; then         bash /scripts/docker/install_airflow.sh;     fi;     if [[ -n "${ADDITIONAL_PYTHON_DEPS}" ]]; then         bash /scripts/docker/install_additional_dependencies.sh;     fi;     find /root/.local/ -name '*.pyc' -print0 | xargs -0 rm -r || true ;     find /root/.local/ -type d -name '__pycache__' -print0 | xargs -0 rm -r || true ;     find /root/.local -executable -print0 | xargs --null chmod g+x;     find /root/.local -print0 | xargs --null chmod g+rw]: exit code: 1

--------------------- 编辑 -3 ----------------

我在 python 3.8 上并成功构建和安装，但在调用 airflow

时出错

Answer 1

您尚未显示完整的错误消息。您也没有发布您使用的命令、您拥有的操作系​​统以及您的文件系统/Docker 引擎是什么。所有这些都很重要。

真正的错误可能在下面。我认为问题在于您正在 Windows 上构建它并且您的系统不支持可执行标志。在 WSL2 或 Linux VM 上构建它，它应该可以正常工作。

如果您收到消息的其余部分，我什至可以修复它，以便它也可以在 Windows 上运行。因此，请随时更新它并在 Airflow Slack 上联系我，或者如果您转储了您拥有的整个错误消息，请在此处联系我。

顺便说一句。这里有一整章关于使用气流 dockerfile 构建自定义图像：https://airflow.apache.org/docs/docker-stack/build.html#customizing-the-image