如何在 Kubernetes 部署中使用 DockerOperator 运行任务？

Question

上下文

我正在运行 Airflow，并尝试使用 Airflow 的 DockerOperator 为 Docker 容器运行概念验证。我正在部署到 Kubernetes (EKS)，但还没有使用 Kubernetes Executor。鉴于我正在运行 pod，通过使用DockerOperator，我将在 Docker 中运行（据我所知）Docker。

每当我运行我的任务时，我都会收到错误消息：ERROR - Error while fetching server API version。该错误发生在 docker-compose 和 EKS (kubernetes) 上。

我现在的状态

这就是我的气流 Dockerfile 的样子：

FROM apache/airflow:1.10.14-python3.8
# Use airflow user for pip installs and other things.
USER root
# Copying Airflow requirements
USER airflow
COPY requirements.txt /tmp/requirements.txt
# Installing requirements. Using airflow user (docs: https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html)
RUN pip install --no-cache-dir -r /tmp/requirements.txt

这就是我尝试运行的 dag 的样子：

with DAG(
        dag_id='task',
        default_args=dict(
            start_date=days_ago(0)
        ),
        schedule_interval='@daily'
) as dag:

    task_1 = DockerOperator(
        dag=dag,
        task_id='docker_task',
        image='centos:latest',
        api_version="auto",
        docker_url='unix://var/run/docker.sock',
        command='/bin/sleep 30'
    )

这是我得到的错误的堆栈跟踪：

[2020-12-29 14:18:52,601] {taskinstance.py:1150} ERROR - Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 410, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/docker_operator.py", line 260, in execute
    self.cli = APIClient(
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

我尝试过的

1. 将套接字挂载到 docker compose /var/run/docker.sock:/var/run/docker.sock:ro

首先，这给了我一个新的错误：

ERROR - Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))

其次，我如何才能在 Kubernetes 上挂载？我猜这会使事情复杂化

1. 在我的容器中安装 docker 并尝试为气流用户授予权限，例如：

FROM apache/airflow:1.10.14-python3.8
# Use airflow user for pip installs and other things.
USER root
# Docker
RUN curl -sSL https://get.docker.com/ | sh  
# Usermod
RUN usermod -aG docker airflow
# Copying Airflow requirements
USER airflow
COPY requirements.txt /tmp/requirements.txt
# Installing requirements. Using airflow user (docs: https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html)
RUN pip install --no-cache-dir -r /tmp/requirements.txt

但这也没有用。

1. 将套接字安装到 DockerOperator 任务中，例如：

    task_1 = DockerOperator(
        dag=dag,
        task_id='docker_task',
        image='centos:latest',
        api_version="auto",
        docker_url='unix://var/run/docker.sock',
        command='/bin/sleep 30',
        volumes=['/var/run/docker.sock:/var/run/docker.sock:ro'],
    )

但这也没有效果

Answer 1

从评论中抄下来：

直接的问题可能是 docker 控制套接字文件由主机上的 root:docker 之类的东西所有，并且假设 Airflow 没有以 root 身份运行（它不应该），那么您需要专门设置它以与主机上的 docker 组相同的 gid 运行（容器内可能没有匹配的组，这可能会混淆其他东西）。

真正的答案是改用 KubernetesPodOperator。一旦您停止在底层主机上使用 Docker，DockerOperator 将根本无法工作，这将很快发生，因为 dockershim 已被弃用。