【问题标题】:Running "docker-compose run" hangs when it's run in cron在 cron 中运行时,运行“docker-compose run”会挂起
【发布时间】:2021-05-30 02:54:15
【问题描述】:

环境

▶ unname -a
Linux ubuntu 5.4.0-66-generic #74-Ubuntu SMP Wed Jan 27 22:54:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

▶ docker -v
Docker 版本 20.10.4,构建 d3cb89e

▶ docker-compose -v
docker-compose 版本 1.28.4,构建 cabd5cfb

▶ lsb_release -a
没有可用的 LSB 模块。
分销商 ID:Ubuntu
说明:Ubuntu 20.04.2 LTS
发布时间:20.04
代号:焦点

问题

我一直在使用 ubuntu 18.04 版,我的 docker 和 docker-compose 运行良好。 我没有定期更新服务器,但就在两天前,我做了sudo apt-get update && apt-get upgrade。然后一切都开始变得奇怪了。

我曾经像这样从主机的 cron 调用 docker-compose 服务。

0 * * * * cd /project/folder && docker-compose run --rm service_name

但是,在更新升级之后,这些 cron 作业开始挂起。

  • 当我检查top 命令时,我可以看到进程悬空
  • 我可以看到容器程序(命令)甚至没有启动

因此,每次 cron 作业尝试调用这些服务时,进程都会堆积起来。

我做过的事情

我卸载了 docker、docker-compose、重新启动并再次重新安装它们,但没有帮助。我什至将操作系统从 18.04 升级到 20.04,但也没有用。

https://docs.docker.com/engine/install/ubuntu/#install-using-the-repository
https://docs.docker.com/compose/install/

do-release-upgrade # to upgrade from v18 to v20

为了进一步调查,我像这样添加了--verbose 并记录了它。

0 * * * * cd /project/folder && docker-compose run --rm --verbose service_name >> /project/folder/cron.log 2>&1

它似乎挂在/usr/bin/docker start --attach --interactive 上,实际上什么也没发生。

当我从终端运行相同的命令cd /project/folder && docker-compose run --rm --verbose service_name >> /project/folder/cron.log 2>&1 时,它会按预期运行。

我像下面这样简单,但是通过 cron 运行它仍然挂在中间。

有人知道是什么原因造成的吗?

docker-compose.yml

services:

  test:
    build:
      context: ./test
    command: ash -c "date"

Dockerfile

FROM alpine:latest

日志

compose.config.config.find: Using configuration files: ./docker-compose.yml
compose.cli.docker_client.get_client: docker-compose version 1.28.4, build cabd5cfb
docker-py version: 4.4.3
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019
compose.cli.docker_client.get_client: Docker base_url: http+docker://localhost
compose.cli.docker_client.get_client: Docker version: Platform={'Name': 'Docker Engine - Community'}, Components=[{'Name': 'Engine', 'Version': '20.10.4', 'Details': {'ApiVersion': '1.41', 'Arch': 'amd64', 'BuildTime': '2021-02-25T07:03:23.000000000+00:00', 'Experimental': 'false', 'GitCommit': '363e9a8', 'GoVersion': 'go1.13.15', 'KernelVersion': '5.4.0-66-generic', 'MinAPIVersion': '1.12', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': '1.4.3', 'Details': {'GitCommit': '269548fa27e0089a8b8278fc4fc781d7f65a939b'}}, {'Name': 'runc', 'Version': '1.0.0-rc92', 'Details': {'GitCommit': 'ff819c7e9184c13b7c2607fe6c30ae19403a7aff'}}, {'Name': 'docker-init', 'Version': '0.19.0', 'Details': {'GitCommit': 'de40ad0'}}], Version=20.10.4, ApiVersion=1.41, MinAPIVersion=1.12, GitCommit=363e9a8, GoVersion=go1.13.15, Os=linux, Arch=amd64, KernelVersion=5.4.0-66-generic, BuildTime=2021-02-25T07:03:23.000000000+00:00
compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('aaa_default')
compose.cli.verbose_proxy.proxy_callable: docker inspect_network -> {'Attachable': True,
 'ConfigFrom': {'Network': ''},
 'ConfigOnly': False,
 'Containers': {},
 'Created': '2020-07-19T12:11:28.401034027+09:00',
 'Driver': 'bridge',
 'EnableIPv6': False,
 'IPAM': {'Config': [{'Gateway': '192.168.80.1', 'Subnet': '192.168.80.0/20'}],
          'Driver': 'default',
          'Options': None},
...
compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('aaa_default')
compose.cli.verbose_proxy.proxy_callable: docker inspect_network -> {'Attachable': True,
 'ConfigFrom': {'Network': ''},
 'ConfigOnly': False,
 'Containers': {},
 'Created': '2020-07-19T12:11:28.401034027+09:00',
 'Driver': 'bridge',
 'EnableIPv6': False,
 'IPAM': {'Config': [{'Gateway': '192.168.80.1', 'Subnet': '192.168.80.0/20'}],
          'Driver': 'default',
          'Options': None},
...
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('aaa_test')
compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64',
 'Author': '',
 'Comment': '',
 'Config': {'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['/bin/sh'],
            'Domainname': '',
            'Entrypoint': None,
            'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'],
...
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.parallel.feed_queue: Pending: {<Service: test>}
compose.parallel.feed_queue: Starting producer thread for <Service: test>
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=True, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
Creating aaa_test_run ... 
compose.parallel.feed_queue: Pending: {ServiceName(project='aaa', service='test', number=1)}
compose.parallel.feed_queue: Starting producer thread for ServiceName(project='aaa', service='test', number=1)
compose.cli.verbose_proxy.proxy_callable: docker inspect_image <- ('aaa_test')
compose.cli.verbose_proxy.proxy_callable: docker inspect_image -> {'Architecture': 'amd64',
 'Author': '',
 'Comment': '',
 'Config': {'AttachStderr': False,
            'AttachStdin': False,
            'AttachStdout': False,
            'Cmd': ['/bin/sh'],
            'Domainname': '',
            'Entrypoint': None,
            'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'],
...
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker containers <- (all=False, filters={'label': ['com.docker.compose.project=aaa', 'com.docker.compose.service=test', 'com.docker.compose.oneoff=False']})
compose.cli.verbose_proxy.proxy_callable: docker containers -> (list with 0 items)
compose.cli.verbose_proxy.proxy_callable: docker create_host_config <- (links=[], port_bindings={}, binds=[], volumes_from=[], privileged=False, network_mode='aaa_default', devices=None, device_requests=None, dns=None, dns_opt=None, dns_search=None, restart_policy=None, runtime=None, cap_add=None, cap_drop=None, mem_limit=None, mem_reservation=None, memswap_limit=None, ulimits=None, log_config={'Type': '', 'Config': {}}, extra_hosts=None, read_only=None, pid_mode=None, security_opt=None, ipc_mode=None, cgroup_parent=None, cpu_quota=None, shm_size=None, sysctls=None, pids_limit=None, tmpfs=None, oom_kill_disable=None, oom_score_adj=None, mem_swappiness=None, group_add=None, userns_mode=None, init=None, init_path=None, isolation=None, cpu_count=None, cpu_percent=None, nano_cpus=None, volume_driver=None, cpuset_cpus=None, cpu_shares=None, storage_opt=None, blkio_weight=None, blkio_weight_device=None, device_read_bps=None, device_read_iops=None, device_write_bps=None, device_write_iops=None, mounts=None, device_cgroup_rules=None, cpu_period=None, cpu_rt_period=None, cpu_rt_runtime=None)
compose.cli.verbose_proxy.proxy_callable: docker create_host_config -> {'Binds': [],
 'Links': [],
 'LogConfig': {'Config': {}, 'Type': ''},
 'NetworkMode': 'aaa_default',
 'PortBindings': {},
 'VolumesFrom': []}
compose.cli.verbose_proxy.proxy_callable: docker create_container <- (command='ash -c "date"', volumes={}, tty=False, stdin_open=True, detach=False, ports=[], name='aaa_test_run_b01e0a8c11e3', environment=[], labels={'com.docker.compose.project': 'aaa', 'com.docker.compose.service': 'test', 'com.docker.compose.oneoff': 'True', 'com.docker.compose.project.working_dir': '/home/ac/projects/playground/aaa', 'com.docker.compose.project.config_files': 'docker-compose.yml', 'com.docker.compose.slug': 'b01e0a8c11e319685038ac27761c14b08d5fa025dee67abab9da546afadc28f', 'com.docker.compose.version': '1.28.4'}, image='aaa_test', host_config={'NetworkMode': 'aaa_default', 'VolumesFrom': [], 'Binds': [], 'PortBindings': {}, 'Links': [], 'LogConfig': {'Type': '', 'Config': {}}}, networking_config={'EndpointsConfig': {'aaa_default': {'Aliases': ['test'], 'IPAMConfig': {}}}})
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.cli.verbose_proxy.proxy_callable: docker create_container -> {'Id': 'c743a18406c5ac561f808086b84a8bf867c89f0b07526ca0c60b17fa527d6b1b',
 'Warnings': []}
compose.cli.verbose_proxy.proxy_callable: docker inspect_container <- ('c743a18406c5ac561f808086b84a8bf867c89f0b07526ca0c60b17fa527d6b1b')
compose.cli.verbose_proxy.proxy_callable: docker inspect_container -> {'AppArmorProfile': '',
 'Args': ['-c', 'date'],
 'Config': {'AttachStderr': True,
            'AttachStdin': True,
            'AttachStdout': True,
            'Cmd': ['ash', '-c', 'date'],
            'Domainname': '',
            'Entrypoint': None,
            'Env': ['PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'],
            'Hostname': 'c743a18406c5',
...
compose.parallel.parallel_execute_iter: Finished processing: ServiceName(project='aaa', service='test', number=1)
Creating aaa_test_run ... done
compose.parallel.feed_queue: Pending: set()
compose.parallel.parallel_execute_iter: Finished processing: <Service: test>
compose.parallel.feed_queue: Pending: set()
compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network <- ('c743a18406c5ac561f808086b84a8bf867c89f0b07526ca0c60b17fa527d6b1b', 'aaa_default')
compose.cli.verbose_proxy.proxy_callable: docker disconnect_container_from_network -> None
compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network <- ('c743a18406c5ac561f808086b84a8bf867c89f0b07526ca0c60b17fa527d6b1b', 'aaa_default', aliases=[], ipv4_address=None, ipv6_address=None, links=[], link_local_ips=None)
compose.cli.verbose_proxy.proxy_callable: docker connect_container_to_network -> None
compose.cli.main.call_docker: /usr/bin/docker start --attach --interactive c743a18406c5ac561f808086b84a8bf867c89f0b07526ca0c60b17fa527d6b1b

【问题讨论】:

    标签: docker ubuntu docker-compose cron


    【解决方案1】:

    更新:这应该在 Docker 20.10.5 中修复,因此升级可能比降级更可取!

    如果您降级到 Docker 20.10.3,该问题应该会得到解决。我相信这是此版本中引入的一个错误,如下所述:https://github.com/moby/moby/issues/42093

    讨厌的问题!希望尽快发布修复程序...

    如果您无法选择降级,您可以尝试按照here 的讨论设置COMPOSE_INTERACTIVE_NO_CLI environment variable(设置为1)——例如前置:

    COMPOSE_INTERACTIVE_NO_CLI=1
    

    到您的 crontab 配置中,以便通过 cron 启动的 docker-compose 任务应该绕过此错误作为短期解决方法

    【讨论】:

    • 我把它们都卸载了,然后sudo apt-get install docker-ce=5:20.10.3~3-0~ubuntu-focal docker-ce-cli=5:20.10.3~3-0~ubuntu-focal containerd.io 降级了。它就像一个魅力!谢谢
    • 不用担心@kukrt,如果您想继续使用最新版本,只需在我的答案中添加一个解决方法
    • 哇...我在过去一个小时内正在调查这个问题,但无法弄清楚 我做了什么来解决这个问题! 。非常感谢您的解决方法! +1
    猜你喜欢
    • 2019-04-21
    • 2022-08-21
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2021-06-09
    • 2018-03-27
    • 2018-05-11
    • 1970-01-01
    相关资源
    最近更新 更多