无法将 Celery 服务器连接到本地主机上的 RabbitMQ

Question

我使用 Celery 和 RabbitMQ 作为消息队列，每个都封装在自己的 Docker 映像中。当它们使用 Docker 中的--link 参数连接时，一切正常。我已经让这个设置工作了一段时间。我想将它们分开，以便它们在不同的主机上运行，​​因此我不能再为此使用--link 参数。当我尝试使用 AMQP 连接但不明白为什么时，我收到了 gaierror: [Errno -2] Name or service not known。

服务器只是使用 DockerHub 上的 rabbitmq 容器：

docker run --rm --name=qrabbit -p 5672:5672 rabbitmq

我可以 telnet 成功：

$ telnet 192.168.99.100 5672
Trying 192.168.99.100...
Connected to 192.168.99.100.
Escape character is '^]'.
abc
^D
AMQP    Connection closed by foreign host.
$

...所以我知道服务器正在运行。

我的客户是这样的：

import os

from logging import getLogger, StreamHandler, DEBUG
from serverlib import QueueServer, CeleryMonitor
from celery import Celery
from argparse import ArgumentParser

log = getLogger('server')
log.addHandler(StreamHandler())
log.setLevel(DEBUG)

broker_service_host = os.environ.get('MESSAGE_QUEUE_SERVICE_SERVICE_HOST')

broker = 'amqp://{0}'.format(broker_service_host)
host = ''
port = 8000
retry = 5

if __name__ == '__main__':
    log.info('connecting to {0}, {1}:{2}, retry={3}'.format(broker, host, port, retry))
    app = Celery(broker=broker)
    monitor = CeleryMonitor(app, retry=retry)
    server = QueueServer((host, port), app)
    monitor.start()
    try:
        log.info('listening on {0}:{1}'.format(host, port))
        server.serve_forever()
    except KeyboardInterrupt:
        log.info('shutdown requested')
    except BaseException as e:
        log.error(e)
    finally:
        monitor.shutdown()

我有点确定外部模块（QueueServer 和 CeleryMonitor）不是问题的一部分，因为当我执行以下操作时它可以正常运行：

 $ docker run --rm --name=qmaster -e "MESSAGE_QUEUE_SERVICE_SERVICE_HOST=localhost" --link qrabbit:rabbit -p 80:8000 render-task-master
connecting to amqp://localhost, :8000, retry=5
listening on :8000
^Cshutdown requested
$

...但如果我执行以下操作（没有--link 参数）：

$  docker run --rm --name=qmaster -e "MESSAGE_QUEUE_SERVICE_SERVICE_HOST=localhost" -p 80:8000 render-task-master
connecting to amqp://localhost, :8000, retry=5
listening on :8000
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/home/celery/serverlib/celerymonitor.py", line 68, in run
    '*': self.__state.event
  File "/usr/local/lib/python2.7/site-packages/celery/events/__init__.py", line 287, in __init__
    self.channel = maybe_channel(channel)
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 1054, in maybe_channel
    return channel.default_channel
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 756, in default_channel
    self.connection
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
    self._connection = self._establish_connection()
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
    conn = self.transport.establish_connection()
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
    conn = self.Connection(**opts)
  File "/usr/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
    self.transport = self.Transport(host, connect_timeout, ssl)
  File "/usr/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
    return create_transport(host, connect_timeout, ssl)
  File "/usr/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
    return TCPTransport(host, connect_timeout)
  File "/usr/local/lib/python2.7/site-packages/amqp/transport.py", line 75, in __init__
    socket.SOCK_STREAM, SOL_TCP):
gaierror: [Errno -2] Name or service not known

^Cshutdown requested
$

使用和不使用可能导致此错误的--link 参数有什么区别？

更新：

我已将其缩小为我创建的监视器类中的错误：

recv = self.app.events.Receiver(connection, handlers={
    'task-received': self.registerTask,
    'task-failed': self.retryTask,
    'task-succeeded': self.deregisterTask,
    # should process all events to have state up to date
    '*': self.__state.event 
})

当它被调用时，它会等待几秒钟（超时？）然后抛出异常。知道为什么这不喜欢指定为 amqp://localhost 的 amqp URL，但是当我使用 --link 参数时一切正常？

这是调用的整个方法，以获取更多上下文：

def run(self):
    log.info('run')
    self.__state = self.app.events.State()
    with self.app.connection() as connection:
        log.info('got a connection')
        recv = self.app.events.Receiver(connection, handlers={
            'task-received': self.registerTask,
            'task-failed': self.retryTask,
            'task-succeeded': self.deregisterTask,
            # should process all events to have state up to date
            '*': self.__state.event 
        })
        log.info('received receiver')
        # Capture until shutdown requested
        while not self.__shutdown:
            log.info('main run loop')
            try:
                recv.capture(limit=None, timeout=1, wakeup=True)
            except timeout:
                # timeout exception is fired when nothing occurs
                # during timeout. Just ignore it.
                pass

Answer 1

我发现了问题：我在容器运行的 Docker 环境中设置了CELERY_BROKER_URL，这导致后端尝试连接到不存在的主机名。一旦我取消设置变量，一切都在我的环境中正确连接。

$ docker inspect server
<... removed ...>
        "Env": [
        "MESSAGE_QUEUE_SERVICE_SERVICE_HOST=192.168.99.100",
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
        "LANG=C.UTF-8",
        "PYTHON_VERSION=2.7.10",
        "PYTHON_PIP_VERSION=7.1.2",
        "CELERY_VERSION=3.1.18",
        "CELERY_BROKER_URL=amqp://guest@rabbit"
    ],
<... removed ...>