在新项目中重用 Django Celery 应用程序，找不到其他项目模块或 celery_app答案

【问题标题】：Reusing Django Celery app in new project, fails to find other project modules or celery_app在新项目中重用 Django Celery 应用程序，找不到其他项目模块或 celery_app
【发布时间】：2018-09-27 19:19:25
【问题描述】：

我有一个使用 celery 执行后台文件传输和其他数据收集任务（闪烁）的工作应用程序，我想在我的新项目中使用这个应用程序的一部分。我已经尝试了几乎所有我能想到的配置排列，但问题仍然存在。从这里开始是运行时的错误转储：

服务 celery 启动

 Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3.6/site-packages/celery/__main__.py", line 54, in <module>
    main()
  File "/usr/lib/python3.6/site-packages/celery/__main__.py", line 30, in main
    main()
  File "/usr/lib/python3.6/site-packages/celery/bin/celery.py", line 81, in main
    cmd.execute_from_commandline(argv)
  File "/usr/lib/python3.6/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/usr/lib/python3.6/site-packages/celery/bin/base.py", line 309, in execute_from_commandline
    argv = self.setup_app_from_commandline(argv)
  File "/usr/lib/python3.6/site-packages/celery/bin/base.py", line 469, in setup_app_from_commandline
    self.app = self.find_app(app)
  File "/usr/lib/python3.6/site-packages/celery/bin/base.py", line 489, in find_app
    return find_app(app, symbol_by_name=self.symbol_by_name)
  File "/usr/lib/python3.6/site-packages/celery/app/utils.py", line 235, in find_app
    sym = symbol_by_name(app, imp=imp)
  File "/usr/lib/python3.6/site-packages/celery/bin/base.py", line 492, in symbol_by_name
    return symbol_by_name(name, imp=imp)
  File "/usr/lib/python3.6/site-packages/kombu/utils/__init__.py", line 96, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/usr/lib/python3.6/site-packages/celery/utils/imports.py", line 101, in import_from_cwd
    return imp(module, package=package)
  File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/cloudscheduler/web_frontend/cloudscheduler/glintwebui/celery_app.py", line 26, in <module>
    django.setup()
  File "/usr/lib64/python3.6/site-packages/django/__init__.py", line 22, in setup
    configure_logging(settings.LOGGING_CONFIG, settings.LOGGING)
  File "/usr/lib64/python3.6/site-packages/django/conf/__init__.py", line 56, in __getattr__
    self._setup(name)
  File "/usr/lib64/python3.6/site-packages/django/conf/__init__.py", line 41, in _setup
    self._wrapped = Settings(settings_module)
  File "/usr/lib64/python3.6/site-packages/django/conf/__init__.py", line 110, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
  File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 941, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'cloudscheduler_web'
    > w4@csv2.heprc.uvic.ca: * Child terminated with errorcode 1
FAILED

我的服务文件如下所示：

[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=celery
Group=celery
EnvironmentFile=-/etc/cloudscheduler/celery
WorkingDirectory=/opt/cloudscheduler/web_frontend/cloudscheduler/
ExecStart=${CELERY_BIN} multi start $CELERYD_NODES -A \
    $CELERY_APP -logfile=${CELERYD_LOG_FILE} \
    --pidfile=${CELERYD_PID_FILE} $CELERYD_OPTS
ExecStop=${CELERY_BIN} multi stopwait $CELERYD_NODES \
    --pidfile=${CELERYD_PID_FILE}
ExecReload=${CELERY_BIN} multi restart $CELERYD_NODES -A \
    $CELERY_APP --pidfile=${CELERYD_PID_FILE} --logfile=${CELERYD_LOG_FILE} \
    --loglevel="${CELERYD_LOG_LEVEL}" $CELERYD_OPTS

[Install]
WantedBy=multi-user.target

环境文件：

# Name of nodes to start
# here we have a single node
#CELERYD_NODES="w1"
# or we could have three nodes:
CELERYD_NODES="w1 w2 w3 w4"

# We only want each worker to claim 1 job at a time, especially the worker dedicated to img collection
CELERYD_PREFETCH_MULTIPLIER=1

# Absolute path to "manage.py"
CELERY_BIN="/opt/cloudscheduler/web_frontend/cloudscheduler/manage.py"

# Where to chdir at start. This could be the root of a virtualenv.
CELERYD_CHDIR="/opt/cloudscheduler/web_frontend/cloudscheduler/glintwebui"

# App instance to use
# comment out this line if you don't use an app
CELERY_APP="celery_app"
# or fully qualified:
#CELERY_APP="glintwebui:celery_app"


# How to call manage.py
CELERYD_MULTI="celery multi"

# Extra command-line arguments to the worker
CELERYD_OPTS="-Q:w1 image_collection -Q:w2,w3,w4 celery -A celery_app --concurrency=1 -Ofair"

# %N will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%N.log"
CELERYD_PID_FILE="/var/run/celery/%N.pid"

请注意，当我将 CELERYD_CHDIR 更改为顶级项目目录时，它无法找到应用程序。当 chdir 设置为顶级项目时，我为完全限定的 CELERY_APP 尝试了几种不同的表达式，但它永远无法以这种方式找到 celery_app。

这里是 cloudscheduler_web 目录中主要 settings.pt 中的相关设置：

CELERY_ACCEPT_CONTENT = ['json']
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_TIMEZONE = 'Canada/Pacific'
CELERY_BROKER_URL = 'redis://localhost:6379/0'
CELERY_DEFAULT_QUEUE = 'celery'
CELERY_DEFAULT_EXCHANGE = "celery"
CELERY_QUEUES = {
    "celery": {"exchange": "celery"},
    "image_collection": {"exchange": "image_collection"},
}
CELERY_ROUTES = {
    'cloudscheduler.glintwebui.tasks.image_collection': {'queue': 'image_collection'},
}

最后是项目的目录结构：

cloudscheduler
│
├── cloudscheduler_web
│   ├── __init__.py
│   ├── settings.py
│   ├── urls.py
│   └── wsgi.py
├── csv2
│   ├── apps.py
│   ├── cloud_views.py
│   ├── config.py
│   ├── config.pyc
│   ├── csv2_web.yaml
│   ├── group_views.py
│   ├── __init__.py
│   ├── __init__.pyc
│   ├── models.py
│   ├── tests.py
│   ├── urls.py
│   ├── user_views.py
│   ├── views.py
│   └── view_utils.py
├── glintwebui
│   ├── admin.py
│   ├── apps.py
│   ├── celery_app.py
│   ├── config.py
│   ├── forms.py
│   ├── glint_api.py
│   ├── __init__.py
│   ├── models.py
│   ├── tests.py
│   ├── urls.py
│   ├── utils.py
│   ├── __version__.py
│   └── views.py
└── manage.py

我迷路的是 celery 如何找到 celery_app 或其他应用程序模块，但不能同时找到两者。我已经多次阅读了 celery 配置文档，但我觉得我仍然需要遗漏一些东西才能让它发挥作用。

如果有帮助，我可以发布原始应用程序的目录结构（闪烁）。作为奖励，这里是 celery_app.py 文件的一部分：

from __future__ import absolute_import, unicode_literals
import os
import time
import subprocess
import django
from django.conf import settings

from celery import Celery
from celery.utils.log import get_task_logger
#import glintwebui.config as config
import config

from glint_api import repo_connector

logger = get_task_logger(__name__)

# Indicate Celery to use the default Django settings module
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'cloudscheduler_web.settings')

django.setup()

app = Celery('celery_app', broker=config.celery_url, backend=config.celery_backend)
app.config_from_object('django.conf:settings')


@app.task(bind=True)
def debug_task(self):
    logger.debug('Request: {0!r}'.format(self.request))


@app.task(bind=True)
def image_collection(self):

尝试 py-D 的建议后出错：

Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/celery/app/utils.py", line 241, in find_app
        found = sym.app
    AttributeError: module 'glintwebui' has no attribute 'app'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
        "__main__", mod_spec)
      File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/usr/lib/python3.6/site-packages/celery/__main__.py", line 54, in <module>
        main()
      File "/usr/lib/python3.6/site-packages/celery/__main__.py", line 30, in main
        main()
      File "/usr/lib/python3.6/site-packages/celery/bin/celery.py", line 81, in main
        cmd.execute_from_commandline(argv)
      File "/usr/lib/python3.6/site-packages/celery/bin/celery.py", line 793, in execute_from_commandline
        super(CeleryCommand, self).execute_from_commandline(argv)))
      File "/usr/lib/python3.6/site-packages/celery/bin/base.py", line 309, in execute_from_commandline
        argv = self.setup_app_from_commandline(argv)
      File "/usr/lib/python3.6/site-packages/celery/bin/base.py", line 469, in setup_app_from_commandline
        self.app = self.find_app(app)
      File "/usr/lib/python3.6/site-packages/celery/bin/base.py", line 489, in find_app
        return find_app(app, symbol_by_name=self.symbol_by_name)
      File "/usr/lib/python3.6/site-packages/celery/app/utils.py", line 246, in find_app
        found = sym.celery
    AttributeError: module 'glintwebui' has no attribute 'celery'
        > w4@csv2.heprc.uvic.ca: * Child terminated with errorcode 1
    FAILED

【问题讨论】：

@Alasdair 正如我在问题中提到的那样，我已经尝试过了。一旦我更改为顶级目录 celery 似乎失去了对 celery_app 的跟踪。我的想法是问题可能是“CELERY_APP”，但我已经尝试了几乎所有我能想到的排列，结果是不变的：CELERY_APP="celery_app" CELERY_APP="glintwebui.celery_app:app" CELERY_APP="glintwebui. celery_app:celery_app" CELERY_APP="glintwebui:celery_app"
我已经更新了问题以在最后包含有问题的错误转储。

标签： django celery django-celery

【解决方案1】：

在您的情况下，您必须选择 app 标题 glintwebui，而不是 celery_app。只需尝试使用：

app = Celery('glintwebui', broker=config.celery_url, backend=config.celery_backend)

【讨论】：

Hrm，它似乎找到了 glintwebui 模块，但似乎仍然没有找到应用程序定义所在的 celery_app.py 文件。我已将新错误添加到问题的底部。我不确定我需要做什么才能让 find_app 在 celery_app.py 文件中查找。
看看它是否对你有用..stackoverflow.com/questions/43032592/…
查看了那里的所有内容，但没有帮助。在做了很多挖掘之后，我认为整个事情都是版本不匹配的问题。我使用 python 2 开发了原始应用程序，而新应用程序使用的是 python3，所以我想我可能只需要从头开始。（也从 celery 4.0.2 到 celery 4.10）

【解决方案2】：

尝试了一百万件事后，我终于发现我的 init.py 文件缺少 celery 应用程序导入。我想只要你的代码兼容 python2-3，那么你就可以在新的 python3 项目中重用它。

【讨论】：