通过任务执行 db 查询时出现 django 错误的 celery：在线程中创建的 DatabaseWrapper 对象只能在同一线程中使用答案

【问题标题】：celery with django error when doing a db query via task: DatabaseWrapper objects created in a thread can only be used in that same thread通过任务执行 db 查询时出现 django 错误的 celery：在线程中创建的 DatabaseWrapper 对象只能在同一线程中使用
【发布时间】：2020-05-22 08:21:54
【问题描述】：

堆栈：

Django 3.0.2 蟒蛇 3.8.1 芹菜 4.4.0 redis 3.2.0

启动 celery 的命令：celery -A app_project worker -l info

我正在使用 celery 在我的 django 项目中运行后台任务。在开发机器上它运行完美，没有任何错误。开发和生产都在同一个堆栈上运行；我已经手动检查并匹配了它们。然而，我仍然面临这个问题。

问题：尝试对数据库进行查询时，Celery 抛出以下错误。

Traceback (most recent call last):
  File "/webapps/app/.virtualenvs/base38/local/lib/python3.8/site-packages/celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/webapps/app/.virtualenvs/base38/local/lib/python3.8/site-packages/celery/app/trace.py", line 650, in __protected_call__
    return self.run(*args, **kwargs)
  File "/webapps/app/backend/app/tasks.py", line 22, in task_assign_photo_to_dish
    if dishq.exists():
  File "/webapps/app/.virtualenvs/base38/local/lib/python3.8/site-packages/django/db/models/query.py", line 777, in exists
    return self.query.has_results(using=self.db)
  File "/webapps/app/.virtualenvs/base38/local/lib/python3.8/site-packages/django/db/models/sql/query.py", line 537, in has_results
    return compiler.has_results()
  File "/webapps/app/.virtualenvs/base38/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1114, in has_results
    return bool(self.execute_sql(SINGLE))
  File "/webapps/app/.virtualenvs/base38/local/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1142, in execute_sql
    cursor = self.connection.cursor()
  File "/webapps/app/.virtualenvs/base38/local/lib/python3.8/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
  File "/webapps/app/.virtualenvs/base38/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 260, in cursor
    return self._cursor()
  File "/webapps/app/.virtualenvs/base38/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 238, in _cursor
    return self._prepare_cursor(self.create_cursor(name))
  File "/webapps/app/.virtualenvs/base38/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 228, in _prepare_cursor
    self.validate_thread_sharing()
  File "/webapps/app/.virtualenvs/base38/local/lib/python3.8/site-packages/django/db/backends/base/base.py", line 553, in validate_thread_sharing
    raise DatabaseError(
django.db.utils.DatabaseError: DatabaseWrapper objects created in a thread can only be used in that same thread. The object with alias 'default' was created in thread id 139987604641728 and this is thread id 139987115662208.
[2020-02-05 23:04:51,621: ERROR/MainProcess] Signal handler <bound method DjangoWorkerFixup.on_task_postrun of <celery.fixups.django.DjangoWorkerFixup object at 0x7f515c6ebd30>> raised: DatabaseError("DatabaseWrapper objects created in a thread can only be used in that same thread. The object with alias 'default' was created in thread id 139987604641728 and this is thread id 139987115662208.")

更新

@shared_task()
def task_assign_photo_to_dish(id):
    dishq = Dish.objects.filter(pk=id)
    if dishq.exists():
        dish = dishq[0]
        DishFile.objects.filter(dish=dish).delete()
        pq  = Post.objects.filter(dish=dish).annotate(c=Count('liked_by')).order_by('-c')
        for p in pq[:5]:
            instance = DishFile.objects.create(dish=dish, file=p.post_files.all().order_by('?')[0].file)
    return "Done"

【问题讨论】：

你能把你的芹菜任务代码贴出来吗？
@NafeesAnwar 我添加了任务代码。错误发生在执行数据库查询的行上。
我有同样的问题，据我所知，它连接到使用 eventlet。降级到 Django 2 有所帮助。另见github.com/celery/celery/issues/5924

标签： django python-3.x redis celery

【解决方案1】：

你在 Windows 上吗？

我遇到了同样的问题，但是一旦我部署到 heroku（在 Linux 上运行），它就可以工作了。

【讨论】：

【解决方案2】：

最佳实践是在视图中构建一个字典并将其作为参数传递给任务

【讨论】：

【解决方案3】：

使用此命令，它应该可以在 Windows 上运行

celery -A your_project.celery worker --loglevel=info --pool=solo

【讨论】：

【解决方案4】：

在 celery 命令中使用线程池

celery -A uploader worker  -l warning -n p80 --pool=threads --concurrency=100

仅供参考-如果您不指定任何池，默认情况下 celery 将采用 prefork

【讨论】：