如何使用多租户 (django-tenant-schemas) Django 应用程序设置 django-celery-beat。答案

【问题标题】：How to setup django-celery-beat with multi tenant (django-tenant-schemas) Django application.如何使用多租户 (django-tenant-schemas) Django 应用程序设置 django-celery-beat。
【发布时间】：2018-05-04 18:38:14
【问题描述】：

我正在尝试将 celery beat 与 celery 一起运行。在tenant_schemas_celery 的帮助下，Celery 可以很好地与我的多租户 Django 应用程序（使用 django-tenant-schemas）配合使用。但我无法运行 celery beat，因为所有与计划任务/定期任务相关的表都是特定于租户的，因此无法启动 celery beat。

错误跟踪：

celery@Amits-iMac.local v4.0.2 (latentcall)

Darwin-17.5.0-x86_64-i386-64bit 2018-05-04 18:20:22

[config]
.> app:         __main__:0x102ef9f28
.> transport:   amqp://guest:**@localhost:5672//
.> results:
.> concurrency: 4 (prefork)
.> task events: ON

[queues]
.> celery           exchange=celery(direct) key=celery


[tasks]
  . dataflow.tasks.launch_dataflow
  . datasession.tasks.launch_copy

[2018-05-04 18:20:23,035: INFO/Beat] beat: Starting...
[2018-05-04 18:20:23,166: ERROR/Beat] Process Beat
Traceback (most recent call last):
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/kombu/utils/objects.py", line 42, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'scheduler'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
psycopg2.ProgrammingError: relation "django_celery_beat_periodictask" does not exist
LINE 1: ...ango_celery_beat_periodictask"."description" FROM "django_ce...
                                                             ^


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/billiard/process.py", line 306, in _bootstrap
    self.run()
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/celery/beat.py", line 613, in run
    self.service.start(embedded_process=True)
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/celery/beat.py", line 528, in start
    humanize_seconds(self.scheduler.max_interval))
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/celery/beat.py", line 572, in scheduler
    return self.get_scheduler()
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/celery/beat.py", line 567, in get_scheduler
    lazy=lazy,
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django_celery_beat/schedulers.py", line 181, in __init__
    Scheduler.__init__(self, *args, **kwargs)
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/celery/beat.py", line 204, in __init__
    self.setup_schedule()
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django_celery_beat/schedulers.py", line 189, in setup_schedule
    self.install_default_entries(self.schedule)
  File "/Users/amit/thoughtanalytik/koolanch/src/koolanch/beat_schedulers.py", line 27, in schedule
    self._schedule = self.all_as_schedule()
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django_celery_beat/schedulers.py", line 195, in all_as_schedule
    for model in self.Model.objects.enabled():
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/models/query.py", line 250, in __iter__
    self._fetch_all()
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/models/query.py", line 1118, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 894, in execute_sql
    raise original_exception
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 884, in execute_sql
    cursor.execute(sql, params)
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/backends/utils.py", line 80, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: relation "django_celery_beat_periodictask" does not exist
LINE 1: ...ango_celery_beat_periodictask"."description" FROM "django_ce...
                                                             ^

[2018-05-04 18:20:23,219: WARNING/Beat] Process Beat:
[2018-05-04 18:20:23,220: WARNING/Beat] Traceback (most recent call last):
[2018-05-04 18:20:23,220: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/kombu/utils/objects.py", line 42, in __get__
    return obj.__dict__[self.__name__]
[2018-05-04 18:20:23,220: WARNING/Beat] KeyError: 'scheduler'
[2018-05-04 18:20:23,220: WARNING/Beat] During handling of the above exception, another exception occurred:
[2018-05-04 18:20:23,220: WARNING/Beat] Traceback (most recent call last):
[2018-05-04 18:20:23,221: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
[2018-05-04 18:20:23,221: WARNING/Beat] psycopg2.ProgrammingError: relation "django_celery_beat_periodictask" does not exist
LINE 1: ...ango_celery_beat_periodictask"."description" FROM "django_ce...
                                                             ^
[2018-05-04 18:20:23,221: WARNING/Beat] The above exception was the direct cause of the following exception:
[2018-05-04 18:20:23,221: WARNING/Beat] Traceback (most recent call last):
[2018-05-04 18:20:23,221: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/billiard/process.py", line 306, in _bootstrap
    self.run()
[2018-05-04 18:20:23,222: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/celery/beat.py", line 613, in run
    self.service.start(embedded_process=True)
[2018-05-04 18:20:23,222: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/celery/beat.py", line 528, in start
    humanize_seconds(self.scheduler.max_interval))
[2018-05-04 18:20:23,222: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
[2018-05-04 18:20:23,222: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/celery/beat.py", line 572, in scheduler
    return self.get_scheduler()
[2018-05-04 18:20:23,222: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/celery/beat.py", line 567, in get_scheduler
    lazy=lazy,
[2018-05-04 18:20:23,222: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django_celery_beat/schedulers.py", line 181, in __init__
    Scheduler.__init__(self, *args, **kwargs)
[2018-05-04 18:20:23,223: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/celery/beat.py", line 204, in __init__
    self.setup_schedule()
[2018-05-04 18:20:23,223: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django_celery_beat/schedulers.py", line 189, in setup_schedule
    self.install_default_entries(self.schedule)
[2018-05-04 18:20:23,223: WARNING/Beat] File "/Users/amit/thoughtanalytik/koolanch/src/koolanch/beat_schedulers.py", line 27, in schedule
    self._schedule = self.all_as_schedule()
[2018-05-04 18:20:23,223: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django_celery_beat/schedulers.py", line 195, in all_as_schedule
    for model in self.Model.objects.enabled():
[2018-05-04 18:20:23,223: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/models/query.py", line 250, in __iter__
    self._fetch_all()
[2018-05-04 18:20:23,224: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/models/query.py", line 1118, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
[2018-05-04 18:20:23,224: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch)
[2018-05-04 18:20:23,224: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 894, in execute_sql
    raise original_exception
[2018-05-04 18:20:23,224: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 884, in execute_sql
    cursor.execute(sql, params)
[2018-05-04 18:20:23,224: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/backends/utils.py", line 80, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
[2018-05-04 18:20:23,224: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
[2018-05-04 18:20:23,224: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
[2018-05-04 18:20:23,225: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
[2018-05-04 18:20:23,225: WARNING/Beat] File "/Users/amit/koolanch-dev/dev-1/lib/python3.6/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
[2018-05-04 18:20:23,225: WARNING/Beat] django.db.utils.ProgrammingError: relation "django_celery_beat_periodictask" does not exist
LINE 1: ...ango_celery_beat_periodictask"."description" FROM "django_ce...

提前致谢。

【问题讨论】：

标签： django celery

【解决方案1】：

Celery beat（使用 django_celery_beat）默认连接到公共模式并监视 django_celery_beat_periodictask 表的周期性任务。因此，我们必须在公共模式中拥有这些表。所以 django_celery_beat 应用程序应该是公共架构的一部分，所有周期性任务都必须在公共架构中创建。

with schema_context('public'):
            schedule, created = IntervalSchedule.objects.get_or_create(
                every=execution_interval,
                period=IntervalSchedule.MINUTES)

            pt_obj, created = PeriodicTask.objects.get_or_create(name=name,
                                                             task='tasks.task_name')
            pt_obj.interval = schedule
            pt_obj.enabled = False
            pt_obj.args = json.dumps([])
            pt_obj.kwargs = json.dumps({})
            pt_obj.save()

【讨论】：

【解决方案2】：

有一个名为tenant-schemas-celery的包可用here是django-tenant-schemas自己推荐的，上面提到了here。

【讨论】：