如何避免 django 中的“数据库已锁定”sqlite3 错误？

【问题标题】：How can I avoid "database is locked" sqlite3 errors in django?如何避免 django 中的“数据库已锁定”sqlite3 错误？
【发布时间】：2018-05-25 11:40:15
【问题描述】：

Django 1.11 版，sqlite3 3.11 版。

我正在使用 WAL 模式和长时间超时：

from django.apps import AppConfig
from django.db.backends.signals import connection_created


class SQLite3Config(AppConfig):
    name = 'sqlite3_config'

    def ready(self):
        connection_created.connect(configure_sqlite)


# noinspection PyUnusedLocal
def configure_sqlite(sender, connection, **_):
    if connection.vendor == 'sqlite':
        cursor = connection.cursor()
        cursor.execute('PRAGMA journal_mode=WAL;')
        cursor.execute('PRAGMA busy_timeout=5000;')

我想保留 sqlite3 而不是迁移到 mysql 或 postgres，因为该应用程序很小并且由用户安装在多个服务器上。

我相信 WAL 应该通过序列化它们来允许“并发”写入。当一起接收到小的突发（大约六次）时，观察到“数据库被锁定”问题。

我可以用线程在 shell 中重现问题。 django 模型方法只是简单地设置一个标志并保存模型：

def activate(self):
    self.activate = True
    self.save()

当我使用线程时，如果我同时启动几个尝试它的线程，我发现它会失败。没有等待，因此不涉及超时。错误发生在 5 秒繁忙超时过去之前（不到两秒）：

In [2]: [NGThread(notifier_group.id).start() for notifier_group in NotifierGroup.objects.all()[:2]]
Out[2]: [None, None]

In [3]: [NGThread(notifier_group.id).start() for notifier_group in NotifierGroup.objects.all()[:3]]
Out[3]: [None, None, None]

In [4]: [NGThread(notifier_group.id).start() for notifier_group in NotifierGroup.objects.all()[:4]]
Out[4]: [None, None, None, None]

In [5]: Exception in thread Thread-97:
Traceback (most recent call last):
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/home/paul/wk/cliosoft/sosadmin/scratch.py", line 41, in run
    toggle_active(notifier_group)
  File "/home/paul/wk/cliosoft/sosadmin/scratch.py", line 30, in toggle_active
    model.activate()
  File "/home/paul/wk/cliosoft/sosadmin/notifications/models/notifier_group.py", line 67, in activate
    self.save()
  File "/home/paul/wk/cliosoft/sosadmin/notifications/models/notifier_group.py", line 33, in save
    self.verify()
  File "/home/paul/wk/cliosoft/sosadmin/notifications/models/notifier_group.py", line 46, in verify
    self.create_notifier(base_spec, model_set, group_event_condition)
  File "/home/paul/wk/cliosoft/sosadmin/notifications/models/notifier_group.py", line 57, in create_notifier
    notifier.users = self.users.all()
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 534, in __set__
    manager.set(value)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 1004, in set
    self.add(*new_objs)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 931, in add
    self._add_items(self.source_field_name, self.target_field_name, *objs)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 1100, in _add_is
    for obj_id in new_ids
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/query.py", line 442, in bulk_create
    ids = self._batched_insert(objs_without_pk, fields, batch_size)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/query.py", line 1083, in _batched_insert
    self._insert(item, fields=fields, using=self.db)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/query.py", line 1060, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 1099, in execute_sql
    cursor.execute(sql, params)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/utils.py", line 80, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: database is locked

Exception in thread Thread-98:
Traceback (most recent call last):
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.OperationalError: database is locked

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/home/paul/wk/cliosoft/sosadmin/scratch.py", line 41, in run
    toggle_active(notifier_group)
  File "/home/paul/wk/cliosoft/sosadmin/scratch.py", line 28, in toggle_active
    model.deactivate()
  File "/home/paul/wk/cliosoft/sosadmin/notifications/models/notifier_group.py", line 72, in deactivate
    self.save()
  File "/home/paul/wk/cliosoft/sosadmin/notifications/models/notifier_group.py", line 33, in save
    self.verify()
  File "/home/paul/wk/cliosoft/sosadmin/notifications/models/notifier_group.py", line 46, in verify
    self.create_notifier(base_spec, model_set, group_event_condition)
  File "/home/paul/wk/cliosoft/sosadmin/notifications/models/notifier_group.py", line 57, in create_notifier
    notifier.users = self.users.all()
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 534, in __set__
    manager.set(value)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 1004, in set
    self.add(*new_objs)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 931, in add
    self._add_items(self.source_field_name, self.target_field_name, *objs)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/fields/related_descriptors.py", line 1100, in _add_is
    for obj_id in new_ids
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/query.py", line 442, in bulk_create
    ids = self._batched_insert(objs_without_pk, fields, batch_size)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/query.py", line 1083, in _batched_insert
    self._insert(item, fields=fields, using=self.db)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/query.py", line 1060, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 1099, in execute_sql
    cursor.execute(sql, params)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/utils.py", line 80, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/home/paul/.virtualenvs/sosadmin/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: database is locked

【问题讨论】：

  • IHMO，我认为您正在尝试在其设计范围之外很好地使用 sqlite3，即使使用一些调度中间件也是如此。根据 sqlite3 文档，一次只能有 1 个写入者，并且该活动也会停止共享（读取）访问。我明白为什么你对使用“更胖”的全功能数据库不感兴趣，但我认为 sqlite3 最适合用于不与其他任何东西共享文件的小型嵌入式系统或应用程序（基本上是单用户访问）。 ref: sqlite.org/lockingv3.html 我应该注意到我几乎每天都在 STRAUX 接收器上使用 sqlite3 数据库。
  • 你可能是对的，但是在 sqlite3 中使用 WAL 并且这个应用程序读取很多，写入很少（诚然经常是小突发）让我认为 sqlite3 应该处理它美好的。我对文档的阅读表明，如果这个错误真的是 sqlite3 达到了写入限制，我应该看到超时过去了，但我没有看到。
  • 如果您在测试中使用它，这可能是相关的：stackoverflow.com/a/51750516/2015768

标签： django multithreading python-3.x sqlite


【解决方案1】：

我在发行说明中看不到任何内容，但我已经升级到 Django 2.0，这个问题现在已经消失了。在 sqlite3 中设置 WAL 后，现在一切正常。

【讨论】：

  • 有趣。很高兴知道。
【解决方案2】：

随着系统的扩展，此错误再次发生。由于此问题范围之外的原因，我需要使用 sqlite3。

Django 中的超时选项似乎被忽略了。我尝试在 db 游标级别使用 sqlite 的包装器自己实现它，但遇到了同样的问题（在该级别，您无法退出足够远的距离来放弃事务并尝试新连接）。

我会喜欢上下文管理器，例如with db_locked_retries():... 但无法在 with 上下文中进行迭代，因此我创建了一个 db_retry 函数：

import time
import logging
import random

from django.conf import settings
import django.db

logger = logging.getLogger('django.db.backends')


def db_retry(fn, timeout=None):
    """Call fn with no arguments. If OperationalError exception, make retries until timeout has passed"""
    timeout = timeout or settings.DATABASES['default'].get('OPTIONS', dict()).get('timeout', 5)
    now = time.time()
    give_up_time = now + timeout
    retries = 0
    while now < give_up_time:
        now = time.time()
        try:
            result = fn()
            if retries:
                logger.warning(f'db_retry: Succeeded after {retries} retries')
            return result
        except django.db.OperationalError as exception:
            msg = str(exception)
            if 'locked' in msg:  # pragma: no cover
                retries += 1
                wait_time = random.uniform(1, timeout / 10)
                logger.warning(f'db_retry: {msg}: Retrying after {wait_time} seconds')
                django.db.close_old_connections()
                time.sleep(wait_time)
            else:  # pragma: no cover
                logger.exception(f'db_retry: {msg}: Giving up')
                raise

由于在模型方法等中定义传递给该函数的函数非常容易。这解决了问题并成功地使用超时重试访问。在我的特定用例中，这目前正在处理数百个并发使用和每 30 分钟左右重试一次。

到目前为止，我只需要“包装”可能进行多次写入的模型代码。

【讨论】：

    猜你喜欢
    • 2011-05-05
    • 1970-01-01
    • 1970-01-01
    • 2011-09-16
    • 1970-01-01
    • 2015-01-07
    • 1970-01-01
    • 1970-01-01
    • 2010-09-07
    相关资源
    最近更新 更多
    热门标签
    Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode