在 django 中运行太多测试时 MySQL 数据库中断

【问题标题】：MySQL database breaks while running too many tests in django在 django 中运行太多测试时 MySQL 数据库中断
【发布时间】：2022-08-18 05:26:36
【问题描述】：

我有一个包含数百个测试的 Django 项目，在添加了大约 50 个测试之后，当我使用 python manage.py test 一次运行所有测试时，几乎所有测试都会引发以下错误：

Traceback (most recent call last):
  File \".../venv/lib/python3.9/site-packages/django/db/models/sql/compiler.py\", line 1142, in execute_sql
    cursor.execute(sql, params)
  File \".../venv/lib/python3.9/site-packages/django/db/backends/utils.py\", line 67, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File \".../venv/lib/python3.9/site-packages/django/db/backends/utils.py\", line 76, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File \".../venv/lib/python3.9/site-packages/django/db/backends/utils.py\", line 79, in _execute
    self.db.validate_no_broken_transaction()
  File \".../venv/lib/python3.9/site-packages/django/db/backends/base/base.py\", line 437, in validate_no_broken_transaction
    raise TransactionManagementError(
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can\'t execute queries until the end of the \'atomic\' block.

During handling of the above exception, another exception occurred:
  File \".../venv/lib/python3.9/site-packages/django/db/models/manager.py\", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File \".../venv/lib/python3.9/site-packages/django/db/models/query.py\", line 653, in first
    for obj in (self if self.ordered else self.order_by(\'pk\'))[:1]:
  File \".../venv/lib/python3.9/site-packages/django/db/models/query.py\", line 274, in __iter__
    self._fetch_all()
  File \".../venv/lib/python3.9/site-packages/django/db/models/query.py\", line 1242, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File \".../venv/lib/python3.9/site-packages/django/db/models/query.py\", line 55, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File \".../venv/lib/python3.9/site-packages/django/db/models/sql/compiler.py\", line 1145, in execute_sql
    cursor.close()
  File \".../venv/lib/python3.9/site-packages/MySQLdb/cursors.py\", line 83, in close
    while self.nextset():
  File \".../venv/lib/python3.9/site-packages/MySQLdb/cursors.py\", line 137, in nextset
    nr = db.next_result()
MySQLdb._exceptions.OperationalError: (2006, \'\')

但是，如果我单独运行测试，所有测试都会正常运行并成功。所以我假设数据库同时运行多个测试时存在问题，但我无法弄清楚它是什么。我已经尝试过the solutions on this question，但我的问题似乎不一样。我也做了this question，但我决定再做一个，因为当时我不知道问题可能与数据库中的测试数量有关。

我正在使用 python 3.9、Django 3.2 和 MySQL 8.0.23。 我的操作系统是 Ubuntu 22.04

有谁知道如何解决这个问题？对这个项目来说，所有的测试都能成功运行是非常重要的。

MySQL错误日志：

2022-08-04T15:17:47.816192Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 1104
2022-08-04T15:17:47.841975Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-04T15:17:48.487357Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-04T15:17:48.814069Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-04T15:17:48.814094Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-04T15:17:48.866968Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: \'127.0.0.1\' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-08-04T15:17:48.867002Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: \'8.0.30-0ubuntu0.22.04.1\'  socket: \'/var/run/mysqld/mysqld.sock\'  port: 3306  (Ubuntu).
2022-08-05T02:50:34.190929Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30-0ubuntu0.22.04.1).
2022-08-05T02:50:35.768640Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.22.04.1)  (Ubuntu).
2022-08-05T16:01:36.811311Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 1119
2022-08-05T16:01:36.830991Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-05T16:01:37.492375Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-05T16:01:37.799605Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-05T16:01:37.799627Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-05T16:01:37.824311Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: \'127.0.0.1\' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-08-05T16:01:37.824331Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: \'8.0.30-0ubuntu0.22.04.1\'  socket: \'/var/run/mysqld/mysqld.sock\'  port: 3306  (Ubuntu).
2022-08-06T00:32:10.132946Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30-0ubuntu0.22.04.1).
2022-08-06T00:32:12.839068Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.22.04.1)  (Ubuntu).
2022-08-06T21:28:21.218813Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 1134
2022-08-06T21:28:21.243315Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-06T21:28:21.867888Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-06T21:28:22.188538Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-06T21:28:22.188561Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-06T21:28:22.209764Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: \'127.0.0.1\' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-08-06T21:28:22.209797Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: \'8.0.30-0ubuntu0.22.04.1\'  socket: \'/var/run/mysqld/mysqld.sock\'  port: 3306  (Ubuntu).
2022-08-08T11:51:47.706954Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30-0ubuntu0.22.04.1).
2022-08-08T11:51:48.491678Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.22.04.1)  (Ubuntu).
2022-08-08T11:51:50.428184Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 1115
2022-08-08T11:51:50.444404Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-08T11:51:50.956962Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-08T11:51:51.322340Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-08T11:51:51.322369Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-08T11:51:51.359037Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: \'127.0.0.1\' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-08-08T11:51:51.359047Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: \'8.0.30-0ubuntu0.22.04.1\'  socket: \'/var/run/mysqld/mysqld.sock\'  port: 3306  (Ubuntu).
2022-08-08T21:36:15.186489Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30-0ubuntu0.22.04.1).
2022-08-08T21:36:17.194038Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 11  user: \'root\'.
2022-08-08T21:36:17.194179Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 852  user: \'root\'.
2022-08-08T21:36:17.194293Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 853  user: \'root\'.
2022-08-08T21:36:18.316093Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.22.04.1)  (Ubuntu).
2022-08-08T21:42:06.980028Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 5459
2022-08-08T21:42:06.992543Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-08T21:42:07.380065Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-08T21:42:07.648877Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-08T21:42:07.648897Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.                                                
2022-08-08T21:42:07.674744Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: \'127.0.0.1\' port: 33060, socket: /var/run/mysqld/mysqlx.sock                                             
2022-08-08T21:42:07.674766Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: \'8.0.30-0ubuntu0.22.04.1\'  socket: \'/var/run/mysqld/mysqld.sock\'  port: 3306  (Ubuntu).              
2022-08-04T15:17:47.816192Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 1104
2022-08-04T15:17:47.841975Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-04T15:17:48.487357Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-04T15:17:48.814069Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-04T15:17:48.814094Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-04T15:17:48.866968Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: \'127.0.0.1\' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-08-04T15:17:48.867002Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: \'8.0.30-0ubuntu0.22.04.1\'  socket: \'/var/run/mysqld/mysqld.sock\'  port: 3306  (Ubuntu).
2022-08-05T02:50:34.190929Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30-0ubuntu0.22.04.1).
2022-08-05T02:50:35.768640Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.22.04.1)  (Ubuntu).
2022-08-05T16:01:36.811311Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 1119
2022-08-05T16:01:36.830991Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-05T16:01:37.492375Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-05T16:01:37.799605Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-05T16:01:37.799627Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-05T16:01:37.824311Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: \'127.0.0.1\' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-08-05T16:01:37.824331Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: \'8.0.30-0ubuntu0.22.04.1\'  socket: \'/var/run/mysqld/mysqld.sock\'  port: 3306  (Ubuntu).
2022-08-06T00:32:10.132946Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30-0ubuntu0.22.04.1).
2022-08-06T00:32:12.839068Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.22.04.1)  (Ubuntu).
2022-08-06T21:28:21.218813Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 1134
2022-08-06T21:28:21.243315Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-06T21:28:21.867888Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-06T21:28:22.188538Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-06T21:28:22.188561Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-06T21:28:22.209764Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: \'127.0.0.1\' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-08-06T21:28:22.209797Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: \'8.0.30-0ubuntu0.22.04.1\'  socket: \'/var/run/mysqld/mysqld.sock\'  port: 3306  (Ubuntu).
2022-08-08T11:51:47.706954Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30-0ubuntu0.22.04.1).
2022-08-08T11:51:48.491678Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.22.04.1)  (Ubuntu).
2022-08-08T11:51:50.428184Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 1115
2022-08-08T11:51:50.444404Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-08T11:51:50.956962Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-08T11:51:51.322340Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-08T11:51:51.322369Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-08T11:51:51.359037Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: \'127.0.0.1\' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2022-08-08T11:51:51.359047Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: \'8.0.30-0ubuntu0.22.04.1\'  socket: \'/var/run/mysqld/mysqld.sock\'  port: 3306  (Ubuntu).
2022-08-08T21:36:15.186489Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.30-0ubuntu0.22.04.1).
2022-08-08T21:36:17.194038Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 11  user: \'root\'.
2022-08-08T21:36:17.194179Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 852  user: \'root\'.
2022-08-08T21:36:17.194293Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 853  user: \'root\'.
2022-08-08T21:36:18.316093Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30-0ubuntu0.22.04.1)  (Ubuntu).
2022-08-08T21:42:06.980028Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30-0ubuntu0.22.04.1) starting as process 5459
2022-08-08T21:42:06.992543Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-08T21:42:07.380065Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-08T21:42:07.648877Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2022-08-08T21:42:07.648897Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.                                                
2022-08-08T21:42:07.674744Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: \'127.0.0.1\' port: 33060, socket: /var/run/mysqld/mysqlx.sock                                             
2022-08-08T21:42:07.674766Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: \'8.0.30-0ubuntu0.22.04.1\'  socket: \'/var/run/mysqld/mysqld.sock\'  port: 3306  (Ubuntu).

MySQL 配置：

[mysqld]
user        = mysql
# pid-file  = /var/run/mysqld/mysqld.pid
# socket    = /var/run/mysqld/mysqld.sock
# port      = 3306
# datadir   = /var/lib/mysql


# tmpdir        = /tmp
#
bind-address        = 127.0.0.1
mysqlx-bind-address = 127.0.0.1
key_buffer_size     = 16M
# max_allowed_packet    = 64M
# thread_stack      = 256K

# thread_cache_size       = -1

myisam-recover-options  = BACKUP

# max_connections        = 151

# table_open_cache       = 4000

max_allowed_packet = 640M
wait_timeout = 28800000
interactive_timeout = 28800000
innodb_log_file_size = 256MB
innodb_buffer_pool_size = 12000MB
    

mysql> SELECT @@global.wait_timeout, @@session.wait_timeout, @@global.max_allowed_packet, @@global.innodb_log_file_size, @@global.innodb_buffer_pool_size;
+-----------------------+------------------------+-----------------------------+-------------------------------+----------------------------------+
| @@global.wait_timeout | @@session.wait_timeout | @@global.max_allowed_packet | @@global.innodb_log_file_size | @@global.innodb_buffer_pool_size |
+-----------------------+------------------------+-----------------------------+-------------------------------+----------------------------------+
|              28800000 |               28800000 |                   671088640 |                     268435456 |                      12884901888 |
+-----------------------+------------------------+-----------------------------+-------------------------------+----------------------------------+
1 row in set (0,00 sec)
  • 你能分享MySQL日志吗？
  • 我对MySQL的经验不是很丰富，但是我发现的唯一错误日志是在/var/logs/mysql/error.log下，并且没有很多相关信息。无论如何我都会发布它。如果您想查看其他 MySQL 日志，请告诉我
  • 这些日志是在python manage.py test 运行时形成的？
  • 我在运行测试后立即得到了日志

标签： python mysql python-3.x django


【解决方案1】：

使用来自 python 的最后一条消息，您与服务器的连接被中止。您可以在您首选的搜索网站中查找更多关于它的信息，以查找 mysql 2006。

MySQLdb._exceptions.OperationalError: (2006, '')

当我们运行大量查询时会发生这种情况有一些原因，其中之一是查询花费的时间或我们包的大小。

尝试将wait_timeout 和max_allowed_packet 的值增加到更高的值，看看它是否有效。当然，如果它有效，您将需要了解这些值并将其适应您的工作量，或者修复您的工作量。

如果你的桌子是InnoDB并且测试正在运行大量查询，您还可以增加innodb_log_file_size（例如 128MB）和innodb_buffer_pool_size（例如，如果 MySQL 单独运行，则至少 50% 的 RAM 或更多）。最后一个参数通常调整为 RAM 的 80%，但这可能会因您拥有多少 RAM 和其他因素而有很大差异。如果您的测试使数据库过载，这些innodb_ 调整只会使查询运行得更快。

【讨论】：

  • 我在配置中更改了这些变量，但错误仍然存​​在。我更新了问题以显示我使用的值
猜你喜欢
  • 2015-08-06
  • 1970-01-01
  • 1970-01-01
  • 2020-07-02
  • 2013-02-20
  • 1970-01-01
  • 2012-05-09
  • 1970-01-01
  • 2016-03-21
相关资源
最近更新 更多
热门标签
Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode