Openstack自发mysql连接错误2006，“MySQL服务器已经消失

Question

我在 openstack-ansible 部署上运行 openstack queens，最近我发现很多多个组件，如 nova 和 neutron 开始在日志中抛出以下错误堆栈，一切正常，但那个错误让我担心，有人知道吗？这个？

我检查了 F5 LB、MySQL、网络连接等基本内容，看起来都不错。

2018-08-13 10:36:23.552 17533 ERROR oslo_db.sqlalchemy.engines "MySQL server has gone away (%r)" % (e,))
2018-08-13 10:36:23.552 17533 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(104, 'Connection reset by peer'))") [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)
2018-08-13 10:36:34.997 17538 ERROR oslo_db.sqlalchemy.engines [req-c9a4ea4f-3577-42c7-aed3-e34416d93c1a 34205a21a4e4430b8be896c6a6b692cb 2b1447ec414b4751965f75785cab6468 - default default] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(104, 'Connection reset by peer'))") [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)
2018-08-13 10:36:34.997 17538 ERROR oslo_db.sqlalchemy.engines "MySQL server has gone away (%r)" % (e,))
2018-08-13 10:36:34.997 17538 ERROR oslo_db.sqlalchemy.engines DBConnectionError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(104, 'Connection reset by peer'))") [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)
2018-08-13 10:38:23.231 17529 ERROR oslo_db.sqlalchemy.engines [req-ba38ec9d-ee4e-4974-933b-46c8133397c1 34205a21a4e4430b8be896c6a6b692cb 2b1447ec414b4751965f75785cab6468 - default default] Database connection was found disconnected; reconnecting: DBConnectionError: (pymysql.err.OperationalError) (2006, "MySQL server has gone away (error(104, 'Connection reset by peer'))") [SQL: u'SELECT 1'] (Background on this error at: http://sqlalche.me/e/e3q8)

这里可以看到更多日志：http://paste.openstack.org/show/728277/

这是我的 3 节点 Galera 集群设置。

[client]
port = 3306
socket = "/var/lib/mysql/mysql.sock"
[mysqld_safe]
socket = "/var/lib/mysql/mysql.sock"
nice = 0
log_error = /var/log/mysql_logs/galera_server_error.log
[mysql]
default-character-set = utf8
[mysqld]
user = mysql
collation-server = utf8_general_ci
init-connect = 'SET NAMES utf8'
character-set-server = utf8
datadir = /var/lib/mysql
bind-address = ::
server-id = 200
log-queries-not-using-indexes = 0
slow-query-log = 0
slow-query-log-file = /var/log/mysql_logs/mysql-slow.log
log_error = /var/log/mysql_logs/galera_server_error.log
log-bin = /var/lib/mysql/mariadb-bin
log-bin-index = /var/lib/mysql/mariadb-bin.index
expire-logs-days = 7
log_slave_updates = 1
log_bin_trust_function_creators = 1
max-allowed-packet = 16M
max-connect-errors = 1000000
max_connections = 1600
wait_timeout = 3600
tmp-table-size = 32M
max-heap-table-size = 32M
query-cache-type = 0
query-cache-size = 0M
thread-cache-size = 50
open-files-limit = 65535
table-definition-cache = 4096
table-open-cache = 10240
innodb-flush-method = O_DIRECT
innodb-log-file-size = 1024M
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table = 1
innodb-buffer-pool-size = 4096M
innodb-read-io-threads = 4
innodb-write-io-threads = 4
innodb-doublewrite = 1
innodb-log-buffer-size = 128M
innodb-buffer-pool-instances = 8
innodb-log-files-in-group = 2
innodb-thread-concurrency = 64
innodb_stats_on_metadata = 0
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
!includedir /etc/mysql/conf.d/

Answer 1

您收到“服务器已离开”消息的原因有很多。太多无法粘贴。阅读此内容，看看是否适用于您： https://dev.mysql.com/doc/refman/5.6/en/gone-away.html

当我过去遇到这些情况时，

• 应用服务器和数据库之间有防火墙，它会关闭 tcp 会话
• 连接池中有一个过时的连接
• 管理员已终止服务器上的连接，应用程序不会报告它，直到再次使用该连接。

如果使用连接池，请确保连接生存期（或超时等）小于您配置的wait_timeout。

由于失败的查询是SELECT 1，我假设连接池启用了连接验证。这会通过一个简单的查询检查连接，如果失败，它会使用池中的新连接，然后重试。这看起来像是处理丢失连接的正常操作。

编辑

由于您发现 F5 的超时时间很短，因此对于数据库连接，可以将其增加到更现实的值。我见过 1 到 8 个小时，这取决于它是应用程序连接还是来自桌面应用程序。

为了强制您的客户端刷新它的连接，使用 SQLAlchemy，看起来 pool_recycle 是您想要添加的，以便在 F5 超时之前回收连接。无论 DataSource 在 OpenStack 中的何处定义，您都需要为 SQLAlchemy docs.sqlalchemy.org/en/latest/core/engines.html 添加更多配置选项。

但是，我只想将 F5/HAProxy 更新为 1 小时，然后看看您出现这些错误的频率。