SSCursor 上的 SqlAlchemy / PyMysql 连接超时

Question

我正在使用 sqlalchemy 连接 mysql 数据库。我的代码看起来像

engine = sqlalchemy.create_engine(sqlalchemy.engine.url.URL(**url), connect_args={'cursorclass': SSCursor})
conn = engine.connect()

由于需要一次从DB中获取大量数据，所以我使用了SSCursor。我做了一些调查，并创建了一个 60 秒后连接丢失的场景。它是通过简单的代码模拟的，我添加了 timeout(60)。我得到如下回溯：

Exception during reset or similar
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line 687, in _finalize_fairy
    fairy._reset(pool)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/pool.py", line 829, in _reset
    pool._dialect.do_rollback(self)
  File "/usr/local/lib/python3.5/dist-packages/sqlalchemy/dialects/mysql/base.py", line 1598, in do_rollback
    dbapi_connection.rollback()
  File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 788, in rollback
    self._execute_command(COMMAND.COM_QUERY, "ROLLBACK")
  File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1067, in _execute_command
    raise err.InterfaceError("(0, '')")
pymysql.err.InterfaceError: (0, '')
Exception ignored in: <bound method MySQLResult.__del__ of <pymysql.connections.MySQLResult object at 0x7fed3af0d828>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1332, in __del__
  File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1434, in _finish_unbuffered_query
  File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 983, in _read_packet
  File "/usr/local/lib/python3.5/dist-packages/pymysql/connections.py", line 1014, in _read_bytes
AttributeError: 'NoneType' object has no attribute 'settimeout'

我无法弄清楚，如何确保连接在不同的所需时间后超时。我尝试在connect_args 中使用connect_timeout，但行为没有改变。

我应该怎么做才能确保连接？

谢谢

Answer 1

您应该更改 MySQL 服务器选项， 增加net_read_timeout和max_allowed_packet的值

https://dev.mysql.com/doc/refman/5.5/en/error-lost-connection.html

Answer 2

Traceback (most recent call last):
  File "/usr/lib/python3.6/dist-packages/pymysql/connections.py", line 1071, in __del__
  File "/usr/lib/python3.6/dist-packages/pymysql/connections.py", line 1177, in _finish_unbuffered_query
  File "/usr/lib/python3.6/dist-packages/pymysql/connections.py", line 657, in _read_packet
  File "/usr/lib/python3.6/dist-packages/pymysql/connections.py", line 688, in _read_bytes
AttributeError: 'NoneType' object has no attribute 'settimeout'

我昨晚在处理大量数据时遇到了同样的问题。 我知道问题的原因，它一定是内存/网络问题。因此我的解决方案是：

1. 捕获异常
2. 睡眠 60 秒
3. 重试

                try:
                   do_something()
                except pymysql.OperationalError:
                    log.error("error %s: %d %s" % (row["id"], e.args[0], e.args[1]))
                    max_try = 10
                    try_times = 0
                    while con.open is False:
                        if try_times < max_try:
                            try_times += 1
                            time.sleep(60)
                            con.ping()
                    # retry
                    do_something() 

如果你有钱，增加记忆力会有所帮助。如果你有时间，多多学习mysql，一遍遍地翻内存和网络。

如果你只是想享受你的生活，那就做程序员的方式吧。这不是最好的解决方案，但我完成了我的工作。