SQLAlchemy 和 pg8000 KeyError / Broken Pipe on Google App Engine with Postgresql

Question

我的目标是在我的 Postgresql dv（在 Cloud SQL 上）上成功执行 Select 语句，在 Google App Engine 上的 Flask 应用中使用 SQLAlchemy 的 ORM。

我能够成功部署应用程序，它成功连接到数据库并发出选择语句。但是，在应用程序运行大约 1 小时后，我开始从查询中收到以下错误。请注意，这些查询在 1 小时之前成功运行，我能够从 pgAdmin 或本地 python 代码成功地在数据库上运行查询。

有两个错误，第一个是发生在 pg8000/core.py 的第 1778 行的 KeyError。在 git 上，这一行是 https://github.com/tlocke/pg8000/blob/master/pg8000/core.py#L1783

这是一个错误，它由一个 except 块处理，但在该 except 块内引发了 Broken Pip 错误。

File "/env/lib/python3.7/site-packages/pg8000/core.py", line 1778, in execute ps = cache['ps'][key] KeyError: ('SELECT voice_comments.id AS voice_comments_id, voice_comments.store_id AS voice_comments_store_id, voice_comments.comment_date AS voice_comments_comment_date, voice_comments.survey_item AS voice_comments_survey_item, voice_comments.comment_text AS voice_comments_comment_text, voice_comments.overall_satisfaction AS voice_comments_overall_satisfaction, voice_comments.visit_date AS voice_comments_visit_date \nFROM voice_comments \nWHERE voice_comments.store_id = %s AND voice_comments.comment_date >= %s AND voice_comments.comment_date <= %s AND voice_comments.overall_satisfaction = %s AND voice_comments.id NOT IN (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)', ((705, 0, <function Connection.__init__.<locals>.text_out at 0x3eb1a3fb7c20>), (1114, 1, <function timestamp_send_integer at 0x3eb1a4a6d320>), (1114, 1, <function timestamp_send_integer at 0x3eb1a4a6d320>), (21, 1, <built-in method pack of Struct object at 0x3eb1a4a41030>), (21, 1, <built-in method pack of Struct object at 0x3eb1a4a41030>), (21, 1, <built-in method pack of Struct object at 0x3eb1a4a41030>), (21, 1, <built-in method pack of Struct object at 0x3eb1a4a41030>), (21, 1, <built-in method pack of Struct object at 0x3eb1a4a41030>), (21, 1, <built-in method pack of Struct object at 0x3eb1a4a41030>), (21, 1, <built-in method pack of Struct object at 0x3eb1a4a41030>), (21, 1, <built-in method pack of Struct object at 0x3eb1a4a41030>), (21, 1, <built-in method pack of Struct object at 0x3eb1a4a41030>), (21, 1, <built-in method pack of Struct object at 0x3eb1a4a41030>), (21, 1, <built-in method pack of Struct object at 0x3eb1a4a41030>), (21, 1, <built-in method pack of Struct object at 0x3eb1a4a41030>), (21, 1, <built-in method pack of Struct object at 0x3eb1a4a41030>), (21, 1, <built-in method pack of Struct object at 0x3eb1a4a41030>), (21, 1, <built-in method pack of Struct object at 0x3eb1a4a41030>)))

在处理上述 KeyError 的 except 块中，引发了 BrokenPipeError，即：

Traceback (most recent call last): File "/env/lib/python3.7/site-packages/flask/app.py", line 2292, in wsgi_app response = self.full_dispatch_request() File "/env/lib/python3.7/site-packages/flask/app.py", line 1815, in full_dispatch_request rv = self.handle_user_exception(e) File "/env/lib/python3.7/site-packages/flask/app.py", line 1718, in handle_user_exception reraise(exc_type, exc_value, tb) File "/env/lib/python3.7/site-packages/flask/_compat.py", line 35, in reraise raise value File "/env/lib/python3.7/site-packages/flask/app.py", line 1813, in full_dispatch_request rv = self.dispatch_request() File "/env/lib/python3.7/site-packages/flask/app.py", line 1799, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/srv/routes/voice.py", line 26, in voice_comments result = get_voice_comments(store_id, start_date, end_date, ids_to_filter_out=used_comment_ids) File "/srv/routes/pgsql_api.py", line 18, in get_voice_comments voice_comments.id.notin_(ids_to_filter_out)).all() File "/env/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3233, in all return list(self) File "/env/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3389, in __iter__ return self._execute_and_instances(context) File "/env/lib/python3.7/site-packages/sqlalchemy/orm/query.py", line 3414, in _execute_and_instances result = conn.execute(querycontext.statement, self._params) File "/env/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 982, in execute return meth(self, multiparams, params) File "/env/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/env/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1101, in _execute_clauseelement distilled_params, File "/env/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1250, in _execute_context e, statement, parameters, cursor, context File "/env/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1478, in _handle_dbapi_exception util.reraise(*exc_info) File "/env/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 153, in reraise raise value File "/env/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1246, in _execute_context cursor, statement, parameters, context File "/env/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 588, in do_execute cursor.execute(statement, parameters) File "/env/lib/python3.7/site-packages/pg8000/core.py", line 861, in execute self._c.execute(self, operation, args) File "/env/lib/python3.7/site-packages/pg8000/core.py", line 1830, in execute self._flush() File "/opt/python3.7/lib/python3.7/socket.py", line 607, in write return self._sock.send(b) BrokenPipeError: [Errno 32] Broken pipe

我正在考虑这三个部分：

1。 我最近从 MySQL 切换到 PostgresSQL。使用 MySQL，我没有遇到 Broken Pipe 错误……或者至少在我不知道的情况下，它们在幕后得到了优雅的处理。我担心我在高水平上做一些愚蠢的事情......即使用 SQLAlchemy 创建引擎时，我应该设置不同的 pool_recycle 或 connect_timeout 值。我对除 pool_timeout=30, pool_recycle=1800 以外的所有参数使用默认值

由于这发生在应用程序运行大约一个小时后，我想知道连接是否正在断开（由 pg db）但 SQLAlchemy 仍在尝试使用它们？

2。 我应该担心 KeyError 吗？或者这是代码中的预期路径。我倾向于它是一种预期，因为它被 except 块等捕获......

3。 我应该在查询级别处理 BrokenPipeError 吗？即类似：

retry = 0
while retry<3:
try:
    session.query(....
    break
except BrokenPipeError:
    retry += 1
    continue

Answer 1

我没有正确地确定 sqlalchemy 会话的范围，如下所述：

https://flask.palletsprojects.com/en/1.1.x/patterns/sqlalchemy/

具体来说，我使用的是 scoped_session，但我没有使用 @app.teardown_context 装饰器删除会话。

我一直在用 mysql db 做这个，但只是在切换到 pgsql 时忘记了做这个。

解决这个问题可以解决问题。

Answer 2

您可能在执行查询时正在关闭管道。根据关于How to prevent BrokenPipeError 的其他stackoverflow 线程，这是一个常见问题。 这里有一个简短的总结：

BrokenPipeError 是正常的，如所说的幻影，因为读数 进程（头）终止并关闭其管道的末端，而 写入过程（python）仍在尝试写入。