Tornado 引发 StreamClosedError：流已关闭

Question

我使用 tornado 和 asyncmc 来构建异步 API 服务。它运行了几个星期，但今天问题来了。几乎我所有由 asyncmc 缓存的服务都会引发 StreamClosedError。

从龙卷风日志中，它说：

ile "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/web.py", line 1348, in _execute
    result = yield result
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 807, in run
    value = future.result()
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 810, in run
    yielded = self.gen.throw(*sys.exc_info())
  File "/opt/app/python/tv-appstore/handler/base.py", line 100, in wrapper
    result = yield mc.get(self.cache_key)
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 807, in run
    value = future.result()
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 810, in run
    yielded = self.gen.throw(*sys.exc_info())
  File "build/bdist.linux-x86_64/egg/asyncmc/client.py", line 63, in wrapper
    res = yield func(self, conn, *args, **kwargs)
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 807, in run
    value = future.result()
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 810, in run
    yielded = self.gen.throw(*sys.exc_info())
  File "build/bdist.linux-x86_64/egg/asyncmc/client.py", line 238, in get
    result = yield self._multi_get(conn, self._key_type(key=key))
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 807, in run
    value = future.result()
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 810, in run
    yielded = self.gen.throw(*sys.exc_info())
  File "build/bdist.linux-x86_64/egg/asyncmc/client.py", line 278, in _multi_get
    servers_resp = yield conn.send_cmd_all(cmd, stream=True)
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 807, in run
    value = future.result()
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 810, in run
    yielded = self.gen.throw(*sys.exc_info())
  File "build/bdist.linux-x86_64/egg/asyncmc/pool.py", line 91, in send_cmd_all
    server_resp = yield host.send_cmd(cmd, *arg, **kw)
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 807, in run
    value = future.result()
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/concurrent.py", line 209, in result
    raise_exc_info(self._exc_info)
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/gen.py", line 212, in wrapper
    yielded = next(result)
  File "build/bdist.linux-x86_64/egg/asyncmc/host.py", line 77, in send_cmd
    yield self.stream.write(cmd)
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/iostream.py", line 356, in write
    self._check_closed()
  File "/opt/python2.7/lib/python2.7/site-packages/tornado-4.1-py2.7-linux-x86_64.egg/tornado/iostream.py", line 864, in _check_closed
    raise StreamClosedError("Stream is closed")

Answer 1

我也有同样的问题。当您将 minsize（对于 asyncmc.Client）设置为大于 1 并且连接到 ConnectionPool 的某些连接由于使用库以外的原因（tornado、asyncmc）而中断时，可能会发生这种情况。为了解决这个问题，我通过自己的方法从缓存中获取价值：

self.mc_get(key)

而不是

self.mc.get(key)

并调用 self.mc.pool.clear() 以清理连接池并初始“完全重新连接”。

Python 代码：

import tornado
from tornado.iostream import StreamClosedError

class SomeBaseHandler(tornado.web.RequestHandler):

    def mc_get(self, key, default=None):
        """ Additional interlayer for debug and logging requested keys from memcached

        Args:
            key (str): name of key in memcached
            default: default value, returns if key not exists

        Returns:
            obj
        """
        try:
            return yield self.mc.get(key, default)
        except StreamClosedError as e:
            self.mc.pool.clear()
            # Here you can raise own exception
            return default

    # …

我不使用递归调用 mc_get 来避免死循环，也没有在方法中添加额外的代码。