【问题标题】:Overloading RESTful server with Keras (Theano) model through JMeter通过 JMeter 使用 Keras (Theano) 模型重载 RESTful 服务器
【发布时间】:2019-10-09 18:36:43
【问题描述】:

我有一个 keras (Theano) 模型,用于预测话语中的情绪。此外,我使用 Flask 创建了一个 RESTful 服务器,用于将模型包装到其中。 我的目标是将性能与未​​缩放和缩​​放的系统进行比较。首先,我想尝试使用未缩放的。为了使请求过载,我使用了 Apache Jmeter,它通过线程模拟 100 个用户。

代码可在this github 页面获取。

不幸的是,在运行测试时,上面提到的模型会因为这个回溯而崩溃:

Traceback (most recent call last):
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/flask/app.py", line 2328, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/flask/app.py", line 2314, in wsgi_app
    response = self.handle_exception(e)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/flask_restful/__init__.py", line 269, in error_router
    return original_handler(e)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/flask/app.py", line 1760, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value.with_traceback(tb)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/flask/app.py", line 2311, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/flask/app.py", line 1834, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/flask_restful/__init__.py", line 269, in error_router
    return original_handler(e)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/flask/app.py", line 1737, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/flask/_compat.py", line 35, in reraise
    raise value.with_traceback(tb)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/flask/app.py", line 1832, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/flask/app.py", line 1818, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/flask_restful/__init__.py", line 458, in wrapper
    resp = resource(*args, **kwargs)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/flask/views.py", line 88, in view
    return self.dispatch_request(*args, **kwargs)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/flask_restful/__init__.py", line 573, in dispatch_request
    resp = meth(*args, **kwargs)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/sentiment-lstm/predict.py", line 54, in post
    "value": self.predict([question])[0][0]
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/sentiment-lstm/predict.py", line 66, in predict
    pred = self.__model.predict(x=X_test_pad)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/keras/engine/training.py", line 1169, in predict
    steps=steps)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/keras/engine/training_arrays.py", line 294, in predict_loop
    batch_outs = f(ins_batch)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/keras/backend/theano_backend.py", line 1388, in __call__
    return self.function(*inputs)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/theano/compile/function_module.py", line 917, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/theano/gof/link.py", line 325, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/six.py", line 692, in reraise
    raise value.with_traceback(tb)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/theano/compile/function_module.py", line 903, in __call__
    self.fn() if output_subset is None else\
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/theano/scan_module/scan_op.py", line 963, in rval
    r = p(n, [x[0] for x in i], o)
  File "/home/lcarnevale/Dropbox/git/lcarnevale/sentiment-lstm/venv/lib/python3.6/site-packages/theano/scan_module/scan_op.py", line 952, in p
    self, node)
  File "scan_perform.pyx", line 546, in theano.scan_module.scan_perform.perform

TypeError: 'NoneType' object does not support item assignment
Apply node that caused the error: forall_inplace,cpu,scan_fn}(Shape_i{1}.0, Subtensor{int64:int64:int8}.0, IncSubtensor{InplaceSet;:int64:}.0, DeepCopyOp.0, Shape_i{1}.0, Subtensor{::, int64:int64:}.0, InplaceDimShuffle{x,0}.0, Subtensor{::, int64:int64:}.0, Subtensor{::, :int64:}.0, InplaceDimShuffle{x,0}.0, Subtensor{::, :int64:}.0, Subtensor{::, int64:int64:}.0, InplaceDimShuffle{x,0}.0, Subtensor{::, int64:int64:}.0, Subtensor{::, int64::}.0, InplaceDimShuffle{x,0}.0, Subtensor{::, int64::}.0)
Toposort index: 44
Inputs types: [TensorType(int64, scalar), TensorType(float32, 3D), TensorType(float32, (True, False, False)), TensorType(float32, (True, False, False)), TensorType(int64, scalar), TensorType(float32, matrix), TensorType(float32, row), TensorType(float32, matrix), TensorType(float32, matrix), TensorType(float32, row), TensorType(float32, matrix), TensorType(float32, matrix), TensorType(float32, row), TensorType(float32, matrix), TensorType(float32, matrix), TensorType(float32, row), TensorType(float32, matrix)]
Inputs shapes: ['No shapes', 'No shapes', 'No shapes', 'No shapes', 'No shapes', 'No shapes', 'No shapes', 'No shapes', 'No shapes', 'No shapes', 'No shapes', 'No shapes', 'No shapes', 'No shapes', 'No shapes', 'No shapes', 'No shapes']
Inputs strides: ['No strides', 'No strides', 'No strides', 'No strides', 'No strides', 'No strides', 'No strides', 'No strides', 'No strides', 'No strides', 'No strides', 'No strides', 'No strides', 'No strides', 'No strides', 'No strides', 'No strides']
Inputs values: [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]
Outputs clients: [[], [], [InplaceDimShuffle{1,0,2}(forall_inplace,cpu,scan_fn}.2)]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

我不确定这里的错误是什么。会不会是内存不足的问题?还是什么?

洛伦佐。

【问题讨论】:

    标签: flask keras jmeter scale theano


    【解决方案1】:

    在我看来这不是 OOM 问题

    'NoneType' 对象不支持项目分配

    很可能有一个 None 常量,您没有正确使用它,因此它没有连接到高负载,所以我建议您仔细检查您的代码并从功能/unit tests开始

    如果您想在 Python 和操作系统级别跟踪内存泄漏,请考虑使用以下方法:

    【讨论】:

      猜你喜欢
      • 2017-12-19
      • 1970-01-01
      • 2019-02-25
      • 1970-01-01
      • 2017-09-24
      • 2016-04-25
      • 2020-07-26
      • 1970-01-01
      • 2016-09-11
      相关资源
      最近更新 更多