flask：直接写入响应流答案

【问题标题】：flask: Writing directly to response streamflask：直接写入响应流
【发布时间】：2017-01-26 10:16:08
【问题描述】：

大家好，

我有一个烧瓶 restplus 服务器，我正在努力实现将 excel 作为八位字节流提供给我的客户。似乎 pandas.to_excel(..) 在序列化大型 DataFrame 时会消耗大量时间（大约 120k 行需要 30 秒）。

请看下面我当前的实现：

def format(data_frame):
    # Idea is to directly write to the flask output stream, instead of buffering
    # the whole excel as io.BytesIO. Is there a way to do it?
    output = io.BytesIO()
    writer = pandas.ExcelWriter(output, engine='xlsxwriter')
    data_frame_ordered = data_frame.reindex_axis(sorted(data_frame.columns), axis=1)

    # This consumes a lot of time
    data_frame_ordered.to_excel(writer, sheet_name='ML Data', na_rep=0, index=False, encoding='utf-8')

    # This consumes a lot of time, too.
    writer.save()

    return output.getvalue()


@api.route('/excel', methods=['GET'])
class ExcelResource(Resource):
    def get(self, args):
        # Well, thats a huge pandas.DataFrame
        data_frame = ...
        resp = make_response(format(data_frame))
        resp.headers['Content-Length'] = resp.content_length
        resp.headers['Content-Type'] = 'application/octet-stream'

        return resp

有没有办法将 excel 直接写入烧瓶输出流，而不将其缓冲到 BytesIO 实例中？

提前致谢

丹尼斯

【问题讨论】：

标签： python-3.x pandas flask

【解决方案1】：

您可以尝试创建一些类似文件的对象，为您提供流接口，例如：

import threading
from flask import Response
from Queue import Queue


class StreamWriter(object):
    def __init__(self):
        self.queue = Queue()

    def write(self, some):
        self.queue.put(some)

    def read(self):
        return self.queue.get(True)

    def flush(self):
        pass

    def tell(self):
        #probably some code
        pass

    def seek(self):
        #probably some code
        pass

    def close(self):
        self.queue.put(None)

@api.route('/excel', methods=['GET'])
class ExcelResource(Resource):
    def get(self, args):
        def generate():
            output = StreamWriter()

            def do_stuff():
                output = StreamWriter()

                writer = pandas.ExcelWriter(output, engine='xlsxwriter')
                data_frame_ordered = data_frame.reindex_axis(sorted(data_frame.columns), axis=1)

                # This consumes a lot of time
                data_frame_ordered.to_excel(writer, sheet_name='ML Data', na_rep=0, index=False, encoding='utf-8')

                # This consumes a lot of time, too.
                writer.save()
                output.close()

            threading.Thread(target=do_stuff).start()
            while True:
                chunk = output.read()
                if chunk is None:
                    break
                yield chunk

        return Response(generate(), headers={some_headers})

只是一个粗略的想法，这段代码未经测试！

【讨论】：