将 axios 请求通过管道传输到 pkgcloud 存储上传会导致“结束后写入”错误答案

【问题标题】：Piping an axios request to a pkgcloud storage upload causes a "write after end" error将 axios 请求通过管道传输到 pkgcloud 存储上传会导致“结束后写入”错误
【发布时间】：2019-10-19 14:17:03
【问题描述】：

我正在尝试使用 axios 下载外部文件，然后使用 pkgcloud 将其通过管道传输到 openstack 对象存储容器中。上传流似乎在文件完全下载之前结束。

async function upload(url, name, extension, container) {
    const uploadStream = storage.createClient({
        ...
    }).upload({
        container: container,
        remote: name + "." + extension,
    });

    await new Promise(async (resolve) => {
        const response = await axios({
            method: "GET",
            url: url,
            responseType: "stream"
        })

        response.data.pipe(uploadStream)

        response.data.on("end", () => {
            console.log("Download finished");
        })
        response.data.on("error", (error: Error) => {
            console.log("Download error")
            console.log(error);
        })
        uploadStream.on("finish", () => {
            console.log("Upload finished");
            resolve();
        })
        uploadStream.on("error", (error: Error) => {
            console.log("Upload error");
            console.log(error);
        })
    })
}

const url = "https://picsum.photos/id/566/600/600";

(async () => {
    await upload(url, "mountain", "jpg", "dummy_container")
    console.log("Promise resolved");
})()

根据控制台输出的事件顺序如下：

Download finished
Upload finished
Promise resolved
Upload error
Error: write after end
...

【问题讨论】：

标签： node.js async-await axios nodejs-stream pkgcloud

【解决方案1】：

问题来自 pkgcloud 中的一个错误，即它如何为 OpenStack 存储流式传输文件。在https://stackoverflow.com/a/60787524/4685491中有解释

【讨论】：

【解决方案2】：

可以通过使用concat-stream 包将下载流式传输到缓冲区，然后将缓冲区流式传输到上传来解决此问题：

function upload(url, name, extension, container) {
    const uploadStream = storage.createClient({
        ...
    }).upload({
        container,
        remote: name + "." + extension,
    });

    axios({
        method: "GET",
        url,
        responseType: "stream",
    }).then((response) => {
        const concatStream = concat((buffer) => {
            const bufferStream = new Stream.PassThrough();
            bufferStream.end(buffer);
            bufferStream.pipe(uploadStream);
        })
        response.data.pipe(concatStream);
    })
}

const url = "https://picsum.photos/id/566/600/600";
upload(url, "mountain", "jpg", "dummy_container")

【讨论】：