Node JS 创建大量 CSV 文件而没有内存不足错误

Question

我正在尝试在 Node Js 中创建一个 CSV 文件（使用 fast-csv），但即使在使用 --max-old-space-size=2000000 之后，我也会收到 FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed 因为有大号（比如 3000 个，每个大小为 1.5mb）。

Node js 按逻辑顺序创建每个文件，但为什么创建时大小为零，并且在所有文件生成后分配内存

任何可能的方式来单独创建文件和分配内存？

提前致谢

Answer 1

其实 CSV 创建并不难自己做。在创建大型 CSV 文件时，最好逐行写入文件，而不是先在内存中创建所有内容。

这是我创建（大）CSV 文件的代码：

const fs = require('fs');
const path = require('path');

const separator = ',';

var data = [
    {name: 'Sam'  , age: 33.5, place: 'some place, with a separator in'},
    {name: 'Sofie', age: 31  , place: 'some place with "double quotes" in'},
    {name: 'Nic'  , age: 1   , place: 'some place with\na newline in'}
];

// construct csv output stream:
const outputPath = path.join(__dirname, '/test.csv');
const output = fs.createWriteStream(outputPath, { encoding: 'utf8' });

// add separator indication (So Excel knows what the CSV separator is):
output.write(`sep=${separator}\n`);

// if no data, end creation of file:
if(data.length == 0) {
    output.end(() => {
        console.log('done');
    });
    return;
}

// get headers from first entry:
var headers = Object.keys(data[0]);

// write headers to file:
output.write(`${constructCsvLine(headers)}\n`);

// write the rest of the data:
for (var i = 0; i < data.length; i++) {
    var entry = data[i];

    var line = [];

    // only get fields that we have in our headers:
    for (var j = 0; j < headers.length; j++) {
        var key = headers[j];

        var field = entry[key];
        if(field === undefined) field = '';

        line.push(field);
    }

    output.write(`${constructCsvLine(line)}\n`);
}

output.end(() => {
    console.log('done');
});



function constructCsvLine(fields) {
    var encodedFields = [];
    for (var i = 0; i < fields.length; i++) {
        var field = '' + fields[i];

        if(field.includes('"')) field = `"${field.replace(/\"/g, '""')}"`; // replace single quotes with double quotes and enclose in quotes
        if(field.includes(separator)) field = `"${field}"`; // enclose in quotes when separator occurs in field
        if(field.includes("\n")) field = `"${field}"`; // enclose in quotes when newline occrus in field
        field = field.replace(/\n/g, "\r"); // seems to be parsed correctly as newline by Excel

        encodedFields.push(field);
    }
    return encodedFields.join(separator);
}

Answer 2

尝试增加你的内存限制：

node --max-old-space-size=8192 fileName.js 

或者使用像 Mongodb 这样的数据库。