【问题标题】:Splitting a text file based on number of lines in node.js根据 node.js 中的行数拆分文本文件
【发布时间】:2016-03-29 11:42:12
【问题描述】:

在这里达到了我的理解极限。我有一个大文本文件,我需要将其分成 200 行的块(使用 mime.types 练习)。我正在使用readline,但它似乎忽略了我的pause / resume 语句。我错过了什么:

// Splits a given file into smaller subfiles by line number
var infileName = 'mime.types';
var fileCount = 1;
var count = 0;
var fs = require('fs');
var outfileName = infileName + '.' + fileCount;
var inStream = fs.createReadStream(infileName);
var outStream = fs.createWriteStream(outfileName);
var lineReader = require('readline').createInterface({
  input: inStream
});

lineReader.on('line', function(line) {
  count++;
  lineReader.pause();
  outStream.write(line + '\n');
  if (count >= 200) {
    fileCount++;
    console.log('file ', outfileName, count);
    outStream.close();
    outfileName = infileName + '.' + fileCount;
    outStream = fs.createWriteStream(outfileName);
    count = 0;
  }
  lineReader.resume();
});

lineReader.on('close', function() {
  if (count > 0) {
    console.log('Final close:', outfileName, count);
  }
  inStream.close();
  outStream.close();
  console.log('Done');
});

我也试过把stream的变化放到写操作的回调中:

lineReader.on('line', function(line) {
  count++;
  lineReader.pause();
  outStream.write(line + '\n', function() {
    if (count >= 200) {
      fileCount++;
      console.log('file ', outfileName, count);
      outStream.close();
      outfileName = infileName + '.' + fileCount;
      outStream = fs.createWriteStream(outfileName);
      count = 0;
    }
    lineReader.resume();
  });
});

两种情况的结果都是一样的:
node split file mime.types.1 1588 Done

理想情况下,我想要一个不依赖额外 npm 的解决方案

【问题讨论】:

  • 您使用的是哪条阅读线?核心或npmjs.com/package/readline ?
  • 仅限核心。想知道这是否可以使用核心解决
  • line 事件被调度了多少次?可以设置断点吗?
  • 1588 次 :-)。具有讽刺意味的是......我注释掉了 .write 事件,然后它会创建预期数量的文件 - 当然是空的。你可以试试。你肯定会在磁盘的某个地方有文件 mime.types(我只是用它来测试,因为它很好,很长而且只有文本)
  • 如果我删除 pause() 和 resume(),它会按预期工作...file mime.types.1 200 file mime.types.2 200 file mime.types.3 200 file mime.types.4 200 file mime.types.5 200 file mime.types.6 200 file mime.types.7 200 Final close: mime.types.8 188

标签: node.js


【解决方案1】:

我摆脱了暂停/恢复。它们似乎不受 readline 流的支持。我所做的只是每 200 行提供一个新的可写流。

// Splits a given file into smaller subfiles by line number
var infileName = 'mime.types';
var fileCount = 1;
var count = 0;
var fs = require('fs');
var outStream;
var outfileName = infileName + '.' + fileCount;
newWriteStream();
var inStream = fs.createReadStream(infileName);

var lineReader = require('readline').createInterface({
    input: inStream
});

function newWriteStream(){
    outfileName = infileName + '.' + fileCount;
    outStream = fs.createWriteStream(outfileName);
    count = 0;
}

lineReader.on('line', function(line) {
    count++;
    outStream.write(line + '\n');
    if (count >= 200) {
        fileCount++;
        console.log('file ', outfileName, count);
        outStream.end();
        newWriteStream();
    }
});

lineReader.on('close', function() {
    if (count > 0) {
        console.log('Final close:', outfileName, count);
    }
    inStream.close();
    outStream.end();
    console.log('Done');
});

【讨论】:

猜你喜欢
  • 2022-12-07
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 2017-04-27
  • 1970-01-01
  • 2015-03-24
  • 1970-01-01
相关资源
最近更新 更多