在 perl 中对 50GB 文件执行一些正则表达式 [重复]

Question

如何在 perl 中流式传输一个 50GB 的大文件来为每一行（或块）执行一些正则表达式？我试过普通香草

for $line (<FH>) {
   # do regex
}

我也尝试过 Tie::File 和 File::Stream，但 perl 总是试图将整个文件加载到内存中，这根本不可能。

#!/usr/bin/perl

use IO::Handle;
use Tie::File;
use File::Stream;
#tie @array, 'Tie::File', $ARGV[0] or die "could not open file";

STDOUT->autoflush(1);

$file=$ARGV[0];
open(INFO, "< $file") or die("Could not open  file.");

print "opening ... \n";
my $stream = File::Stream->new(<INFO>);

#$out = $ARGV[1];
#open(my $OH, '>', $out) or die "Could not open file '$out' $!";
print "starting ... \n";
while (<$stream>)  {
    $line = $_;
    $line =~ s/\n/\[!BR!\]/g;
    $line =~ s/<page>/\n<page>/g;
    $line =~ s/<\/page>/<\/page>\n/g;
    print $line;

    #STDOUT->flush();
}

close(INFO);

Answer 1

正确的“plain vanilla”语法是

while (my $line = <FH>) { ...

您的for 循环确实会导致 Perl 首先将整个文件读入内存。

Answer 2

我建议使用PerlMonks page 中概述的方法。

这是该页面的示例：

# Set the character which will be used to indicate the end of a line.
# This defaults to the system's end of line character, but it doesn't
# hurt to set it explicitly, just in case some other part of your code
# has altered it from the default.
local $/ = "\n";

# Open the file for read access:
open my $filehandle, '<', 'myfile.txt';

my $line_number = 0;

# Loop through each line:
while (defined($line = <$filehandle>))
{
  # The text of the line, including the linebreak
  # is now in the variable $line.

  # Keep track of line numbers
  $line_number++;

  # Strip the linebreak character at the end.
  chomp $line;

  # Do something with the line.
  do_something($line);

  # Perhaps bail out of the loop
  if ($line =~ m/^ERROR/)
  {
    warn "Error on line $line_number - skipping rest of file";
    last;
  }
}

编辑：要获取行号，可以省略$line_number，直接使用$.（见http://perldoc.perl.org/perlvar.html）