使用 C++ 删除文本文件中重复行的最节省内存的方法

Answer 1

您可以使用每行的哈希快速查找重复行，如其他答案所示。但是，如果您只存储散列，则假定没有散列冲突。如果您使用std::hash，那将是不正确的。如果您使用良好的加密散列，您可能会侥幸逃脱。

由于您的输入只有 10G，我建议采用不同的方法。好吧，除了琐碎。 10G 是您可能只需加载到内存中并将每行存储为现代系统上的字符串的东西。

但是让我们节省一些内存：

• 首先，您应该对文件进行 mmap 映射，以便可以从 C++ 访问其所有数据，而无需同时将其全部加载到内存中。
• 创建一个std::unordered_multimap<std::size_t, std::string_view> lines; 来跟踪输入中已经看到的行
• 循环输入文件并为每一行文本创建一个string_view，计算哈希值并在lines中查找。如果散列存在，则将该行与具有相同散列的其他行进行比较。如果该行是唯一的，则将其添加到lines 并输出。

我认为这将使用每行（唯一）行 32 字节的内存。因此，对于短行，所需的内存可能比输入文件更多。另一方面，对于短线，独特的线可能要少得多。

PS：您可以通过仅存储每行的开头来节省内存。如果您估计（唯一）行的数量，您可以使用具有不同冲突策略（没有 bin）的哈希表将其降低到每行 8 个字节。

Answer 2

当然，您可以在 O(n^2) 时间内完成此操作，只需使用保存两行所需的内存量、一个布尔标志和两个文件偏移量。

基本算法将是：

• 打开输入文件
• 打开输出文件
• 将标志设置为假
• 将预读偏移设置为 0
• 虽然输入更多：
  • 读取第一个输入行
  • 将当前输入文件偏移保存为读取后偏移
  • 寻找输入文件到偏移量 0
  • 当当前输入文件偏移量小于预读偏移量时：
    • 读取第二个输入行
    • 如果第一个输入行等于第二个输入行：
      • 将标志设置为真
      • 休息
  • 如果标志为假：
    • 将第一行输入写入输出文件
  • 将标志设置为假
  • 寻找输入文件到读后偏移
  • 将预读偏移设置为读后偏移

当然，这是非常省时的，但它与内存效率差不多。

可能的 C++ 实现：

std::ifstream input(inputFilePath);
std::ofstream output(outputFilePath);

std::streamoff offset_before = 0;
std::streamoff offset_after = 0;
bool found_dupe = false;

std::string line1;
while (std::getline(input, line1)) {
    offset_after = input.tellg();

    input.seekg(0);
    std::string line2;
    while (input.tellg() < offset_before && std::getline(input, line2)) {
        if (line1 == line2) {
            found_dupe = true;
            break;
        }
    }

    if (!found_dupe) {
        output << line1 << '\n';
    }

    found_dupe = false;
    input.seekg(offset_after);
    offset_before = offset_after;
}

Live Demo

Answer 3

我知道现在回答这个问题有点晚了，但只是为了好玩，我编写了一个我认为非常节省内存同时仍然具有合理性能的实现。

特别是，此解决方案在O(N*log(N)) 时间运行，并且（在我的机器上）仅使用 360 KB（！）的堆内存，同时对包含 99,990,000 个随机排列的重复行的 100,000,000 行（5 GB）文本文件进行重复数据删除，并且6分32秒结束。

当然，它确实有点作弊，因为它将临时索引文件写入磁盘（索引包含输入文件中每一行的哈希值，并且该行在输入文件中的位置）。每个文本行的索引文件需要 16 个字节，所以在我的测试中它达到了 ~1.4GB。

为进行重复数据删除，程序mmap() 将索引文件放入 RAM，按哈希码对其内容进行排序，然后扫描索引并使任何具有相同哈希码且引用相同字符串的现在相邻条目无效输入文件。

之后，它按字节偏移量对索引文件进行重新排序，然后再对索引进行一次迭代以生成去重输出文件。

我的测试运行（在 2018 Intel Mac Mini 上）的输出如下：

Jeremys-Mac-mini-2:~ jaf$ time ./a.out
Step 1:  Read input file [big_input_file.txt] and write index file [index_file.tmp]
Step 2:  mmap() index file [index_file.tmp] into RAM and sort its entries by hash-code
Step 3:  Iterate through index file [index_file.tmp] and invalidate any duplicate entries
Step 4:  Re-sort the index file [index_file.tmp] by byte-offset in preparation for generating output
Step 5:  Write output file [big_output_file.txt]
Step 6:  Delete index file and exit
Final result:  Out of 100000001 lines read, 99990000 duplicate lines detected and removed.
real        6m32.800s
user        3m39.215s
sys         2m41.448s

源代码如下（我用g++ -std=c++20 ./dedup_file.cpp编译）：

#include <fcntl.h>
#include <stdint.h>
#include <unistd.h>
#include <sys/mman.h>

#include <array>
#include <fstream>
#include <iostream>
#include <span>
#include <string>

using SizeTPair = std::array<size_t, 2>;

static const SizeTPair INVALID_INDEX_ENTRY = {(std::size_t)-1, (std::size_t)-1};  // special value meaning "not a valid index entry"

// Given a pointer into the contents of the input file, returns a string_view representing
// the line of text starting there.  (This is necessary since we can't modify the input file
// and the lines in the input file are not NUL-terminated)
static std::string_view GetStringAtOffset(const char * inputFileMap, size_t offset)
{
   if (offset == (size_t)-1) return "";

   const char * s  = &inputFileMap[offset];
   const char * nl = strchr(s, '\n');
   return nl ? std::string_view(s, nl-s) : std::string_view(s);
}

// Comparison functor to sort SizeTPairs by the text they point to
// breaks ties by sorting by line-number (so that if a duplicate line is
// detected in the text, it will always be the second instance of that line that
// is excluded from our program's output, not the first instance)
class SortIndicesByStringCompareFunctor
{
public:
   SortIndicesByStringCompareFunctor(const char * inputFileMap) : _inputFileMap(inputFileMap) {/* empty */}

   bool operator()(const SizeTPair & a, const SizeTPair & b) const
   {
      const std::string_view textA = GetStringAtOffset(_inputFileMap, a[0]);
      const std::string_view textB = GetStringAtOffset(_inputFileMap, b[0]);
      if (textA != textB) return (textA < textB);
      return (a[1] < b[1]); // sub-sort by line number
   }

private:
   const char * _inputFileMap;
};

static void WriteEntryToIndexFile(std::ofstream & indexFile, const SizeTPair & entry, size_t & indexSizeItems)
{
   indexFile.write(reinterpret_cast<const char *>(&entry), 2*sizeof(size_t));
   indexSizeItems++;
}

int main(int, char **)
{
   const char * bigInputFileName  = "big_input_file.txt";
   const char * indexFileName     = "index_file.tmp";
   const char * bigOutputFileName = "big_output_file.txt";

   std::cout << "Step 1:  Read input file [" << bigInputFileName << "] and write index file [" << indexFileName << "]" << std::endl;

   // Step 1.  Read through the big input-text file, and generate a binary
   // index-file containing (for each line) that line's hash-code and also
   // its location in the input file
   size_t indexSizeItems = 0;
   size_t inputFileSizeBytes = 0;
   {
      std::ifstream inputFile;
      inputFile.open(bigInputFileName, std::ios_base::binary | std::ios_base::ate);  // binary only so we can get valid file-offsets out of tellg()
      inputFileSizeBytes = inputFile.tellg();  // get file size
      inputFile.seekg(0, std::ios_base::beg);  // then go back to the beginning of the file so we can read it

      std::ofstream indexFile;
      indexFile.open(indexFileName, std::ios_base::binary);

      std::string nextLine;
      while(inputFile.good())
      {
         const std::streampos curFileOffset = inputFile.tellg();  // do this before reading the line:  record our current read-offset into the file
         std::getline(inputFile, nextLine);
         WriteEntryToIndexFile(indexFile, {std::hash<std::string>()(nextLine), (std::size_t)curFileOffset}, indexSizeItems);
      }

      // Add a final dummy-entry to the end of the index, just to force the flushing of any
      // final text-line(s) in our for-loop in step (3)
      WriteEntryToIndexFile(indexFile, INVALID_INDEX_ENTRY, indexSizeItems);
   }

   std::cout << "Step 2:  mmap() index file [" << indexFileName << "] into RAM and sort its entries by hash-code" << std::endl;

   // Step 2.  mmap() the index-file we just generated, and sort its contents by hash-code (sub-sort by byte-offset)
   const int indexFD = open(indexFileName, O_RDWR, (mode_t)0666);
   if (indexFD < 0) {std::cerr << "Couldn't open() index file!" << std::endl; exit(10);}

   char * indexFileMap = (char *) mmap(0, indexSizeItems*(2*sizeof(size_t)), PROT_READ | PROT_WRITE, MAP_SHARED, indexFD, 0);
   if (indexFileMap == MAP_FAILED) {std::cerr << "mmap() of index file failed!" << std::endl; exit(10);}

   SizeTPair * index = reinterpret_cast<SizeTPair *>(indexFileMap);
   std::span<SizeTPair> indexSpan(index, index+indexSizeItems);
   std::sort(std::begin(indexSpan), std::end(indexSpan));

   std::cout << "Step 3:  Iterate through index file [" << indexFileName << "] and invalidate any duplicate entries" << std::endl;

   // Step 3.  Go through the index file and invalidate any duplicate
   // entries (i.e. any entries that have the same hash code and same
   // underlying string as a previous entry)
   const int inputFD = open(bigInputFileName, O_RDONLY, (mode_t)0666);
   if (inputFD < 0) {std::cerr << "Couldn't open() input file!" << std::endl; exit(10);}

   const char * inputFileMap = (const char *) mmap(0, inputFileSizeBytes, PROT_READ, MAP_SHARED, inputFD, 0);
   if (indexFileMap == MAP_FAILED) {std::cerr << "mmap() of index file failed!" << std::endl; exit(10);}

   size_t dupesRemoved = 0;
   ssize_t runStartIdx = -1;
   for (size_t i=0; i<indexSizeItems; i++)
   {
      SizeTPair & curEntry = index[i];

      // swap to put the line number in [0] and the hash in [1], since in the future
      // we will want to sort by line number and this will make it easier to do that.
      std::swap(curEntry[0], curEntry[1]);

      const size_t curByteOffset = curEntry[0];
      const size_t curHash       = curEntry[1];

      if (runStartIdx >= 0)
      {
         if (curHash != index[runStartIdx][1])
         {
            // A run of identical hashes started at (runStartIdx) and ended just before (i)
            if ((i-runStartIdx)>1)
            {
               // Re-sort the index-entries-with-identical-hashes by the strings they represent
               // so that we can find and remove any duplicate-strings easily.  (We have to do this
               // because the same hash could, at least in principle, be associted with two different strings)
               std::span<SizeTPair> duplicateHashesSpan(index+runStartIdx, index+i);
               std::sort(std::begin(duplicateHashesSpan), std::end(duplicateHashesSpan), SortIndicesByStringCompareFunctor(inputFileMap));
               std::string_view previousEntryTextLine;
               for (size_t j=runStartIdx; j<i; j++)
               {
                  const std::string_view curEntryTextLine = GetStringAtOffset(inputFileMap, index[j][0]);
                  if (curEntryTextLine == previousEntryTextLine)
                  {
                     index[j] = INVALID_INDEX_ENTRY;
                     dupesRemoved++;
                  }
                  previousEntryTextLine = curEntryTextLine;
               }
            }
            runStartIdx = i;
         }
      }
      else runStartIdx = i;
   }

   std::cout << "Step 4:  Re-sort the index file [" << indexFileName << "] by byte-offset in preparation for generating output" << std::endl;

   // Step 4.  Re-sort the index file by byte-offset (note that each line's byte-offset is stored
   //          as the first entry in its SizeTPair now!)
   std::sort(std::begin(indexSpan), std::end(indexSpan));

   std::cout << "Step 5:  Write output file [" << bigOutputFileName << "]" << std::endl;

   // Step 5.  Read through the big text file one more time, and
   // write out only those lines that still exist in the index file
   std::ofstream outputFile;
   outputFile.open(bigOutputFileName);
   for (size_t i=0; i<indexSizeItems; i++)
   {
      const SizeTPair & curEntry = index[i];
      if (curEntry == INVALID_INDEX_ENTRY) break;  // these will all have been sorted to the end so we can stop now
                                      else outputFile << GetStringAtOffset(inputFileMap, curEntry[0]) << std::endl;
   }
   outputFile.close();

   // Step 6.  Clean up our mess and exit
   std::cout << "Step 6:  Delete index file and exit" << std::endl;
   close(inputFD);
   close(indexFD);
   remove(indexFileName);

   std::cout << "Final result:  Out of " << (indexSizeItems-1) << " lines read, " << dupesRemoved << " duplicate lines detected and removed. " << std::endl;
   return 0;
}

Answer 4

此代码逐行读取输入文件，仅将字符串的哈希值存储在内存中。如果之前没有看到该行，它将结果写入输出文件。如果之前看到过这条线，它不会做任何事情。

它使用sparsepp 来减少内存占用。

输入数据：

• 12 GB 文件大小
• ~197.000.000 不同的行
• 行长 < 120 个字符

建造：

• C++20
• 发布版本
• 不在 Visual Studio 中运行（未附加调试器）

加工：

• AMD锐龙2700X
• 32 GB 内存
• 希捷固态硬盘
• 190 秒
• 954 MB 虚拟内存峰值

这够好吗？我不能说，因为您的性能要求非常模糊，并且您没有提供适当的性能比较数据。这可能取决于您的机器、您的数据、您的 RAM 大小、您的硬盘速度……

#include <chrono>
#include <iostream>
#include <fstream>
#include <algorithm>
#include <array>
#include <cstring>
#include <functional>
#include <random>
#include <string>
#include "spp.h"
int main()
{
    std::chrono::steady_clock::time_point begin = std::chrono::steady_clock::now();
    std::ifstream input;
    std::ofstream output;
    input.open("10gb.txt");
    output.open("10gb_nodupes.txt");
    std::string inputline;
    spp::sparse_hash_map<size_t, size_t> hashes;
    while (std::getline(input, inputline))
    {
        std::size_t hash = std::hash<std::string>{}(inputline);
        if (!hashes.contains(hash))
        {
            output << inputline << '\n';
            hashes[hash]=0;
        }
    }
    input.close();
    output.close();
    std::chrono::steady_clock::time_point end = std::chrono::steady_clock::now();
    std::cout << "Time difference = " << std::chrono::duration_cast<std::chrono::seconds>(end - begin).count() << "[s]" << std::endl;
    std::cout << "Done";
}