从 MongoDB Node.js 驱动程序使用批量插入时内存泄漏答案

【问题标题】：memory leak when using bulk insert from MongoDB Node.js driver从 MongoDB Node.js 驱动程序使用批量插入时内存泄漏
【发布时间】：2020-09-02 17:55:23
【问题描述】：

我正在使用 node 12.16.3 和 mongodb 包 3.5.7。

我正在尝试将 1000 万个文档插入 mongodb。我的第一个想法是使用以下代码逐个插入每个文档：

const MongoClient = require("mongodb").MongoClient;
const url = "mongodb://localhost:27017";
const dbName = "myDb";
const client = new MongoClient(url);

client.connect(async function () {
  const db = client.db(dbName);
  await insertDocuments(db, 10000000);
  client.close();
});

const insertDocuments = async function (db, nbDocuments) {
  const collection = db.collection("documents");

  for (let i = 1; i <= nbDocuments; i++) {
    await collection.insert({ number: i });
  }
};

这似乎有效，但需要花费大量时间（大约超过 40 分钟，并且一旦将所有文档插入 db，节点进程就会冻结，并且不会按预期结束，并在控制台中显示一条消息： 在 xx 秒内完成）

我的第二个想法是使用批量插入，所以我更新了insertDocuments：

const insertDocuments = async function (db, nbDocuments) {
  const collection = db.collection("documents");

  const bulk = collection.initializeOrderedBulkOp();

  for (let i = 1; i <= nbDocuments; i++) {
    collection.insert({ number: i });
  }

  await bulk.execute()
};

但是有了这个，我遇到了内存泄漏：

<--- Last few GCs --->

[3649:0x2ef6970]    21379 ms: Mark-sweep 2047.7 (2051.5) -> 2046.8 (2051.3) MB, 1364.6 / 0.0 ms  (+ 115.4 ms in 23 steps since start of marking, biggest step 5.5 ms, walltime since start of marking 1497 ms) (average mu = 0.118, current mu = 0.022) allocat[3649:0x2ef6970]    22603 ms: Mark-sweep 2048.9 (2052.3) -> 2048.8 (2053.3) MB, 1098.4 / 0.0 ms  (+ 111.7 ms in 21 steps since start of marking, biggest step 14.0 ms, walltime since start of marking 1224 ms) (average mu = 0.067, current mu = 0.011) alloca

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x13c5b79]
Security context: 0x24a9afdc08d1 <JSObject>
    1: /* anonymous */(aka /* anonymous */) [0x32acdc77d799] [/home/rcheck/node_ts_starter/build/index.js:~87] [pc=0x2416a87dd0b9](this=0x3dab77d404b1 <undefined>,0x32acdc77d829 <Object map = 0x123f2f4420a9>)
    2: step(aka step) [0x32acdc77d8c9] [/home/rcheck/node_ts_starter/build/index.js:33] [bytecode=0xc3b5024c5c9 offset=751](this=0x3dab77d404b1 <undefined>,...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20200516.225751.3649.0.001.json
Node.js report completed
 1: 0xa09830 node::Abort() [node]
 2: 0xa09c55 node::OnFatalError(char const*, char const*) [node]
 3: 0xb7d71e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb7da99 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd2a1f5  [node]
 6: 0xd2a886 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xd37105 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node] 8: 0xd37fb5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xd3aa6c v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xd0163b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [node]
11: 0x104316c v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long*, v8::internal::Isolate*) [node]
12: 0x13c5b79  [node]
Aborted (core dumped)
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

所以我想知道是否有更有效的方法将 1000 万个文档插入到 mongodb 中（而不是一个一个地插入它们，因为它需要很多时间）。

提前致谢

【问题讨论】：

标签： node.js mongodb memory-leaks

【解决方案1】：

我终于做到了，通过制作多个批量（每个批量包含 100 万个文档），整个操作需要 64 秒：

const 集合 = await db.collection("documents");

  const documents: {}[] = [];

  for (let i = 1; i <= 10000000; i++) {
    documents.push({ number: i });
    if (i > 0 && i % 1000000 === 0) {
      await collection.insertMany(documents);
      documents.length = 0;
    }
  }

【讨论】：