Neo4j 导入工具 - OutOfMemory 错误：超出 GC 开销限制

【问题标题】：Neo4j import tool - OutOfMemory error: GC overhead limit exceededNeo4j 导入工具 - OutOfMemory 错误：超出 GC 开销限制
【发布时间】：2016-05-25 19:39:58
【问题描述】：

我正在使用 neo4j-import 工具 (Windows) 导入具有约 2000 万关系的约 100 万个节点，所有这些都应该是唯一的。该过程顺利进行，直到它到达“关系计数”任务，在那里它一直加载到 20M（似乎是所有关系），但随后它会挂起一段时间（30 分钟到 1 小时），最终返回“java.lang”。 lang.OutOfMemoryError：超出 GC 开销限制”。

我之前已经成功加载过大型图形数据库（39M 节点，21M 关系），所以我不确定问题出在哪里。是不是因为图数据库比我之前加载的数据库连接的更密集？

或者，可能存在内存泄漏？在我的任务管理器中，Java Platform SE 二进制进程在导入加载时需要越来越大的内存（16GB RAM 中最多 12-13GB），尤其是在接近尾声时。这看起来很可疑，尤其是因为 39M 节点/21M 关系图数据库能够使用导入工具相对较快地成功导入（没有挂在关系计数上）。

对可能出现的问题有任何想法吗？提前致谢！

如果查看我的节点/关系文件有帮助，这里是它们的链接： https://drive.google.com/open?id=0Bw7N-SlJA3ZCei0ycEhoa2YwNUU

这里是 neo4j shell 输出：

C:Users\Username\Documents\Neo4j>neo4jImport -into graphDB1.graphdb --nodes D:\concept.csv --relationships D:\predicate.csv --stacktrace --idtype integer
WARNING! This batch script has been deprecated. Please use the provided PowerShell scripts instead: http://neo4j.com/docs/stable/powershell.html
The system cannot find the path specified.
Importing the contents of these files into graphDB1.graphdb:
Nodes:
  D:\concept.csv
Relationships:
      D:\predicate.csv

Available memory:
  Free machine memory: 13.50 GB
  Max heap memory : 12.75 GB

    Nodes
[>:|PR|NOD|*LABEL SCAN---------------------------------|v:6.79 MB/s----------------------------]  1M
Done in 40s 562ms
Prepare node index
[*DETECT:20.37 MB------------------------------------------------------------------------------]  1M
Done in 802ms
Calculate dense nodes
[*>:59.38 MB/s----------------------------------|PREPARE(3)====================================] 20M
Done in 12s 566ms
Relationships
[>:2.01 |PREPARE-----------|P|RELATIONSHI|*v:4.05 MB/s-----------------------------------------] 20M
Done in 6m 3s 655ms
Node --> Relationship
[>:3.19 MB/s--------------------------|L|*v:2.39 MB/s------------------------------------------]  1M
Done in 8s 421ms
Relationship --> Relationship
[*>:6.82 MB/s--------------------------------------|LINK-----------|v:6.82 MB/s----------------] 20M
Done in 1m 36s 849ms
Node counts
[*COUNT:91.55 MB-------------------------------------------------------------------------------]  1M
Done in 3m 35s 21ms
Relationship counts
[*>:8.62 MB/s-----------------------------------------------------------|COUNT-----------------] 20MException in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOf(Unknown Source)
        at java.util.ArrayList.toArray(Unknown Source)
        at java.util.ArrayList.<init>(Unknown Source)
        at org.neo4j.unsafe.impl.batchimport.stats.StepStats.<init>(StepStats.java:39)
        at org.neo4j.unsafe.impl.batchimport.staging.AbstractStep.stats(AbstractStep.java:220)
        at org.neo4j.unsafe.impl.batchimport.staging.StageExecution$1.compare(StageExecution.java:123)
        at org.neo4j.unsafe.impl.batchimport.staging.StageExecution$1.compare(StageExecution.java:118)
        at java.util.TimSort.countRunAndMakeAscending(Unknown Source)
        at java.util.TimSort.sort(Unknown Source)
        at java.util.TimSort.sort(Unknown Source)
        at java.util.Arrays.sort(Unknown Source)
        at java.util.Collections.sort(Unknown Source)
        at org.neo4j.unsafe.impl.batchimport.staging.StageExecution.stepsOrderedBy(StageExecution.java:117)
        at org.neo4j.unsafe.impl.batchimport.staging.DynamicProcessorAssigner.assignProcessorsToPotentialBottleNeck(DynamicProcessorAssigner.java:94)
        at org.neo4j.unsafe.impl.batchimport.staging.DynamicProcessorAssigner.check(DynamicProcessorAssigner.java:81)
        at org.neo4j.unsafe.impl.batchimport.staging.MultiExecutionMonitor.check(MultiExecutionMonitor.java:106)
        at org.neo4j.unsafe.impl.batchimport.staging.ExecutionSupervisor.supervise(ExecutionSupervisor.java:65)
        at org.neo4j.unsafe.impl.batchimport.staging.ExecutionSupervisors.superviseExecution(ExecutionSupervisors.java:80)
        at org.neo4j.unsafe.impl.batchimport.ParallelBatchImporter.executeStages(ParallelBatchImporter.java:224)
        at org.neo4j.unsafe.impl.batchimport.ParallelBatchImporter.doImport(ParallelBatchImporter.java:185)
        at org.neo4j.tooling.ImportTool.main(ImportTool.java:363)
        at org.neo4j.tooling.ImportTool.main(ImportTool.java:279)

更新1：

这是导入在关系计数处挂起时的线程转储：

2016-02-17 08:28:12
Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.80-b11 mixed mode):

"MuninnPageCache[1]-FlushTask" daemon prio=6 tid=0x0000000026855800 nid=0xfe0 waiting on condition [0x00000000288fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c0189810> (a org.neo4j.io.pagecache.impl.muninn.MuninnPageCache)
        at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.continuouslyFlushPages(MuninnPageCache.java:909)
        at org.neo4j.io.pagecache.impl.muninn.FlushTask.run(FlushTask.java:36)
        at org.neo4j.io.pagecache.impl.muninn.BackgroundTask.run(BackgroundTask.java:45)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"MuninnPageCache[1]-EvictionTask" daemon prio=6 tid=0x0000000026904000 nid=0x3bd4 runnable [0x00000000287fe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000004c0189810> (a org.neo4j.io.pagecache.impl.muninn.MuninnPageCache)
        at java.util.concurrent.locks.LockSupport.parkNanos(Unknown Source)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.parkEvictor(MuninnPageCache.java:697)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.parkUntilEvictionRequired(MuninnPageCache.java:751)
        at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.continuouslySweepPages(MuninnPageCache.java:732)
        at org.neo4j.io.pagecache.impl.muninn.EvictionTask.run(EvictionTask.java:39)
        at org.neo4j.io.pagecache.impl.muninn.BackgroundTask.run(BackgroundTask.java:45)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)

"Service Thread" daemon prio=6 tid=0x0000000024ee8000 nid=0x301c runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x0000000024ee6000 nid=0x3060 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x0000000024ee2800 nid=0x2198 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" daemon prio=10 tid=0x0000000024ee2000 nid=0x1ae4 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x0000000024ee1000 nid=0x135c waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=8 tid=0x0000000024ed9000 nid=0x3480 in Object.wait() [0x00000000278ff000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000004c000d4b0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        - locked <0x00000004c000d4b0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(Unknown Source)
        at java.lang.ref.Finalizer$FinalizerThread.run(Unknown Source)

"Reference Handler" daemon prio=10 tid=0x0000000024ed8000 nid=0x1ae8 in Object.wait() [0x00000000277ff000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000004c000d300> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:503)
        at java.lang.ref.Reference$ReferenceHandler.run(Unknown Source)
        - locked <0x00000004c000d300> (a java.lang.ref.Reference$Lock)

"main" prio=6 tid=0x00000000023c2800 nid=0x2e7c waiting on condition [0x00000000023bf000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.neo4j.io.fs.FileUtils.waitAndThenTriggerGC(FileUtils.java:253)
        at org.neo4j.io.fs.FileUtils.deleteFile(FileUtils.java:110)
        at org.neo4j.io.fs.DefaultFileSystemAbstraction.deleteFile(DefaultFileSystemAbstraction.java:127)
        at org.neo4j.kernel.impl.storemigration.FileOperation$3.perform(FileOperation.java:93)
        at org.neo4j.kernel.impl.storemigration.StoreFile.fileOperation(StoreFile.java:267)
        at org.neo4j.tooling.ImportTool.main(ImportTool.java:389)
        at org.neo4j.tooling.ImportTool.main(ImportTool.java:279)

"VM Thread" prio=10 tid=0x0000000024ed1800 nid=0x3058 runnable

"GC task thread#0 (ParallelGC)" prio=6 tid=0x00000000023d7000 nid=0x313c runnable

"GC task thread#1 (ParallelGC)" prio=6 tid=0x00000000023d9000 nid=0x3144 runnable

"GC task thread#2 (ParallelGC)" prio=6 tid=0x00000000023da800 nid=0x974 runnable

"GC task thread#3 (ParallelGC)" prio=6 tid=0x00000000023dc000 nid=0x3a3c runnable

"GC task thread#4 (ParallelGC)" prio=6 tid=0x00000000023de800 nid=0x3684 runnable

"GC task thread#5 (ParallelGC)" prio=6 tid=0x00000000023e1000 nid=0x35b8 runnable

"GC task thread#6 (ParallelGC)" prio=6 tid=0x00000000023e4000 nid=0x3950 runnable

"GC task thread#7 (ParallelGC)" prio=6 tid=0x00000000023e5800 nid=0x318c runnable

"GC task thread#8 (ParallelGC)" prio=6 tid=0x00000000023e8800 nid=0x30b8 runnable

"GC task thread#9 (ParallelGC)" prio=6 tid=0x00000000023e9800 nid=0x32dc runnable

"VM Periodic Task Thread" prio=10 tid=0x0000000024eed800 nid=0x3710 waiting on condition

JNI global references: 377

Heap
 PSYoungGen      total 2071552K, used 0K [0x0000000780000000, 0x0000000800000000, 0x0000000800000000)
  eden space 2043904K, 0% used [0x0000000780000000,0x0000000780000000,0x00000007fcc00000)
  from space 27648K, 0% used [0x00000007fe500000,0x00000007fe500000,0x0000000800000000)
  to   space 25600K, 0% used [0x00000007fcc00000,0x00000007fcc00000,0x00000007fe500000)
 ParOldGen       total 11534336K, used 10982258K [0x00000004c0000000, 0x0000000780000000, 0x0000000780000000)
  object space 11534336K, 95% used [0x00000004c0000000,0x000000075e4dcb50,0x0000000780000000)
 PSPermGen       total 21504K, used 13521K [0x00000004bae00000, 0x00000004bc300000, 0x00000004c0000000)
  object space 21504K, 62% used [0x00000004bae00000,0x00000004bbb34588,0x00000004bc300000)

2016-02-17 08:28:20

【问题讨论】：

    标签： java csv import neo4j garbage-collection


    【解决方案1】：

    在这么小的数据集上这很奇怪。您希望在这个数据集中有多少独特的关系和标签？您还可以在暂停时提供线程转储吗？

    编辑：问题是包含属性值的列被用作LABEL。这会错误地生成大量标签，而计数与此不成比例。

    【讨论】：

    • 所有节点都是唯一的，我刚刚更新了导出为 CSV 的 MySQL 表，因此数据文件中的 (START_ID, TYPE, END_ID) 关系三元组也应该是唯一的。这确实导致关系 csv 从 ~21M 减少到 ~20M 行，所以我重新运行了导入。不幸的是，它仍然挂在关系计数上。另外，我在问题中添加了线程转储。
    • 另外，不确定它是否有用，但我添加了指向节点和关系 csv 文件的链接。
    • 非常感谢。线程转储似乎来自异常发生之后，这并不是真正有用，因为它不会告诉异常发生之前它在哪里。确保在异常发生之前抓住一个。
    • 哦，等一下，您正在使用PREFERRED_NAME1:LABEL 列作为标签，这意味着大多数节点将有不同的标签，就像每个新节点都有新的唯一标签一样。这就是导致计数阶段爆炸的原因。您应该在数据库中有几个（数十、数百左右）标签/关系类型。
    • 啊，这就是问题所在。我删除了该列并且导入工作。谢谢！如果你现在想编辑你的答案，我会继续接受它。
    猜你喜欢
    • 2017-11-12
    • 2021-03-14
    • 1970-01-01
    • 1970-01-01
    • 2016-12-03
    • 2010-11-26
    • 1970-01-01
    • 2018-04-10
    相关资源
    最近更新 更多
    热门标签
    Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode