【发布时间】：2017-06-03 17:21:30
【问题描述】：
从最近几天开始，我在 cassandra 集群中的一些节点上看到由于以下错误，DSE 正在关闭，我需要终止 Java 进程并重新启动 DSE 服务。
我已经交叉检查了读取、写入和压缩，看起来没有任何可疑之处，但我看到这些服务器上的 Gc 完全暂停，就在问题发生之前。这可能是什么原因如何调试？当这些节点像集群的其他部分一样服务相同的请求时，如何细化为什么在这些特定节点上会发生这种情况？？
这是因为 Full Gc 没有正确执行，我们使用 G1GC 和 DSE 4.8.3
ERROR [SharedPool-Worker-25] 2016-12-27 10:14:26,100 JVMStabilityInspector.java:117 - JVM state determined to be unstable. Exiting forcefully due to:java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3181) ~[na:1.8.0_74]
at org.apache.cassandra.db.RangeTombstoneList.copy(RangeTombstoneList.java:112) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.DeletionInfo.copy(DeletionInfo.java:104) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.AtomicBTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:217) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.Memtable.put(Memtable.java:210) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1230) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_74]
at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
ERROR [SharedPool-Worker-25] 2016-12-27 10:14:28,100 SEPWorker.java:141 - Failed to execute task, unexpected exception killed worker: {}
java.lang.IllegalStateException: Shutdown in progress
at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[na:1.8.0_74]
at java.lang.Runtime.removeShutdownHook(Runtime.java:239) ~[na:1.8.0_74]
at org.apache.cassandra.service.StorageService.removeShutdownHook(StorageService.java:764) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:119) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:109) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:68) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:168) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131]
at
java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
INFO [Thread-6] 2016-12-27 10:14:56,150 DseDaemon.java:420 - DSE shutting down...
【问题讨论】：
-
内存不足异常，你的堆有多大？
-
@phact 堆为 12Gb，新堆为 4Gb。
-
您在使用搜索吗？
-
检查堆转储的一个好方法是检查堆中的内容。 .Hprof 文件
-
您好，我也遇到了同样的问题。你有解决办法吗？
标签： java garbage-collection datastax datastax-enterprise cassandra-2.1