【问题标题】:Cassandra 1.1.1 crashes while inserting heavy data using Hector 1.0.5Cassandra 1.1.1 在使用 Hector 1.0.5 插入大量数据时崩溃
【发布时间】:2012-07-05 14:08:36
【问题描述】:

我正在使用 Cassandra 1.1.1 和使用 Hector 1.0.5,我正在尝试将数据(大量)插入到列族中。在我的程序执行期间,cassandra 服务器崩溃并显示 Out-of-memory 错误。之后我别无选择,只能退出服务器。对于我试图存储 html 文件内容的一个列族,这会重复,但我从来没有机会完成它。 html 文件内容从 225 KB 数据到一行 700 KB 数据不等,我试图插入近 1000 条记录。


在程序中它抛出以下内容

Exception in thread "main" me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client.
    at me.prettyprint.cassandra.connection.HConnectionManager.getClientFromLBPolicy(HConnectionManager.java:393)
    at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:249)
    at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
    at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
    at com.epocrates.soa.rx.util.DiseaseImporter.insertDisease(DiseaseImporter.java:207)
    at com.epocrates.soa.rx.util.DiseaseImporter.batchProcess(DiseaseImporter.java:81)
    at com.epocrates.soa.rx.

util.DiseaseImporter.main(DiseaseImporter.java:37)

在 System.log 中,我找到以下内容

java.io.IOError: java.io.IOException: Map failed
    at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
    at org.apache.cassandra.db.commitlog.CommitLogSegment.freshSegment(CommitLogSegment.java:80)
    at org.apache.cassandra.db.commitlog.CommitLogAllocator.createFreshSegment(CommitLogAllocator.java:244)
    at org.apache.cassandra.db.commitlog.CommitLogAllocator.access$500(CommitLogAllocator.java:49)
    at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:104)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Map failed
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)
    at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:119)
    ... 6 more
Caused by: java.lang.OutOfMemoryError: Map failed
    at sun.nio.ch.FileChannelImpl.map0(Native Method)
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:755)
    ... 7 more

java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down
    at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
    at org.apache.cassandra.service.StorageProxy.insertLocal(StorageProxy.java:457)
    at org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:314)
    at org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:119)
    at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:260)
    at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:193)
    at org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:637)
    at org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:587)
    at org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:595)
    at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3112)
    at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3100)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
    at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

【问题讨论】:

    标签: cassandra hector


    【解决方案1】:

    这意味着您已经用完了将提交日志段映射到的地址空间。

    最佳解决方案:升级到 64 位 JVM。

    更糟糕的解决方案:在 cassandra.yaml 中,将 commitlog_segment_size_in_mb 和 commitlog_total_space_in_mb 都设置为 16。

    这不是第一次出现;我已经打开 https://issues.apache.org/jira/browse/CASSANDRA-4422 来改进默认设置。

    【讨论】:

    • 感谢您的回复约翰。我选择了更糟糕的解决方案,因为我需要一些时间来测试最佳解决方案。当我说“commitlog_total_space_in_mb”到 16 时,它成功了。
    猜你喜欢
    • 2013-08-27
    • 1970-01-01
    • 2012-03-04
    • 1970-01-01
    • 2012-08-04
    • 1970-01-01
    • 2021-12-11
    • 2012-04-25
    • 2012-03-12
    相关资源
    最近更新 更多