由于 sstable 损坏，无法启动 Cassandra答案

【问题标题】：Unable to start Cassandra due to corrupted sstable由于 sstable 损坏，无法启动 Cassandra
【发布时间】：2016-09-29 00:40:36
【问题描述】：

运行后：- sudo service cassandra start 后跟 sudo service cassandra status 我得到一个无法访问 Cassandra 的 pidfile。

当我检查它给我的日志时，我认为它是稳定的损坏，但找不到任何解决方法。

ERROR [SSTableBatchOpen:1] 2016-05-30 23:17:42,301 FileUtils.java:447 - 
Exiting forcefully due to file system exception on startup, disk failure policy "stop"

org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFException
    at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:131) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:85) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:79) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Builder.complete(CompressedPoolingSegmentedFile.java:72) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:169) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:741) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:692) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:480) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:376) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.sstable.SSTableReader$4.run(SSTableReader.java:523) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_80]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) [na:1.7.0_80]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_80]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_80]
    at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]

Caused by: java.io.EOFException: null
    at java.io.DataInputStream.readUnsignedShort(DataInputStream.java:340) ~[na:1.7.0_80]
    at java.io.DataInputStream.readUTF(DataInputStream.java:589) ~[na:1.7.0_80]
    at java.io.DataInputStream.readUTF(DataInputStream.java:564) ~[na:1.7.0_80]
    at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:106) ~[apache-cassandra-2.1.11.jar:2.1.11]
    ... 14 common frames omitted

删除 sstables 后，我收到了这个额外的错误。

ERROR [SSTableBatchOpen:2] 2016-06-13 22:44:59,177 CassandraDaemon.java:227 - Exception in thread Thread[SSTableBatchOpen:2,5,main]
java.lang.IllegalStateException: Shutdown in progress
    at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[na:1.7.0_80]
    at java.lang.Runtime.removeShutdownHook(Runtime.java:239) ~[na:1.7.0_80]
    at org.apache.cassandra.service.StorageService.removeShutdownHook(StorageService.java:758) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:119) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.utils.JVMStabilityInspector.killCurrentJVM(JVMStabilityInspector.java:88) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.io.util.FileUtils.handleStartupFSError(FileUtils.java:450) ~[apache-cassandra-2.1.11.jar:2.1.11]

我也意识到其他节点上的错误，但我能够启动它们。 NullPointerException

ERROR [GossipStage:1] 2016-06-13 23:06:31,317 CassandraDaemon.java:227 - Exception in thread Thread[GossipStage:1,5,main]
java.lang.NullPointerException: null
    at org.apache.cassandra.service.StorageService.getApplicationStateValue(StorageService.java:1624) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.service.StorageService.getTokensFor(StorageService.java:1632) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.service.StorageService.handleStateNormal(StorageService.java:1686) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.service.StorageService.onChange(StorageService.java:1510) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.service.StorageService.onJoin(StorageService.java:2161) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.gms.Gossiper.handleMajorStateChange(Gossiper.java:1042) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.gms.Gossiper.applyStateLocally(Gossiper.java:1115) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.gms.GossipDigestAck2VerbHandler.doVerb(GossipDigestAck2VerbHandler.java:49) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[apache-cassandra-2.1.11.jar:2.1.11]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_80]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_80]
    at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_80]

【问题讨论】：

我会假设一些事情，所以我可能是错的。如果只是安装 cassandra，需要先擦除一些临时数据才能运行：sudo rm -rf /var/lib/cassandra/data/system/*
我已尝试删除该文件夹中的文件，但仍然出现相同的错误
issues.apache.org/jira/browse/CASSANDRA-10534 很可能是您遇到的问题。找到“CompressionInfo.db”组件大小为 0 的 sstables 并删除它们。备份节点后，请确保运行修复
删除 sstables 后，我仍然遇到同样的错误，还有一个错误。还有其他解决方案吗？
我有同样的问题，我关注了engineering.gosquared.com/dealing-corrupt-sstable-cassandra博客，它已经解决了我的问题。

标签： ubuntu cassandra

【解决方案1】：

在 ubuntu 16.04+cassandra 3.4.0 上，我对这个问题的解决方案是删除异常指向的中间文件。我没有丢失任何开发数据，但我现在正在做备份。

【讨论】：