Cassandra EOFException 在电源故障后在某些查询上引发

Question

  `INFO 11:44:29,874 Listening for thrift clients...
ERROR 11:47:01,471 Exception in thread Thread[ReadStage:33,5,main]
java.lang.RuntimeException: org.apache.cassandra.io.sstable.CorruptSSTableExcept
ion: java.io.EOFException: EOF after 2 bytes out of 29545
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(Stora
geProxy.java:1582)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFE
xception: EOF after 2 bytes out of 29545
        at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(Simpl
eSliceReader.java:89)
        at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createRea
der(SSTableSliceIterator.java:68)
        at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SS
TableSliceIterator.java:44)
        at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnItera
tor(SliceQueryFilter.java:101)
        at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(Q
ueryFilter.java:68)
        at org.apache.cassandra.db.CollationController.collectAllData(CollationC
ontroller.java:275)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(Collat
ionController.java:65)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFa
milyStore.java:1363)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamil
yStore.java:1220)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamil
yStore.java:1132)
        at org.apache.cassandra.db.Table.getRow(Table.java:355)
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadComm
and.java:70)
        at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThr
ow(StorageProxy.java:1052)
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(Stora
geProxy.java:1578)
        ... 3 more
Caused by: java.io.EOFException: EOF after 2 bytes out of 29545
        at org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:
350)
        at org.apache.cassandra.utils.ByteBufferUtil.skipShortLength(ByteBufferU
til.java:382)
        at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(Simpl
eSliceReader.java:71)
        ... 16 more
ERROR 11:47:12,900 Exception in thread Thread[ReadStage:35,5,main]
java.lang.RuntimeException: org.apache.cassandra.io.sstable.CorruptSSTableExcept
ion: java.io.EOFException: EOF after 2 bytes out of 29545
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(Stora
geProxy.java:1582)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.cassandra.io.sstable.CorruptSSTableException: java.io.EOFE
xception: EOF after 2 bytes out of 29545
        at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(Simpl
eSliceReader.java:89)
        at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createRea
der(SSTableSliceIterator.java:68)
        at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SS
TableSliceIterator.java:44)
        at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnItera
tor(SliceQueryFilter.java:101)
        at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(Q
ueryFilter.java:68)
        at org.apache.cassandra.db.CollationController.collectAllData(CollationC
ontroller.java:275)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(Collat
ionController.java:65)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFa
milyStore.java:1363)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamil
yStore.java:1220)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamil
yStore.java:1132)
        at org.apache.cassandra.db.Table.getRow(Table.java:355)
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadComm
and.java:70)
        at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThr
ow(StorageProxy.java:1052)
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(Stora
geProxy.java:1578)
        ... 3 more
Caused by: java.io.EOFException: EOF after 2 bytes out of 29545
        at org.apache.cassandra.io.util.FileUtils.skipBytesFully(FileUtils.java:
350)
        at org.apache.cassandra.utils.ByteBufferUtil.skipShortLength(ByteBufferU
til.java:382)
        at org.apache.cassandra.db.columniterator.SimpleSliceReader.<init>(Simpl
eSliceReader.java:71)
        ... 16 more`

在 Hector 中，我得到一个 HTimedOutException，我假设这是因为 cassandra 中的 SSTable 损坏，因为它在电源故障之前运行良好。我该如何解决？这也是开发数据库，​​我之前遇到过这个问题，最终放弃了一些列族，但我不希望这是每次断电或系统崩溃时唯一的解决方案。我正在使用 cassandra 1.2.2

Answer 1

您可以删除损坏的 SSTable，然后运行修复以从副本中恢复数据，而不是删除整个 CF。问题是错误不会在日志中打印文件名，因此很难找出哪个文件已损坏。您可以通过在调试模式下运行 Cassandra 来获得它（编辑 log4j-server.properties）。请参阅其他 SO 帖子：

Error messages while starting Cassandra node

他们是如何做到的。

Answer 2

也许首先要尝试的是nodetool scrubthis cf（例如，参见this post for v.1.2）。