cassandra 启动时遇到异常：索引（1）必须小于大小（1）

【问题标题】：cassandra Exception encountered during startup: index (1) must be less than size (1)cassandra 启动时遇到异常：索引（1）必须小于大小（1）
【发布时间】：2013-10-15 18:11:30
【问题描述】：

我的 Cassandra 集群中的所有 4 个节点都停电了。我已经带回了所有的盒子，但是现在当我尝试启动 Cassandra（方法 bin/Cassandra）时，在重播日志后，我在 4 个中的 3 个上收到以下消息。

Exception encountered during startup: index (1) must be less than size (1)
 INFO 10:11:00,479 CFS(Keyspace='system', ColumnFamily='peers') liveRatio is 13.10204081632653 (just-counted was 13.10204081632653).  calculation took 18ms for 21 columns

（这只是结束，如果有帮助，我可以发布更多信息）

出现的一个只为读取而运行，我假设因为我在所有键空间上都有复制因子 3，所以如果没有其他键空间它就无法工作。

关于如何检索情况的任何想法。我在数据库中有很多有价值的信息。

Cassandra v1.2.6，

CentOs v6.4

更新：

在重放提交日志期间，所有三个死节点似乎都失败了。我已将它们到达的行加粗。我最初的想法是删除该提交日志并尝试重新启动.. 但我有点害怕后果！

 INFO 10:46:48,600 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1379365208291.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208292.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208293.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208294.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208295.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208296.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208297.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208298.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208299.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208301.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208302.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208304.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208305.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208306.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208307.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208308.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208309.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208310.log, /var/lib/cassandra/commitlog/CommitLog-2-1379365208311.log, /var/lib/cassandra/commitlog/CommitLog-2-1381251642075.log, /var/lib/cassandra/commitlog/CommitLog-2-1381251731119.log
 INFO 10:46:48,614 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1379365208291.log
 INFO 10:46:49,598 GC for ParNew: 605 ms for 2 collections, 50055912 used; max is 1046937600
 INFO 10:46:51,775 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1379365208291.log
 INFO 10:46:51,776 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1379365208292.log
 INFO 10:46:53,995 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1379365208292.log
 INFO 10:46:53,995 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1379365208293.log
 **INFO 10:46:54,087 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1379365208293.log**
ERROR 10:46:54,088 Exception encountered during startup
java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
    at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
    at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
    at com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45)
    at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:94)
    at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76)
    at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
    at java.util.TreeMap.compare(TreeMap.java:1188)
    at java.util.TreeMap.put(TreeMap.java:531)
    at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:102)
    at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:88)
    at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114)
    at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109)
    at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:101)
    at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:397)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:202)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:97)
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146)
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:298)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:441)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:484)
java.lang.IndexOutOfBoundsException: index (1) must be less than size (1)
    at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:305)
    at com.google.common.base.Preconditions.checkElementIndex(Preconditions.java:284)
    at com.google.common.collect.SingletonImmutableList.get(SingletonImmutableList.java:45)
    at org.apache.cassandra.db.marshal.CompositeType.getComparator(CompositeType.java:94)
    at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:76)
    at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
    at java.util.TreeMap.compare(TreeMap.java:1188)
    at java.util.TreeMap.put(TreeMap.java:531)
    at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:102)
    at org.apache.cassandra.db.TreeMapBackedSortedColumns.addColumn(TreeMapBackedSortedColumns.java:88)
    at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114)
    at org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:109)
    at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:101)
    at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:397)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:202)
    at org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:97)
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146)
    at org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126)
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:298)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:441)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:484)
Exception encountered during startup: index (1) must be less than size (1)
 INFO 10:46:54,110 CFS(Keyspace='system', ColumnFamily='peers') liveRatio is 13.10204081632653 (just-counted was 13.10204081632653).  calculation took 19ms for 21 columns

【问题讨论】：

  • 删除提交日志意味着可能会丢失任何已写入但未刷新到 cassandra 的数据。这似乎是known issue (this as well)。
  • 是的，我认为可能是这样。为了将来参考，我是否应该定期刷新节点，即 nodetool 刷新？
  • C* 定期进行刷新（我的头顶上是 4 小时）

标签： cassandra


【解决方案1】：

我最好的理论是，这是删除并重新创建同名表 (https://issues.apache.org/jira/browse/CASSANDRA-5905) 的可能结果。我们的目标是修复 2.1 (https://issues.apache.org/jira/browse/CASSANDRA-5202)；同时，更喜欢 TRUNCATE 而不是 drop/recreate。

【讨论】：

  • 谢谢@peter。这非常有趣，因为我确实遇到了这种情况。我已经在 cassandra 上运行了 hadoop，并且对于许多作业，他们需要在运行之前清除目标输出表，这是通过删除并重新创建表来完成的。我最终通过删除受影响的文件让所有死节点启动并运行（我知道我很可能有一些数据丢失，但我们走了）。我以为文件因为崩溃而损坏了，但它们不是最后的日志文件，而是散落在各处。我想这会支持你的理论？
【解决方案2】：

虽然已经很晚了，但您也必须解决此问题。对于其他有类似问题的人，试试这个

sudo bash -c 'rm -rf /var/lib/cassandra/data/system/*'

请记住，它仅用于开发目的。 此命令将删除您的所有数据。

【讨论】：

    猜你喜欢
    • 1970-01-01
    • 2015-05-07
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    • 2016-04-09
    • 1970-01-01
    • 1970-01-01
    • 1970-01-01
    相关资源
    最近更新 更多
    热门标签
    Java Python linux javascript Mysql C# Docker 算法 前端 SpringBoot Redis Vue spring 设计模式 .net core .net kubernetes c++ 数据库 数据结构 大数据 js 机器学习 微服务 Android Go 程序员 面试 JVM ASP.net core 云原生 人工智能 后端 PHP git CSS golang k8s Nginx Django mybatis 深度学习 多线程 React 架构 devops 爬虫 云计算 Spring Boot LeetCode