【问题标题】:Not able to start cassandra无法启动 cassandra
【发布时间】:2017-11-21 18:23:38
【问题描述】:

我遇到了 Cassandra 的问题,每当我尝试启动 Cassandra 时,我都会收到打开文件过多的错误。

我已将文件描述符增加到1000000,仍然是同样的错误。

[更新]

我查看了调试日志,一开始它打开了许多 sstables。这是日志

调试 [SSTableBatchOpen:2] 2017-06-20 11:03:40,635 SSTableReader.java:479 - 打开 /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-159921-big(60 字节) 调试 [SSTableBatchOpen:1] 2017-06-20 11:03:40,635 SSTableReader.java:479 - 打开 /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-166961-big(49 字节) 调试 [SSTableBatchOpen:4] 2017-06-20 11:03:40,635 SSTableReader.java:479 - 打开 /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-181150-big(57 字节) 调试 [SSTableBatchOpen:3] 2017-06-20 11:03:40,635 SSTableReader.java:479 - 打开 /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-188190-big(49 字节) 调试 [SSTableBatchOpen:2] 2017-06-20 11:03:40,635 SSTableReader.java:479 - 打开 /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-173741-big(54 字节) 调试 [SSTableBatchOpen:1] 2017-06-20 11:03:40,635 SSTableReader.java:479 - 打开 /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-159987-big(45 字节) 调试 [SSTableBatchOpen:3] 2017-06-20 11:03:40,635 SSTableReader.java:479 - 打开 /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-167027-big(49 字节) 调试 [SSTableBatchOpen:4] 2017-06-20 11:03:40,635 SSTableReader.java:479 - 打开 /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-198208-big(53 字节) 调试 [SSTableBatchOpen:1] 2017-06-20 11:03:40,636 SSTableReader.java:479 - 打开 /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-177001-big(48 字节) 调试 [SSTableBatchOpen:2] 2017-06-20 11:03:40,636 SSTableReader.java:479 - 打开 /cassandra/cass/data/system/IndexInfo-9f5c6374d48532299a0a5094af9ad1e3/mc-184041-big (57 bytes 以下是系统日志:

ERROR [SSTableBatchOpen:1] 2017-06-19 19:08:40,175 CassandraDaemon.java:205 - Exception in thread Thread[SSTableBatchOpen:1,5,main]
java.lang.RuntimeException: java.io.FileNotFoundException: /cassandra/cass/data/crownit/activitylog-60fcc250bc7211e6995a87b62bcc4eac/.controller_idx/mc-1033-big-CompressionInfo.db (Too many open files)
        at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:127) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.compress.CompressionMetadata.create(CompressionMetadata.java:91) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.metadata(CompressedSegmentedFile.java:125) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.util.CompressedSegmentedFile$Builder.complete(CompressedSegmentedFile.java:132) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.util.SegmentedFile$Builder.complete(SegmentedFile.java:177) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.util.SegmentedFile$Builder.buildData(SegmentedFile.java:193) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:745) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.sstable.format.SSTableReader.load(SSTableReader.java:706) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:492) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:375) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at org.apache.cassandra.io.sstable.format.SSTableReader$4.run(SSTableReader.java:534) ~[apache-cassandra-3.0.9.jar:3.0.9]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_101]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_101]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Caused by: java.io.FileNotFoundException: /cassandra/cass/data/crownit/activitylog-60fcc250bc7211e6995a87b62bcc4eac/.controller_idx/mc-1033-big-CompressionInfo.db (Too many open files)
        at java.io.FileInputStream.open0(Native Method) ~[na:1.8.0_101]
        at java.io.FileInputStream.open(FileInputStream.java:195) ~[na:1.8.0_101]
        at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[na:1.8.0_101]
        at java.io.FileInputStream.<init>(FileInputStream.java:93) ~[na:1.8.0_101]
        at org.apache.cassandra.io.compress.CompressionMetadata.<init>(CompressionMetadata.java:100) ~[apache-cassandra-3.0.9.jar:3.0.9]
        ... 15 common frames omitted
ERROR [SSTableBatchOpen:1] 2017-06-19 19:08:40,177 JVMStabilityInspector.java:140 - JVM state determined to be unstable.  Exiting forcefully due to:
java.io.FileNotFoundException: /cassandra/cass/data/crownit/activitylog-60fcc250bc7211e6995a87b62bcc4eac/.controller_idx/mc-1033-big-CompressionInfo.db (Too many open files)

【问题讨论】:

  • 检查此文件是否存在/cassandra/cass/data/crownit/activitylog-60fcc250bc7211e6995a87b62bcc4eac/.controller_idx/mc-1033-big-CompressionInfo.db
  • 文件存在
  • @Arvind 似乎是 java 错误而不是 cassendra : 1. 尝试重新启动您的机器,以释放空间。 2. 如果,不关注 1.尝试杀死你的本地 java 线程并重新启动你的 IDE 3.stackoverflow.com/questions/13706409/… 签出更新 ulimit 的链接。我遇到了同样的错误,我重新启动工作正常。
  • 15GB 空间可用,我已尝试重新启动机器,但对我不起作用。我已经增加了 Ulimit。

标签: java cassandra


【解决方案1】:

由于我无法评论上一个答案,这里有个小提示:

您的 cassandra 是如何启动的?它是如何安装的?您的 ulimit 更改可能不会影响用户 cassandra 运行(在您的数据目录中仔细检查 ls -l 以查看文件是谁)。使用 debian 包 cassandra 以用户 cassandra 运行,限制设置为

cassandra01:/etc$ cat security/limits.d/cassandra.conf
# Provided by the cassandra package
cassandra  -  memlock  unlimited
cassandra  -  nofile   100000
cassandra  -  as       unlimited
cassandra  -  nproc    8096
cassandra01:/etc$

您的数据目录中有多少个 sstable?

尝试找出在崩溃之前打开了多少文件,如下所示:

lsof -n | grep java

【讨论】:

  • 我还更新了 cassandra.conf,在崩溃前打开了大约 20000 个文件,我有超过 40000 个 ssTables
  • 40000 个 SSTable 真的很多 - 特别是如果你记住,每个 sstable 都有一些文件(索引、目录、布隆过滤器等)你能检查一下输出 'cat /proc/ sys/fs/file-max` 产生? fs.file-max 是系统范围的文件句柄限制 - 您可能会遇到这个限制。还有你有多少个cloumnfamilies?有很多很多小的 sstables (*-Data.db)?
猜你喜欢
  • 2021-02-25
  • 1970-01-01
  • 2012-08-07
  • 2011-04-21
  • 2018-08-17
  • 2015-04-22
  • 1970-01-01
  • 2017-08-05
  • 2018-10-25
相关资源
最近更新 更多