【发布时间】：2016-01-06 12:02:03
【问题描述】：
Cassandra 2.1.5 的性能非常差。我对此并不陌生，因此将不胜感激有关如何调试的任何建议。这是我的桌子的样子：
Keyspace: nt_live_october x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Read Count: 6 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Read Latency: 20837.149166666666 ms. x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Write Count: 39799 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Write Latency: 0.45696595391844014 ms. x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Pending Flushes: 0 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Table: nt x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
SSTable count: 12 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Space used (live): 15903191275 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Space used (total): 15971044770 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Space used by snapshots (total): 0 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Off heap memory used (total): 14468424 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
SSTable Compression Ratio: 0.1308103413354315 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Number of keys (estimate): 740 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Memtable cell count: 43483 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Memtable data size: 9272510 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Memtable off heap memory used: 0 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Memtable switch count: 17 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Local read count: 6 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Local read latency: 20837.150 ms x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Local write count: 39801 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Local write latency: 0.457 ms x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Pending flushes: 0 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Bloom filter false positives: 0 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Bloom filter false ratio: 0.00000 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Bloom filter space used: 4832 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Bloom filter off heap memory used: 4736 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Index summary off heap memory used: 576 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Compression metadata off heap memory used: 14463112 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Compacted partition minimum bytes: 6867 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Compacted partition maximum bytes: 30753941057 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Compacted partition mean bytes: 44147544 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Average live cells per slice (last five minutes): 0.0 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Maximum live cells per slice (last five minutes): 0.0 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Average tombstones per slice (last five minutes): 0.0 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Maximum tombstones per slice (last five minutes): 0.0
我通过 cqlsh 发出以下查询：
cassandra@cqlsh> TRACING ON; Tracing is already enabled. Use TRACING OFF to disable. x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cassandra@cqlsh> CONSISTENCY; x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Current consistency level is ONE. x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
cassandra@cqlsh> select * from nt_live_october.nt where group_id='254358' and epoch >=1444313898 and epoch<=1444348800 LIMIT 1; x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
OperationTimedOut: errors={}, last_host=XXX.203
Statement trace did not complete within 10 seconds
这是 system_traces.events 显示的内容：
xxx.xxx.xxx.203 |第1281章解析 select * from nt_live_october.nt where group_id='254358'\nand epoch >=1443916800 and epoch xxx.xxx.xxx.203 | 2604 |准备声明
xxx.xxx.xxx.203 | 8454 |对用户执行单分区查询
xxx.xxx.xxx.203 | 8474 |获取 sstable 引用
xxx.xxx.xxx.203 | 8547 |合并 memtable 墓碑
xxx.xxx.xxx.203 | 8675 | sstable 1 的键缓存命中
xxx.xxx.xxx.203 | 8685 |寻找从数据文件开始的分区
xxx.xxx.xxx.203 | 9040 |跳过 0/1 个非切片相交的 sstable，由于墓碑而包含 0 个
xxx.xxx.xxx.203 | 9056 |合并来自 memtables 和 1 个 sstables 的数据
xxx.xxx.xxx.203 | 9120 |读取 1 个活的和 0 个墓碑单元
xxx.xxx.xxx.203 | 9854 |读取修复 DC_LOCAL
xxx.xxx.xxx.203 | 10033 |对用户执行单分区查询
xxx.xxx.xxx.203 | 10046 |获取 sstable 引用
xxx.xxx.xxx.203 | 10105 |合并 memtable 墓碑
xxx.xxx.xxx.203 | 10189 | sstable 1 的键缓存命中
xxx.xxx.xxx.203 | 10198 |寻找从数据文件开始的分区
xxx.xxx.xxx.203 | 10248 |跳过 0/1 个非切片相交的 sstable，由于墓碑而包含 0 个
xxx.xxx.xxx.203 | 10261 |合并来自 memtables 和 1 个 sstables 的数据
xxx.xxx.xxx.203 | 10296 |读取 1 个活的和 0 个墓碑单元
xxx.xxx.xxx.203 | 12511 |在nt上执行单分区查询
xxx.xxx.xxx.203 | 12525 |获取 sstable 引用
xxx.xxx.xxx.203 | 12587 |合并 memtable 墓碑
xxx.xxx.xxx.203 | 18067 |推测 /xxx.xxx.xxx.205 上的读取重试
xxx.xxx.xxx.203 | 18577 |发送 READ 消息到 xxx.xxx.xxx.205/xxx.xxx.xxx.205
xxx.xxx.xxx.203 | 25534 |为 sstable 8885 找到包含 6093 个条目的分区索引
xxx.xxx.xxx.203 | 25571 |寻求对数据文件中的索引部分进行分区
xxx.xxx.xxx.203 | 34989 |为 sstable 8524 找到包含 5327 个条目的分区索引
xxx.xxx.xxx.203 | 35022 |寻求对数据文件中的索引部分进行分区
xxx.xxx.xxx.203 | 36322 |为 sstable 8477 找到包含 333 个条目的分区索引
xxx.xxx.xxx.203 | 36336 |寻求对数据文件中的索引部分进行分区
xxx.xxx.xxx.203 | 714242 |为 sstable 8541 找到包含 299251 个条目的分区索引
xxx.xxx.xxx.203 | 714279 |寻求对数据文件中的索引部分进行分区
xxx.xxx.xxx.203 | 715717 |为 sstable 8217 找到 501 个条目的分区索引
xxx.xxx.xxx.203 | 715745 |寻求对数据文件中的索引部分进行分区
xxx.xxx.xxx.203 | 716232 |为 sstable 8888 找到包含 252 个条目的分区索引
xxx.xxx.xxx.203 | 716245 |寻求对数据文件中的索引部分进行分区
xxx.xxx.xxx.205 | 87 |从 /xxx.xxx.xxx.203
收到的 READ 消息 xxx.xxx.xxx.205 | 50427 |在nt上执行单分区查询
xxx.xxx.xxx.205 | 50535 |获取 sstable 引用
xxx.xxx.xxx.205 | 50628 |合并 memtable 墓碑
xxx.xxx.xxx.205 | 170441 |为 sstable 6332 找到具有 35650 个条目的分区索引
xxx.xxx.xxx.203 | 30718026 |为 sstable 5958 找到包含 199905 个条目的分区索引
xxx.xxx.xxx.203 | 30718077 |寻求对数据文件中的索引部分进行分区
xxx.xxx.xxx.205 | 170499 |寻求对数据文件中的索引部分进行分区
xxx.xxx.xxx.205 | 248898 |为 sstable 6797 找到包含 30958 个条目的分区索引
xxx.xxx.xxx.205 | 248962 |寻求对数据文件中的索引部分进行分区
xxx.xxx.xxx.203 | 67814573 |读取超时：org.apache.cassandra.exceptions.ReadTimeoutException：操作超时 - 仅收到 0 个响应。
xxx.xxx.xxx.203 | 67814675 |时间到;收到 0 个回复，共 1 个回复
我有 4 个节点，复制因子为 3（一个节点非常轻，但不是 .203）我尝试读取的数据不是很多——即使 LIMIT 1 没有被推送到远程节点，间隔的低端应该是大约 3 小时前（我没有超过当前时间的纪元）
关于如何解决此问题/可能出现什么问题的任何提示？我的 cassandra 版本是 2.1.9，主要使用默认值运行
表架构如下（出于隐私原因，我无法发布整个架构，但显示我希望最重要的关键）
PRIMARY KEY (group_id, epoch, group_name, auto_generated_uuid_field)
) WITH CLUSTERING ORDER BY (epoch ASC, group_name ASC, auto_generated_uuid_field ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 7776000
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
_______________编辑_____________ 回答以下问题：
状态输出：
-- Address Load Tokens Owns Host ID Rack x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
DN xxx.xxx.xxx.204 15.8 GB 1 ? 32ed196b-f6eb-4e93-b759 r1 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
UN xxx.xxx.xxx.205 20.38 GB 1 ? 446d71aa-e9cd-4ca9-a6ac r1 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
UN xxx.xxx.xxx.202 1.48 GB 1 ? 2a6670b2-63f2-43be-b672 r1 x~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
UN xxx.xxx.xxx.203 15.72 GB 1 ? dd26dfee-82da-454b-8db2 r1
system.log 比较棘手，因为我在那里登录了很多...我看到的一个可疑的事情是
WARN [CompactionExecutor:6] 2015-10-08 19:44:16,595 SSTableWriter.java (line 240) Compacting large partition nt_live_october/nt:254358 (230692316 bytes)
但这只是一个警告……在我看到之后不久
INFO [CompactionExecutor:6] 2015-10-08 19:44:16,642 CompactionTask.java (line 274) Compacted 4 sstables to [/cassandra/data_dir_d/nt_live_october/nt-72813b106b9111e58f1ea1f0942ab78d/nt_live_october-nt-ka-9024,]. 35,733,701 bytes to 30,186,394 (~84% of original) in 34,907ms = 0.824705MB/s. 21 total partitions merged to 18. Partition merge counts were {1:17, 4:1, }
我在日志中看到了很多这样的对...但没有错误级别的消息。压缩似乎进展顺利..它确实说这是最大的列族，但所有消息都是 INFO 级别....
【问题讨论】：
-
您能否经常检查 /var/log/cassandra/ 中的 system.log 并查找大分区、大量墓碑和节点上下移动的任何警告？ nodetool status 的输出也可能对我们有所帮助。