【发布时间】：2016-05-04 13:21:06
【问题描述】：
Apache Canssandra 2.2.5 版
我们有一个 cassandra 集群，它有 10 个音符。总数据大小为 10TB。有一个非常大的表格收件箱。这张表上的几个查询非常非常慢。 95%的查询可以在10ms内返回。但很少有查询读取超时（> 5s）。
收件箱表的架构是：
CREATE TABLE feeds.inbox (
owner_id bigint,
activity_id bigint,
insert_time timestamp,
PRIMARY KEY (owner_id, activity_id)
) WITH CLUSTERING ORDER BY (activity_id ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = ''
AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
其中一个缓慢的查询是：
select activity_id, insert_time FROM inbox WHERE owner_id=87500212012802048 ORDER BY activity_id desc;
此分区只有 44 行 (owner_id=87500212012802048)。
查询跟踪：
activity | timestamp | source | source_elapsed
--------------------------------------------------------------------------------------------------------------------------------------+----------------------------+--------------+----------------
Execute CQL3 query | 2016-05-04 20:58:53.970000 | 172.31.8.188 | 0
Parsing select activity_id, insert_time FROM inbox WHERE owner_id=87500212012802048 ORDER BY activity_id desc; [SharedPool-Worker-2] | 2016-05-04 20:58:53.971000 | 172.31.8.188 | 147
Preparing statement [SharedPool-Worker-2] | 2016-05-04 20:58:53.971000 | 172.31.8.188 | 213
reading data from /172.31.8.187 [SharedPool-Worker-2] | 2016-05-04 20:58:53.971000 | 172.31.8.188 | 369
Sending READ message to /172.31.8.187 [MessagingService-Outgoing-/172.31.8.187] | 2016-05-04 20:58:53.971000 | 172.31.8.188 | 739
READ message received from /172.31.8.188 [MessagingService-Incoming-/172.31.8.188] | 2016-05-04 20:58:53.972000 | 172.31.8.187 | 19
Executing single-partition query on inbox [SharedPool-Worker-4] | 2016-05-04 20:58:53.972000 | 172.31.8.187 | 350
Acquiring sstable references [SharedPool-Worker-4] | 2016-05-04 20:58:53.972000 | 172.31.8.187 | 372
Merging memtable tombstones [SharedPool-Worker-4] | 2016-05-04 20:58:53.972000 | 172.31.8.187 | 414
Bloom filter allows skipping sstable 353339 [SharedPool-Worker-4] | 2016-05-04 20:58:53.972000 | 172.31.8.187 | 446
Key cache hit for sstable 353313 [SharedPool-Worker-4] | 2016-05-04 20:58:53.972000 | 172.31.8.187 | 467
Seeking to partition indexed section in data file [SharedPool-Worker-4] | 2016-05-04 20:58:53.972001 | 172.31.8.187 | 475
Bloom filter allows skipping sstable 352862 [SharedPool-Worker-4] | 2016-05-04 20:58:53.973000 | 172.31.8.187 | 590
Key cache hit for sstable 352122 [SharedPool-Worker-4] | 2016-05-04 20:58:53.973000 | 172.31.8.187 | 612
Seeking to partition indexed section in data file [SharedPool-Worker-4] | 2016-05-04 20:58:53.973000 | 172.31.8.187 | 621
Key cache hit for sstable 345513 [SharedPool-Worker-4] | 2016-05-04 20:58:53.973000 | 172.31.8.187 | 779
Seeking to partition indexed section in data file [SharedPool-Worker-4] | 2016-05-04 20:58:53.973000 | 172.31.8.187 | 820
Skipped 0/5 non-slice-intersecting sstables, included 0 due to tombstones [SharedPool-Worker-4] | 2016-05-04 20:58:53.973000 | 172.31.8.187 | 837
Merging data from memtables and 3 sstables [SharedPool-Worker-4] | 2016-05-04 20:58:53.973001 | 172.31.8.187 | 848
Read 44 live and 0 tombstone cells [SharedPool-Worker-4] | 2016-05-04 20:58:57.546000 | 172.31.8.187 | 574196
Enqueuing response to /172.31.8.188 [SharedPool-Worker-4] | 2016-05-04 20:58:57.549000 | 172.31.8.187 | 577130
Sending REQUEST_RESPONSE message to /172.31.8.188 [MessagingService-Outgoing-/172.31.8.188] | 2016-05-04 20:58:57.551000 | 172.31.8.187 | 579411
REQUEST_RESPONSE message received from /172.31.8.187 [MessagingService-Incoming-/172.31.8.187] | 2016-05-04 20:58:57.557000 | 172.31.8.188 | 586750
Processing response from /172.31.8.187 [SharedPool-Worker-4] | 2016-05-04 20:58:57.557000 | 172.31.8.188 | 586930
Request complete | 2016-05-04 20:58:57.560568 | 172.31.8.188 | 590568
我们可以看到“读取 44 个活的和 0 个墓碑单元 [SharedPool-Worker-4]”这一步需要超过 50 秒！为什么查询这么慢，虽然这个分区只有44行。
当我将聚类键查询顺序更改为 asc 时，查询提升了一点，对于在线功能来说仍然太慢了。
activity | timestamp | source | source_elapsed
-------------------------------------------------------------------------------------------------------------------------------------+----------------------------+--------------+----------------
Execute CQL3 query | 2016-05-04 21:19:29.275000 | 172.31.8.188 | 0
Parsing select activity_id, insert_time FROM inbox WHERE owner_id=87500212012802048 ORDER BY activity_id asc; [SharedPool-Worker-2] | 2016-05-04 21:19:29.275000 | 172.31.8.188 | 96
Preparing statement [SharedPool-Worker-2] | 2016-05-04 21:19:29.275000 | 172.31.8.188 | 150
Executing single-partition query on inbox [SharedPool-Worker-3] | 2016-05-04 21:19:29.275000 | 172.31.8.188 | 393
Acquiring sstable references [SharedPool-Worker-3] | 2016-05-04 21:19:29.275000 | 172.31.8.188 | 419
Merging memtable tombstones [SharedPool-Worker-3] | 2016-05-04 21:19:29.275000 | 172.31.8.188 | 450
Partition index lookup allows skipping sstable 324509 [SharedPool-Worker-3] | 2016-05-04 21:19:29.275001 | 172.31.8.188 | 510
Key cache hit for sstable 324457 [SharedPool-Worker-3] | 2016-05-04 21:19:29.275001 | 172.31.8.188 | 534
Seeking to partition beginning in data file [SharedPool-Worker-3] | 2016-05-04 21:19:29.275001 | 172.31.8.188 | 546
Key cache hit for sstable 324018 [SharedPool-Worker-3] | 2016-05-04 21:19:29.275001 | 172.31.8.188 | 640
Seeking to partition beginning in data file [SharedPool-Worker-3] | 2016-05-04 21:19:29.275001 | 172.31.8.188 | 685
Key cache hit for sstable 321829 [SharedPool-Worker-3] | 2016-05-04 21:19:29.276000 | 172.31.8.188 | 899
Seeking to partition beginning in data file [SharedPool-Worker-3] | 2016-05-04 21:19:29.276000 | 172.31.8.188 | 926
Key cache hit for sstable 314770 [SharedPool-Worker-3] | 2016-05-04 21:19:29.276000 | 172.31.8.188 | 1171
Seeking to partition beginning in data file [SharedPool-Worker-3] | 2016-05-04 21:19:29.276000 | 172.31.8.188 | 1199
Skipped 0/5 non-slice-intersecting sstables, included 0 due to tombstones [SharedPool-Worker-3] | 2016-05-04 21:19:29.276000 | 172.31.8.188 | 1456
Merging data from memtables and 4 sstables [SharedPool-Worker-3] | 2016-05-04 21:19:29.276000 | 172.31.8.188 | 1488
Read 44 live and 0 tombstone cells [SharedPool-Worker-3] | 2016-05-04 21:19:29.440000 | 172.31.8.188 | 165319
Request complete | 2016-05-04 21:19:29.448062 | 172.31.8.188 | 173062
【问题讨论】：
-
inbox的架构是什么？里面真的有大细胞吗？ -
如果您使用的每个节点的令牌数为 1，那么它总是会产生问题。因为 cassandra 无法将负载平均分配给每个节点。因此，您必须在每个节点 256 中使用 no of tokens。如果读取和写入发生在同一个节点上，则如果有 1 个令牌，则会产生缓慢的结果。
-
@ChrisLohfink 里面没有大单元格。只有 owner_id(long)、actor_id(long) 和 insert_time(date)
-
@mahendrasingh 我正在使用虚拟节点。每个节点 256 个令牌。
标签： cassandra