【问题标题】:Maximum memory usage reached (4294967296), cannot allocate chunk of 1048576达到最大内存使用量 (4294967296),无法分配 1048576 块
【发布时间】:2019-12-14 05:09:41
【问题描述】:

我在 Cassandra 3.11.4 debug.log 上遇到如下错误。

可能是什么原因?如何避免这些错误。??

我可以看到分区大小是 1.9 MB。堆大小 16GB。

================

free -m
              total        used        free      shared  buff/cache   available
Mem:          64264       23748        3279         619       37236       33190
Swap:             0           0           0

=========================

lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    1
Core(s) per socket:    1
Socket(s):             16
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 85
Model name:            Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping:              4
CPU MHz:               2394.482
BogoMIPS:              4788.96
Virtualization:        VT-x
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              1024K
L3 cache:              28160K
NUMA node0 CPU(s):     0-15
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl eagerfpu pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f rdseed adx smap clflushopt clwb avx512cd xsaveopt xsavec xgetbv1 arat
nodetool tablestats test_data.local_data -H
Total number of tables: 37
----------------
Keyspace : test_data
        Read Count: 0
        Read Latency: NaN ms
        Write Count: 48776898
        Write Latency: 1.534334614042082 ms
        Pending Flushes: 2
                Table: local_data
                SSTable count: 84
                Space used (live): 115.53 GiB
                Space used (total): 115.53 GiB
                Space used by snapshots (total): 165.45 KiB
                Off heap memory used (total): 216.71 MiB
                SSTable Compression Ratio: 0.5440445152549487
                Number of partitions (estimate): 139647
                Memtable cell count: 1495900
                Memtable data size: 167.87 MiB
                Memtable off heap memory used: 174.38 MiB
                Memtable switch count: 2016
                Local read count: 0
                Local read latency: NaN ms
                Local write count: 48777246
                Local write latency: 1.580 ms
                Pending flushes: 2
                Percent repaired: 0.0
                Bloom filter false positives: 0
                Bloom filter false ratio: 0.00000
                Bloom filter space used: 287.65 KiB
                Bloom filter off heap memory used: 431.88 KiB
                Index summary off heap memory used: 160.4 KiB
                Compression metadata off heap memory used: 41.75 MiB
                Compacted partition minimum bytes: 3974
                Compacted partition maximum bytes: 1955666
                Compacted partition mean bytes: 1109275
                Average live cells per slice (last five minutes): NaN
                Maximum live cells per slice (last five minutes): 0
                Average tombstones per slice (last five minutes): NaN
                Maximum tombstones per slice (last five minutes): 0
                Dropped Mutations: 6 bytes

=========================================

nodetool tablehistograms test_data local_data
test_data/local_data histograms
Percentile  SSTables     Write Latency      Read Latency    Partition Size        Cell Count
                              (micros)          (micros)           (bytes)
50%             0.00            315.85              0.00           1358102             61214
75%             0.00            943.13              0.00           1955666            152321
95%             0.00           2816.16              0.00           1955666            152321
98%             0.00           8409.01              0.00           1955666            152321
99%             0.00          36157.19              0.00           1955666            152321
Min             0.00             73.46              0.00              3974               373
Max             0.00         557074.61              0.00           1955666            152321

======================================

nodetool info
ID                     : 2985f5ef-5b84-4654-91f6-dc459741e9fc
Gossip active          : true
Thrift active          : false
Native Transport active: true
Load                   : 117.25 GiB
Generation No          : 1576230314
Uptime (seconds)       : 15745
Heap Memory (MB)       : 8040.00 / 16384.00
Off Heap Memory (MB)   : 287.19
Data Center            : dc1
Rack                   : rack1
Exceptions             : 0
Key Cache              : entries 36568, size 66.79 MiB, capacity 100 MiB, 26555 hits, 63045 requests, 0.421 recent hit rate, 14400 save period in seconds
Row Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache          : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Chunk Cache            : entries 65024, size 3.97 GiB, capacity 3.97 GiB, 12839660 misses, 15296535 requests, 0.161 recent hit rate, 468.900 microseconds miss latency
Percent Repaired       : 1.8637552068836277E-7%
Token                  : (invoke with -T/--tokens to see all 256 tokens)

===================================

nodetool netstats
Mode: NORMAL
Not sending any streams.
Read Repair Statistics:
Attempted: 0
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name                    Active   Pending      Completed   Dropped
Large messages                  n/a         2              0         2
Small messages                  n/a         0       66196004         4
Gossip messages                 n/a         0          54691        30

===========================

INFO  [CompactionExecutor:8] 2019-12-13 17:49:53,408 NoSpamLogger.java:91 - Maximum memory usage reached (4294967296), cannot allocate chunk of 1048576
INFO  [CompactionExecutor:9] 2019-12-13 18:04:55,683 NoSpamLogger.java:91 - Maximum memory usage reached (4294967296), cannot allocate chunk of 1048576
INFO  [CompactionExecutor:6] 2019-12-13 18:21:39,420 NoSpamLogger.java:91 - Maximum memory usage reached (4294967296), cannot allocate chunk of 1048576
INFO  [CompactionExecutor:8] 2019-12-13 18:36:45,011 NoSpamLogger.java:91 - Maximum memory usage reached (4294967296), cannot allocate chunk of 1048576
INFO  [CompactionExecutor:9] 2019-12-13 18:51:53,479 NoSpamLogger.java:91 - Maximum memory usage reached (4294967296), cannot allocate chunk of 1048576
INFO  [CompactionExecutor:7] 2019-12-13 19:06:55,898 NoSpamLogger.java:91 - Maximum memory usage reached (4294967296), cannot allocate chunk of 1048576

【问题讨论】:

  • 您可以使用free 命令的输出来编辑您的答案吗?
  • free -m total used free shared buff/cache available Mem: 64264 23748 3279 619 37236 33190 Swap: 0 0 0 ================= ===========
  • 你好 sateesh,欢迎来到 SO。尝试编辑您的问题添加更多详细信息,而不是将它们放入 cmets:这些不是为此目的,通常不会在其中找到有用的信息。

标签: cassandra


【解决方案1】:

块缓存有点新。它由 cassandra.yaml 中的 file_cache_size_in_mb 设置控制。如果您运行“nodetool info”命令,您将看到块缓存的大小。我相信默认值是 (MemTotal - Heap)/2。它用于在读取时缓存表块,帮助优化读取性能。在读取内容时,内容会不断进出块缓存。如果读入它的速度比出去的快,它就会“填满”,因此你会得到消息。您可以增加块缓存或直接忽略该消息。

仅供参考, 如果您正在寻找一篇关于读/写如何工作(使用块)的好文章,请参阅这篇文章:

https://thelastpickle.com/blog/2018/08/08/compression_performance.html

-吉姆

【讨论】:

  • 谢谢 Jim,如何增加块缓存?
  • 在cassandra.yaml文件中增加这个参数:file_cache_size_in_mb
猜你喜欢
  • 2017-08-07
  • 2019-02-26
  • 2018-03-29
  • 1970-01-01
  • 2019-01-06
  • 2013-08-09
  • 1970-01-01
  • 2010-10-10
  • 2023-03-20
相关资源
最近更新 更多