【问题标题】:HSET vs SET memory usage?HSET 与 SET 内存使用情况?
【发布时间】:2012-10-08 09:58:36
【问题描述】:

我正在阅读this article,其中提到在 redis 中存储 100 万个密钥将使用 17GB 内存。然而,当切换到散列时,将它们分块为 1k(例如:HSET "mediabucket:1155" "1155315" "939")允许它们将 1M 存储在 5GB 中,这是一个相当大的节省。

我阅读了redis memory-optimization,但我不太了解其中的区别。它说 HGET 不是 O(1),但足够接近,并且在使用 hset 时提到了更多的 cpu 使用率。我不明白为什么会有更多的 cpu 使用率(确保空间交易时间。但是如何/什么?)。它提到了“编码”,但没有提到它们是如何编码的。

它也只提到了字符串,但我不知道只有字符串是什么意思。是哈希字段吗?这是否意味着哈希字段?我在HSET 中没有看到任何关于它的信息。究竟会编码什么,为什么编码比使用 SET 更有效?

HSET "mediabucket:1155" "1155315" "939" 怎么可能比SET "mediabucket:1155315" "939" 更有效率? SET 中的数据较少(使用 1155315 和 1155 而不是 1155315)。我个人会尝试使用二进制密钥,但我认为这与 HSET 更高效的原因无关。

编辑:

交叉发布在 redis-db 邮件列表中:https://groups.google.com/d/topic/redis-db/90K3UqciAx0/discussion

【问题讨论】:

    标签: memory redis


    【解决方案1】:

    根据 hash-max-ziplist-entries 和 hash-max-ziplist-value 参数的值,将小散列对象编码为 ziplist。这是简单的数据序列化。

    一个ziplist定义如下(摘自Redis源码):

    /* The ziplist is a specially encoded dually linked list that is designed
    * to be very memory efficient. It stores both strings and integer values,
    * where integers are encoded as actual integers instead of a series of
    * characters. It allows push and pop operations on either side of the list
    * in O(1) time. However, because every operation requires a reallocation of
    * the memory used by the ziplist, the actual complexity is related to the
    * amount of memory used by the ziplist.
    *
    * ----------------------------------------------------------------------------
    *
    * ZIPLIST OVERALL LAYOUT:
    * The general layout of the ziplist is as follows:
    * <zlbytes><zltail><zllen><entry><entry><zlend>
    *
    * <zlbytes> is an unsigned integer to hold the number of bytes that the
    * ziplist occupies. This value needs to be stored to be able to resize the
    * entire structure without the need to traverse it first.
    *
    * <zltail> is the offset to the last entry in the list. This allows a pop
    * operation on the far side of the list without the need for full traversal.
    *
    * <zllen> is the number of entries.When this value is larger than 2**16-2,
    * we need to traverse the entire list to know how many items it holds.
    *
    * <zlend> is a single byte special value, equal to 255, which indicates the
    * end of the list.
    *
    * ZIPLIST ENTRIES:
    * Every entry in the ziplist is prefixed by a header that contains two pieces
    * of information. First, the length of the previous entry is stored to be
    * able to traverse the list from back to front. Second, the encoding with an
    * optional string length of the entry itself is stored.
    *
    * The length of the previous entry is encoded in the following way:
    * If this length is smaller than 254 bytes, it will only consume a single
    * byte that takes the length as value. When the length is greater than or
    * equal to 254, it will consume 5 bytes. The first byte is set to 254 to
    * indicate a larger value is following. The remaining 4 bytes take the
    * length of the previous entry as value.
    *
    * The other header field of the entry itself depends on the contents of the
    * entry. When the entry is a string, the first 2 bits of this header will hold
    * the type of encoding used to store the length of the string, followed by the
    * actual length of the string. When the entry is an integer the first 2 bits
    * are both set to 1. The following 2 bits are used to specify what kind of
    * integer will be stored after this header. An overview of the different
    * types and encodings is as follows:
    *
    * |00pppppp| - 1 byte
    *      String value with length less than or equal to 63 bytes (6 bits).
    * |01pppppp|qqqqqqqq| - 2 bytes
    *      String value with length less than or equal to 16383 bytes (14 bits).
    * |10______|qqqqqqqq|rrrrrrrr|ssssssss|tttttttt| - 5 bytes
    *      String value with length greater than or equal to 16384 bytes.
    * |11000000| - 1 byte
    *      Integer encoded as int16_t (2 bytes).
    * |11010000| - 1 byte
    *      Integer encoded as int32_t (4 bytes).
    * |11100000| - 1 byte
    *      Integer encoded as int64_t (8 bytes).
    * |11110000| - 1 byte
    *      Integer encoded as 24 bit signed (3 bytes).
    * |11111110| - 1 byte
    *      Integer encoded as 8 bit signed (1 byte).
    * |1111xxxx| - (with xxxx between 0000 and 1101) immediate 4 bit integer.
    *      Unsigned integer from 0 to 12. The encoded value is actually from
    *      1 to 13 because 0000 and 1111 can not be used, so 1 should be
    *      subtracted from the encoded 4 bit value to obtain the right value.
    * |11111111| - End of ziplist.
    *
    * All the integers are represented in little endian byte order.
    */
    

    散列对象中的每个项目都表示为 ziplist 中的键/值对(2 个连续条目)。键和值都可以存储为简单的字符串或整数。这种格式在内存中更紧凑,因为它节省了大量实现动态数据结构(如真正的哈希表)所需的指针(每个 8 字节)。

    缺点是当应用于 ziplist 时,HSET/HGET 操作实际上是 O(N)。这就是为什么 ziplist 必须保持较小的原因。当 ziplist 数据适合 L1 CPU 缓存时,尽管具有线性复杂度,但相应的算法仍然足够快。

    您可能需要参考以下链接以获取更多信息:

    Redis 10x more memory usage than data

    Redis Data Structure Space Requirements

    这些答案涉及其他数据结构(如集合、列表或排序集合),但它们的概念完全相同。

    【讨论】:

      猜你喜欢
      • 2015-12-08
      • 1970-01-01
      • 2012-01-13
      • 1970-01-01
      • 1970-01-01
      • 1970-01-01
      • 2011-01-29
      • 2014-09-04
      • 1970-01-01
      相关资源
      最近更新 更多