Ruby 中的内存高效 2D 位存储（100M 项）

Question

我想在 Ruby 中使用大型数据结构来表示位（0 或 1）或布尔值（真/假）。

在下面的示例代码中，我使用大小为 10^4 * 10^4 的二维数组来存储布尔数据。

require 'get_process_mem'

num = 10000
twod =  Array.new(num) { Array.new(num, false)}
for i in 0..num-1 do
  for j in 0..num-1 do
    twod[i][j] = i>j
  end
end

mem = GetProcessMem.new
puts mem.inspect

输出显示它使用了 ~778 MB。

#<GetProcessMem:0x00000078 @mb=777.93359375 @gb=0.7597007751464844 @kb=796604.0 @bytes=0.815722496e9>

我刚刚尝试在代码中将数据类型更改为整数，内存使用情况报告相同的值。

更新代码：

require 'get_process_mem'

num = 10000
twod =  Array.new(num) { Array.new(num, 500)}
for i in 0..num-1 do
  for j in 0..num-1 do
    twod[i][j] = 1000+i+j
  end
end

mem = GetProcessMem.new
puts mem.inspect

输出：

#<GetProcessMem:0x00000078 @mb=777.6015625 @gb=0.7593765258789062 @kb=796264.0 @bytes=0.815374336e9>

我原以为布尔数组会比整数数组使用更少的内存，但事实并非如此！ 有没有其他优化的方式来存储位或布尔值？

Answer 1

您可以在 FFI:Buffer 中构建字节数组以获得良好的大小/性能 https://rubydoc.info/github/ffi/ffi/FFI/Buffer 并对每个字节进行位操作：

#!/usr/bin/env ruby
require 'get_process_mem'
require 'ffi'
BIT_OP = {
  get: 1,
  set: 2,
  clear: 3,
}
class BitStore
  attr_accessor :buf, :size
  def initialize(shape) # ex [10000,10000] [10,20,30], etc
    @shp = shape
    @size = 1;
    shape.each do | sz |
      @size *= sz
    end
    @size = max(next8(@size)/8, 8) + 1 # min size needed
    clear = true
    @buf = FFI::Buffer.new(@size,1,clear)
  end
  def max(x,y)
    x > y ? x : y
  end
  def next8(val)
    val % 8 == 0 ? val+8 : ((val / 8).to_i + 1)*8
  end
  def write_buf_to_file(fname)
    fout = File.open(fname, "wb")
    fout.write(@buf.get_bytes(0,@size))
    fout.close
  end
  def check_position(coord_arr)
    if coord_arr.size != @shp.size
      raise "coord_arr size != shape size"
    end
    coord_arr.each_with_index do | coord, i |
      if coord_arr[i] > @shp[i]-1
        raise "coord index out of bounds "
      end
    end
  end
  def get_position(coord_arr)
    position = coord_arr[0]
    (1..coord_arr.size-1).each do | i |
      position += coord_arr[i] * @shp.reverse[i]
    end
    return position
  end
  def bit_op(coord_arr, op)
    check_position(coord_arr)
    position = get_position(coord_arr)
    offset = (position / 8).to_i
    bit_pos = (position % 8)
    bit_val = 1 << bit_pos
    val = @buf.get_string(offset, 1)
    numeric_value = val == "" ? 0 : val.unpack("C")[0]
    case op
    when BIT_OP[:get]
      numeric_value = numeric_value & bit_val
      return numeric_value > 0 ? "set" : "clear"
    when BIT_OP[:set]
      numeric_value = numeric_value | bit_val
    when BIT_OP[:clear]
      numeric_value = numeric_value & (bit_val ^ 0xff)
    end
    @buf.put_string(offset,[numeric_value].pack("C"))
    return ""
  end
end
def main
  begin
    rows = 10000
    cols = 10000
    b = BitStore.new([rows,cols])
    
    puts "setting [3,0]"
    b.bit_op([3,0],BIT_OP[:set])
    is_set = b.bit_op([3,0],BIT_OP[:get])
    puts is_set

    puts "setting [8000,8000]"
    b.bit_op([8000,8000],BIT_OP[:set])
    is_set = b.bit_op([8000,8000],BIT_OP[:get])
    puts is_set

    puts "clearing [8000,8000]"
    b.bit_op([8000,8000],BIT_OP[:clear])
    is_set = b.bit_op([8000,8000],BIT_OP[:get])
    puts is_set

    mem = GetProcessMem.new
    puts mem.inspect
  rescue NoMemoryError => e
    puts "NoMemoryError"
    p e
    p e.backtrace.inspect
  end
end
main

内存使用量为 26MB：

user1@debian10 /home/user1/rubynew > ./pack_testb.rb 
setting [3,0]
set
setting [8000,8000]
set
clearing [8000,8000]
clear
#<GetProcessMem:0x000000a0 @mb=26.6953125 @gb=0.02606964111328125 @kb=27336.0 @bytes=0.27992064e8>

磁盘上的文件结构：

b = BitStore.new([7,6])
puts "setting [3,0]"
b.bit_op([3,0],BIT_OP[:set])
is_set = b.bit_op([3,0],BIT_OP[:get])
puts is_set
b.write_buf_to_file("/tmp/the_buf.bin")
system("xxd /tmp/the_buf.bin")

puts "setting [6,5]"
b.bit_op([6,5],BIT_OP[:set])
is_set = b.bit_op([6,5],BIT_OP[:get])
puts is_set
b.write_buf_to_file("/tmp/the_buf.bin")
system("xxd /tmp/the_buf.bin")

setting [3,0]
00000000: 0800 0000 0000 0000 00
setting [6,5]
00000000: 0000 0000 0002 0000 00