内存映射缓冲区的增量分配和 1 GB 大小的故障

Question

我创建了以下演示来查看 MMF begaviour（我想将它用作一个非常大的 long 值数组）。

import java.nio._, java.io._, java.nio.channels.FileChannel

object Index extends App {

    val formatter = java.text.NumberFormat.getIntegerInstance
    def format(l: Number) = formatter.format(l)

    val raf = new RandomAccessFile("""C:\Users\...\Temp\96837624\mmf""", "rw")
    raf.setLength(20)
    def newBuf(capacity: Int) = {
      var bytes= 8.toLong*capacity
      println("new buf " + format(capacity) + " words = " + format(bytes) + " bytes")

      // java.io.IOException: "Map failed" at the following line
      raf.getChannel.map(FileChannel.MapMode.READ_WRITE, 0, bytes).asLongBuffer()
    }

    (1 to 100 * 1000 * 1000).foldLeft(newBuf(2): LongBuffer){ case(buf, i) =>
        if (Math.random < 0.000009) println(format(buf.get(buf.position()/2)))
        (if (buf.position == buf.capacity) {
            val p = buf.position
            val b = newBuf(buf.capacity * 2)
            b.position(p) ; b
        } else buf).put(i)

    }

    raf.close

输出失败

16,692,145
16,741,940
new buf 67,108,864
[error] (run-main-1) java.io.IOException: Map failed
java.io.IOException: Map failed
        at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:907)

我看到创建了一个 512 MB 的文件，但系统似乎无法将其扩展至 1 GB。

但是，如果我使用 64M 长字 newBuf(64*1024*1027) 而不是 2 个长字 foldLeft(newBuf(2)) 的初始大小，则运行时会成功创建 1GB 文件，但在尝试创建 2GB 文件时会失败

new buf 268 435 458 words = 2 147 483 664 bytes
java.lang.IllegalArgumentException: Size exceeds Integer.MAX_VALUE
        at sun.nio.ch.FileChannelImpl.map(Unknown Source)

我用 64 位 jvm 运行它。

我也不确定如何关闭缓冲区以释放它以供以后在 sbt 中应用，并确保数据最终会出现在文件中。这个机制看起来完全不可靠。

Answer 1

好吧，一天的实验表明，无论如何，32 位 JVM 都会失败，IOException: Map failed 在 1 GB 上。为了规避 Size exceeds Integer.MAX_VALUE 在 64 位机器上的映射，应该使用多个可承受大小的缓冲区，例如每个 100 mb 都可以。那是因为buffers are addressed by integer。

关于这个问题，您可以在内存中同时保持所有此类缓冲区打开，即在分配下一个有效增加文件大小之前无需关闭一个缓冲区 => null，如下面的演示所示

import Utils._, java.nio._, java.io._, java.nio.channels.FileChannel

object MmfDemo extends App {

    val bufAddrWidth = 25 /*in bits*/ // Every element of the buff addresses a long
    val BUF_SIZE_WORDS = 1 << bufAddrWidth ; val BUF_SIZE_BYTES = BUF_SIZE_WORDS << 3
    val bufBitMask = BUF_SIZE_WORDS - 1
    var buffers = Vector[LongBuffer]()
    var capacity = 0 ; var pos = 0
    def select(pos: Int) = {
        val bufn = pos >> bufAddrWidth // higher bits of address denote the buffer number
        //println(s"accessing $pos = " + (pos - buf * wordsPerBuf) + " in " + buf)
        while (buffers.length <= bufn) expand
        pass(buffers(bufn)){_.position(pos & bufBitMask)}
    }
    def get(address: Int = pos) = {
        pos = address +1
        select(address).get
    }
    def put(value: Long) {
        //println("writing " + value + " to " + pos)
        select(pos).put(value) ; pos += 1
    }
    def expand = {
        val fromByte = buffers.length.toLong  * BUF_SIZE_BYTES
        println("adding " + buffers.length + "th buffer, total size expected " + format(fromByte + BUF_SIZE_BYTES) + " bytes")

        // 32bit JVM: java.io.IOException: "Map failed" at the following line if buf size requested is larger than 512 mb
        // 64bit JVM: IllegalArgumentException: Size exceeds Integer.MAX_VALUE
        buffers :+= fc.map(FileChannel.MapMode.READ_WRITE, fromByte, BUF_SIZE_BYTES).asLongBuffer()
        capacity += BUF_SIZE_WORDS
    }

    def rdAll(get: Int => Long) {
        var firstMismatch = -1
        val failures = (0 until parse(args(1))).foldLeft(0) { case(failures, i) =>
            val got = get(i)
            if (got != i && firstMismatch == -1) {firstMismatch = i; println("first mismatch at " +format(i) + ", value = " + format(got))}
            failures + ?(got != i, 1, 0)
        } ; println(format(failures) + " mismatches")
    }

    val raf = new RandomAccessFile("""C:\Temp\mmf""", "rw")
    val fc = raf.getChannel
    try {

        if (args.length < 1) {
            println ("usage1: buf_gen <len in long words>")
            println ("usage1: raf_gen <len in long words>")
            println("example: buf_gen 30m")
            println("usage2: raf_rd <size in words>")
            println("usage3: buf_rd <size in words>")
        } else {
            val t1 = System.currentTimeMillis
            args(0) match {
                case "buf_gen" => raf.setLength(0)
                    (0 until parse(args(1))) foreach {i => put(i.toLong)}
                case "raf_gen" => raf.setLength(0)
                    (0 until parse(args(1))) foreach {i =>raf.writeLong(i.toLong)}
                        //fc.force(true)
                case "rd_raf" => rdAll{i => raf.seek(i.toLong * 8) ; raf.readLong()}
                case "rd_buf" => rdAll(get)
                case u =>println("unknown command " + u)
            } ; println("finished in " + (System.currentTimeMillis - t1) + " ms")
        }
    } finally {
        raf.close ; fc.close

        buffers = null ; System.gc /*GC needs to close the buffer*/}

}

object Utils {
    val formatter = java.text.NumberFormat.getIntegerInstance
    def format(l: Number) = formatter.format(l)

    def ?[T](sel: Boolean, a: => T, b: => T) = if (sel) a else b
    def parse(s: String) = {
        val lc = s.toLowerCase()
        lc.filter(_.isDigit).toInt *
            ?(lc.contains("k"), 1000, 1) *
            ?(lc.contains("m"), 1000*1000, 1)
    }
    def eqa[T](a: T, b: T) = assert(a == b, s"$a != $b")
    def pass[T](a: T)(code: T => Unit) = {code(a) ; a}
}

至少在 Windows 中。使用这个程序，我设法创建了比我的机器内存大的 mmf 文件（更不用说 JVM 的 -Xmx，它在这些问题上根本没有任何作用）。只需用鼠标在 Windows 控制台中选择一些文本来减慢文件生成速度（程序将暂停，直到您释放选择），否则 Windows 会将所有其他性能关键人员逐出页面文件，您的 PC 将在抖动中死机。

顺便说一句，尽管我只写入文件末尾并且 Windows 可能会驱逐我未使用的千兆字节块，但 PC 仍会在抖动中死机。另外，我注意到我正在写的块实际上是读取的

以下输出

adding 38th buffer, total size expected 12,480,000,000 bytes
adding 39th buffer, total size expected 12,800,000,000 bytes

伴随着以下系统请求

5:24,java,"QueryStandardInformationFile",mmf,"SUCCESS","AllocationSize: 12 480 000 000, EndOfFile: 12 480 000 000, NumberOfLinks: 1, DeletePending: False, Directory: False"
5:24,java,"SetEndOfFileInformationFile",mmf,"SUCCESS","EndOfFile: 12 800 000 000"
5:24,java,"SetAllocationInformationFile",mmf,"SUCCESS","AllocationSize: 12 800 000 000"
5:24,java,"CreateFileMapping",mmf,"FILE LOCKED WITH WRITERS","SyncType: SyncTypeCreateSection, PageProtection: "
5:24,java,"QueryStandardInformationFile",mmf,"SUCCESS","AllocationSize: 12 800 000 000, EndOfFile: 12 800 000 000, NumberOfLinks: 1, DeletePending: False, Directory: False"
5:24,java,"CreateFileMapping",mmf,"SUCCESS","SyncType: SyncTypeOther"
5:24,java,"ReadFile",mmf,"SUCCESS","Offset: 12 480 000 000, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
5:24,java,"ReadFile",mmf,"SUCCESS","Offset: 12 480 032 768, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
5:24,java,"ReadFile",mmf,"SUCCESS","Offset: 12 480 065 536, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
5:24,java,"ReadFile",mmf,"SUCCESS","Offset: 12 480 098 304, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
5:24,java,"ReadFile",mmf,"SUCCESS","Offset: 12 480 131 072, Length: 20 480, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"

skipped 9000 reads

5:25,java,"ReadFile",mmf,"SUCCESS","Offset: 12 799 836 160, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
5:25,java,"ReadFile",mmf,"SUCCESS","Offset: 12 799 868 928, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
5:25,java,"ReadFile",mmf,"SUCCESS","Offset: 12 799 901 696, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
5:25,java,"ReadFile",mmf,"SUCCESS","Offset: 12 799 934 464, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"
5:25,java,"ReadFile",mmf,"SUCCESS","Offset: 12 799 967 232, Length: 32 768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal"

但那是另一回事了。

事实证明，这个答案与Peter Lawrey's 重复，除了我的问题专门用于映射大型缓冲区时的“映射失败”和“超出整数范围”，而原始问题与 JVM 中的 OutOfMem 相关，这与处理 I/O。