在不读取内存的情况下向 JSON 添加属性的最佳方法

Question

我有一个大型 JSON 和复杂文件 (~100MB)。我需要在不读取内存的情况下为其添加一个属性。如果不读取内存中的全部内容，我无法找到附加到 JSON 的选项。

我找不到任何合适的例子。

我能想到的最好的办法是使用StreamReader(s) 和StreamReader(s) 将最后一个} 替换为"key":"value"}

JSON 是通过遗留应用程序读取的，我们正在将其集成到 Web 应用程序中，因此这正在成为一个瓶颈。

Answer 1

如果你不愿意解析数据，我认为你应该使用Java.io.RandomAccessFile。该包将允许您查找文件的末尾（或末尾前的 1），并写入新数据。请记住，作者会覆盖字符串，它不会插入它们，因此假设您要添加"key": property,，您需要记住插入,"key": property}

Answer 2

我正在考虑从输入流到输出流的全功能流式传输（因此从一开始就按令牌读取和写入令牌），但是既然您提到您可以访问文件，@Carson 的建议确实比我最初的想法好。我只是进一步发展了这个想法：

public final class JsonAppender
        extends Writer {

    private final BufferedWriter writer;
    private final char terminator;

    private boolean isAboutToWrite = true;

    private JsonAppender(final BufferedWriter writer, final char terminator) {
        this.writer = writer;
        this.terminator = terminator;
    }

    public static Writer appendAtEnd(final RandomAccessFile randomAccessFile)
            throws IOException {
        long pos = randomAccessFile.length() - 1;
        char terminator = '\u0000';
        outer_whitespace:
        for ( ; pos >= 0; pos-- ) {
            randomAccessFile.seek(pos);
            final char ch = (char) randomAccessFile.readByte();
            switch ( ch ) {
// @formatter:off
            case ' ': case '\r': case '\n': case '\t':
// @formatter:on
                continue;
// @formatter:off
            case ']': case '}':
// @formatter:on
                terminator = ch;
                break outer_whitespace;
            default:
                throw new IOException("Unexpected " + ch + " at " + pos);
            }
        }
        if ( pos < 0 ) {
            throw new IOException("No object or array begin found");
        }
        inner_whitespace:
        for ( pos -= 1; pos >= 0; pos-- ) {
            randomAccessFile.seek(pos);
            final char ch = (char) randomAccessFile.readByte();
            switch ( ch ) {
// @formatter:off
            case ' ': case '\r': case '\n': case '\t':
// @formatter:on
                continue;
// @formatter:off
            case '}': case ']':
            case '\"':
            case '0': case '1': case '2': case '3': case '4': case '5': case '6': case '7': case '8': case '9':
            case 'e': // for both true and false
            case 'l': // for null
// @formatter:on
                break inner_whitespace;
            default:
                throw new IOException("Unexpected " + ch + " at " + pos);
            }
        }
        return new JsonAppender(new BufferedWriter(new OutputStreamWriter(new FileOutputStream(randomAccessFile.getFD()))), terminator);
    }

    @Override
    public void write(final char[] buffer, final int offset, final int length)
            throws IOException {
        if ( isAboutToWrite ) {
            isAboutToWrite = false;
            writer.write(',');
        }
        writer.write(buffer, offset, length);
    }

    @Override
    public void flush()
            throws IOException {
        writer.flush();
    }

    @Override
    public void close()
            throws IOException {
        writer.write(terminator);
        writer.close();
    }

}

还有一个示例测试（假设测试在秒表友好的环境中运行，例如 IntelliJ IDEA -- 见下文）：

public final class JsonAppenderTest {

    @RepeatedTest(10)
    public void testAppendAtEnd()
            throws IOException {
        try ( final RandomAccessFile randomAccessFile = new RandomAccessFile(LARGE_JSON_PATH, "rw");
                final Writer writer = JsonAppender.appendAtEnd(randomAccessFile) ) {
            final String json = new JSONObject(ImmutableMap.of("foo", "bar")).toString();
            CharStreams.copy(new StringReader(json), writer);
        }
    }

}

它是如何工作的：附加写入器首先从给定文件的末尾开始尝试跳过可能的空格，然后尝试检测终止字符（} 或 ]，取决于文档） ，然后它尝试检测最后一个值以在之后附加一个新值（可能有空格）。之后，一旦找到写入位置，它只会为已配置为在该位置写入的给定文件描述符创建一个缓冲读取器。并且一旦 writer 关闭，终止字符将写入文件输出流，然后底层流也将关闭。

这里是连续 10 次附加的示例结果（测试在适当的基准测试方面写得很差，但这在这里已经足够好了），输入文件大小约为 24 MB（比你的大三倍）：

• ~30..40ms 用于第一次测试运行（预热 JVM 等）
• 从第 2 次到第 10 次每次测试运行约 1..2 毫秒
• 总共 10 次测试运行约 40 毫秒

运行您的解决方案每次测试运行大约产生约 500 毫秒（总共 10 次测试运行大约需要 5 秒）。另请注意，您的解决方案在每次运行时都会创建一个新的大文件，因此非常耗费资源且不是最优的（随着输入文件大小的增长甚至会更慢），而使用 RandomAccessFile 进行一些低级解析非常快且仅追加必要的数据到最后。

再次感谢 Carson 的好主意！

Answer 3

围绕@fluffy 和@Carson 的答案创建一个包装器。

public class JsonStream extends Writer {
    private final BufferedWriter writer;
    private final char terminator;

    private boolean isAboutToWrite = true;

    private JsonStream(final BufferedWriter writer, final char terminator) {
        this.writer = writer;
        this.terminator = terminator;
    }

    public static Writer appendAtEnd(final RandomAccessFile randomAccessFile)
            throws IOException {
        long pos = randomAccessFile.length() - 1;
        char terminator = '\u0000';
        outer_whitespace:
        for ( ; pos >= 0; pos-- ) {
            randomAccessFile.seek(pos);
            final char ch = (char) randomAccessFile.readByte();
            switch ( ch ) {
// @formatter:off
                case ' ': case '\r': case '\n': case '\t':
// @formatter:on
                    continue;
// @formatter:off
                case ']': case '}':
// @formatter:on
                    terminator = ch;
                    break outer_whitespace;
                default:
                    throw new IOException("Unexpected " + ch + " at " + pos);
            }
        }
        if ( pos < 0 ) {
            throw new IOException("No object or array begin found");
        }
        inner_whitespace:
        for ( pos -= 1; pos >= 0; pos-- ) {
            randomAccessFile.seek(pos);
            final char ch = (char) randomAccessFile.readByte();
            switch ( ch ) {
// @formatter:off
                case ' ': case '\r': case '\n': case '\t':
// @formatter:on
                    continue;
// @formatter:off
                case '}': case ']':
                case '\"':
                case '0': case '1': case '2': case '3': case '4': case '5': case '6': case '7': case '8': case '9':
                case 'e': // for both true and false
                case 'l': // for null
// @formatter:on
                    break inner_whitespace;
                default:
                    throw new IOException("Unexpected " + ch + " at " + pos);
            }
        }
        return new JsonStream(new BufferedWriter(new OutputStreamWriter(new FileOutputStream(randomAccessFile.getFD()))), terminator);
    }

    @Override
    public void write(final char[] buffer, final int offset, final int length)
            throws IOException {
        if ( isAboutToWrite ) {
            isAboutToWrite = false;
            writer.write(',');
        }
        writer.write(buffer, offset, length);
    }

    @Override
    public void flush()
            throws IOException {
        writer.flush();
    }

    @Override
    public void close()
            throws IOException {
        writer.write(terminator);
        writer.close();
    }

}

public class JsonStreamAppender extends Writer {

    final File file;
    final Map<String,Object> map;
    final Writer writer;
    public JsonStreamAppender(File file) throws IOException {
        this.file=file;
        map=new LinkedHashMap<>();
        final RandomAccessFile randomAccessFile = new RandomAccessFile(file.getAbsolutePath(), "rw");
        writer= JsonStream.appendAtEnd(randomAccessFile);
    }

    public void add(String key,int value) throws IOException {
        map.put(key,value);
        this.append();
    }



    private void append() throws IOException {
        final String json = new JSONObject(ImmutableMap.copyOf(map)).toString();
        CharStreams.copy(new StringReader(json.substring(1,json.length()-1)), writer);
    }

    /**
     * Writes a portion of an array of characters.
     *
     * @param cbuf Array of characters
     * @param off  Offset from which to start writing characters
     * @param len  Number of characters to write
     * @throws IOException If an I/O error occurs
     */
    @Override
    public void write(char[] cbuf, int off, int len) throws IOException {
        writer.write(cbuf,off,len);
    }

    /**
     * Flushes the stream.  If the stream has saved any characters from the
     * various write() methods in a buffer, write them immediately to their
     * intended destination.  Then, if that destination is another character or
     * byte stream, flush it.  Thus one flush() invocation will flush all the
     * buffers in a chain of Writers and OutputStreams.
     *
     * <p> If the intended destination of this stream is an abstraction provided
     * by the underlying operating system, for example a file, then flushing the
     * stream guarantees only that bytes previously written to the stream are
     * passed to the operating system for writing; it does not guarantee that
     * they are actually written to a physical device such as a disk drive.
     *
     * @throws IOException If an I/O error occurs
     */
    @Override
    public void flush() throws IOException {
        writer.flush();
    }

    /**
     * Closes the stream, flushing it first. Once the stream has been closed,
     * further write() or flush() invocations will cause an IOException to be
     * thrown. Closing a previously closed stream has no effect.
     *
     * @throws IOException If an I/O error occurs
     */
    @Override
    public void close() throws IOException {
        writer.close();
    }
}

测试用例

    @RepeatedTest(10)
    public void testAppendAtEndAppender(){
        try(JsonStreamAppender jsonStreamAppender=new JsonStreamAppender(new File(LARGE_JSON_PATH))){
            jsonStreamAppender.add("operationalLife",-1);
        } catch (IOException e) {
            e.printStackTrace();
        }


    }