解析大于内存的 JSON 字符串答案

【问题标题】：Parsing a JSON string that is bigger than the memory解析大于内存的 JSON 字符串
【发布时间】：2017-10-01 12:56:28
【问题描述】：

我正在开发的平台具有非常严格的内存限制，我正在尝试找到一种方法来解析大型 JSON 字符串，而无需将最多几百个字节加载到内存中。 JSON 字符串存储在更大芯片（闪存）上的文件中。

有两件事我真的找不到好的解决方案：

通过指定像foo["bar"][2]这样的“路径”来访问某个值。
（如果结果是一个数组/对象，那么我们应该只返回它是一个数组/对象的事实，也可能返回它是否为空。）
遍历 JSON 中的任何对象/数组。

所以基本上我需要在调用时逐步解析 json 并且只保存我们实际需要继续解析的部分的函数。

对于界面，我认为不可能有类似exampleJson["aa"].2.["gg] 的东西，但我设法非常接近：exampleJson["aa"].2.["gg"]()。这将导致调用一个函数，然后该函数可以轻松访问 {'aa',2,'gg'} 并从文件中读取/解析 json。

这是我目前的代码，但我真的不知道如何继续：
https://repl.it/HfwS/2

-- Looks complicated, but is pretty simple. Using meta tables we create a json interface that can almost be accessed as if it was a lua table.
-- E.g. example["aa"][2]["gg"]() ; the only difference is that we have to use parentheses at the end
-- The problematic part starts where it says `THIS IS WHERE THE JSON PARSING WOULD HAPPEN`
json = {}
setmetatable(json, {
    __call = function(path)
        local jsonFile = _file.open(filePath)
        local fileLen = jsonFile:stat().size

        local patternTable = {} -- Will store `{'aa',2,'gg'}` for `example.['aa'].[2]['gg']()`

        local fakeJson = {}
        setmetatable(fakeJson, { 
            __index = function (t, k)
                patternTable[#patternTable+1] = k
                return fakeJson
            end;
            __call = function()

                -- THIS IS WHERE THE JSON PARSING WOULD HAPPEN --

                -- The patternTable contains {'aa',2,'gg'} at this point 

                -- Loop through the json file char by char
                local valueToReturn = ''
                local filePos = 0
                for i=1, fileLen do
                    jsonFile:seek("set", filePos)
                    local currentChar = jsonFile:read(1) -- read character at current position
                    filePos = filePos + 1
                    -- print(currentChar)

                    -- Now the question is, how do we parse the json?
                    print('Magic to parse the json')
                    -- valueToReturn = ?
                end

                patternTable = {} -- Reset the patternTable
                return valueToReturn
            end;
        })
      return fakeJson
    end;
})


local fakeParsedJson = json('example.json')
local value = fakeParsedJson["aa"][2]["gg"]() -- Notice the `()` in the end

print(value)

【问题讨论】：

标签： json parsing memory lua lua-5.1

【解决方案1】：

如果您想解码单个 JSON 元素（对象、数组等）而不是解码整个 JSON，您需要具有两个功能的 JSON 库：

“遍历”功能（试运行解码，无需创建 Lua 对象）
能够将 JSON 作为小部分序列传递（而不是将整个 JSON 预加载为巨大的 Lua 字符串）。

示例：
如何使用this module 部分解码 JSON：

-- This is content of data.txt file:
-- {"aa":["qq",{"k1":23,"gg":"YAY","Fermat_primes":[3, 5, 17, 257, 65537]}]}
-- We want to extract as Lua values only "Fermat_primes" array and "gg" string
local json = require('json')

-- Open file
local file = assert(io.open('data.txt', 'r'))

-- Define loader function which will read the file in 64-byte chunks
local function my_json_loader()
   return file:read(64)
end

local FP, gg
-- Prepare callback function for traverse with partial decode
local function my_callback (path, json_type, value)
   path = table.concat(path, '/')
   if path == "aa/2/Fermat_primes" then
      FP = value
      return true  -- we want to decode this array instead of traverse through it
   elseif path == "aa/2/gg" then 
      gg = value
   end
end

json.traverse(my_json_loader, my_callback)

-- Close file
file:close()

-- Display the results
print('aa.2.gg = '..gg)
print('aa.2.Fermat_primes:')
for k, v in ipairs(FP) do print(k, v) end

输出：

 aa.2.gg = YAY
 aa.2.Fermat_primes:
 1  3
 2  5
 3  17
 4  257
 5  65537

【讨论】：

【解决方案2】：

我花了更多时间思考如何实现这一目标，并最终成功实现了这一目标。检索值和迭代数组/对象就像一个魅力。如果您知道更好的方法，请告诉我。（我对代码不太满意；看起来它可能会更简洁。）但是，它可以工作。

如果你想尝试一下，这里有一个小提琴： https://repl.it/HfwS/31

json = {}
setmetatable(json, {
    __call = function(filePath)
        local jsonFile = _file.open(filePath)
        local fileLen = jsonFile:stat().size

        local jsonPath = {} -- Would store `{'aa',2,'gg'}` for `example['aa'][2]['gg']()`

        local fakeJson = {}
        setmetatable(fakeJson, { 
            __index = function (t, k)
                jsonPath[#jsonPath+1] = k
                return fakeJson
            end;
            __call = function()

                -- THIS IS WHERE THE JSON PARSING WOULD HAPPEN --

                -- The jsonPath contains {'aa',2,'gg'} at this point 

                local brcStack = {} -- will be used to push/pop braces/brackets
                local jsonPathDim = 1 -- table dimension (['a'] ==  1; ['a']['b'] == 2; ...)
                -- Loop through the json file char by char
                local valueToReturn
                local filePos = 0
                local nextChar = function()
                    jsonFile:seek("set", filePos)
                    filePos = filePos + 1
                    local char = jsonFile:read(1)
                    --print(char)
                    return char
                end
                local jsonValid = true
                for o=1, fileLen do -- infinite
                    if jsonPathDim > #jsonPath then -- jsonPath followed. Now we can extract the value.
                        while true do
                            local currentChar = nextChar()
                            if currentChar == '"' then -- string
                                valueToReturn = ''
                                for i=1, fileLen do
                                    currentChar = nextChar()
                                    if currentChar == '"' then
                                        break
                                    elseif currentChar == nil then
                                        jsonValid = false
                                        break
                                    else
                                        valueToReturn = valueToReturn .. currentChar
                                    end
                                end
                                break
                            elseif string.find(currentChar,'[%d.]') then -- numbers 0.3, .3, 99 etc
                                local rawValue = ''
                                if currentChar == '.' then
                                    rawValue = '0'
                                end
                                for i=1, fileLen do
                                    if string.find(currentChar, '[%s,\r\n%]%}]') then
                                        break
                                    elseif filePos > fileLen then
                                        jsonValid = false
                                        break
                                    else
                                        rawValue = rawValue .. currentChar
                                    end
                                    currentChar = nextChar()
                                end
                                valueToReturn = tonumber(rawValue)
                                break
                            elseif currentChar == 't' then -- true
                                valueToReturn = true
                                break
                            elseif currentChar == 'f' then -- false
                                valueToReturn = false
                                break
                            elseif currentChar == 'n' then -- null
                                valueToReturn = nil -- ?
                                break
                            elseif currentChar == '{' then -- null
                                valueToReturn = {}
                                brcStack[#brcStack+1] = '{'
                                local origBrcLvl = #brcStack
                                while true do
                                    currentChar = nextChar()
                                    if filePos > fileLen then
                                        jsonValid = false
                                        break
                                    elseif currentChar == '\\' then
                                        nextChar()
                                        -- Continue
                                    elseif origBrcLvl == #brcStack and currentChar == '"' then
                                        local keyToPush = ''
                                        while true do
                                            currentChar = nextChar()
                                            if currentChar == '"' then
                                                while true do
                                                    currentChar = nextChar()
                                                    if currentChar == ':' then
                                                        valueToReturn[keyToPush] = 0
                                                        break
                                                    elseif filePos > fileLen then
                                                        break
                                                    end
                                                end
                                                break
                                            elseif filePos > fileLen then
                                                jsonValid = false
                                                break
                                            else
                                                keyToPush = keyToPush .. currentChar
                                            end
                                        end
                                        break
                                    elseif currentChar == '[' or currentChar == '{' then
                                        brcStack[#brcStack+1] = currentChar
                                    elseif currentChar == ']' then
                                        if brcStack[#brcStack] == ']' then
                                            brcStack[#brcStack] = nil
                                        else
                                            jsonValid = false
                                            break
                                        end
                                    elseif currentChar == '}' then
                                        if brcStack[#brcStack] == '}' then
                                            brcStack[#brcStack] = nil
                                        else
                                            jsonValid = false
                                            break
                                        end
                                    end
                                end
                                break
                            elseif currentChar == '[' then
                                brcStack[#brcStack+1] = '['
                                valueToReturn = {} 
                                local origBrcLvl = #brcStack
                                while true do
                                    currentChar = nextChar()

                                    if origBrcLvl == #brcStack and #valueToReturn == 0 and not string.find(currentChar, '[%s\r\n%]]') then
                                        valueToReturn[#valueToReturn+1] = 0
                                    end
                                    if filePos > fileLen then
                                        jsonValid = false
                                        break
                                    elseif currentChar == '\\' then
                                        nextChar()
                                        -- Continue
                                    elseif origBrcLvl == #brcStack and currentChar == ',' then
                                        valueToReturn[#valueToReturn+1] = 0
                                    elseif currentChar == '[' or currentChar == '{' then
                                        brcStack[#brcStack+1] = currentChar
                                    elseif currentChar == ']' then
                                        if brcStack[#brcStack] == ']' then
                                            brcStack[#brcStack] = nil
                                        else
                                            jsonValid = false
                                            break
                                        end
                                    elseif currentChar == '}' then
                                        if brcStack[#brcStack] == '}' then
                                            brcStack[#brcStack] = nil
                                        else
                                            jsonValid = false
                                            break
                                        end
                                    end
                                end
                                break
                            end
                        end
                        break
                    end
                    local currentKey = jsonPath[jsonPathDim]
                    local currentKeyLen = string.len(currentKey)
                    if type(jsonPath[jsonPathDim]) == 'string' then -- Parsing { object
                        while true do
                            local currentChar = nextChar()
                            if currentChar == '{' then
                                brcStack[#brcStack+1] = '{'
                                local origBrcLvl = #brcStack
                                local keyFound = true
                                for z=1, fileLen do -- loop over keys until we find it
                                    currentChar = nextChar()
                                    if currentChar == '\\' then
                                        nextChar()
                                        -- Continue
                                    elseif origBrcLvl == #brcStack and currentChar == '"' then
                                        local keyMatched = false
                                        for i=1, fileLen do
                                            local expectedChar = string.sub(currentKey,i,i)
                                            if nextChar() == expectedChar then
                                                if i == currentKeyLen and nextChar() == '"' then
                                                    keyMatched = true
                                                    while true do 
                                                        currentChar = nextChar()
                                                        if currentChar == ':' then
                                                            break
                                                        elseif currentChar == nil then
                                                            jsonValid = false
                                                            break
                                                        end
                                                    end
                                                    break
                                                end
                                                -- Continue
                                            else
                                                keyMatched = false
                                                break
                                            end
                                        end
                                        if keyMatched then
                                            keyFound = true
                                            break
                                        end
                                    elseif currentChar == '[' or currentChar == '{' then
                                        brcStack[#brcStack+1] = currentChar
                                    elseif currentChar == ']' then
                                        if brcStack[#brcStack] == ']' then
                                            brcStack[#brcStack] = nil
                                        else
                                            jsonValid = false
                                            break
                                        end
                                    elseif currentChar == '}' then
                                        if brcStack[#brcStack] == '}' then
                                            brcStack[#brcStack] = nil
                                        else
                                            jsonValid = false
                                            break
                                        end
                                    end
                                end
                                if keyFound then
                                    jsonPathDim = jsonPathDim+1
                                end
                                break
                            elseif currentChar == nil then
                                jsonValid = false
                                break
                            end
                        end
                    elseif type(jsonPath[jsonPathDim]) == 'number' then -- Parsing [ array
                        while true do
                            local currentChar = nextChar()
                            if currentChar == '[' then
                                brcStack[#brcStack+1] = '['
                                local origBrcLvl = #brcStack
                                local currentIndex = 1
                                -- currentKey
                                local keyMatched = true
                                for i=1, fileLen do
                                    currentChar = nextChar()
                                    if currentChar == '\\' then
                                        nextChar()
                                        -- Continue
                                    elseif origBrcLvl == #brcStack and currentChar == ',' then
                                        currentIndex = currentIndex +1
                                        if currentIndex == currentKey then
                                            jsonPathDim = jsonPathDim+1
                                            break
                                        end
                                    elseif currentChar == '[' or currentChar == '{' then
                                        brcStack[#brcStack+1] = currentChar
                                    elseif currentChar == ']' then
                                        if brcStack[#brcStack] == ']' then
                                            brcStack[#brcStack] = nil
                                        else
                                            jsonValid = false
                                            break
                                        end
                                    elseif currentChar == '}' then
                                        if brcStack[#brcStack] == '}' then
                                            brcStack[#brcStack] = nil
                                        else
                                            jsonValid = false
                                            break
                                        end
                                    else
                                        -- Continue
                                    end
                                end
                                break
                            elseif currentChar == nil then
                                jsonValid = false
                                break
                            end
                        end
                    else
                        jsonValid = false
                        break -- Invalid json
                    end
                end
                jsonPath = {} -- Reset the jsonPath
                return valueToReturn
            end;
        })
      return fakeJson
    end;
})



local example =  json('example.json')

-- Read a value
local value = example["aa"][2]['k1']()
print(value)

-- Loop over a key value table and print the keys and values
for key, value in pairs(example["aa"][2]()) do
    print('key: ' .. key, 'value: ' .. example["aa"][2][key]())
end

JSON 验证可能会更好，但如果您提供无效的 json 数据，那么无论如何您都不应该期待任何东西。

【讨论】：