如何将 BIG JSON 数据反序列化为 DataGridView

Question

我想将一个大的 JSON 文件 (~600MB) 反序列化为 DataGridView 而不会遇到“maxJsonLength”错误。

我的代码适用于小型 JSON 文件。但对于更大的 JSON 文件，我得到“maxJsonLength”错误。由于我不是最有经验的编码员，是否有解决此问题的解决方法或简单方法？

Private Sub BtnOpenFile_Click(sender As Object, e As EventArgs) Handles BtnOpenFileOld.Click
        OpenFileDialog1.InitialDirectory = "C:\"

        If OpenFileDialog1.ShowDialog = DialogResult.Cancel Then
        End If

        Dim JSonFilePath As String = File.ReadAllText(OpenFileDialog1.FileName)
        LblFilePath.Text = OpenFileDialog1.FileName

        DataGridView1.Rows.Clear()
        DataGridView1.Refresh()

        Dim dict As Object = New JavaScriptSerializer().Deserialize(Of List(Of Object))(JSonFilePath)

        For Each item As Object In dict
            DataGridView1.Rows.Add(item("EMail").ToString, item("Timestamp").ToString, item("Number").ToString)
        Next

    End Sub

我的 JSON 文件看起来像

[
  {
    "EMail": "one@mail.com",
    "Timestamp": "2019-05-25T21:24:06.799381+02:00",
    "Number": 206074,
    "randomtrash1": "notneeded",
    "randomtrash2": "notneeded",
    "randomtrash3": "notneeded",
    "randomtrash4": "notneeded",
    "randomtrash5": "notneeded",
    "randomtrash6": "notneeded",
    "randomtrash7": "notneeded",
    "randomtrash8": "notneeded",
    "randomtrash9": "notneeded"
  },
  {
    "EMail": "two@mail.com",
    "Timestamp": "2019-05-25T21:24:06.8273826+02:00",
    "Number": 7397,
    "randomtrash1": "notneeded",
    "randomtrash2": "notneeded",
    "randomtrash3": "notneeded",
    "randomtrash4": "notneeded",
    "randomtrash5": "notneeded",
    "randomtrash6": "notneeded",
    "randomtrash7": "notneeded",
    "randomtrash8": "notneeded",
    "randomtrash9": "notneeded",
    "randomtrash10": "notneeded",
    "randomtrash11": "notneeded",
    "randomtrash12": "notneeded",
    "randomtrash13": "notneeded",
    "randomtrash14": "notneeded",
    "randomtrash15": "notneeded",
    "randomtrash16": "notneeded",
    "randomtrash17": "notneeded",
    "randomtrash18": "notneeded",
    "randomtrash19": "notneeded",
    "randomtrash20": "notneeded",
    "randomtrash21": "notneeded"
  }
]

Answer 1

避免maxJsonLength 错误的最小修复方法是将JavaScriptSerializer.MaxJsonLength = int.MaxValue 设置为this answer 中的The length of the string exceeds the value set on the maxJsonLength property 中的Taha Rehman Siddiqui：

Dim serializer As JavaScriptSerializer = New JavaScriptSerializer()
serializer.MaxJsonLength = Int32.MaxValue
Dim dict As Object = serializer.Deserialize(Of List(Of Object))(JSonFilePath)

但是，您当前正在将一个 600MB 的文件加载到一个 600-1,200MB 的字符串中，该字符串正在接近 .Net maximum string length。如果你的文件变得有点大，你将开始出现内存不足的异常。更好的方法是使用流解决方案直接从文件反序列化，但是JavaScriptSerializer 不支持流。因此，我建议切换到json.net，确实如此。

首先，安装Json.NET，如图here。接下来，由于您似乎不想为 JSON 定义显式数据模型，请创建以下静态方法：

Public Module JsonExtensions
    Public Function LoadAnonymousType(Of T)(ByVal path as String, ByVal anonymousTypeObject as T, Optional ByVal settings as JsonSerializerSettings = Nothing) as T
        Using streamReader As TextReader = File.OpenText(path)
            Dim serializer = JsonSerializer.CreateDefault(settings)
            return CType(serializer.Deserialize(streamreader, GetType(T)), T)
        End Using
    End Function
End Module

这样做的目的是将path 参数指定位置的文件的内容反序列化为anonymousTypeObject 参数指定类型的数据模型。

现在您可以将 JSON 从文件中直接反序列化为 anonymous type objects 数组，如下所示：

Dim array = JsonExtensions.LoadAnonymousType( _
    FileName, _
    { New With {.Email = CType(Nothing, String), .Timestamp = CType(Nothing, String), .Number = CType(Nothing, Long) }} _
)
For Each item In array
    ' DataGridView1.Rows.Add(item("EMail").ToString, item("Timestamp").ToString, item("Number").ToString)
    Console.WriteLine("Email = {0}, Timestamp = {1}, Number = {2}", item.Email, item.Timestamp, item.Number)
Next

或者，您可以按如下方式创建显式数据模型：

Public Class RootObject
    Public Property Email As String
    Public Property Timestamp As String
    Public Property Number As Long
End Class

并反序列化如Deserialize JSON from a file所示：

Dim list As List(Of RootObject) = Nothing
Using streamReader As TextReader = File.OpenText(FileName)
    list = CType(JsonSerializer.CreateDefault().Deserialize(streamReader, GetType(List(Of RootObject))), List(Of RootObject))
End Using
For Each item In list
    ' DataGridView1.Rows.Add(item("EMail").ToString, item("Timestamp").ToString, item("Number").ToString)
    Console.WriteLine("Email = {0}, Timestamp = {1}, Number = {2}", item.Email, item.Timestamp, item.Number)
Next

演示小提琴here.

Answer 2

Imports Newtonsoft.Json

Public Class Form1

Private Sub Form1_Load(sender As Object, e As EventArgs) Handles MyBase.Load

    Dim str As String = _
            "[" + _
            "  {" + _
            "    ""EMail"": ""one@mail.com""," + _
            "    ""Timestamp"": ""2019-05-25T21:24:06.799381+02:00""," + _
            "    ""Number"": 206074," + _
            "  }," + _
            "  {" + _
            "    ""EMail"": ""two@mail.com""," + _
            "    ""Timestamp"": ""2019-05-25T21:24:06.8273826+02:00""," + _
            "    ""Number"": 7397," + _
            "  }," + _
            "]"

    Try
        Dim list As List(Of jsnn) = JsonConvert.DeserializeObject(Of List(Of jsnn))(str)

        For Each item As jsnn In list
            Console.WriteLine(item.EMail & ": " & item.Timestamp & ": " & item.Number)
        Next

    Catch ex As Exception
        MsgBox(ex.Message)
    End Try

End Sub

End Class

Public Class jsnn
    Public Property EMail As String
    Public Property Timestamp As DateTime
    Public Property Number As Integer
End Class

这是截图