更快地创建 CSV 文件

Question

我通过获取一个数据表然后循环遍历该数据表并写入 CSV 文件的每一行来创建一个 CSV 文件。我的数据源中通常有大约 65,000 行。此过程需要几分钟才能从浏览器下载。问题是在本地和开发上它不会花费太长时间，但在客户端它们通常会超时。

有没有更快的方法来生成这个文件？

Function GenerateCSVFile() As String
    Dim stuPro As New studentProvider.StudentProvider
    Dim emailCenter As New EmailCenter
    Dim strFileName As String = System.IO.Path.GetRandomFileName().Replace(".", "")
    Dim strResult As String = ""

    Dim dtStudent As Data.DataTable
    Dim paymentYear As String = ""
    dtStudent = stuPro.generateDataFile()

    If dtStudent.Rows.Count > 0 Then

        Using sw As New System.IO.StreamWriter(Server.MapPath("Temp/" + strFileName + ".csv"))
            Try
                Dim lineValue As String = ""

                lineValue += "Academic Year, StudentID, SSN, First, Middle, Last"

                sw.WriteLine(lineValue)

                For i As Integer = 0 To dtStudent.Rows.Count - 1

                    lineValue = dtStudent.Rows(i)("fy").ToString
                    lineValue += "," & dtStudent.Rows(i)("uniq_stu_id").ToString
                    lineValue += "," & dtStudent.Rows(i)("ssn").ToString
                    lineValue += "," & dtStudent.Rows(i)("fname").ToString
                    lineValue += "," & dtStudent.Rows(i)("mname").ToString
                    lineValue += "," & dtStudent.Rows(i)("lname").ToString
                    sw.WriteLine(lineValue)

                Next
            Catch ex As Exception
                strResult += ex.ToString
            Finally
                sw.Close()
            End Try

        End Using

        Dim strFriendlyName As String = Date.Now.ToString("MM-dd-yyyy") & ".csv"

        If String.IsNullOrEmpty(strResult) Then

            Dim fs As System.IO.FileStream = Nothing

            fs = System.IO.File.Open(Server.MapPath("Temp/" + strFileName + ".csv"), System.IO.FileMode.Open)
            Dim btFile(fs.Length) As Byte
            fs.Read(btFile, 0, fs.Length)
            fs.Close()

            With Response
                .AddHeader("Content-disposition", "attachment;filename=" & strFriendlyName)
                .ContentType = "application/octet-stream"
                .BinaryWrite(btFile)
                .End()
            End With
        End If
    Else
        strResult = "No records found for specified academic year"
    End If

    Return strResult
End Function

更新代码

Function GenerateCSVFile() As String
    Dim startDate As Date = Date.Now
    Dim enddate As Date = Nothing
    Dim stuPro As New studentProvider.StudentProvider
    Dim emailCenter As New EmailCenter
    Dim strFileName As String = System.IO.Path.GetRandomFileName().Replace(".", "")
    Dim strResult As String = ""

    Dim dtStudent As Data.DataTable
    Dim paymentYear As String = ""
    dtStudent = stuPro.generateDataFile(Session("VendorID"), txtAcademicYear.Text.Trim)

    If dtStudent.Rows.Count > 0 Then

        With Response

            Dim strFriendlyName As String = Date.Now.ToString("MM-dd-yyyy") & ".csv"
            .AddHeader("Content-disposition", "attachment;filename=" & strFriendlyName)
            .ContentType = "application/octet-stream"

            Dim lineValue As StringBuilder = New StringBuilder

            lineValue.Append("Academic Year, StudentID, SSN, First, Middle, Last")

            .Write(lineValue.ToString)

            For i As Integer = 0 To dtStudent.Rows.Count - 1

                lineValue = New StringBuilder
                lineValue.Append(dtStudent.Rows(i)("fy").ToString)
                lineValue.Append("," & dtStudent.Rows(i)("uniq_stu_id").ToString)
                lineValue.Append("," & dtStudent.Rows(i)("ssn").ToString)
                lineValue.Append("," & dtStudent.Rows(i)("fname").ToString)
                lineValue.Append("," & dtStudent.Rows(i)("mname").ToString)
                lineValue.Append("," & dtStudent.Rows(i)("lname").ToString)

                .Write(lineValue.ToString)

            Next
            enddate = Date.Now

            MsgBox(DateDiff(DateInterval.Second, startDate, enddate))

            .End()
        End With
    Else
        strResult = "No records found for specified academic year"
    End If
    Return strResult
End Function

Answer 1

您正在写入临时文件，读取该文件，然后他们将该文件的内容写入响应中。跳过临时文件并一次一行直接写入响应。这将使浏览器不会认为您的服务器正在超时，使事情变得更快，并减少您的应用程序消耗的内存量。

之后，研究如何在此 Web 请求上启用缓存，以便 ASP.NET 在多个用户在短时间内请求时不必重新创建 CSV。

Answer 2

有一些选项可以加快速度：

• 不要从 StreamWriter 写入文件到 Response，直接写入页面 Response。
• 寻找使用数据读取器而不是数据表来循环数据的解决方案，使用这么多数据可能会更快。它还会降低内存使用量（不会将整个表加载到内存中）。
• 如果连接多个字符串使用 StringBuilder，或者在这种情况下使用 String.Join 轻松创建整行。

String.Join 示例：

For Each row in dtStudent.Rows
    Dim line as new List(of String)
    line.Add(row("fy").ToString)
    line.Add(row("uniq_stu_id").ToString)
    line.Add(-etc-)

    Response.Write(String.Join(",", line.ToArray) & vbcrlf )
Next

Answer 3

您应该使用 StringBuilder 而不是串联。

Answer 4

除了@Robert Levy 的建议之外，请注意如何使用字符串变量。在这些行上使用 stringbuilder 会更好地为您服务：

        dim sbTemp as new StringBuilder()
        For i As Integer = 0 To dtStudent.Rows.Count - 1

            sbTemp.Append(dtStudent.Rows(i)("fy").ToString)
            sbTemp.Append(",") 
            sbTemp.Append(dtStudent.Rows(i)("uniq_stu_id").ToString)

            'etc
            sw.WriteLine(lineValue)

        Next

Answer 5

您可以研究的一件事是生产者/消费者设计模式。这可以让您做的是让一个（或多个）线程提供一个包含需要写入 csv 文件的数据的队列和另一个（或多个）执行实际写入的线程。