IIS 在控制器操作之前缓冲请求内容

Question

我正在尝试设置一个“外部”Web api，它将能够接收大型 http 帖子 (+1GB) 并将流转发到另一个将请求内容写入文件的“内部”Web api。我根据使用自定义 WebHostBufferPolicySelector 和在控制器方法中使用 UseBufferedInputStream 方法的示例对我的实现进行了建模。它在使用 IIS Express 时按预期工作，内存占用量没有显着增加，但是一旦我的代码部署到 IIS，内存占用量就会很大并导致 OOM。

我已经在我的控制器方法和我的 WebHostBufferPolicySelector.UseBufferedInputStream 中放置了跟踪语句，并验证了 UseBufferedInputStream 总是返回 false 并且我的控制器方法受到了打击。我注意到的唯一区别是，当我调试时，UseBufferedInputStream 和我的控制器方法之间的时间戳非常接近。在 IIS 上托管的地方，时间戳相距甚远，这表明在调用 UseBufferedInputStream 和调用我的控制器方法之间的某些东西正在完全缓冲请求。

我正在寻找一些提示，以找出导致请求被缓冲的原因以及如何让它不缓冲并一直使用流媒体。

客户端将进入外部 Web api，其内容类型为 application/octet-stream，传输编码为 Chucked。

用于构建实现

https://forums.asp.net/t/2018289.aspx?Web+API2+WebHostBufferPolicySelector+UseBufferedInputStream+override

https://www.strathweb.com/2012/09/dealing-with-large-files-in-asp-net-web-api/

代理 Web Api 控制器方法

        <HttpPost, Route("postLargeFile")>
        Protected Overridable Async Function PostLargeFile() As Threading.Tasks.Task(Of IHttpActionResult)
            Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyProxyController)}", "Started {0}", NameOf(MyProxyController.PostLargeFile))

            Dim internalHttpClient As HttpClient
            Dim fowardingContent As StreamContent = Nothing
            Dim fowardingMessage As HttpRequestMessage = Nothing
            Dim fowardingResponse As HttpResponseMessage = Nothing
            Dim externalResponse As HttpResponseMessage = Nothing

            Try
                internalHttpClient = New HttpClient()
                internalHttpClient.BaseAddress = "https://myinternalService.com"

                fowardingMessage = New HttpRequestMessage(HttpMethod.Post, "https://myinternalService.com/saveLargeFile")
                fowardingContent = New StreamContent(HttpContext.Current.Request.GetBufferlessInputStream(True))
                CopyContentHeaders(Request.Content, fowardingContent)

                fowardingMessage.Headers.TransferEncodingChunked = True
                fowardingMessage.Content = fowardingContent

                fowardingResponse = Await internalHttpClient.SendAsync(fowardingMessage, HttpCompletionOption.ResponseHeadersRead)

                externalResponse = New HttpResponseMessage(fowardingResponse.StatusCode)
                externalResponse.Content = New StreamContent(Await fowardingResponse.Content.ReadAsStreamAsync)
                CopyContentHeaders(fowardingResponse.Content, externalResponse.Content)

                Return New Results.ResponseMessageResult(externalResponse)

            Catch ex As Exception
                Return InternalServerError(ex)
            Finally
                Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyProxyController)}", "Finished {0}", NameOf(MyProxyController.PostLargeFile))
            End Try
        End Function

内部 Web Api 控制器方法

        <HttpPost, Route("saveLargeFile")>
        Protected Overridable Async Function SaveLargeFile() As Threading.Tasks.Task(Of IHttpActionResult)
            Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyInternalController)}", "Started {0}", NameOf(MyInternalController.PostLargeFile))

            Dim bufferlessStream As IO.Stream
            Dim fowardingContent As StreamContent = Nothing

            Try

                bufferlessStream = HttpContext.Current.Request.GetBufferlessInputStream()
                Using fileStream As IO.FileStream = IO.File.Create("MyFile.txt")
                    bufferlessStream.CopyTo(fileStream)
                    fileStream.Flush()
                End Using

                Return New Results.StatusCodeResult(Net.HttpStatusCode.Created, Me)

            Catch ex As Exception
                Return InternalServerError(ex)
            Finally
                Configuration.Services.GetTraceWriter.Info(Request, $"{Me.GetType.Namespace}.{NameOf(MyInternalController)}", "Finished {0}", NameOf(MyInternalController.PostLargeFile))
            End Try
        End Function

策略选择器配置

Public Class MyBufferPolicySelector
    Inherits Http.WebHost.WebHostBufferPolicySelector

    Public Property Tracer As ITraceWriter

    Public Overrides Function UseBufferedInputStream(hostContext As Object) As Boolean
        UseBufferedInputStream = False
        Tracer?.Info(Nothing, $"{Me.GetType.Namespace}.{NameOf(MyBufferPolicySelector)}", "{0} UseBufferedInputStream={1}", HttpContext.Current?.Request?.Url?.AbsoluteUri, UseBufferedInputStream)
        Return UseBufferedInputStream
    End Function
End Class

用于内部和外部 Web API 的 WebApiConfig

Public Module WebApiConfig

    Public Sub Register(ByVal config As HttpConfiguration)
        Dim tracer As SystemDiagnosticsTraceWriter

        ' Web API configuration and services

        ' Web API routes
        config.MapHttpAttributeRoutes()

        tracer = config.EnableSystemDiagnosticsTracing
        tracer.IsVerbose = True
        tracer.MinimumLevel = Tracing.TraceLevel.Debug

        GlobalConfiguration.Configuration.Services.Replace(GetType(IHostBufferPolicySelector), New MyBufferPolicySelector() With {.Tracer = tracer})

    End Sub

End Module

Answer 1

我能够找出导致 IIS 中缓冲的原因。下面的链接将我带到 IIS 中的 uploadReadAheadSize 设置。这被最大化了。因此，这将导致 IIS 在将请求传递到存在 web api 管道的模块（web api 控制器）之前完全读入/缓冲请求。将其设置为默认值后，我看到我的大文件帖子没有被缓冲，应用程序池内存占用仍然很低，不再出现内存不足异常，并且性能大幅提升。太好了！

但现在我遇到了与以下链接中描述的相同的问题。当需要 SSL 时，在 IIS 中设置，在我们的非开发环境中需要它，uploadReadAheadSize 需要增加，以便 ssl 可以在我猜的 ssl 模块中工作。它可能与某些 SSL 重新协商有关。

谁能描述一种方法来防止 SSL 中的缓冲以保持低内存占用并防止大型 http 帖子的内存不足异常？

Large file upload when using ssl and client certificates (uploadReadAheadSize) but dont want all data to be readahead