无法从网站捕获数据

Question

我正在尝试捕获由以下 URL 提供的 JSON： https://www.gasunietransportservices.nl/en/shippers/balancing-regime/sbs-and-pos/graphactualjson/M3G

我为我的问题找到了各种文章和解决方案，我将在下面逐一介绍。它们似乎都不起作用。

简单案例

HttpWebRequest req =
                (HttpWebRequest)
                    WebRequest.Create(
                        "https://www.gasunietransportservices.nl/en/shippers/balancing-regime/sbs-and-pos/graphactualjson/M3G");

            var resp = req.GetResponse();

给出以下 Web 异常：

An unhandled exception of type 'System.Net.WebException' occurred in System.dll

Additional information: The underlying connection was closed: An unexpected error occurred on a send.

下一次尝试：TLS 版本

来自..The underlying connection was closed: An unexpected error occurred on a receive，我最初尝试了简单的答案，设置 TLS 版本和 Expect100。代码变为（有或没有 KeepAlive 的结果相同）：

    ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
    ServicePointManager.Expect100Continue = true;

    HttpWebRequest req =
        (HttpWebRequest)
            WebRequest.Create(
                "https://www.gasunietransportservices.nl/en/shippers/balancing-regime/sbs-and-pos/graphactualjson/M3G");

    req.KeepAlive = false;

    var resp = req.GetResponse();

给出例外：

An unhandled exception of type 'System.Net.WebException' occurred in System.dll

Additional information: The underlying connection was closed: An unexpected error occurred on a send.

尝试 3：MSDN 解决方案

查看以下 MSD 文章“https://support.microsoft.com/en-us/help/915599/you-receive-one-or-more-error-messages-when-you-try-to-make-an-http-re”（由其他线程上的另一个答案链接到，建议以下内容，所有这些都提供了前面已经说明的相同异常：

解决方案 A：更新到最新框架。

我在 .Net 4.5 之前已经尝试过，结果相同。最初为 .Net 1.1 编写的文章。由于我的环境限制，我无法升级任何其他版本。

解决方案 D：将 KeepAlive 设置为 false

如上面的摘录所示，这没有区别。

解决方案 E：设置 MaxIdleTime 属性

我也试过了，结果没有区别。代码变成：

ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
            ServicePointManager.Expect100Continue = true;

            ServicePointManager.MaxServicePoints = 4;
            ServicePointManager.MaxServicePointIdleTime = 1000;

            HttpWebRequest req =
                (HttpWebRequest)
                    WebRequest.Create(
                        "https://www.gasunietransportservices.nl/en/shippers/balancing-regime/sbs-and-pos/graphactualjson/M3G");

            req.KeepAlive = false;

            var resp = req.GetResponse();

结果 F：修改服务器超时

网络服务器是外部的，不受我的控制。

解决方案 0：HTTP-100 标头

我已经尝试了上面列出的代码，但都没有'Expect100Continue 属性。结果相同。

我已经尝试了上述所有解决方案，无论是联合还是单独尝试，都没有成功。如果有人能够提供任何建议，所提供的代码应该“按原样”运行。

最后，我尝试使用 WebClient 类，而不是 HttpWebRequest - 再次得到相同的结果。

谢谢。

编辑 1 - 代理 我可以从同一台机器上的浏览器访问 URL。我还添加了以下代码：

req.Proxy = new WebProxy("http://MY-PROXY-URL");
            req.Proxy.Credentials = CredentialCache.DefaultCredentials;

结果不受影响。

编辑 2 - 相同的代码，不同的环境

使用“下一次尝试：Tls 版本”中的代码，但是从我的服务器运行，会产生不同的异常： System.Net.WebException：请求被中止：无法创建 SSL/TLS 安全通道。

编辑 3 - 内部异常

我桌面上初始代码的完整错误跟踪：

System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host\r\n   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)\r\n   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)\r\n   --- End of inner exception stack trace ---\r\n   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)\r\n   at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)\r\n   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)\r\n   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)\r\n   at System.Ne
t.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)\r\n   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)\r\n   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)\r\n   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)\r\n   at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)\r\n   at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)\r\n   at System.Net.ConnectStream.WriteHeaders(Boolean async)\r\n   --- End of inner exception stack trace ---\r\n   at System.Net.HttpWebRequest.GetResponse()\r\n   at GasUnieSBS.Program.Main(String[] args) in c:\\dev\\Sandpit\\MessagingPlay\\GasUnieSBS\\Program.cs:line 37"

Answer 1

希望这对其他人有所帮助。

通过在 Windows 10 上发布代码示例（“下一次尝试 - TLS 版本”）解决了这个问题。我读过的所有内容听起来都像 Windows 7 和 .Net 4.5 应该支持完整的 TLS1.2。

但是，对目标 URL 的扫描表明，他们非常受制于所允许的内容。我们的一位安全工程师建议这是我们遇到的第二个类似问题，它与我们自己的 Windows-7 构建配置有关。

我还没有解决服务器上的问题（Windows 2008 R2）。