多线程应用程序中的 HttpWebRequest.GetResponse 性能问题

Question

当从不同的线程为不同的 URL 调用 HttpWebRequest.GetResponse 方法时，我的性能非常差。 例如，如果我们有一个线程并执行 url1，它需要 3 秒。 如果我们并行处理 url1 和 url2 需要 10 秒，第一个请求在 8 秒后结束，第二个在 10 秒后结束。

如果我们执行 10 个 URL url1, url2, ... url0 它需要 1 分 4 秒！！！第一个请求在 50 秒后结束！

我使用 GetResponse 方法。 我试过 te set DefaultConnectionLimit 但它没有帮助。 如果使用 BeginGetRequest/EndGetResponse 方法，它的工作速度非常快，但前提是从一个线程调用此方法。如果从不同的地方它也很慢。 我需要同时执行来自多个线程的 Http 请求。

在每个线程中执行相同的代码。如果只有一个线程 GetResponse 方法工作得非常快。 每个线程的 URL 中的 IP 地址也不同。 如果您编译并运行以下代码，您将看到请求被一一处理。拳头执行 3 秒，第二个 8 秒，第三个 15 秒.... 即多线程没有任何好处。

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.IO;
using System.Xml;
using System.Reflection;
using System.Threading;
using System.Collections.Generic;
using System.Collections.Specialized;
using System.Collections;

namespace HttpRequestExample
{

    class HttpPerformer
    {
        private Thread thread = null;
        HttpWebRequest httpRequest = null;

        public void start(string url)
        {
            thread = new Thread(new ThreadStart(WorkerThread));
            thread.Name = url;
            thread.Start();

        }

        public void WorkerThread()
        {
            try
            {
                HttpWebRequest req = (HttpWebRequest)WebRequest.Create((string)thread.Name);
                Console.WriteLine(DateTime.Now + " : before get response " + thread.Name);
                HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
                Console.WriteLine(DateTime.Now + " : after get response " + thread.Name);

            }
            catch (Exception e)
            {
                Console.WriteLine(DateTime.Now + " : Exception : " + e.Message + thread.Name);
            }
        }
    }

    class HttpAccessUtils
    {
        public static bool SetAllowUnsafeHeaderParsing20()
        {
            //Get the assembly that contains the internal class 
            Assembly aNetAssembly = Assembly.GetAssembly(typeof(System.Net.Configuration.SettingsSection));
            if (aNetAssembly != null)
            {
                //Use the assembly in order to get the internal type for the internal class 
                Type aSettingsType = aNetAssembly.GetType("System.Net.Configuration.SettingsSectionInternal");
                if (aSettingsType != null)
                {
                    //Use the internal static property to get an instance of the internal settings class. 
                    //If the static instance isn't created allready the property will create it for us. 
                    object anInstance = aSettingsType.InvokeMember("Section",
                    BindingFlags.Static | BindingFlags.GetProperty | BindingFlags.NonPublic, null, null, new object[] { });

                    if (anInstance != null)
                    {
                        //Locate the private bool field that tells the framework is unsafe header parsing should be allowed or not 
                        FieldInfo aUseUnsafeHeaderParsing = aSettingsType.GetField("useUnsafeHeaderParsing", BindingFlags.NonPublic | BindingFlags.Instance);
                        if (aUseUnsafeHeaderParsing != null)
                        {
                            aUseUnsafeHeaderParsing.SetValue(anInstance, true);
                            return true;
                        }
                    }
                }
            }
            return false;
        }
    }

    class Program
    {

        static void Main(string[] args)
        {
            HttpAccessUtils.SetAllowUnsafeHeaderParsing20();
            ServicePointManager.UseNagleAlgorithm = true;
            ServicePointManager.Expect100Continue = true;
            ServicePointManager.CheckCertificateRevocationList = true;
            ServicePointManager.DefaultConnectionLimit = 200; //ServicePointManager.DefaultPersistentConnectionLimit;
            ServicePointManager.MaxServicePoints = 100;
            Console.WriteLine(ServicePointManager.MaxServicePoints);


            ArrayList a = new ArrayList(150);

            for (int i = 100; i < 220; i++)
            {
                a.Add("http://207.242.7." + i.ToString());
            }

            for (int i = 0; i < a.Count; i++)
            {
                HttpPerformer hp = new HttpPerformer();
                hp.start((string)a[i]);
            }

        }

        static void performRequest(object url)
        {
            try
            {
                HttpWebRequest req = (HttpWebRequest)WebRequest.Create((string)url);
                Console.WriteLine(DateTime.Now + " : before get response " + url);
                HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
                Console.WriteLine(DateTime.Now + " : after get response " + url);
            }
            catch (Exception e)
            {
                Console.WriteLine(DateTime.Now + " : Exception : " + e.Message + (string)url);
            }

        }
    }
}

С 有人遇到过这样的问题吗？感谢您的任何建议。

Answer 1

你需要在得到响应流后关闭连接，否则连接会长时间保持打开状态。这可能是缓慢的原因。

Answer 2

您需要在收到响应后关闭连接。这似乎导致了问题。