并行 for 循环和 httpclient 死锁并引发异常

Question

我很困惑为什么我的 parallels.for 循环在 httpclient 调用中不断地爆炸。 该代码适用于大约 10-15 个请求，然后会挂起很长时间，并出现 System.AggregateException 错误 我尝试了多种不同的变体，包括 webclient。 请考虑以下几点：

class Program
{
    static void Main(string[] args)
    {
        Parallel.For(0, 250, i =>
        {

            var task = GetPages.CallHttp();
            task.Wait();
            var content = task.Result;
            TestObject obj = content.ToTestObject(); //take the string and find some values in it. 
            Console.WriteLine(obj.H1Tag);
        });
    }
}

public static class GetPages
{
    private static readonly HttpClient client = new HttpClient() {Timeout = new TimeSpan(0,5,0)};

   public static async Task<string> CallHttp()
    {

        client.DefaultRequestHeaders.UserAgent.ParseAdd("customAgent/1.0");
        string astr = await client.GetStringAsync("the url I am testing").ConfigureAwait(false);
        return astr;
    }
}

public static class StringExtensions
{
    private static readonly object objLock = new object();
    public static TestObject ToTestObject(this string content)
    {
        lock (objLock)
        {
            var obj = new TestObject();
// creates a bunch of properties inspecting the html string
            var result = new HtmlExtractions(content); 
            obj.PageTitle = result.PageTitle;
            obj.H1Tag = result.H1Tag;
           ...
            return obj;
        }
    }

}


public class HtmlExtractions
{

    internal HtmlDocument doc;

    public HtmlExtractions(string contentToRead)
    {
        doc = new HtmlDocument();
        doc.LoadHtml(contentToRead);
    }

    public string PageTitle => doc.DocumentNode.Descendants("title").FirstOrDefault()?.InnerHtml.Replace("&amp;", "&").Trim();
...
}

结果是抛出以下异常。

    System.AggregateException was unhandled by user code
  HResult=-2146233088
  Message=One or more errors occurred.
Source=mscorlib
StackTrace:
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at System.Threading.Tasks.Task.Wait()
   at ConsoleApplication1.Program.<>c.<Main>b__0_0(Int32 i) in c:\users\username\documents\visual studio 2015\Projects\ConsoleApplication1\Program.cs:line 27
   at System.Threading.Tasks.Parallel.<>c__DisplayClass17_0`1.<ForWorker>b__1()
InnerException: 
   HResult=-2146233029
   Message=A task was canceled.
   InnerException: Id = 50, Status = Canceled, Method = "{null}", Result = "{Not yet computed}"

******按照 Todd 的建议更新 // 代码中的 cmets。太令人沮丧了。即使扩展到 5 个请求也会导致挂起。 ******

    static async void RunPagesAsync()
    {
        Console.WriteLine("getting contents");
        var tasks = Enumerable.Range(0, 5).Select(i => GetPages.CallHttp());
        var contents = await Task.WhenAll(tasks);

        Console.WriteLine("Got Contents..continuing");
        foreach (var content in contents)
        {
            TestObject obj = content.ToTestObject(); //take the string and find some values in it. 
            Console.WriteLine(obj.H1Tag);
        }
        Console.WriteLine("completed");
    }

    static void Main(string[] args)
    {

      //Task.Run(() => RunPagesAsync()); //doesn't work. 
      //  RunPagesAsync(); //just hangs after what looks like 2 itterations

      //  var tasks = Enumerable.Range(0, 5).Select(i => GetPages.CallHttp());
      //  var contents = await Task.WhenAll(tasks); //won't compile under synch Main dues to await 
      //  foreach (var content in contents)
      //  {
      //      TestObject obj = content.ToTestObject(); //take the string and find some values in it. 
      //      Console.WriteLine(obj.H1Tag);
      //  }

        var tasks1 = Enumerable.Range(0, 5).Select(i => GetPages.CallHttp());
        var contents1 =  Task.WhenAll(tasks1); 
        contents1.Wait();
        foreach (var content in contents1.Result)
        {
            TestObject obj = content.ToTestObject(); //take the string and find some values in it. 
            Console.WriteLine(obj.H1Tag);
        }

Answer 1

异步任务（释放线程）和Parallel.For（强制使用多个线程）往往不能很好地混合。 .Wait() 和 .Result 在与异步任务一起使用时都会阻塞调用并引发死锁。尝试使用 Task.WhenAll 重写您的 Main 方法并避免阻塞调用：

var tasks = Enumerable.Range(0, 250).Select(i => GetPages.CallHttp());
var contents = await Task.WhenAll(tasks);
foreach (var content in contents)
{
    TestObject obj = content.ToTestObject(); //take the string and find some values in it. 
    Console.WriteLine(obj.H1Tag);
}