下载多个文件的更快方法

Question

我需要从 SEC 网站下载大约 200 万个文件。每个文件都有一个唯一的 url，平均为 10kB。这是我目前的实现：

    List<string> urls = new List<string>();
    // ... initialize urls ...
    WebBrowser browser = new WebBrowser();
    foreach (string url in urls)
    {
        browser.Navigate(url);
        while (browser.ReadyState != WebBrowserReadyState.Complete) Application.DoEvents();
        StreamReader sr = new StreamReader(browser.DocumentStream);
        StreamWriter sw = new StreamWriter(), url.Substring(url.LastIndexOf('/')));
        sw.Write(sr.ReadToEnd());
        sr.Close();
        sw.Close();
    }

预计时间约为 12 天...有更快的方法吗？

编辑：顺便说一句，本地文件处理只需要 7% 的时间

编辑：这是我的最终实现：

    void Main(void)
    {
        ServicePointManager.DefaultConnectionLimit = 10000;
        List<string> urls = new List<string>();
        // ... initialize urls ...
        int retries = urls.AsParallel().WithDegreeOfParallelism(8).Sum(arg => downloadFile(arg));
    }

    public int downloadFile(string url)
    {
        int retries = 0;

        retry:
        try
        {
            HttpWebRequest webrequest = (HttpWebRequest)WebRequest.Create(url);
            webrequest.Timeout = 10000;
            webrequest.ReadWriteTimeout = 10000;
            webrequest.Proxy = null;
            webrequest.KeepAlive = false;
            webresponse = (HttpWebResponse)webrequest.GetResponse();

            using (Stream sr = webrequest.GetResponse().GetResponseStream())
            using (FileStream sw = File.Create(url.Substring(url.LastIndexOf('/'))))
            {
                sr.CopyTo(sw);
            }
        }

        catch (Exception ee)
        {
            if (ee.Message != "The remote server returned an error: (404) Not Found." && ee.Message != "The remote server returned an error: (403) Forbidden.")
            {
                if (ee.Message.StartsWith("The operation has timed out") || ee.Message == "Unable to connect to the remote server" || ee.Message.StartsWith("The request was aborted: ") || ee.Message.StartsWith("Unable to read data from the trans­port con­nec­tion: ") || ee.Message == "The remote server returned an error: (408) Request Timeout.") retries++;
                else MessageBox.Show(ee.Message, "Error", MessageBoxButtons.OK, MessageBoxIcon.Error);
                goto retry;
            }
        }

        return retries;
    }

Answer 1

在多个线程中下载文件。线程数取决于您的吞吐量。另外，请查看 WebClient 和 HttpWebRequest 类。简单示例：

var list = new[] 
{ 
    "http://google.com", 
    "http://yahoo.com", 
    "http://stackoverflow.com" 
}; 

var tasks = Parallel.ForEach(list,
        s =>
        {
            using (var client = new WebClient())
            {
                Console.WriteLine($"starting to download {s}");
                string result = client.DownloadString((string)s);
                Console.WriteLine($"finished downloading {s}");
            }
        });

Answer 2

同时执行下载而不是顺序执行，并设置一个合理的 MaxDegreeOfParallelism 否则您将尝试同时发出太多请求，这看起来像是 DOS 攻击：

    public static void Main(string[] args)
    {
        var urls = new List<string>();
        Parallel.ForEach(
            urls, 
            new ParallelOptions{MaxDegreeOfParallelism = 10},
            DownloadFile);
    }

    public static void DownloadFile(string url)
    {
        using(var sr = new StreamReader(HttpWebRequest.Create(url)                                               
           .GetResponse().GetResponseStream()))
        using(var sw = new StreamWriter(url.Substring(url.LastIndexOf('/'))))
        {
            sw.Write(sr.ReadToEnd());
        }
    }

Answer 3

我会并行使用多个线程，WebClient。我建议将最大并行度设置为所需的线程数，因为未指定的并行度不适用于长时间运行的任务。我在我的一个项目中使用了 50 次并行下载没有问题，但根据个人下载的速度，低得多的下载可能就足够了。

如果您从同一台服务器并行下载多个文件，默认情况下您只能进行少量（2 或 4 个）并行下载。虽然 http 标准规定了如此低的限制，但许多服务器并未强制执行。使用ServicePointManager.DefaultConnectionLimit = 10000; 增加限制。