如何在 Python 中结合使用 Request 和 BeautifulSoup 来加速 Webscraping？

Question

目标是使用BeautifulSoup 抓取多个pages，其输入来自requests.get 模块。

步骤如下：

首先使用requests加载html

page = requests.get('https://oatd.org/oatd/' + url_to_pass)

然后，使用以下定义抓取html 内容：

def get_each_page(page_soup):
    return dict(paper_author=page_soup.find(attrs={"itemprop": "name"}).text,
                paper_title=page_soup.find(attrs={"itemprop": "name"}).text)

比如说，我们有一百个唯一的url要抓取 ['record?record=handle\:11012\%2F16478&q=eeg'] * 100，整个过程可以通过下面的代码完成：

import requests
from bs4 import BeautifulSoup as Soup

def get_each_page(page_soup):
    return dict(paper_author=page_soup.find(attrs={"itemprop": "name"}).text,
                paper_title=page_soup.find(attrs={"itemprop": "name"}).text)

list_of_url = ['record?record=handle\:11012\%2F16478&q=eeg'] * 100 # In practice, there will be 100 diffrent unique sub-href. But for illustration purpose, we purposely duplicate the url
all_website_scrape = []
for url_to_pass in list_of_url:

    page = requests.get('https://oatd.org/oatd/' + url_to_pass)
    if page.status_code == 200:
        all_website_scrape.append(get_each_page(Soup(page.text, 'html.parser')))

但是，每个 url 都被请求并一次抓取一个，因此原则上很耗时。

我不知道是否有其他方法可以提高我不知道的上述代码的性能？

Answer 1

您也许可以使用线程模块。 您可以使脚本多线程并运行得更快。 https://docs.python.org/3/library/threading.html

但如果你愿意改变主意，我推荐scrapy

Answer 2

realpython.com 有一篇很好的文章，介绍了如何通过并发加速 Python 脚本。

https://realpython.com/python-concurrency/

使用他们的线程示例，您可以设置执行多个线程的工作线程数，从而增加您一次可以发出的请求数。

    from bs4 import BeautifulSoup as Soup
    import concurrent.futures
    import requests
    import threading
    import time
    
    def get_each_page(page_soup):
        return dict(paper_author=page_soup.find(attrs={"itemprop": "name"}).text,
                    paper_title=page_soup.find(attrs={"itemprop": "name"}).text)
    
    def get_session():
        if not hasattr(thread_local, "session"):
            thread_local.session = requests.Session()
        return thread_local.session
    
    def download_site(url_to_pass):
        session = get_session()
        page = session.get('https://oatd.org/oatd/' + url_to_pass, timeout=10)
        print(f"{page.status_code}: {page.reason}")
        if page.status_code == 200:
            all_website_scrape.append(get_each_page(Soup(page.text, 'html.parser')))
    
    def download_all_sites(sites):
        with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
            executor.map(download_site, sites)
    
    if __name__ == "__main__":
        list_of_url = ['record?record=handle\:11012\%2F16478&q=eeg'] * 100  # In practice, there will be 100 diffrent unique sub-href. But for illustration purpose, we purposely duplicate the url
        all_website_scrape = []
        thread_local = threading.local()
        start_time = time.time()
        download_all_sites(list_of_url)
        duration = time.time() - start_time
        print(f"Downloaded {len(all_website_scrape)} in {duration} seconds")