如何使用 BeautifulSoup 查找所有下一个链接答案

【问题标题】：How to use BeautifulSoup to find all the next links如何使用 BeautifulSoup 查找所有下一个链接
【发布时间】：2017-08-21 21:35:55
【问题描述】：

我目前正在通过预设一个名为 number_of_pages 的变量来抓取特定网站的所有页面。在添加我不知道的新页面之前，预设此变量有效。例如下面的代码是 3 页，但网站现在有 4 页。

base_url = 'https://securityadvisories.paloaltonetworks.com/Home/Index/?page='
number_of_pages = 3
for i in range(1, number_of_pages, 1):
   url_to_scrape = (base_url + str(i))

我想使用 BeautifulSoup 来查找网站上所有下一个要抓取的链接。下面的代码找到第二个 URL，但不是第三个或第四个。如何在抓取之前构建所有页面的列表？

base_url = 'https://securityadvisories.paloaltonetworks.com/Home/Index/?page='
CrawlRequest = requests.get(base_url)
raw_html = CrawlRequest.text
linkSoupParser = BeautifulSoup(raw_html, 'html.parser')
page = linkSoupParser.find('div', {'class': 'pagination'})
for list_of_links in page.find('a', href=True, text='next'):
  nextURL = 'https://securityadvisories.paloaltonetworks.com' + list_of_links.parent['href']
print (nextURL)

【问题讨论】：

标签： python python-3.x web-scraping beautifulsoup

【解决方案1】：

有几种不同的方法来处理分页。这是其中之一。

这个想法是初始化一个无限循环并在没有“下一个”链接时打破它：

from urllib.parse import urljoin

from bs4 import BeautifulSoup
import requests


with requests.Session() as session:
    page_number = 1
    url = 'https://securityadvisories.paloaltonetworks.com/Home/Index/?page='
    while True:
        print("Processing page: #{page_number}; url: {url}".format(page_number=page_number, url=url))
        response = session.get(url)
        soup = BeautifulSoup(response.content, 'html.parser')

        # check if there is next page, break if not
        next_link = soup.find("a", text="next")
        if next_link is None:
            break

        url = urljoin(url, next_link["href"])
        page_number += 1

print("Done.")

如果你执行它，你会看到打印以下信息：

Processing page: #1; url: https://securityadvisories.paloaltonetworks.com/Home/Index/?page=
Processing page: #2; url: https://securityadvisories.paloaltonetworks.com/Home/Index/?page=2
Processing page: #3; url: https://securityadvisories.paloaltonetworks.com/Home/Index/?page=3
Processing page: #4; url: https://securityadvisories.paloaltonetworks.com/Home/Index/?page=4
Done.

请注意，为了提高性能并在请求中保留 cookie，我们正在与 requests.Session 保持网络抓取会话。

【讨论】：