使用python列出网页中的链接

Question

我正在尝试编写一个 python 脚本来列出网页中包含某些子字符串的所有链接。我遇到的问题是该网页有多个“页面”，因此它不会弄乱整个屏幕。以https://www.go-hero.net/jam/17/solutions/1/1/C++ 为例。

这是我目前所拥有的：

import requests
from bs4 import BeautifulSoup
url = "https://www.go-hero.net/jam/17/solutions/1/1/C++"
response = requests.get(url)
soup = BeautifulSoup(response.content, "html5lib")
links = soup.find_all('a')

for tag in links:
  link = tag.get('href', None)
  if link is not None and 'GetSource' in link:
    print(link)

关于如何让它发挥作用的任何建议？提前致谢。

Answer 1

编辑/更新： 使用Selenium，您可以在抓取 html 之前单击页面链接以将所有内容收集到 html 中。当您单击页面时，许多/大多数带有分页的网站不会收集 html 中的所有文本，但我注意到您提供的示例确实如此。看看at this SO question 的一个快速示例，让 Selenium 与 BeautifulSoup 一起工作。以下是在代码中使用它的方法：

import requests
from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Firefox()
original_url = "https://www.go-hero.net/jam/17/solutions/1/1/C++"
driver.get(original_url)

# click the links for pages 1-29
for i in range(1, 30):
    path_string = '/jam/17/solutions/1/1/C++#page-' + str(i)
    driver.find_element_by_xpath('//a[@href=' + path_string + ']').click()

# scrape from the accumulated html
html = driver.page_source
soup = BeautifulSoup(html)
links = soup.find_all('a')

# proceed as normal from here
for tag in links:
    link = tag.get('href', None)
    if link is not None and 'GetSource' in link:
        print(link)

原始答案：对于您上面提供的link，您可以简单地遍历可能的网址并在循环中运行您的抓取代码：

import requests
from bs4 import BeautifulSoup
original_url = "https://www.go-hero.net/jam/17/solutions/1/1/C++"

# scrape from the original page (has no page number)
response = requests.get(original_url)
soup = BeautifulSoup(response.content, "html5lib")
links = soup.find_all('a')

# prepare to scrape from the pages numbered 1-29
# (note that the original page is not numbered, and the next page is "#page-1")
url_suffix = '#page-'

for i in range(1, 30):
    # add page number to the url
    paginated_url = original_url + url_suffix + str(i)
    response = requests.get(paginated_url)
    soup = BeautifulSoup(response.content, "html5lib")
    # append resulting list to 'links' list
    links += soup.find_all('a')

# proceed as normal from here
for tag in links:
    link = tag.get('href', None)
    if link is not None and 'GetSource' in link:
        print(link)

我不知道您是否介意结果中会出现重复。您将在 link 列表中得到重复的结果，就像代码当前一样，但您可以将链接添加到 Set 或其他东西以轻松解决此问题。