使用 Beautiful Soup 计算请求页面上的链接

Question

这应该相当简单。我想计算通过网页搜索创建的链接。在此示例中，search 代表 Stack Overflow 上的“gwen stefani”。截至撰写本文时，结果数为 15。

import bs4 #  beautiful soup 4
import requests
import webbrowser

url = "https://stackoverflow.com/search?q=gwen+stefani"

myURL = url
webbrowser.open(myURL)

page = requests.get(url).text
r = requests.get(myURL)
html_content = r.text

soup = bs4.BeautifulSoup(html_content, "html.parser")

print soup.title

for link in soup.find_all("a"):
    print(link.get("href"))

当链接被打印出来时，它不包含任何提到的结果。我是汤的新手，现在我不确定我哪里出错了。

Answer 1

我使用的是 python 3.x，因此您可能需要对此进行调整，但我得到了所有 15 个链接。

from bs4 import BeautifulSoup
import requests

url = 'https://stackoverflow.com/search?q=gwen+stefani'
page = requests.get(url)
soup = BeautifulSoup(page.text, 'hmtl.parser')
for link in soup.findAll('div', class_='result-link'):
    print('https://stackoverflow.com'+link.a['href'])

Answer 2

您也可以尝试下面的代码，您不需要使用 div 元素的类。

只需检查页面并找到问题链接的类别。

import bs4 #  beautiful soup 4
import requests
import webbrowser
import json

url = "https://stackoverflow.com/search?q=gwen+stefani"

webbrowser.open(url)

r = requests.get(url)
html_content = r.text

# with open('response.html', 'w', encoding="utf-8") as f:
#   f.write(html_content)

soup = bs4.BeautifulSoup(html_content, "html.parser")

print(soup.title)
links = soup.find_all("a", class_='question-hyperlink')

valid_links = {}

for i, link in enumerate(links):
    href = link.get('href').strip()

    if href.startswith('/questions/'):
        valid_links[href] = link.text.strip()

print(json.dumps(valid_links, indent=4)) # pretty printing dictionary
print(len(valid_links)) # 15

输出

<title>Posts containing 'gwen stefani' - Stack Overflow</title>
{
    "/questions/39268369/what-does-minus-minus-do-in-excel": "Q: What does \u2014 (minus minus) do in Excel? [duplicate]",
    "/questions/53264513/using-beautiful-soup-to-count-links-on-requested-page": "Q: Using Beautiful Soup to count links on requested page",
    "/questions/31074289/is-there-a-script-that-can-transfer-text-from-an-excel-file-into-an-adobe-design/31100563#31100563": "A: Is there a script that can transfer text from an excel file into an Adobe design program?",
    "/questions/39268369/what-does-minus-minus-do-in-excel/39268800#39268800": "A: What does \u2014 (minus minus) do in Excel?",
    "/questions/1668447/launch-failed-binary-not-found-snow-leopard-and-eclipse-c-c-ide-issue/8463357#8463357": "A: \u201cLaunch Failed. Binary Not Found.\u201d Snow Leopard and Eclipse C/C++ IDE issue",
    "/questions/33023818/split-and-rejoin-path-without-trailing-backslash": "Q: Split and rejoin path without trailing backslash",
    "/questions/36986461/regex-match-return-remaining-rest-of-string": "Q: Regex match, return remaining rest of string",
    "/questions/44686123/pass-variable-from-javascript-to-windows-batch-file": "Q: Pass variable from JavaScript to Windows batch file",
    "/questions/44686123/pass-variable-from-javascript-to-windows-batch-file/44686309#44686309": "A: Pass variable from JavaScript to Windows batch file",
    "/questions/52465425/reversing-a-list-with-single-element-gives-none": "Q: Reversing a list with single element gives None [duplicate]",
    "/questions/22196612/array-length-outside-of-a-method": "Q: Array length outside of a method",
    "/questions/13300815/not-getting-expected-results-from-select-query/13300920#13300920": "A: Not getting expected results from SELECT query",
    "/questions/32884087/slicing-string-from-start": "Q: Slicing string from start [duplicate]",
    "/questions/53264513/using-beautiful-soup-to-count-links-on-requested-page/53265048#53265048": "A: Using Beautiful Soup to count links on requested page",
    "/questions/23337218/recursive-conditions-missing-base-case": "Q: Recursive conditions - missing base case"
}
15