如何通过使用 BS4 和 Requests 进行谷歌搜索来获取完整的 URL答案

【问题标题】：How to get the complete URL by making a google search with BS4 and Requests如何通过使用 BS4 和 Requests 进行谷歌搜索来获取完整的 URL
【发布时间】：2020-12-28 05:58:44
【问题描述】：

所以，我正在制作一个程序，它将搜索 google 并获取给定关键字的所有结果。我想获取所有 URL 并将它们打印到屏幕上，我决定为此使用 BS4，我就是这样做的：

r = requests.get(f'https://www.google.com/search?q={dork}&start={page}',headers={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:54.0) Gecko/20100101 Firefox/54.0'})
soup = BeautifulSoup(r.text, "html.parser")
urls = soup.find_all('div', attrs={'class': 'BNeawe UPmit AP7Wnd'})
for url in urls:
url = url.split('<div class="BNeawe UPmit AP7Wnd">')[1].split('</div>')[0]
url = url.replace(' › ','/')
print(f'{Fore.GREEN}{url}{Fore.WHITE}')
open(f'results/{timeLol}/urls.txt', "a")

但是，它并没有返回完整的 URL，如果 URL 很大，它会在一些 URL 之后返回 ...，即使它没有使用 BS4 和请求，有没有办法获得完整的 URL .

【问题讨论】：

标签： url beautifulsoup python-requests

【解决方案1】：

任何搜索查询示例都将不胜感激。

虽然您不提供查询示例，但您可以尝试使用bs4 css selectors (css selectors reference)：

for result in soup.select('.tF2Cxc'):
  link = result.select_one('.yuRUbf a')['href']

# https://spicysouthernkitchen.com/best-way-to-cook-corn-on-the-cob/
# other URLs below...

抓取更多的代码和example in the online IDE：

import requests, lxml
from bs4 import BeautifulSoup

headers = {
    "User-Agent":
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3538.102 Safari/537.36 Edge/18.19582"
}

params = {'q': 'how to cook best corn on the cob'}

html = requests.get('https://www.google.com/search', headers=headers, params=params)
soup = BeautifulSoup(html.text, 'lxml')

# container with all needed data
for result in soup.select('.tF2Cxc'):
  link = result.select_one('.yuRUbf a')['href']
  print(link)

---------
'''
https://spicysouthernkitchen.com/best-way-to-cook-corn-on-the-cob/
https://www.allrecipes.com/recipe/222352/jamies-sweet-and-easy-corn-on-the-cob/
https://www.delish.com/cooking/a22487458/corn-on-the-cob/
https://www.thekitchn.com/best-method-cook-corn-skills-showdown-23045869
https://natashaskitchen.com/15-minute-corn-on-the-cob/
https://www.thegunnysack.com/how-long-to-boil-corn-on-the-cob/
https://www.epicurious.com/recipes/food/views/basic-method-for-cooking-corn-on-the-cob-40047
https://houseofnasheats.com/the-best-boiled-corn-on-the-cob/
https://www.tasteofhome.com/article/perfect-corn-on-the-cob/
'''

或者，您可以使用来自 SerpApi 的 Google 搜索结果 API 执行相同的操作，但无需考虑如何解析内容，因为它已经为最终用户完成。所需要做的只是迭代结构化的 JSON 字符串。

这是一个带有免费计划的付费 API。

要集成的代码：

from serpapi import GoogleSearch
import os

params = {
  "api_key": os.getenv("API_KEY"),
  "engine": "google",
  "q": "how to cook best corn on the cob",
  "hl": "en",
}

search = GoogleSearch(params)
results = search.get_dict()

for result in results['organic_results']:
    link = result['link']
    print(link)

----------
'''
https://spicysouthernkitchen.com/best-way-to-cook-corn-on-the-cob/
https://www.allrecipes.com/recipe/222352/jamies-sweet-and-easy-corn-on-the-cob/
https://www.delish.com/cooking/a22487458/corn-on-the-cob/
https://www.thekitchn.com/best-method-cook-corn-skills-showdown-23045869
https://natashaskitchen.com/15-minute-corn-on-the-cob/
https://www.thegunnysack.com/how-long-to-boil-corn-on-the-cob/
https://www.epicurious.com/recipes/food/views/basic-method-for-cooking-corn-on-the-cob-40047
https://houseofnasheats.com/the-best-boiled-corn-on-the-cob/
https://www.tasteofhome.com/article/perfect-corn-on-the-cob/
'''

免责声明，我为 SerpApi 工作。

【讨论】：