使用 Beautiful Soup 从 Google 搜索中排除一些 URL

Question

我的任务是从 Google 搜索中获取公司的 URL。示例：如果公司是“facoboom Sachs”，我应该返回网址“facebook.com”。我正在以一种在搜索中找到所有链接并仅使用第一个 url 的方式来处理它（请参见下面的代码）：

Answer 1

确保您使用user-agent 来伪造真实的用户访问（请参阅代码中的headers）否则，Google 最终会阻止您的请求。

代码：

import requests, lxml
from bs4 import BeautifulSoup

headers = {
    "User-Agent":
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3538.102 Safari/537.36 Edge/18.19582"
}

params = {'q': 'facebook Sachs'}

html = requests.get(f'https://www.google.com/search',
                    headers=headers,
                    params=params).text
soup = BeautifulSoup(html, 'lxml')

# Container where needed data located
for container in soup.select('.tF2Cxc'):
   # Grabbing link itself
   link = container.a['href']
   print(link)

输出：

https://www.facebook.com/SACHSofficial/
https://www.facebook.com/SachsPerformanceOfficial/
https://www.facebook.com/jeffrey.sachs
https://www.facebook.com/SachsArtsPhilly/
https://www.facebook.com/public/John-Sachs
https://m.facebook.com/SACHSofficial/photos/?ref=page_internal&mt_nav=0
https://www.facebook.com/SachsMedia/
https://www.facebook.com/goldmansachs/
https://www.facebook.com/teamsachs
https://www.facebook.com/SachsAssociates/

或者，您可以使用来自 SerpApi 的 Google Search Engine Results API 来完成此操作。这是一个带有免费计划的付费 API。

要集成的代码：

from serpapi import GoogleSearch

params = {
  "api_key": "YOUR_API_KEY",
  "engine": "google",
  "q": "facebook Sachs",
  "google_domain": "google.com",

}

search = GoogleSearch(params)
results = search.get_dict()

for result in results['organic_results']:
  link = result['link']
  print(link)

输出：

https://www.facebook.com/SACHSofficial/
https://www.facebook.com/SachsPerformanceOfficial/
https://www.facebook.com/jeffrey.sachs
https://www.facebook.com/public/John-Sachs
https://m.facebook.com/SACHSofficial/photos/?ref=page_internal&mt_nav=0
https://www.facebook.com/teamsachs
https://www.facebook.com/SachsMedia/
https://www.facebook.com/goldmansachs/
https://www.facebook.com/SachsArtsPhilly/
https://www.facebook.com/SachsAssociates/

免责声明，我为 SerpApi 工作。

Answer 2

您必须通过在 init 中使用 not 来在您不想获取的 url 列表中搜索 url.contents[0] 以获得提及列表以外的结果。

代码。

for i in range(1):
    # retrieve urls
    soup = BeautifulSoup(driver.page_source, 'html.parser')
    for url in soup.find_all(class_='iUh30'):
        if url.contents[0] not in ['www.instahyre.com', 'in.linkedin.com', 'en.wikipedia.org']:
            one_urls.append(url.contents[0])
            break;

Answer 3

我过去做过类似的事情，我得到了这个：

import requests
import googlesearch

def gsearch_scraper(query, location, country, stop):

  """ This function automates Google searches and
  returns a list of URL's from each result."""

  # Add retries:
  for retry in range(5):

    try:

      # Try making a request:
      stop = int(stop)
      num = 10
      pause = 10.0
      fquery = query + ' ' + location
      print('Attemping connection to the web server...\n')

      # Make requests using the free API:
      response = googlesearch.search(
          query = fquery,
          country = country,
          stop = stop,
          num = num,
          pause = pause,
          )
      
      # Store Query Results:
      print('Storing URL Addresses from Results...\n')
      results = []

      for r in response:
        results.append(r)
        print('{} out of {} results have been fetched. Please wait...'\
              .format(
                  len(results),
                  stop)
              )
      
      print('\nSuccess! End of Script.')
      return results

    except:
      print("Failed to Process Page...\n\nEnd of Script.")
      return []

它的作用是自动化谷歌搜索。我使用 for 循环传递公司名称列表。您可以为每个公司名称指定每个查询需要多少个结果。作为回报，它将为您提供它为该公司收集的所有 URL 的列表。您还可以添加'New York City, NY' 之类的位置以获得更好的结果以及指定国家/地区。

例子：

gsearch_scraper('goldman sachs', 'US', 'US', 5)

Output:

Attemping connection to the web server...

Storing URL Addresses from Results...

1 out of 5 results have been fetched. Please wait...
2 out of 5 results have been fetched. Please wait...
3 out of 5 results have been fetched. Please wait...
4 out of 5 results have been fetched. Please wait...
5 out of 5 results have been fetched. Please wait...

Success! End of Script.
['https://www.goldmansachs.com/',
 'https://en.wikipedia.org/wiki/Goldman_Sachs',
 'https://en.wikipedia.org/wiki/Goldman_Sachs#History',
 'https://en.wikipedia.org/wiki/Goldman_Sachs#Current_operations',
 'https://en.wikipedia.org/wiki/Goldman_Sachs#Philanthropy']

然后，您可以对结果应用一些正则表达式来提取您需要的网址或排除您不需要的网址。