如何使用 BeautifulSoup 从页面中抓取

Question

问的问题很简单，但对我来说，它不起作用，我不知道！

我想用 BeautifulSoup 从这个页面 https://www.brewersfriend.com/homebrew/recipe/view/16367/southern-tier-pumking-clone 刮掉评分啤酒，但它不起作用。

这是我的代码：

import requests
import bs4
from bs4 import BeautifulSoup



url = 'https://www.brewersfriend.com/homebrew/recipe/view/16367/southern-tier-pumking-clone'

test_html = requests.get(url).text

soup = BeautifulSoup(test_html, "lxml")

rating = soup.findAll("span", class_="ratingValue")

rating

当我完成时，它不起作用，但如果我对另一个页面做同样的事情......我不知道。有人可以帮助我吗？评分结果为 4.58

谢谢大家！

Answer 1

如果您打印 test_html，您会发现收到 403 禁止响应。

您应该在您的 GET 请求中添加一个标头（至少是一个用户代理：））。

import requests
from bs4 import BeautifulSoup


headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36'
}

url = 'https://www.brewersfriend.com/homebrew/recipe/view/16367/southern-tier-pumking-clone'

test_html = requests.get(url, headers=headers).text

soup = BeautifulSoup(test_html, 'html5lib')

rating = soup.find('span', {'itemprop': 'ratingValue'})

print(rating.text)

# 4.58

Answer 2

获得禁止状态代码（HTTP 错误 403）背后的原因，这意味着尽管了解响应，但服务器不会满足您的请求。如果您尝试抓取许多具有安全功能以防止僵尸程序的更受欢迎的网站，您肯定会收到此错误。所以你需要伪装你的请求！

1. 为此，您需要使用 Headers。
2. 您还需要更正您要获取其数据的标签属性，即itemprop

3. 使用 lxml 作为您的树生成器，或任何其他选择

   import requests
   from bs4 import BeautifulSoup
   
   
   url = 'https://www.brewersfriend.com/homebrew/recipe/view/16367/southern-tier-pumking-clone'
   
   # Add this 
   headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36'}
   
   test_html = requests.get(url, headers=headers).text      
   
   soup = BeautifulSoup(test_html, 'lxml')
   
   rating = soup.find('span', {'itemprop':'ratingValue'})
   
   print(rating.text)
   

Answer 3

您请求响应的页面为 403 禁止，因此您可能不会收到错误，但它会为您提供 [] 形式的空白结果。为避免这种情况，我们添加了用户代理，此代码将为您提供所需的结果。

import urllib.request
from bs4 import BeautifulSoup

user_agent = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.7) Gecko/2009021910 Firefox/3.0.7'

url = "https://www.brewersfriend.com/homebrew/recipe/view/16367/southern-tier-pumking-clone"
headers={'User-Agent':user_agent} 

request=urllib.request.Request(url,None,headers) #The assembled request
response = urllib.request.urlopen(request)
soup = BeautifulSoup(response, "lxml")

rating = soup.find('span', {'itemprop':'ratingValue'})

rating.text

Answer 4

    import requests
    from bs4 import BeautifulSoup


    headers = {
   'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) 
 AppleWebKit/537.36 
   (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36'
   }

 url = 'https://www.brewersfriend.com/homebrew/recipe/view/16367/southerntier-pumking 
clone'

test_html = requests.get(url, headers=headers).text

soup = BeautifulSoup(test_html, 'html5lib')

rating = soup.find('span', {'itemprop': 'ratingValue'})

 print(rating.text)

Answer 5

你正面临这个错误，因为一些网站不能被美丽的汤刮掉。所以对于这类网站，你必须使用 selenium

• 根据你的操作系统从link下载最新的chrome驱动
• 通过这个命令“pip install selenium”安装selenium驱动

# import required modules 
import selenium
from selenium import webdriver
from bs4 import BeautifulSoup
import time, os

curren_dir  = os.getcwd()
print(curren_dir)

# concatinate web driver with your current dir && if you are using window change "/" to '\' .

# make sure , you placed chromedriver in current directory 
driver = webdriver.Chrome(curren_dir+'/chromedriver')
# driver.get open url on your browser 
driver.get('https://www.brewersfriend.com/homebrew/recipe/view/16367/southern-tier-pumking-clone')
time.sleep(1)

# it fetch data html data from driver
super_html = driver.page_source

# now convert raw data with 'html.parser'

soup=BeautifulSoup(super_html,"html.parser")
rating = soup.findAll("span",itemprop="ratingValue")
rating[0].text