使用 beautifulsoup 在 pandas 数据框中抓取问题/错误答案

【问题标题】：Problem/Error with scraping in a pandas data frame with beautifulsoup使用 beautifulsoup 在 pandas 数据框中抓取问题/错误
【发布时间】：2019-06-13 16:29:52
【问题描述】：

我正在处理这个 csv (https://www.kaggle.com/jtrofe/beer-recipes)，我想抓取数据框中的每个 URL，但我不能，因为我有问题/错误，我无法抓取所有 URL，如果我尝试使用 1 个 URL，没关系，但功能有问题...有人可以帮助我吗？

这是我的代码：

import requests
from bs4 import BeautifulSoup
from time import sleep 


headers = {
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36'
}

base = 'https://www.brewersfriend.com'
links = [f'{base}{r}' for r in df['URL']]

while True:
    try:
        r = requests.get(links, headers=headers, stream=False, timeout=8).text
        break
    except:
        if r.status_code == 404:
            print("Client error")
            r.raise_for_status()
        sleep(1)


soup = BeautifulSoup(r, 'html5lib')

rating = soup.find('span', {'itemprop': 'ratingValue'})

DEFAULT_VALUE = 'NaN'

if rating is None:
    rating = DEFAULT_VALUE
    
print(rating.text)

我已经知道在某些页面中没有评分，因此我使用 Not a Number 创建了 DEFAULT_VALURE，但也可能是一个错误。

这段代码前面有数据框，我就不放了。

希望有人能帮帮我！

非常感谢

【问题讨论】：

标签： python pandas dataframe beautifulsoup screen-scraping

【解决方案1】：

这里各种乱七八糟的东西。我不会全部介绍，但我看到的一件事是您正在尝试print (rating.text)。如果你的评分是'NaN'，一个错误是你不能做rating.text

这不是我要写的方式，而是脱离你的初始编码：

import pandas as pd
import requests
from bs4 import BeautifulSoup
from time import sleep 


df = pd.read_csv('C:/recipeData/recipeData.csv', encoding = 'ISO-8859-1')
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.108 Safari/537.36'}
base = 'https://www.brewersfriend.com'

links = [f'{base}{r}' for r in df['URL']]
for link in links:
    try:
        r = requests.get(link, headers=headers, stream=False, timeout=8)

        if r.status_code == 404:
            print("Client error")
            r.raise_for_status()
            continue
        else:
            r = r.text     
    except:
        continue


    soup = BeautifulSoup(r, 'html5lib')
    rating = soup.find('span', {'itemprop': 'ratingValue'}).text
    DEFAULT_VALUE = 'NaN'

    if rating is None:
        rating = DEFAULT_VALUE

    print('%s: %s' %(link,rating))

【讨论】：

您的代码也存在一些其他问题。今天晚些时候我会分解它，只是昨天没有太多时间来深入了解它。

【解决方案2】：

这是一种完成整个过程的方法

import requests, re
import pandas as pd
from bs4 import BeautifulSoup as bs

p = re.compile(r'dataviewToken":"(.*?)"')
p1 = re.compile(r'"rowCount":(\d+)')
results = []
i = 0

with requests.Session() as s:
    r = s.get('https://www.kaggle.com/jtrofe/beer-recipes')   
    token = p.findall(r.text)[0]
    rows = int(p1.findall(r.text)[0])
    data = {"jwe":{"encryptedToken": token},"source":{"type":3,"dataset":{"url":"jtrofe/beer-recipes","tableType":1,"csv":{"fileName":"recipeData.csv","delimiter":",","headerRows":1}}},"select":["BeerID","Name","URL","Style","StyleID","Size(L)","OG","FG","ABV","IBU","Color","BoilSize","BoilTime","BoilGravity","Efficiency","MashThickness","SugarScale","BrewMethod","PitchRate","PrimaryTemp"],"skip":0,"take": rows}
    base = 'https://www.brewersfriend.com'
    r = s.post('https://www.kaggleusercontent.com/services/datasets/kaggle.dataview.v1.DataViewer/GetDataView', json = data).json()
    names, links = zip(*[(row['text'][1], base + row['text'][2]) for row in r['dataView']['rows']])

    for link in links:
        r = s.get(link, headers = {'User-Agent' : 'Mozilla/5.0'})
        if r.status_code == 403:
            rating = 'N/A'
        else:
            soup = bs(r.content, 'lxml')
            rating = soup.select_one('[itemprop=ratingValue]')
            if rating is None:
                rating = 'N/A'
            else:
                rating = rating.text
        row = [names[i], rating]
        results.append(row)
        i+=1

df = pd.DataFrame(results, columns = ['Name', 'Rating'])
print(df.head())
df.to_csv(r'C:\Users\User\Desktop\Data.csv', sep=',', encoding='utf-8-sig',index = False )

【讨论】：