使用 Python 和 Beautiful Soup 进行 HTML 抓取（来自 IMDb）答案

【问题标题】：HTML scraping using Python and Beautiful Soup (from IMDb)使用 Python 和 Beautiful Soup 进行 HTML 抓取（来自 IMDb）
【发布时间】：2023-03-20 10:40:02
【问题描述】：

例如，我想从this page 获取电影评分并逐行打印评分，我用 BS4 提取了名称和发布年份，但不知道如何处理评级...

import requests
from bs4 import BeautifulSoup
import urllib.request


url = urllib.request.urlopen('http://imdb.com/list/ls097228983/')
content = url.read()
soup = BeautifulSoup(content, 'lxml')



for div in soup.findAll('h3', attrs={'class':'lister-item-header'}):
    #print(div.find('a')['href'])
    #print("**")
    #print(div)
    year = div.find('span', attrs={'class':'lister-item-year text-muted unbold'})
    year = str(year)
    year = year.replace('<span class="lister-item-year text-muted unbold">', '')
    year = year.replace('</span>', '')
    name = div.find('a').contents[0]
    print(name + ' ' + year)

    >> I want: Solaris (1972) 8.1

【问题讨论】：

切向，当然，您是否考虑过 programmatically accessing the data 并消除这些与抓取相关的头痛......？
是的，但我需要这种方式。

标签： python web-scraping beautifulsoup scrape imdb

【解决方案1】：

您需要将 'class':'lister-item-header' 更改为 'class':'lister-item-content' 父类才能获得评级。

import requests
from bs4 import BeautifulSoup
import urllib.request


url = urllib.request.urlopen('http://imdb.com/list/ls097228983/')
content = url.read()
soup = BeautifulSoup(content, 'lxml')

for div in soup.findAll('div', {'class':'lister-item-content'}):
    #print(div.find('a')['href'])
    #print("**")
    #print(div)
    year = div.find('span', attrs={'class':'lister-item-year text-muted unbold'})
    year = str(year)
    year = year.replace('<span class="lister-item-year text-muted unbold">', '')
    year = year.replace('</span>', '')
    name = div.find('a').contents[0]
    rating = div.find('span',class_='ipl-rating-star__rating').text
    # print(rating)
    # you could also format string.
    print(f'{name} {year} {rating}'.format(name, year, rating))
    print(name + ' ' + year + " " +rating)

【讨论】：