使用 Beautiful Soup 访问 javascript 文本答案

【问题标题】：Access javascript text using Beautiful Soup使用 Beautiful Soup 访问 javascript 文本
【发布时间】：2020-04-06 05:54:26
【问题描述】：

我想在 IMDB 网站上保存一些奖项信息，但我无法访问所需的 javascript 文本。

import pandas as pd
import numpy as np
import requests
from bs4 import BeautifulSoup

urls = [
    'https://www.imdb.com/event/ev0000003/2000',
    'https://www.imdb.com/event/ev0000003/2001',
]

for url in urls:
    response = requests.get(url_test).content
    soup = BeautifulSoup(response, 'html.parser')
    soup.find_all('script', {'type':'text/javascript'})

现在，我怎样才能只访问类别信息：

"categories":[{"categoryName":"Best Actor in a Leading Role","nominations":[{"primaryNominees":[{"name":"Kevin Spacey","note":null,"imageUrl":.....

由于我必须为不同的奖项和年份这样做，我的想法是将它们保存在一个 json 文件中：

{"award": "oscars",  
 "year": "2000",  
 "data": [{"categoryName":"Best Actor in a Leading Role","nominations":[{"primaryNominees":[{"name":"Kevin Spacey","note":null,"imageUrl":.....  
}

【问题讨论】：

标签： python json web-scraping beautifulsoup

【解决方案1】：

数据存储在页面中的 javascript 中，因此您可以通过例如正则表达式访问它。要解析数据，您可以使用json 模块。

例如：

import re
import json
import requests

urls = [
    'https://www.imdb.com/event/ev0000003/2000',
    'https://www.imdb.com/event/ev0000003/2001',
]

for url in urls:
    response = requests.get(url).text

    data = json.loads( re.findall(r'IMDbReactWidgets\.NomineesWidget\.push.*?(\{.*\})', response)[0] )

    # print(json.dumps(data, indent=4)) # <-- comment this out to print all data

    for award in data['nomineesWidgetModel']['eventEditionSummary']['awards']:
        if award['awardName'] != 'Oscar':
            continue
        for category in award['categories']:
            print(category['categoryName'])

    print('-' * 80)

打印：

Best Actor in a Leading Role
Best Actor in a Supporting Role
Best Actress in a Leading Role
Best Actress in a Supporting Role
Best Art Direction-Set Decoration
Best Cinematography
Best Costume Design
Best Director
Best Documentary, Features
Best Documentary, Short Subjects
Best Effects, Sound Effects Editing
Best Effects, Visual Effects
Best Film Editing
Best Foreign Language Film

...and so on.

【讨论】：