用 BeautifulSoup 提取 <script> 的内容

Question

1/ 我正在尝试使用漂亮的汤提取脚本的一部分，但它什么也没打印。怎么了？

URL = "http://www.reuters.com/video/2014/08/30/woman-who-drank-restaurants-tainted-tea?videoId=341712453"
oururl= urllib2.urlopen(URL).read()
soup = BeautifulSoup(oururl)

for script in soup("script"):
        script.extract()

list_of_scripts = soup.findAll("script")
print list_of_scripts

2/ 目标是提取属性“transcript”的值：

<script type="application/ld+json">
{
    "@context": "http://schema.org",
    "@type": "VideoObject",
    "video": {
        "@type": "VideoObject",
        "headline": "Woman who drank restaurant&#039;s tainted tea hopes for industry...",
        "caption": "Woman who drank restaurant&#039;s tainted tea hopes for industry...",  
        "transcript": "Jan Harding is speaking out for the first time about the ordeal that changed her life.               SOUNDBITE: JAN HARDING, DRANK TAINTED TEA, SAYING:               \"Immediately my whole mouth was on fire.\"               The Utah woman was critically burned in her mouth and esophagus after taking a sip of sweet tea laced with a toxic cleaning solution at Dickey's BBQ.               SOUNDBITE: JAN HARDING, DRANK TAINTED TEA, SAYING:               \"It was like a fire beyond anything you can imagine. I mean, it was not like drinking hot coffee.\"               Authorities say an employee mistakenly mixed the industrial cleaning solution containing lye into the tea thinking it was sugar.               The Hardings hope the incident will bring changes in the restaurant industry to avoid such dangerous mixups.               SOUNDBITE: JIM HARDING, HUSBAND, SAYING:               \"Bottom line, so no one ever has to go through this again.\"               The district attorney's office is expected to decide in the coming week whether criminal charges will be filed.",

Answer 1

来自documentation：

从 Beautiful Soup 版本 4.9.0 开始，当使用 lxml 或 html.parser 时，<script>、<style> 和 <template> 标签的内容不被视为“文本”，因为这些标签不是页面的人类可见内容的一部分。

所以基本上上面 falsetru 接受的答案都很好，但是在较新版本的 Beautiful Soup 中使用 .string 而不是 .text，否则你会像我一样对 @ 感到困惑987654327@ 总是为<script> 标签返回None。

Answer 2

extract 从 dom 中删除标签。这就是你得到空列表的原因。

---

使用type="application/ld+json" 属性查找script 并使用json.loads 对其进行解码。然后，您可以像 Python 数据结构一样访问数据。 （dict 用于给定数据）

import json
import urllib2

from bs4 import BeautifulSoup

URL = ("http://www.reuters.com/video/2014/08/30/"
       "woman-who-drank-restaurants-tainted-tea?videoId=341712453")
oururl= urllib2.urlopen(URL).read()
soup = BeautifulSoup(oururl)

data = json.loads(soup.find('script', type='application/ld+json').text)
print data['video']['transcript']

Answer 3

感谢您的启发。我已经尝试了几个小时如何做到这一点。但是让我告诉你，由于 Python3 不再与 urllib2 一起使用，我们必须使用 requests 库而不是 urllib2。我只是把更新的版本放在这里。享受;)

import json
import requests
from bs4 import BeautifulSoup

url = input('Enter url:')
html = requests.get(url)
soup = BeautifulSoup(html.text,'html.parser')

data = json.loads(soup.find('script', type='application/ld+json').text)
print(data['articleBody'])