从交互式图表中抓取 JSON 数据 - 额外数据错误 `json.load()`

Question

我一直试图弄清楚如何使用这篇文章作为参考来抓取网络数据：How to scrape json data from an interactive chart?

我已从网站获得 JSON 图表数据，但在最后一步使用 json.load() 加载文件时遇到了困难。我不断收到错误消息：JSONDecodeError: Extra data: line 1 column 67193 (char 67192)

任何建议或帮助将不胜感激。

    import requests
    import json
    import codecs
    from pandas.io.json import json_normalize
    from bs4 import BeautifulSoup
    
    url = "https://docs.google.com/spreadsheets/d/e/2PACX-1vQaftrT4JJh6dqfFHaUuxiOE5CiFDTt2YIbiJe9sATHQrBILnfUVpSnDTKg26yTgmZKci4NVxtsByD7/pubchart?oid=1258306542&format=interactive"
    response = requests.get(url, verify=False)
    soup = BeautifulSoup(response.text, features="html.parser")
    scripts = soup.find_all('script')[1]
    
    for script in scripts:
        if 'chartJson' in script:
            encoded_string = script
            encoded_string = encoded_string.split("'chartJson': '",1)[-1]
            encoded_string = encoded_string.split("', 'fallbackUri'",1)[0]
            jsonStr = codecs.getdecoder('unicode-escape')(encoded_string)[0]
            jsonStr = jsonStr.split(',\\"parsedNumHeaders\\":1',1)[0]
            jsonStr = jsonStr.split('"rows":',1)[1]
            jsonObj = json.loads(jsonStr)

Answer 1

您可以将jsonStr 粘贴到here 以查找问题。这段代码对我有用。您在 "rows" 键上进行了拆分，但该级别中还有另一个 "cols" 键。然后它们在末尾也是一个额外的逗号。

jsonStr = codecs.getdecoder('unicode-escape')(encoded_string)[0] 也没有处理 unicode 转义，所以我添加了jsonStr = jsonStr.encode('utf8').decode('unicode_escape')

import requests
import json
import codecs
from pandas.io.json import json_normalize
from bs4 import BeautifulSoup

import unicodedata

url = "https://docs.google.com/spreadsheets/d/e/2PACX-1vQaftrT4JJh6dqfFHaUuxiOE5CiFDTt2YIbiJe9sATHQrBILnfUVpSnDTKg26yTgmZKci4NVxtsByD7/pubchart?oid=1258306542&format=interactive"
response = requests.get(url, verify=False)
soup = BeautifulSoup(response.text, features="html.parser")
scripts = soup.find_all('script')[1]

for script in scripts:
    if 'chartJson' in script:
        encoded_string = script
        encoded_string = encoded_string.split("'chartJson': '",1)[-1]
        encoded_string = encoded_string.split("', 'fallbackUri'",1)[0]
        jsonStr = codecs.getdecoder('unicode-escape')(encoded_string)[0]
        jsonStr = jsonStr.split(',\\"parsedNumHeaders\\":1',1)[0]
        jsonStr = jsonStr.split('"rows":',1)[1]
        jsonStr = jsonStr.split('"cols":',1)[0].rstrip(',')
        jsonStr = jsonStr.encode('utf8').decode('unicode_escape')
        jsonObj = json.loads(jsonStr)