从 BeautifulSoup 的 JSON 脚本输出返回 Pandas DataFrame答案

【问题标题】：Return Pandas DataFrame from BeautifulSoup's JSON script output从 BeautifulSoup 的 JSON 脚本输出返回 Pandas DataFrame
【发布时间】：2018-07-01 14:00:53
【问题描述】：

使用下面的代码，我想从 html 输出中返回一个 Python DataFrame。这是可以从 Python 中的包中完成的吗？表格格式见网页链接。

from bs4 import BeautifulSoup
import urllib.request
r = urllib.request.urlopen("https://www.zacks.com/zrank/sector-industry-classification.php").read()
soup = BeautifulSoup(r, "html.parser")

soup.find_all("script")[16]

输出脚本：

<script>window.app_data =
                {

                    columns : [

                    { "mDataProp"   : "Sector Group"
                    , "sTitle"      : "Sector Group"
                    , "sClass"      : "alpha"
                    , "bSortable"   : true 
                    }
                    ,
                    {
                      "mDataProp"   : "Sector Code"
                    , "sTitle"      : "Sector Code"
                    , "sClass"      : ""
                    , "bSortable"   : false 
                    }
                    ,
                    {
                      "mDataProp"   : "Medium(M) Industry Group"
                    , "sTitle"      : "Medium(M) Industry Group"
                    , "sClass"      : "alpha"
                    , "bSortable"   : false 
                    }

数据包含以下内容：

data"  : [  { "Sector Group"               :  "<span title=\"Index\" >Index</span>", "Sector Code"                :  "0", "Medium(M) Industry Group"   :  "<span title=\"Indices\" >Indices</span>", "Medium(M) Industry Code"    :  "0", "Expanded(X) Industry Group" :  "<span title=\"Indicies\" >Indicies</span>", "Expanded(X) Industry Code"  :  "400" } ,  { "Sector Group"               :  "<span title=\"Consumer Staples\" >Consumer Staple...</span>", "Sector Code"                :  "1", "Medium(M) Industry Group"   :  "<span title=\"Food\" >Food</span>", "Medium(M) Industry Code"    :  "3", "Expanded(X) Industry Group" :  "<span title=\"Food - Meat Products\" >Food - Meat Pro...</span>", "Expanded(X) Industry Code"  :  "75" } ,  { "Sector Group"               :  "<span title=\"Consumer Staples\" >Consumer Staple...</span>", "Sector Code"                :  "1", "Medium(M) Industry Group"   :  "<span title=\"Cons Prod-misc Staples\" >Cons Prod-misc...</span>", "Medium(M) Industry Code"    :  "7", "Expanded(X) Industry Group" :  "<span title=\"Funeral Services\" >Funeral Service...</span>", "Expanded(X) Industry Code"  :  "78" } ,  { "Sector Group"               :  "<span title=\"Consumer Staples\" >Consumer Staple...</span>", "Sector Code"                :  "1", "Medium(M) Industry Group"   :  "<span title=\"Food\" >Food</span>", "Medium(M) Industry Code"    :  "3", "Expanded(X) Industry Group" :  "<span title=\"Food - Confectionery\" >Food - Confecti...</span>", "Expanded(X) Industry Code"  :  "72" } ,  { "Sector Group"

注意：此处粘贴的数据过多。我也尝试了以下方法，因为其他答案提出了类似的方法，除了我选择了所有使用：

import re
pattern = re.compile("'.*': '.*'")
fields = dict(re.findall(pattern, soup))
print(fields)

输出为{}

【问题讨论】：

标签： python pandas beautifulsoup

【解决方案1】：

熊猫已经准备好了

pd.read_html('https://www.zacks.com/zrank/sector-industry-classification.php')

【讨论】：

这很有帮助，但不幸的是它没有返回带有行业代码的脚本表。请参阅soup.find_all("script")[16] 数据。

【解决方案2】：

我相信有更好的方法来实现这一点。但是，嘿，它给了你你想要的。此外，最好将 Selenium+PhantomJS 用于此类任务。

from bs4 import BeautifulSoup
import requests
import json
import pandas as pd

request = requests.get('https://www.zacks.com/zrank/sector-industry-classification.php')
soup = BeautifulSoup(request.text, 'lxml')

#Tweaked the string for parsing. It's ugly solution. I have failed with regular expressions.
#You can achieve this with way better way.
data = soup.find_all("script")[16].text.split('data"')[1].strip()[3:].rstrip()[:-7]

json_data = json.loads('[' + data)

def get_title(key):
    return BeautifulSoup(data[key],'lxml').find('span').attrs['title']

d = []

for data in json_data:
    sector_group = get_title('Sector Group')
    sector_code = data['Sector Code']
    medium_industry_group =get_title('Medium(M) Industry Group')
    medium_industry_code = data['Medium(M) Industry Code']
    expanded_industry_group = get_title('Expanded(X) Industry Group')
    expanded_industry_code = data['Expanded(X) Industry Code']

    d.append((sector_group,sector_code,medium_industry_group,medium_industry_code,expanded_industry_group,expanded_industry_code))

print(pd.DataFrame(d,columns=('Sector Group','Sector Code','Medium(M) Industry Group','Medium(M) Industry Code','Expanded(X) Industry Group','Expanded(X) Industry Code')))

【讨论】：

这是一个创造性的解决方案。您如何建议将其放入 DataFrame
更新了我的答案。看看吧。