抓取不同的 Javascript 表答案

【问题标题】：Scraping different Javascript Table抓取不同的 Javascript 表
【发布时间】：2018-01-10 19:47:22
【问题描述】：

是否可以用漂亮的汤来刮掉下一页中的“BOND INDEXES”表而不是刮掉默认的“CDS INDEXES”表？：

https://web.apps.markit.com/

或者这是一项需要硒的任务？

我不知道如何研究这个问题，因为我不知道如何调用更改表选项

【问题讨论】：

此网页中没有 JavaScript、表格甚至文本，您确定您的 URL 正确吗？
确定链接，你能看到表格上方不同的TAB吗？我不确定 JavaScript 是否用于在两个不同的表之间切换
wsj.com/mdc/public/npage/2_3023_creditdervs.html

标签： python selenium automation beautifulsoup

【解决方案1】：

您需要一个来自主页上加载的iframe 的cookie。您可以通过创建请求会话并请求主页，然后是 iframe 应用程序来获得它。这样您就拥有了访问最后一段所需的 cookie 和 url。

下面获取表的表头和每一行：

import requests
from bs4 import BeautifulSoup
import json

with requests.Session() as sess:
    # Get the data:
    response = sess.get('http://www.wsj.com/mdc/public/npage/2_3023_creditdervs.html')
    sess.get(BeautifulSoup(response.text, 'lxml').find('iframe').attrs['src'])
    response = sess.post(
        'https://web.apps.markit.com/AppsApi/GetIndexData',
        data={'indexOrBond': 'bond', 'ClientCode': 'WSJ'}
    )
    table = BeautifulSoup(json.loads(response.text)['html'], 'lxml').find('table', {'id': 'BondIndexTable'})
    header = [cell.text for cell in table.find('thead').find_all('tr')[-1].find_all('th')]
    data = list()
    for row in table.find_all('tr'):
        row = [cell.text for cell in row.find_all('td')]
        if len(row) > 2:
            data.append(row)

    # Do something with the data:
    print(header)
    for row in data:
        print(row)

这会产生：

['Bond Indexes', 'Daily', 'Monthly', 'YTD', '1Y', '3Y']
['Markit iBoxx USD Overall', '0.18%', '0.41%', '3.45%', '-0.60%', '8.98%']
['Markit iBoxx USD Treasuries', '0.20%', '0.46%', '2.60%', '-2.26%', '7.56%']
['Markit iBoxx USD Liquid Investment Grade Index', '0.17%', '0.47%', '5.62%', '1.54%', '14.07%']
['Markit iBoxx USD Liquid High Yield Index', '-0.07%', '0.00%', '5.55%', '9.85%', '14.23%']
['Markit iBoxx EUR Overall', '0.13%', '0.50%', '0.15%', '-2.25%', '8.67%']
['Markit iBoxx EUR Corporates', '0.09%', '0.40%', '1.79%', '0.67%', '8.79%']
['Markit iBoxx EUR Sovereigns', '0.15%', '0.60%', '-0.24%', '-3.29%', '9.68%']
['Markit iBoxx GBP Overall', '0.68%', '0.66%', '2.00%', '-0.80%', '23.18%']
['Markit iBoxx GBP Corporates', '0.55%', '0.57%', '4.13%', '2.95%', '24.88%']
['Markit iBoxx GBP Gilts', '0.74%', '0.72%', '1.36%', '-2.03%', '23.39%']
['Markit iBoxx Asia', '0.00%', '-0.02%', '2.23%', '-1.00%', '8.27%']
['Markit iBoxx Global Inflation-Linked Index All USD', '0.58%', '0.71%', '-0.45%', '-0.63%', '9.81%']
['Markit iBoxx GEMX USD', '0.06%', '0.10%', '2.70%', '1.44%', '6.92%']
['Markit iBoxx USD Corporates', '0.16%', '0.39%', '4.96%', '1.92%', '11.88%']

这可以与 pandas 或其他一些数据操作工具一起使用：

import pandas as pd
df = pd.DataFrame(data, columns=header)
for col in df.columns:
    if col != 'Bond Indexes':
        df[col] = pd.to_numeric(df[col].replace(regex=True, to_replace='%', value=''))/100
print(df)

【讨论】：

【解决方案2】：

您需要从第一页获取唯一的 URL。然后访问此 URL 以获取 cookie。然后您可以发出 POST 请求以在 json 对象中获取所需的 HTML。像这样：

import requests
from bs4 import BeautifulSoup
import json

s = requests.Session() 
t = s.get('http://www.wsj.com/mdc/public/npage/2_3023_creditdervs.html')
soup1 = BeautifulSoup(t.text, "html.parser")
# Get the unique URL.
url = soup1.find('iframe').get("src")
# Make a request to set cookies.
s.get(url)
# I'm not sure all these headers are needed but some are.
headers = {'X-Requested-With': 'XMLHttpRequest',
           'Accept': 'application/json, text/javascript, */*; q=0.01',
           'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:54.0) Gecko/20100101 Firefox/54.0',
           'Content-Type': 'application/x-www-form-urlencoded',
           'referer': url}
data = {"ClientCode": "WSJ","indexOrBond": "bond"}
# Get the data as json.
r = s.post('https://web.apps.markit.com/AppsApi/GetIndexData',  data = data, headers = headers)
jn = r.json()
# Get the HTML from the json.
print (jn['html'])

如果你想进一步解析它，你可以从那里将 HTML 加载到 BeautifulSoup 中。

输出：

<Div><div class="tableDesc"><div class="firstComponentNote">Green indicates rising index levels; red indicates declining index levels.</div></div><table id="BondIndexTable" class="dataTable"><thead><tr><th class="col1"></th><th class="col23456 topCell centerAlign" colspan="5" style="text-align: center;">Total Return</th></tr><tr><th class="col1 leftCell">Bond Indexes</th><th class="col2">Daily</th><th class="col3">Monthly</th><th class="col4">YTD</th><th class="col5">1Y</th><th class="col6">3Y</th></tr></thead><tbody><tr><td class="col1 leftCell"><div class="qname">Markit iBoxx USD Overall</div></td><td class="col2 yellowBack"><span class="pos">0.18%</span></td><td class="col3 yellowBack"><span class="pos">0.41%</span></td><td class="col4 yellowBack"><span class="pos">3.45%</span></td><td class="col5 yellowBack"><span class="neg">-0.60%</span></td><td class="col6 yellowBack"><span class="pos">8.98%</span></td></tr><tr><td class="col1 leftCell"><div class="qname">Markit iBoxx USD Treasuries</div></td><td class="col2 yellowBack"><span class="pos">0.20%</span></td><td class="col3 yellowBack"><span class="pos">0.46%</span></td><td class="col4 yellowBack"><span class="pos">2.60%</span></td><td class="col5 yellowBack"><span class="neg">-2.26%</span></td><td class="col6 yellowBack"><span class="pos">7.56%</span></td></tr><tr><td class="col1 leftCell"><div class="qname">Markit iBoxx USD Liquid Investment Grade Index</div></td><td class="col2 yellowBack"><span class="pos">0.17%</span></td><td class="col3 yellowBack"><span class="pos">0.47%</span></td><td class="col4 yellowBack"><span class="pos">5.62%</span></td><td class="col5 yellowBack"><span class="pos">1.54%</span></td><td class="col6 yellowBack"><span class="pos">14.07%</span></td></tr><tr><td class="col1 leftCell"><div class="qname">Markit iBoxx USD Liquid High Yield Index</div></td><td class="col2 yellowBack"><span class="neg">-0.07%</span></td><td class="col3 yellowBack"><span class="pos">0.00%</span></td><td class="col4 yellowBack"><span class="pos">5.55%</span></td><td class="col5 yellowBack"><span class="pos">9.85%</span></td><td class="col6 yellowBack"><span class="pos">14.23%</span></td></tr><tr><td class="col1 leftCell"><div class="qname">Markit iBoxx EUR Overall</div></td><td class="col2 yellowBack"><span class="pos">0.13%</span></td><td class="col3 yellowBack"><span class="pos">0.50%</span></td><td class="col4 yellowBack"><span class="pos">0.15%</span></td><td class="col5 yellowBack"><span class="neg">-2.25%</span></td><td class="col6 yellowBack"><span class="pos">8.67%</span></td></tr><tr><td class="col1 leftCell"><div class="qname">Markit iBoxx EUR Corporates</div></td><td class="col2 yellowBack"><span class="pos">0.09%</span></td><td class="col3 yellowBack"><span class="pos">0.40%</span></td><td class="col4 yellowBack"><span class="pos">1.79%</span></td><td class="col5 yellowBack"><span class="pos">0.67%</span></td><td class="col6 yellowBack"><span class="pos">8.79%</span></td></tr><tr><td class="col1 leftCell"><div class="qname">Markit iBoxx EUR Sovereigns</div></td><td class="col2 yellowBack"><span class="pos">0.15%</span></td><td class="col3 yellowBack"><span class="pos">0.60%</span></td><td class="col4 yellowBack"><span class="neg">-0.24%</span></td><td class="col5 yellowBack"><span class="neg">-3.29%</span></td><td class="col6 yellowBack"><span class="pos">9.68%</span></td></tr><tr><td class="col1 leftCell"><div class="qname">Markit iBoxx GBP Overall</div></td><td class="col2 yellowBack"><span class="pos">0.68%</span></td><td class="col3 yellowBack"><span class="pos">0.66%</span></td><td class="col4 yellowBack"><span class="pos">2.00%</span></td><td class="col5 yellowBack"><span class="neg">-0.80%</span></td><td class="col6 yellowBack"><span class="pos">23.18%</span></td></tr><tr><td class="col1 leftCell"><div class="qname">Markit iBoxx GBP Corporates</div></td><td class="col2 yellowBack"><span class="pos">0.55%</span></td><td class="col3 yellowBack"><span class="pos">0.57%</span></td><td class="col4 yellowBack"><span class="pos">4.13%</span></td><td class="col5 yellowBack"><span class="pos">2.95%</span></td><td class="col6 yellowBack"><span class="pos">24.88%</span></td></tr><tr><td class="col1 leftCell"><div class="qname">Markit iBoxx GBP Gilts</div></td><td class="col2 yellowBack"><span class="pos">0.74%</span></td><td class="col3 yellowBack"><span class="pos">0.72%</span></td><td class="col4 yellowBack"><span class="pos">1.36%</span></td><td class="col5 yellowBack"><span class="neg">-2.03%</span></td><td class="col6 yellowBack"><span class="pos">23.39%</span></td></tr><tr><td class="col1 leftCell"><div class="qname">Markit iBoxx Asia</div></td><td class="col2 yellowBack"><span class="neg">0.00%</span></td><td class="col3 yellowBack"><span class="neg">-0.02%</span></td><td class="col4 yellowBack"><span class="pos">2.23%</span></td><td class="col5 yellowBack"><span class="neg">-1.00%</span></td><td class="col6 yellowBack"><span class="pos">8.27%</span></td></tr><tr><td class="col1 leftCell"><div class="qname">Markit iBoxx Global Inflation-Linked Index All USD</div></td><td class="col2 yellowBack"><span class="pos">0.58%</span></td><td class="col3 yellowBack"><span class="pos">0.71%</span></td><td class="col4 yellowBack"><span class="neg">-0.45%</span></td><td class="col5 yellowBack"><span class="neg">-0.63%</span></td><td class="col6 yellowBack"><span class="pos">9.81%</span></td></tr><tr><td class="col1 leftCell"><div class="qname">Markit iBoxx GEMX USD</div></td><td class="col2 yellowBack"><span class="pos">0.06%</span></td><td class="col3 yellowBack"><span class="pos">0.10%</span></td><td class="col4 yellowBack"><span class="pos">2.70%</span></td><td class="col5 yellowBack"><span class="pos">1.44%</span></td><td class="col6 yellowBack"><span class="pos">6.92%</span></td></tr><tr><td class="col1 leftCell"><div class="qname">Markit iBoxx USD Corporates</div></td><td class="col2 yellowBack"><span class="pos">0.16%</span></td><td class="col3 yellowBack"><span class="pos">0.39%</span></td><td class="col4 yellowBack"><span class="pos">4.96%</span></td><td class="col5 yellowBack"><span class="pos">1.92%</span></td><td class="col6 yellowBack"><span class="pos">11.88%</span></td></tr></tbody><tfoot><tr><td colspan="6">Markit iBoxx indexes track the performance of the global sovereign- and corporate-bond markets. These benchmark indexes are calculated using prices contributed by multiple financial institutions. The indexes are owned, calculated and administered by Markit. For more information visit <a href="http://indices.markit.com" target="_blank">indices.markit.com</a></td></tr></tfoot></table></Div>

【讨论】：