如何使用 pd.read_html 抓取具有 % 值的 HTML 表格？

Question

我正在尝试使用以下代码从以下 URL：https://markets.ft.com/data/funds/tearsheet/summary?s=LU0526609390:EUR 中抓取“个人资料和投资”表：

import requests
import pandas as pd

# Define all urls required for data scraping from the FT Website - if new fund is added simply add the appropriate Fund ID to the List
List = ['LU0526609390:EUR', 'IE00BHBX0Z19:EUR', 'LU1076093779:EUR', 'LU1116896363:EUR']
df = pd.DataFrame(List, columns=['List'])
urls = 'https://markets.ft.com/data/funds/tearsheet/summary?s='+ df['List']

for url in urls:
    r = requests.get(url).content
    df = pd.read_html(r)[0]
    print (df)

但是，当我使用pd.read_html 函数时，我收到以下错误代码：

ValueError: int() 以 10 为底的无效文字：'100%'

因为该表在% 中有条目。有没有办法让 Pandas 接受 % 值？

我需要的输出是得到一个格式如下的表格：

    Fund_ID          Fund_type     Income_treatment     Morningstar category ......
LU0526609390:EUR        ...              ...                    ....
IE00BHBX0Z19:EUR        ...              ...                    ....
LU1076093779:EUR        ...              ...                    ....
LU1116896363:EUR        ...              ...                    ....

Answer 1

问题是该站点使用'colspan' 属性并使用% 而不是int。 AsishM 在comments 中提到：

浏览器通常对 % 之类的东西更宽容，但 colspan 的 html 规范清楚地提到这应该是一个整数。浏览器将 100% 视为 100。mdn link。这本身不是熊猫问题。

这些应该是 int 的形式，虽然有些浏览器会适应这种情况，但 pandas 特别希望它是适当的语法：

<td colspan="number">

解决方法是：

1. 使用 BeautifulSoup 修复这些属性

2. 由于它不在您实际要解析的表中，因此请使用 BeautifulSoup 获取第一个表，然后无需担心。

3. 查看表是否具有特定属性，并可以将其作为参数添加到.read_html()，以便仅获取该特定表。

我在这里选择了选项 2：

import requests
import pandas as pd
from bs4 import BeautifulSoup

# Define all urls required for data __scraping__ from the FT Website - if new fund is added simply add the appropriate Fund ID to the List
List = ['LU0526609390:EUR', 'IE00BHBX0Z19:EUR', 'LU1076093779:EUR', 'LU1116896363:EUR']
df = pd.DataFrame(List, columns=['List'])
urls = 'https://markets.ft.com/data/funds/tearsheet/summary?s='+ df['List']

results = pd.DataFrame()
for url in urls:
    print(url)
    r = requests.get(url).content
    soup = BeautifulSoup(r, 'html.parser')
    table = soup.find('table')
    df = pd.read_html(str(table), index_col=0)[0].T
    results = results.append(df, sort=False)
    
results = results.reset_index(drop=True)
print (results)

输出：

print(results.to_string())
0                      Fund type Income treatment       Morningstar category IMA sector  Launch date Price currency    Domicile          ISIN                                                        Manager & start date                            Investment style (bonds)                 Investment style (stocks)
0                          SICAV           Income   Global Bond - EUR Hedged         --  06 Aug 2010            GBP  Luxembourg  LU0526609390  Jonathan Gregory01 Nov 2012Vivek Acharya09 Dec 2015Simon Foster01 Nov 2012                                                 NaN                                       NaN
1  Open Ended Investment Company           Income       EUR Diversified Bond         --  21 Feb 2014            EUR     Ireland  IE00BHBX0Z19                         Lorenzo Pagani12 May 2017Konstantin Veit01 Jul 2019  Credit Quality: HighInterest-Rate Sensitivity: Mod                                       NaN
2                          SICAV           Income  Eurozone Large-Cap Equity         --  11 Jul 2014            GBP  Luxembourg  LU1076093779                                                                         NaN                                                 NaN  Market Cap: LargeInvestment Style: Blend
3                          SICAV           Income          EUR Flexible Bond         --  01 Dec 2014            EUR  Luxembourg  LU1116896363                                                                         NaN                                                 NaN                                       NaN

以下是使用 BeautifulSoup 修复 colspan 属性的方法。

import requests
import pandas as pd
from bs4 import BeautifulSoup

# Define all urls required for data scraping from the FT Website - if new fund is added simply add the appropriate Fund ID to the List
List = ['LU0526609390:EUR', 'IE00BHBX0Z19:EUR', 'LU1076093779:EUR', 'LU1116896363:EUR']
df = pd.DataFrame(List, columns=['List'])
urls = 'https://markets.ft.com/data/funds/tearsheet/summary?s='+ df['List']


for url in urls:
    print(url)
    r = requests.get(url).content
    soup = BeautifulSoup(r, 'html.parser')
    
    all_colspan = soup.find_all(attrs={'colspan':True})
    for colspan in all_colspan:
        colspan.attrs['colspan'] = colspan.attrs['colspan'].replace('%', '')
        
    df = pd.read_html(str(soup))