如何抓取行中包含行的 web 表？答案

【问题标题】：How to scrape web table that has rows within rows?如何抓取行中包含行的 web 表？
【发布时间】：2021-11-09 04:38:13
【问题描述】：

我正在尝试抓取一张表格，而无论是谁设置它以将一堆信息放在一个单列表格中，并且在每一行内，都有很多行。

我想从行内抓取每个 for 并创建一个数据框，并将其作为一行。我还想将位于<strong> </strong> 中的信息设置为整个数据框的列。

有没有办法用 python 做到这一点？我一直在使用 selenium 和 pandas read_html，但我想我在这里碰壁了。最终，我想将所有这些信息整合到一个数据框中。

HTML 看起来像这样。

<td>
    <strong>    Important Information1  </strong>
    <br>    Some information
    <br>    Some information
    <br>    Some information
    <br>    Some information
    <br>    Some information
    <br>    Some information
</td>
<td>
    <strong>    Important Information 2 </strong>
    <br>    Some information 2
    <br>    Some information 2
    <br>    Some information 2
    <br>    Some information 2
    <br>    Some information 2  
    <br>    Some information 2
    <br>    Some information 2  
    <br>    Some information 2
    <br>    Some information 2  
    <br>    Some information 2  
</td>
<td>
    <strong>    Important Information 3 </strong>
    <br>    Some information 3
    <br>    Some information 3
    <br>    Some information 3
    <br>    Some information 3  
</td>

预期结果：

           Important Header Some Information Header
0   Important Information1         Some information
1   Important Information1         Some information
2   Important Information1         Some information
3   Important Information1         Some information
4   Important Information1         Some information
5   Important Information1         Some information
6    Important Information2      Some information 2
7    Important Information2      Some information 2
8    Important Information2      Some information 2
9    Important Information2      Some information 2
10   Important Information2      Some information 2
11   Important Information2      Some information 2
12   Important Information2      Some information 2
13   Important Information2      Some information 2
14   Important Information2      Some information 2
15   Important Information2      Some information 2
16   Important Information3      Some information 3
17   Important Information3      Some information 3
18   Important Information3      Some information 3
19   Important Information3      Some information 3

【问题讨论】：

标签： python selenium web-scraping beautifulsoup

【解决方案1】：

如果我理解正确，您希望将 HTML 文档转换为 3 列 pandas DataFrame：

import pandas as pd
from bs4 import BeautifulSoup

html_doc = """
<td>
    <strong>    Important Information1  </strong>
    <br>    Some information
    <br>    Some information
    <br>    Some information
    <br>    Some information
    <br>    Some information
    <br>    Some information
</td>
<td>
    <strong>    Important Information 2 </strong>
    <br>    Some information 2
    <br>    Some information 2
    <br>    Some information 2
    <br>    Some information 2
    <br>    Some information 2  
    <br>    Some information 2
    <br>    Some information 2  
    <br>    Some information 2
    <br>    Some information 2  
    <br>    Some information 2  
</td>
<td>
    <strong>    Important Information 3 </strong>
    <br>    Some information 3
    <br>    Some information 3
    <br>    Some information 3
    <br>    Some information 3  
</td>
"""

soup = BeautifulSoup(html_doc, "html.parser")

cols = []
for td in soup.select("td"):
    col_name, *data = td.get_text(strip=True, separator="|").split("|")
    cols.append(pd.Series(data, name=col_name))

print(pd.concat(cols, axis=1))

打印：

  Important Information1 Important Information 2 Important Information 3
0       Some information      Some information 2      Some information 3
1       Some information      Some information 2      Some information 3
2       Some information      Some information 2      Some information 3
3       Some information      Some information 2      Some information 3
4       Some information      Some information 2                     NaN
5       Some information      Some information 2                     NaN
6                    NaN      Some information 2                     NaN
7                    NaN      Some information 2                     NaN
8                    NaN      Some information 2                     NaN
9                    NaN      Some information 2                     NaN

【讨论】：

这很接近，但我想在轴 0 上连接它们。所以标题将是重要信息，一些信息
@pkpto39 您能否编辑您的问题并将预期结果放在那里（格式正确）？
请注意这个答案。我不得不将轴 = 1 更改为轴 = 0，但这让我非常接近我所需要的。

【解决方案2】：

如果没有您计划如何抓取这些元素的示例，很难确切地说出什么最适合您，但如果我假设您是从头开始，我建议您获取一个元素，然后获取该元素的子元素。

它可能需要错误处理才能健壮。许多人更喜欢使用 css 选择器作为标识符，但我个人更喜欢 xpaths。

它可能看起来像：

elements_you_want = driver.find_elements_by_xpath('xpath to parent')
for child in element:
     # do something

一些逻辑需要选择每个父元素，但这实际上取决于您要抓取的特定页面。

此 stackoverflow 帖子中更详细地显示了这一点： Get all child elements

【讨论】：

【解决方案3】：

确保导入环境：

# >> Get Ready: Importing Programming Environment Package Are You Using
import os
from selenium import webdriver

# >> Setup chrome browser
chromedriver = "C:\Program Files\Python39\Scripts\chromedriver"
os.environ["webdriver.chrome.driver"] = chromedriver
driver = webdriver.Chrome(chromedriver)

代码 sn-p 到抓取：

# - Programing: Scraping
element_list = driver.find_elements_by_tag_name('td')
_i_ = 0
Data = [[]]
for _item_ in element_list:
    _i_ += 1
    Title = _item_.find_element_by_xpath('//td['+str(_i_)+']/strong').text.strip()
    Data.append([_i_, Title])
    for _element_ in _item_.find_elements_by_xpath('//td['+str(_i_)+']/br'):
        Value = _element_.text.strip()
        Data[_i_ + 1].extend(Value) #or Try if the fill array data program not true: Data[_i_].extend(Value)
    
# - Show results:
print('- Data[1] = ', Data[0])
print('- Data[2] = ', Data[1])
print('- Data[3] = ', Data[2])

更新：代码导出 csv

import csv

def pad(data):
    max_n = max([len(x) for x in data.values()])
    for field in data:
        data[field] += [''] * (max_n - len(data[field]))
    return data

def merge_dicts(*dict_args):
    """
    Given any number of dictionaries, shallow copy and merge into a new dict,
    precedence goes to key-value pairs in latter dictionaries.
    """
    result = {}
    for dictionary in dict_args:
        result.update(dictionary)
    return result

Data_1 = Data[0]
Data_2 = Data[0]
Data_3 = Data[0]

sdata_1 = {"Data_1":Data_1, "Data_2":Data_2}
sdata_2 = { "Data_3":Data_3}
data = merge_dicts(sdata_1, sdata_2)
print(data)

import pandas as pd
df = pd.DataFrame(pad(data))
df.to_csv("output.csv", index=False)

print('>> Finish export to CSV')

【讨论】：