使用 BeautifulSoup 解析 h1 标签下的表并存储在 df

Question

我想在 h1“表格”标题下从多个网站中提取表格。每个网站都有多个 h1 标题，但“表格”在所有网站上都是一致的。尽管我已经设法将它们提取到一个列表中，但每个表还带有一个不同站点的 h2 标题。

我当前的代码解析所有表格，甚至是那些在 h1 'Tables' 标题之前的表格。如何排除这些表？ html与下面的类似：

<h2>I don't care about this table</h2>
<table class="foo">
  <tr>
    <td>Key A</td>
  </tr>
  <tr>
    <td>A value I don't want</td>
  </tr>
</table>

<h1>Tables</h1>
<p> A description I don't care about </p>
<h2>First good table</h2>
<table class="foo">
  <tr>
    <td>Key B</td>
  </tr>
  <tr>
    <td>A value I want</td>
  </tr>
</table>


<h2>Second good table</h2>
<table class="foo">
  <tr>
    <td>Key C</td>
  </tr>
  <tr>
    <td>A value I want</td>
  </tr>
</table>

我目前的做法：

soup = BeautifulSoup(self.body, features="lxml")
headers = [tags.text for tags in soup.find_all(["h1", "h2"])]

try:
    # Find h2 table headers under h1 'Tables' header
    target_index = headers.index("Tables")
    table_headers = headers[target_index + 1 :]

except ValueError:
    print("Page doesn't contain tables")

# This includes all tables. How do we make sure we only include those under the 'Tables' header?
tables_raw = [[[cell.text for cell in row("th") + row("td")] for row in table("tr")]for table in soup("table")]

# Create dfs and assign a name
tables_df = [pd.DataFrame(table) for table in tables_raw]
tables_and_names = list(zip(table_headers, tables_df))

我确实看过 this solution，但不知道如何获得我目前拥有的 df 输出。任何帮助将不胜感激。

Answer 1

您可以将 CSS 选择器与 ~ 一起使用：

for table in soup.select("h1:-soup-contains(Tables) ~ table"):
    for row in table.select("tr"):
        print(*[td.get_text(strip=True) for td in row.select("td")])

打印：

Key B
A value I want
Key C
A value I want

---

或者没有 CSS 选择器：

h1 = soup.find("h1", text="Tables")
for table in h1.find_next_siblings("table"):
    for row in table.select("tr"):
        print(*[td.get_text(strip=True) for td in row.select("td")])

Answer 2

应该这样做

header = soup.find('h1')
for sibling in header.next_siblings:
    if sibling.name == 'table':
        do_stuff()