没有id的网页抓取

Question

我需要有关网页抓取的帮助。这是html示例：

<div class="content" name="content-name">
   <h2 class="Topic">First Topic</h2>
   <ul>
      <li>This Data 1</li>
      <li>This Data 2</li>
      <li>This Data 3</li>
   </ul>
   <h2 class="Topic">Second Topic</h2>
   <ul>
      <li>That Data 1</li>
      <li>That Data 2</li>
      <li>That Data 3</li>
   </ul>
   <h2 class="Topic">Third Topic</h2>
   <ul>
      <li>Their Data 1</li>
      <li>Their Data 2</li>
      <li>Their Data 3</li>
   </ul>
</div>

使用 BeautifulSoup，我可以获得 name="content-name" 的 html div 标签。但是，我如何在 h2 标签之后的 ul 中获取所有带有 li 标签的文本，其中包含“第二个主题”文本？因为所有这些都在同一个 div 标签中，没有特定的类、id 或名称。 提前致谢。

Answer 1

from bs4 import BeautifulSoup

src = """
<div class="content" name="content-name">
    <h2 class="Topic">First Topic</h2>
    <ul>
        <li>This Data 1</li>
        <li>This Data 2</li>
        <li>This Data 3</li>
    </ul>
    <h2 class="Topic">Second Topic</h2>
    <ul>
        <li>That Data 1</li>
        <li>That Data 2</li>
        <li>That Data 3</li>
    </ul>
    <h2 class="Topic">Third Topic</h2>
    <ul>
        <li>Their Data 1</li>
        <li>Their Data 2</li>
        <li>Their Data 3</li>
    </ul>
</div>
"""

soup = BeautifulSoup(src, 'lxml')

content = soup.find_all("div", class_="content")[0]


second_topic = content.find_all("h2", class_="Topic", string="Second Topic")[0]

ul = second_topic.next_sibling.next_sibling

li = ul.find_all("li")
for i in li:
    print(i.string)

Answer 2

标签没有id、类或父标签时总是比较困难的。

您可以使用find_previous_sibling

from bs4 import BeautifulSoup
html = """
<div class="content" name="content-name">
   <h2 class="Topic">First Topic</h2>
   <ul>
      <li>This Data 1</li>
      <li>This Data 2</li>
      <li>This Data 3</li>
   </ul>
   <h2 class="Topic">Second Topic</h2>
   <ul>
      <li>That Data 1</li>
      <li>That Data 2</li>
      <li>That Data 3</li>
   </ul>
   <h2 class="Topic">Third Topic</h2>
   <ul>
      <li>Their Data 1</li>
      <li>Their Data 2</li>
      <li>Their Data 3</li>
   </ul>
</div>
"""
soup = BeautifulSoup(html, 'html.parser')

for ul in soup.find_all('ul'):
    if ul.find_previous_sibling('h2').text == 'Second Topic':
        for li in ul.find_all('li'):
            print(li.text)

返回

That Data 1
That Data 2
That Data 3