用 BeautifulSoup 刮新闻

Question

我正在尝试从网站上抓取新闻文章。我只对包含<span class="news_headline"> 和文本“Transfers”的文章感兴趣。从这篇文章中，我想从<div class="news_text"> 中提取跨度内的文本。结果应以 csv 文件结尾，如下所示：

R.Wolf; wechselt für 167.000 von Computer zu; Hauke  
Weiner; wechselt für 167.000 von Computer zu; Hauke  
Gonther; wechselt für 770.000 von Computer zu; Hauke 

或

3378; wechselt für 167.000 von Computer zu; 514102  
3605; wechselt für 167.000 von Computer zu; 514102  
1197; wechselt für 770.000 von Computer zu; 514102

我对编程很陌生，所以我希望任何人都可以提供帮助。

<div class="single_news_container">
    <div class="news_body_right single_news_content">
        <div class="cont_news_headlines">
          <span class="wrapper_news_headlines">
            <span class="news_headline">Transfers</span>
          </span>
        </div>
        <div class="news_text">
            <div>
                <p>
                    <span><a href="/2.bundesliga/players/R. Wolf-3378">R. Wolf</a> wechselt für 167.000 von Computer zu <a href="/users/514102">Hauke</a></span>
                </p>
                <p>
                    <span><a href="/2.bundesliga/players/Weiner-3605">Weiner</a> wechselt für 167.000 von Computer zu <a href="/users/514102">Hauke</a></span>
                </p>
                <p>
                    <span><a href="/2.bundesliga/players/Gonther-1197">Gonther</a> wechselt für 770.000 von Computer zu <a href="/users/514096">Christoph</a></span>
                </p>
            </div>
        </div>
    </div>
</div>

Answer 1

首先，检查 html 代码的嵌套结构。您会看到您要抓取的数据没有包含在您提到的div 中，而是它们都包含在<div class="news_body_right single_news_content"> 中。所以你应该在div 上运行find_all，然后循环结果以检查这些div 中的新闻标题是否包含“Transfers”。只有这样您才能提取数据，例如填充一个空列表，然后将其加载到pandas 并将其保存到csv：

find_all 返回一个列表

from bs4 import BeautifulSoup
import pandas as pd

html='''<div class="single_news_container">
<div class="news_body_right single_news_content">
    <div class="cont_news_headlines">
      <span class="wrapper_news_headlines">
        <span class="news_headline">Transfers</span>
      </span>
    </div>
    <div class="news_text">
        <div>
            <p>
                <span><a href="/2.bundesliga/players/R. Wolf-3378">R. Wolf</a> wechselt für 167.000 von Computer zu <a href="/users/514102">Hauke</a></span>
            </p>
            <p>
                <span><a href="/2.bundesliga/players/Weiner-3605">Weiner</a> wechselt für 167.000 von Computer zu <a href="/users/514102">Hauke</a></span>
            </p>
            <p>
                <span><a href="/2.bundesliga/players/Gonther-1197">Gonther</a> wechselt für 770.000 von Computer zu <a href="/users/514096">Christoph</a></span>
            </p>
        </div>
    </div>
</div>'''

soup = BeautifulSoup(html,'html.parser')

data = []

for news in soup.find_all("div", class_="news_body_right single_news_content"):
  if 'Transfers' in news.find("span", class_="news_headline"):
    for i in news.find("div", class_="news_text").find_all('span'):
      subject = i.find_all('a')[0].get_text()
      amount = i.get_text().split('für ', 1)[1].split(' von')[0].replace('.','').replace(',','.')
      from_player = i.get_text().split('von ', 1)[1].split(' zu')[0]
      to_player = i.find_all('a')[1].get_text()
      data.append({'subject': subject, 'amount': amount, 'from_player': from_player, 'to_player': to_player})

df = pd.DataFrame(data)
df.to_csv('output.csv')

结果：

	subject	amount	from_player	to_player
0	R. Wolf	167000	Computer	Hauke
1	Weiner	167000	Computer	Hauke
2	Gonther	770000	Computer	Christoph