Python：网页抓取标签导航 wiki 表答案

【问题标题】：Python: Web Scraping Tag Navigation wiki tablePython：网页抓取标签导航 wiki 表
【发布时间】：2019-11-24 07:50:00
【问题描述】：

我正在尝试仅使用 bs4 隔离球员所效力球队的“职业历史”列表 - NFL Qbs 表格的部分：

我想要的输出是：

['St. Louis Rams (2005–2006)', 'Cincinnati Bengals (2007–2008)', 'Buffalo Bills (2009–2012)', 'Tennessee Titans (2013)', 'Houston Texans (2014)', 'New York Jets (2015–2016)', 'Tampa Bay Buccaneers (2017–2018)', 'Miami Dolphins (2019–present)']

我的代码是：

url = 'https://en.wikipedia.org/wiki/Ryan_Fitzpatrick'
table = BeautifulSoup(player_wiki.text , 'html.parser')

for tr  in table.find('tbody').find_all('ul'):
  v = [li.text for li in tr.find_all('li')]
  print(v)

当前输出：

['St. Louis Rams (2005–2006)', 'Cincinnati Bengals (2007–2008)', 'Buffalo Bills (2009–2012)', 'Tennessee Titans (2013)', 'Houston Texans (2014)', 'New York Jets (2015–2016)', 'Tampa Bay Buccaneers (2017–2018)', 'Miami Dolphins (2019–present)']
['Ivy League Player of the Year (2004)', 'First-team All–Ivy League (2004)', 'George H. “Bulger” Lowe Award (2004)']

我确定这是我的外循环的“ul”标签。如何缩小我的 find_all() 的范围以防止不需要的数据？有小费吗？我是网络抓取的新手。

【问题讨论】：

标签： python web-scraping beautifulsoup

【解决方案1】：

方法 1 - 使用 requests 和 beautifulsoup4：

    import requests
    from bs4 import BeautifulSoup

    r = requests.get('https://en.wikipedia.org/wiki/Ryan_Fitzpatrick')
    soup = BeautifulSoup(r.text, 'html.parser')

    for item in soup.find('tbody').findAll('ul'):
        for href in item.findAll('a'):
            print(href.get_text())

方法 2- 使用 wikipedia 模块：

    from bs4 import BeautifulSoup
    import wikipedia

    ry = wikipedia.page('Ryan_Fitzpatrick')
    soup = BeautifulSoup(ry.html(), 'html.parser')
    career_history = []
    for tr in soup.find('tbody').find_all('ul'):
        for li in tr.find_all('li'):
          career_history.append(li.text)

    print(career_history)

输出：

['St. Louis Rams (2005–2006)', 'Cincinnati Bengals (2007–2008)', 'Buffalo Bills (2009–2012)', 'Tennessee Titans (2013)', 'Houston Texans (2014)', 'New York Jets (2015–2016)', 'Tampa Bay Buccaneers (2017–2018)', 'Miami Dolphins (2019–present)', 'Ivy League Player of the Year (2004)', 'First-team All–Ivy League (2004)', 'George H. “Bulger” LoweAward (2004)']

【讨论】：

【解决方案2】：

你可以使用soup.find_all:

import requests
from bs4 import BeautifulSoup as soup
d = soup(requests.get('https://en.wikipedia.org/wiki/Ryan_Fitzpatrick').text, 'html.parser')
result = [i.get_text(strip=True) for i in d.find('table', {'class':'infobox vcard'}).find_all('tr')[12].find_all('li')]

输出：

['St. Louis Rams(2005–2006)', 'Cincinnati Bengals(2007–2008)', 'Buffalo Bills(2009–2012)', 'Tennessee Titans(2013)', 'Houston Texans(2014)', 'New York Jets(2015–2016)', 'Tampa Bay Buccaneers(2017–2018)', 'Miami Dolphins(2019–present)']

【讨论】：