Beautiful Soup，从 Wikipedia 获取表格数据

Question

我正在关注 Seppe vanden Broucke 和 Bart Baesens 所著的“Practical Web Scraping for Data Science Best Practices and examples with Python”一书。

下一个代码应该从 Wikipedia 获取数据，即权力的游戏剧集列表：

import requests
from bs4 import BeautifulSoup
url = 'https://en.wikipedia.org/w/index.php' + \
'?title=List_of_Game_of_Thrones_episodes&oldid=802553687'
r = requests.get(url)
html_contents = r.text
html_soup = BeautifulSoup(html_contents, 'html.parser')
# We'll use a list to store our episode list
episodes = []
ep_tables = html_soup.find_all('table', class_='wikiepisodetable')
for table in ep_tables:
    headers = []
    rows = table.find_all('tr')
    for header in table.find('tr').find_all('th'):
        headers.append(header.text)
        for row in table.find_all('tr')[1:]:
            values = []
            for col in row.find_all(['th','td']):
                values.append(col.text)
                if values:
                    episode_dict = {headers[i]: values[i] for i in
                                    range(len(values))}
                    episodes.append(episode_dict)
                    for episode in episodes:
                        print(episode)

但在运行代码时会显示下一个错误：

{'No.overall': '1'}

IndexError Traceback（最近一次调用最后一次）

<ipython-input-8-d2e64c7e0540> in <module>
     20                 if values:
     21                     episode_dict = {headers[i]: values[i] for i in
---> 22                                     range(len(values))}
     23                     episodes.append(episode_dict)
     24                     for episode in episodes:

<ipython-input-8-d2e64c7e0540> in <dictcomp>(.0)
     19                 values.append(col.text)
     20                 if values:
---> 21                     episode_dict = {headers[i]: values[i] for i in
     22                                     range(len(values))}
     23                     episodes.append(episode_dict)

IndexError: list index out of range

谁能告诉为什么会这样？

Answer 1

问题不在于代码，而在于代码的缩进。第三个for 循环应该与第二个并行，而不是在第二个for 循环内。书中是这样显示的。

import requests
from bs4 import BeautifulSoup
url = 'https://en.wikipedia.org/w/index.php' + \
'?title=List_of_Game_of_Thrones_episodes&oldid=802553687'
r = requests.get(url)
html_contents = r.text
html_soup = BeautifulSoup(html_contents, 'html.parser')
# We'll use a list to store our episode list
episodes = []
ep_tables = html_soup.find_all('table', class_='wikitable plainrowheaders wikiepisodetable')
for table in ep_tables:
    headers = []
    rows = table.find_all('tr')
    # Start by fetching the header cells from the first row to determine
    # the field names
    for header in table.find('tr').find_all('th'):
        headers.append(header.text)
    # Then go through all the rows except the first one
    for row in table.find_all('tr')[1:]:
        values = []
        # And get the column cells, the first one being inside a th-tag
        for col in row.find_all(['th','td']):
            values.append(col.text)
        if values:
            episode_dict = {headers[i]: values[i] for i in
        range(len(values))}
        episodes.append(episode_dict)
# Show the results
for episode in episodes:
 print(episode)

Answer 2

你的踪迹是

{'No.overall': '1'}
Traceback (most recent call last):
  File "/Users/karl/code/deleteme/foo.py", line 20, in <module>
    episode_dict = {headers[i]: values[i] for i in
  File "/Users/karl/code/deleteme/foo.py", line 20, in <dictcomp>
    episode_dict = {headers[i]: values[i] for i in
IndexError: list index out of range

代码可能过于缩进，并且在选择变量时有点难以阅读。了解您要准确提取的内容会很有用。剧集列表？ 由于这本书可能表结构已经改变。

如果是，那么每个相关的剧集标题都具有这种形状。

<td class="summary" style="text-align:left">"<a href="/wiki/Stormborn" title="Stormborn">Stormborn</a>"</td>

import requests
from bs4 import BeautifulSoup
url = 'https://en.wikipedia.org/w/index.php?title=List_of_Game_of_Thrones_episodes&oldid=802553687'
r = requests.get(url)
html_contents = r.text
html_soup = BeautifulSoup(html_contents, 'html.parser')
# We'll use a list to store our episode list
episodes = []
ep_tables = html_soup.find_all('table', class_='wikiepisodetable')
for table in ep_tables:
    headers = []
    rows = table.find_all('tr')
    for header in table.find('tr').find_all('th'):
        headers.append(header.text)
        for row in table.find_all('tr')[1:]:
            values = []
            for col in row.find_all('td', class_='summary'):
                print(col.text)