我将如何使用 BeautifulSoup 的索引来抓取多个表？

Question

我正在尝试抓取此 URL，因此我只能获取某些索引。在这种情况下，我的示例显示我可以抓取索引 6，这将为我提供任何以 /wiki/ 开头的 url。这将为我提供所有以 A 开头的 TLD。我想获取所有与我的任务相关的索引。

截至目前，我已尝试将它们列为 [6、7、8 等] 并在引号中列出。不过，我对列表的工作并不多，我需要花更多时间学习。

import requests
from bs4 import BeautifulSoup 

page = requests.get('https://en.wikipedia.org/wiki/List_of_Internet_top-level_domains')
soup = BeautifulSoup(page.text, 'lxml')

table = soup.findAll('table')[6]
for record in table.findAll('tr'):
    for data in record.findAll('td'):
        for link in data.select("a[href^='/wiki/.']"):
            links = link.contents[0]
            print(links)

但是，由于我是编程新手，我不知道如何添加除 6 之外的多个索引。这些是我收到的错误：

======= RESTART: /run/media/sean/The Continuum/Python/wikinamelist.py =======
Traceback (most recent call last):
  File "/run/media/sean/The Continuum/Python/wikinamelist.py", line 7, in <module>
    table_data = soup.find_all('table')["6", "7"]
TypeError: list indices must be integers or slices, not tuple
>>> 
======= RESTART: /run/media/sean/The Continuum/Python/wikinamelist.py =======
Traceback (most recent call last):
  File "/run/media/sean/The Continuum/Python/wikinamelist.py", line 7, in <module>
    table_data = soup.find_all('table')[6, 7];
TypeError: list indices must be integers or slices, not tuple
>>> 
======= RESTART: /run/media/sean/The Continuum/Python/wikinamelist.py =======
Traceback (most recent call last):
  File "/run/media/sean/The Continuum/Python/wikinamelist.py", line 7, in <module>
    table_data = soup.find_all('table')[6, 7, 8];
TypeError: list indices must be integers or slices, not tuple

正如您在上面看到的，我尝试了多种方法，并在错误消息中显示。

任何反馈都将不胜感激，谢谢！

Answer 1

您也许可以使用逗号分隔的第 n 个类型

table:nth-of-type(6), table:nth-of-type(7), table:nth-of-type(8)

所以，

tables = soup.select('table:nth-of-type(6), table:nth-of-type(7), table:nth-of-type(8)')

然后

for table in table:

你也可以浓缩

links = [item['href'] for item in soup.select("table:nth-of-type(6) [href^='/wiki/.'], table:nth-of-type(7) [href^='/wiki/.'], table:nth-of-type(8) [href^='/wiki/.']")

您也可以将table 的类型选择器换成类选择器，例如.wikitable。这样会更快。

---

pd.read_html：

如果pd.read_html 返回表，那么您可以只索引/切片到该列表中以获取所需的表。

Answer 2

import requests
from bs4 import BeautifulSoup 

page = requests.get('https://en.wikipedia.org/wiki/List_of_Internet_top-level_domains')
soup = BeautifulSoup(page.text, 'html.parser')

a = soup.select('table:nth-of-type(5) td:first-of-type')
b = soup.select('table:nth-of-type(6) td:first-of-type')
c = soup.select('table:nth-of-type(7) td:first-of-type')
d = soup.select('table:nth-of-type(8) td:first-of-type')
e = soup.select('table:nth-of-type(9) td:first-of-type')
f = soup.select('table:nth-of-type(10) td:first-of-type')
g = soup.select('table:nth-of-type(11) td:first-of-type')
h = soup.select('table:nth-of-type(12) td:first-of-type')
ij = soup.select('table:nth-of-type(13) td:first-of-type')
k = soup.select('table:nth-of-type(14) td:first-of-type')
l = soup.select('table:nth-of-type(15) td:first-of-type')
m = soup.select('table:nth-of-type(16) td:first-of-type')
n = soup.select('table:nth-of-type(17) td:first-of-type')
opq = soup.select('table:nth-of-type(18) td:first-of-type')
r = soup.select('table:nth-of-type(19) td:first-of-type')
s = soup.select('table:nth-of-type(20) td:first-of-type')
t = soup.select('table:nth-of-type(21) td:first-of-type')
uv = soup.select('table:nth-of-type(22) td:first-of-type')
wxyz = soup.select('table:nth-of-type(23) td:first-of-type')


print(a, b, c, d, e, f, g, h, ij, k, l, m, n, opq, r, s, t, uv, wxyz)

这是在页面上抓取多个列表的最简单方法。这将分别针对第一列，并遍历每个表。

上面的答案确实对我的问题有所帮助！但是，我对上面的建议进行了修改。我没有按照建议压缩代码，而是创建了一个变量列表，用于选择我要求的表。然后我将变量中的信息打印到 STDout。这段代码更易读，更模块化。

变量也对应表的名称。

感谢您的帮助，经过您的建议，这变得非常简单。