页码无序时如何抓取多个页面

Question

我正在尝试使用 BeautifulSoup 从网站上抓取单词列表。抓取第一页很容易，但是要获取所有页面，我必须为每个页面获取页码（精确的字符串），这对我来说非常困难，因为它们不是从传统的 {1-100} 或 {a-z} 开始的，它们是不同的每一页。

例如，this 是为 /a/ 类别中的其余页面存储所有链接的页面。通常它们会像 a/1,a/2,a/3 但在这种情况下它们是：

https://dictionary.cambridge.org/browse/english/a/a
https://dictionary.cambridge.org/browse/english/a/a-conflict-of-interest
https://dictionary.cambridge.org/browse/english/a/a-hard-tough-row-to-hoe
and so on...all the way to /english/z/{}

我的代码：

import requests
from bs4 import BeautifulSoup as bs

url = 'https://dictionary.cambridge.org/browse/english/a/a/'
head = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36'
# regex = 'idiom$'

with open('output.txt', 'w', encoding="utf-8") as f_out:

    soup = bs(requests.get(url,headers={'User-Agent': head}).content, 'html.parser')
    div = soup.find('div', attrs={'class', 'hdf ff-50 lmt-15'})
    span = div.find_all('a')

    for text in span:

        text_str = text.text.strip()
        print(text_str)
        print('{}'.format(text_str), file=f_out)

它按预期获取文本，但之后我不知道如何解析下一页。

Answer 1

您可以遍历字母表，获取所有href 属性，剪掉它的最后一部分（即您的单词或表达），然后将其保存到文件中。

方法如下：

import string

import requests
from bs4 import BeautifulSoup

headers = {
    "user-agent": 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36',
}
letters = string.ascii_lowercase
main_url = "https://dictionary.cambridge.org/browse/english/"

for letter in letters:
    print(f"Fetching words for letter {letter.upper()}...")
    page = requests.get(f"{main_url}{letter}", headers=headers).content
    soup = BeautifulSoup(page, "html.parser").find_all("a", {"class": "dil tcbd"})
    with open(f"{letter}_words.txt", "w") as output:
        output.writelines(
            "\n".join(a["href"].split("/")[-2] for a in soup[1:]) + "\n"
        )

输出：每个字母对应一个文件，例如字母a。

a-conflict-of-interest
a-hard-tough-row-to-hoe
a-meeting-of-minds
a-pretty-fine-kettle-of-fish
a-thing-of-the-past
ab-initio
abduction
abo
abreast
absolute-motion
absurdity
accent
accidental-death-benefit
account-for-sth
acct
acetylcholinesterase
ackee
acrobatics
actionable
actuarial
adapting
adduce
adjective
administration-order
adoration
adumbrated
advertised
aerie
affect
affronting
afters
agender
agit-pop
...