如何抓取网站Python中的所有页面[重复]

Question

我正在尝试使用具有多个页面的 python 抓取一个论坛。

示例：https://www.f150forum.com/f118/would-you-buy-f150-again-463954/

如何去网站的多个页面提取每个用户的cmets？提前谢谢你

url = "https://www.f150forum.com/f118/would-you-buy-f150-again-463954/"
page = requests.get(url, headers=headers)
soup = BeautifulSoup(page.text, 'html.parser')
domains = soup.find_all("div")
posts = soup.find(id = "posts")
comments_class = soup.findAll('div',attrs={"class":"ism-true"})   
comments = [row.get_text() for row in comments_class]

Answer 1

import requests
from bs4 import BeautifulSoup
import re

data = []
with requests.Session() as req:
    for item in range(1, 11):
        print(f"Extracting Page# {item}")
        r = req.get(
            f"https://www.f150forum.com/f118/would-you-buy-f150-again-463954/index{item}/")
        soup = BeautifulSoup(r.text, 'html.parser')
        result = [item.get_text(strip=True, separator=" ") for item in soup.findAll(
            "div", id=re.compile("^post_message_"))]
        data.append(result)

print(data)