Selenium/BeautifulSoup 中的 Python

Question

我正在尝试使用本教程从使用硒和美丽汤的网站中提取房地产列表信息：https://medium.com/@ben.sturm/scraping-house-listing-data-using-selenium-and-beautiful-soup-1cbb94ba9492

目的是在找到“下一页”按钮之前从第一页收集所有href链接，导航到下一页并收集该页面上的所有链接等等。

尝试使用单个函数来实现这一点并为每个页面重复，但无法弄清楚为什么它不起作用。刚开始学习代码，似乎太琐碎而无法找到答案。希望有任何帮助

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import time
import sys
import numpy as np
import pandas as pd
import regex as re


driver = webdriver.Chrome
url = "http://property.shw.co.uk/searchproperties/Level2-0/Level1-0-181-236-167-165/Units/Development-or-House-and-Flat-or-Investment-or-Land-or-Office-or-Other/UnitIds-0/For-Sale"
driver.get(url)
try:
    wait = WebDriverWait(driver, 3)
    wait.until(EC.presence_of_element_located((By.ID, "body1")))
    print("Page is Ready!")
except TimeoutException:
    print("page took too long to load")


def get_house_links(url, driver, pages=3):
    house_links = []
    driver.get(url)
    for i in range(pages):
        soup = BeautifulSoup(driver.page_source, 'html.parser')
        listings = soup.find_all("a", class_="L")
        page_data = [row['href'] for row in listings]
        house_links.append(page_data)
        time.sleep(np.random.lognormal(0, 1))
        next_button = soup.find_all("a", class_="pageingBlock darkBorder")
        next_button_link = ['http://property.shw.co.uk'+row['href'] for row in next_button]
        if i < 3:
            driver.get(next_button_link[0])
    return house_links
get_house_links(url, driver)

Answer 1

class_="pageingBlock darkBorder" 也匹配上一页按钮，所以next_button_link[0] 将您送回上一页。您需要更精确的定位器

next_button = soup.select('img[src*="propNext"]')
if next_button:
    next_button = next_button[0].find_parent('a')
    next_button_link = 'http://property.shw.co.uk' + next_button['href']
    driver.get(next_button_link)