使用 xpath selenium 查找元素

Question

我关注这个tutorial 是为了学习如何构建一个网络爬虫来获取工作列表。现在我正在尝试跟随另一个网站。我遇到了我不知道如何提取各个职位列表的链接的问题。

当我检查页面时，我找到了我需要的元素

当我复制 xpath 并在我的代码中使用它时，我得到一个错误。我究竟做错了什么？

import selenium 
base_url = "https://www.nationalevacaturebank.nl"     
start_url = "https://www.nationalevacaturebank.nl/vacature/zoeken?query=&location=&distance=city&limit=100&sort=relevance&filters%5BcareerLevel%5D%5B%5D=Starter&filters%5BeducationLevel%5D%5B%5D=MBO" 
driver = webdriver.Firefox()    
elem = driver.find_element_by_xpath("//*[@id="search-results-container"]/div/div[1]/div[2]/article/job/a")

>>NoSuchElementException: Message: Unable to locate element: //*[@id="search-results-container"]/div/div[1]/div[2]/article/job/a

Answer 1

首先，您应该连接到您的网站...

其次，你应该使用waits你可以阅读它here

代码应该类似于：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC 


base_url = "https://www.nationalevacaturebank.nl"     
start_url = "https://www.nationalevacaturebank.nl/vacature/zoeken?query=&location=&distance=city&limit=100&sort=relevance&filters%5BcareerLevel%5D%5B%5D=Starter&filters%5BeducationLevel%5D%5B%5D=MBO" 
my_xpath = '//*[@id="search-results-container"]/div/div[1]/div[2]/article/job/a'
driver = webdriver.Firefox()
driver.get(start_url)  
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.XPATH, my_xpath)))

编辑

我为了获取各个职位列表的所有链接，您可以创建链接列表并附加主题：

wait = WebDriverWait(driver, 10)
elements = wait.until(EC.presence_of_all_elements_located((By.XPATH, '//*[@id="search-results-container"]//article/job/a')))
list_of_links = []
for i in elements:
    list_of_links.append(i.get_attribute('href'))
    # print(f"link = {i.get_attribute('href')}")
print(list_of_links)

希望对你有帮助！