Python：抓取工作门户网站

Question

我刚开始学习如何使用 Python 来探索抓取工作门户网站 - 所以请多多包涵，因为我可能会问一些非常基本的问题。

情况： 我设法建立了以下几行

import pandas as pd
from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Chrome('C:/Users/ - Home/Desktop/Web Scraper/chromedriver.exe')
driver.get('https://www.mycareersfuture.gov.sg/search?sortBy=relevancy&page=0')

results =[]
content = driver.page_source
soup = BeautifulSoup(content, 'html.parser')
listing= soup.find('div', class_ = 'card-list')
job = listing.find('p')
print(job)

并发症：我似乎无法从工作卡中提取以下项目：

1. 职务
2. 公司名称
3. 工资

   我查看了几个教程，每个教程都表示要查找具有相应类的 h2 标签或 div。但是，我正在抓取的网站似乎没有明确说明这一点。

   网站链接：https://www.mycareersfuture.gov.sg/search?sortBy=relevancy&page=0

   例如，我检查了 HTML 并发现职位名称在这一行的某处；但是，我似乎无法提取它。

   <span data-cy="job-card__job-title" class="f4-5 fw6 mv0 dib mr2 brand-sec JobCard__jobtitle___3HqOw" style="overflow-wrap: break-word;">2402 - IT Manager [ Amber Rd /   /  5 days ]</span>
   

   我真的很感激这方面的任何帮助。我整晚都在研究解决方案，但无济于事......

Answer 1

可能的解决方案之一：

import csv
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC


options = webdriver.ChromeOptions()
# set headless mode
# options.add_argument("--headless")
# disable chromedriver log message in cmd
options.add_experimental_option("excludeSwitches", ["enable-automation", "enable-logging"])

service = Service(executable_path='path	oyourchromedriver.exe')
driver = webdriver.Chrome(service=service, options=options)

# set an explicit wait (10 sec)
wait = WebDriverWait(driver, 10)

url = 'https://www.mycareersfuture.gov.sg/search?sortBy=relevancy&page=0'

# page where parsing will stop
last_page = 5
# loads a web page
driver.get(url)

while True:
    # waiting(max 10 sec) for least one element with our css selector present on a web page.
    company_names = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'p[data-testid="company-hire-info"]')))
    job_titles = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'span[data-cy="job-card__job-title"]')))
    salaries = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, 'div[class="lh-solid"]')))
    
    # get data from received web elements
    for data in zip(company_names, job_titles, salaries):
        data = {
            'Company name': data[0].text,
            'Job title': data[1].text,
            'Salary': data[2].text
        }
        # save received data in csv
        with open(file='mycareersfuture.csv', mode='a', encoding="utf-8") as f:
            writer = csv.writer(f, lineterminator='
')
            writer.writerow([data['Company name'], data['Job title'], data['Salary']])

    # waiting for an element is present on the DOM of a page. after that click on it
    wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'button[aria-label="Next"]'))).click()
    # if the current page is equal to the last_page stop parsing
    if driver.current_url.endswith(str(last_page)):
        break

driver.quit()

输出 mycareersfuture.csv：

THE SUPREME HR ADVISORY PTE. LTD.,2402 - IT Manager [ Amber Rd / / 5 days ],$6 500to$7 000
TRITON AI PTE. LTD.,"Property Executive, Town Council (Facilities Management)",$2 000to$3 000
PISTACHIO RESTAURANT PTE. LTD.,Service Crew / Supervisor,$1 700to$3 000
THE SUPREME HR ADVISORY PTE. LTD.,2402 - Quantity Surveyor [ Admiralty / 5 days ],$3 000to$3 500
THE SUPREME HR ADVISORY PTE. LTD.,2402 - WSH Co-ordinator [ 5 days / WSQ Advanced Cert ],$2 200to$3 500