仅在 csv 中获取一个条目。 Python beautifulsoup，请求，硒

Question

试图抓取一些数据。用打印检查它并获得多张打印。但是，CSV 只有一个条目。你能帮忙吗？非常感谢。

import csv
import time
import requests
from bs4 import BeautifulSoup
from selenium.webdriver.chrome.options import Options
from selenium import webdriver


job_Details = []
job_links = []



chrome_options = Options()
'''chrome_options.add_argument("--headless")'''
driver = webdriver.Chrome(executable_path='C:/bin/chromedriver.exe', options=chrome_options)
driver.get(f'https://remotejobs.world/')
'''SCROLL_PAUSE_TIME = 20'''
last_height = driver.execute_script("return document.body.scrollHeight")
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
divs = driver.find_elements_by_tag_name('h2')
for div in divs:
    job_Details.append(div)
    link = div.find_element_by_tag_name('a')
    job_links.append(link)

for job_detail, job_link in zip(job_Details, job_links):
    if job_detail and job_link:
        print(job_link.get_attribute('href'))
        print(job_detail.text)
        url = job_link.get_attribute('href')
        new_page = requests.get(url).text
        time.sleep(2)
        soup = BeautifulSoup(new_page, 'html.parser')
        job_desc = soup.find('div', class_='w-full md:w-2/3')
        if job_desc:
            print(job_desc.text) #Successful Prints.
            dict = {'Job_title and Company': job_detail.text, "Job link": job_link.get_attribute('href'),
                        "Job Details": job_desc.text}
            with open('remoteWORLD.csv', 'w') as f:
                w = csv.DictWriter(f, dict.keys())
                w.writeheader()
                w.writerow(dict)

Answer 1

请看下面我的建议是多么简单。

with open('remoteWORLD.csv', 'w') as f:
    w = csv.DictWriter(f, ['Job_title and Company', "Job link", "Job Details"])
    w.writeheader()
    for job_detail, job_link in zip(job_Details, job_links):
        if job_detail and job_link:
            print(job_link.get_attribute('href'))
            print(job_detail.text)
            url = job_link.get_attribute('href')
            new_page = requests.get(url).text
            time.sleep(2)
            soup = BeautifulSoup(new_page, 'html.parser')
            job_desc = soup.find('div', class_='w-full md:w-2/3')
            if job_desc:
                print(job_desc.text) #Successful Prints.
                dict = {'Job_title and Company': job_detail.text, "Job link": job_link.get_attribute('href'),
                            "Job Details": job_desc.text}
                w.writerow(dict)

Answer 2

尝试使用：

with open('remoteWORLD.csv', 'a+') as f:

w 每次都会重写文件。 a 表示您可以附加（+ 表示它可以读取和写入）。

查看此处了解更多说明： Difference between modes a, a+, w, w+, and r+ in built-in open function?

编辑：或者正如贾斯汀所说，将其移出循环并将您的列表 job_Details 和 job_links 写入其中