写入硒数据时为空 CSV

Question

我正在第一次体验 Selenium，并在一个著名的视频平台上做一些教程。它在大多数情况下都非常可靠。但是，我在创建 CSV 但没有导出数据的几页上遇到了问题。 CSV 被“触摸”，但它不会导出以正常打印显示的数据。

谁能帮我解决这个脚本的问题？

#_*_coding: utf-8_*_


from selenium import webdriver
import selenium
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import csv
import os

os.chdir("C:\Selenium")
PATH = "chromedriver.exe"
driver = webdriver.Chrome(PATH)

driver.get("https://twitter-trends.iamrohit.in/")

try: 
    main = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, "panel-body"))
    )
 
    main = (main.text)
 
    f = open('twitter.csv', 'wb')
    print(main, file = f)
    f.close()
    
    #print(main)

except:
    driver.quit()

driver.quit()

Python 版本 3.7.4，Selenium 版本 3.141.0，Windows 10

Answer 1

要抓取 Twitter Trends - Worldwide 表，您可以使用 Python Pandas 中的 DataFrame 并使用以下 @987654322 将其写入 csv 文件@：

代码块：

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import pandas as pd

driver.get("https://twitter-trends.iamrohit.in/")
driver.execute_script("return arguments[0].scrollIntoView(true);", WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.XPATH, "//b[text()='Note:']"))))
headers = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "thead > tr > th")))]
ranks = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@id='twitter-trends']//tbody//tr//descendant::th[1]")))]
topics = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@id='twitter-trends']//tbody/tr//descendant::th[2]/a")))]
volumes = [my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@id='twitter-trends']//tbody/tr//descendant::th[3]")))]
df = pd.DataFrame(data=list(zip(ranks, topics, volumes)), columns=headers)
df.to_csv(r'C:\Data_Files\output_files\twitter.csv', index=False)
driver.quit()

CSV 快照：

---

参考文献

您可以在以下位置找到一些相关的详细讨论：

• Pandas DF.output write to columns (current data is written all to one row or one column)
• Selenium: Web-Scraping Historical Data from Coincodex and transform into a Pandas Dataframe
• Scraping and writing the table into dataframe shows me TypeError

Answer 2

调试您的代码，您可以正确获取数据。

   main = (main.text)
   print(main)
   f = open('twitter.csv', 'wb')

所以错误是在您写入输出文件时。替换你的代码

main = (main.text)

with open('twitter.txt', 'wb', encoding='utf-8') as file1:
# Writing data to a file
    file1.writelines(main)

会起作用，如果你检查打印你有中文字符会导致输出文件中的写入失败。