Selenium Python - 获取表格数据而不是 JavaScript 代码

Question

我需要一些关于数据抓取任务的帮助：https://soilhealth.dac.gov.in/NewHomePage/NutriPage 我设法填写下拉菜单并使用此代码单击查看：

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from bs4 import BeautifulSoup

url = "https://soilhealth.dac.gov.in/NewHomePage/NutriPage"
driver = webdriver.Chrome(executable_path='./chromedriver.exe')
driver.get(url)

select = Select(driver.find_element_by_id('NutriCatId'))
select.select_by_visible_text('Sample Wise')
select = Select(driver.find_element_by_id('CycleId'))
select.select_by_visible_text('All Cycle')
select = Select(driver.find_element_by_id('State_Code'))
select.select_by_visible_text('Andaman And Nicobar Islands')
driver.implicitly_wait(5)
select = Select(driver.find_element_by_id('District_Code'))
select.select_by_visible_text('Nicobars')
driver.find_element_by_id('s').click()
driver.implicitly_wait(30)
soup_level1=BeautifulSoup(driver.page_source, 'lxml')

我需要从源代码中抓取表数据，而不是将其放在soup_level1 xml 中，我只有javascript 代码。 了解是否可以使用 Selenium 抓取数据的任何帮助都是可能的，我该怎么做会很糟糕。 感谢您的帮助。

Answer 1

嘿，下面的代码可以完成这项工作。但是它很慢，因为表很大并且需要一些时间来解析。我观察到该报告有一个可用的导出选项，因此请尝试使用 Selenium 直接下载它。哦，为了解释，报告是作为与页面的默认源不同的 iframe 生成的，因此您需要切换到该框架以获取信息。请让我知道任何澄清。所需数据在 df 变量中。

# -*- coding: utf-8 -*-
"""
Created on Tue Mar 24 11:08:32 2020

@author: prakh
"""

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from bs4 import BeautifulSoup
import pandas as pd
import time

url = "https://soilhealth.dac.gov.in/NewHomePage/NutriPage"
driver = webdriver.Chrome(executable_path='C:/Users/prakh/Documents/PythonScripts/chromedriver.exe') 
driver.get(url)

select = Select(driver.find_element_by_id('NutriCatId'))
select.select_by_visible_text('Sample Wise')
select = Select(driver.find_element_by_id('CycleId'))
select.select_by_visible_text('All Cycle')
select = Select(driver.find_element_by_id('State_Code'))
select.select_by_visible_text('Andaman And Nicobar Islands')
driver.implicitly_wait(5)
select = Select(driver.find_element_by_id('District_Code'))
select.select_by_visible_text('Nicobars')
driver.find_element_by_id('s').click()
driver.implicitly_wait(30)
#soup_level1=BeautifulSoup(driver.page_source, 'lxml')

#src = driver.find_element_by_xpath('//*[@id="report"]/iframe').get_attribute("src")

driver.switch_to.frame(driver.find_element_by_xpath('//*[@id="report"]/iframe'))
time.sleep(10)

html = driver.page_source
df_list = pd.read_html(html)
df = df_list[-3]

driver.quit()