selenium 和 python：如何单击 div 类的“OnClick”按钮并提取后续数据

Question

我想在 this 页面上使用 selenium。

我想抓取页面的步骤：

1. type '22663' into the box that says 'search by plant-based food'
2. click 'food-disease association
3. click submit on the bottom of the page
4. click 'plant-disease associations'
5. export the plant-disease table

我写了这段代码：

import sys
import pandas as pd
from bs4 import BeautifulSoup
import selenium
from selenium import webdriver
from selenium.webdriver.support.ui import Select
import csv
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary

#binary = FirefoxBinary('/Users/kela/Desktop/scripts/scraping/geckodriver')
url = 'http://147.8.185.62/services/NutriChem-2.0/'
driver = webdriver.Firefox(executable_path='/Users/kela/Desktop/scripts/scraping/geckodriver')
driver.get(url)

element = driver.find_element_by_id("input_food_name")
element.send_keys("22663")

#click food-disease association
element = driver.find_element_by_xpath("//select[@name='food_search_section']")
#all_options = element.find_elements_by_tag_name("option")

element = Select(driver.find_element_by_css_selector('[name=food_search_section]'))
element.select_by_value('food_disease')

submit_xpath = '/html/body/form/p[2]/input[1]'
destination_page_link = driver.find_element_by_xpath(submit_xpath)
destination_page_link.click()


#this doesn't work for step 4
#xpath2 = '/html/body/table/tbody/tr/td[3]/div'
#destination_page_link = driver.find_element_by_xpath(xpath2)
#destination_page_link.click()

#this doesn't work for step 4
xpath2 = '/html/body/table/tbody/tr/td[3]/div/span'
destination_page_link = driver.find_element_by_xpath(xpath2)
destination_page_link.click()

我在第 4 步和第 5 步中苦苦挣扎。

对于第 4 步， 如何选择 'div 类 -> onclick ClickButton (nutrichem12587_disease.tsv','plant_disease' 按钮？您可以看到我在上面的代码中尝试过的几件事其他 stackoverflow 问题，例如 here 和，我尝试了一些很好的东西，这是两个例子。

然后对于第 5 步，我已经可以预见到会遇到类似的问题，因为我想单击每一行的“展开/向右箭头”（例如箭头 beisde pomegranate/diabetes），并打印出其下方的数据，即

PredictionPMID:22919408 Punica granatum     Diabetes
PredictionPMID:22529479 P. granatum     Diabetes
PredictionPMID:22529479 Punica granatum     Diabetes
PredictionPMID:20020514 Punica granatum     Diabetes

对于随后的每一行。有人可以告诉我如何做到这一点。

编辑 1：对于第 4 步，我已经尝试过这样的事情，但它们返回错误说元素不存在，即使我通过复制 XPath 获得了位置：

#click plant-disease associations
#submit_xpath = '/html/body/table/tbody/tr/td[3]/div/span'
submit_xpath = '/html/body/table/tbody/tr/td[3]'
destination_page_link = driver.find_element_by_xpath(submit_xpath)
destination_page_link.click()

Answer 1

第 4 步

如果您确信网页每次都完全相同，您可以识别包含“植物-疾病关联”按钮的元素，然后手动单击该元素中的 (x, y) 坐标。 as described as the second answer here

第 5 步

尝试先舀整个桌子，而不是单独的向右箭头，然后通过识别所有孩子来手动遍历它。

Answer 2

对于第 4 步，您的代码可能无法正常工作，因为它没有等待页面加载。如果是这种情况，请添加以下导入语句：

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

我发现这个功能对浏览器自动化非常方便，您可以添加脚本：

def wait_for_element(driver, selector, method):
    """Returns element after waiting for page load"""
try:
    wait = WebDriverWait(driver, 10)
    wait.until(
        eval(f'EC.presence_of_element_located((By.{method}, "{selector}"))')
    )
finally:
    element = eval(f'driver.find_element_by_{method.lower()}("{selector}")')

    return element

使用以下方法实现它以查找第 4 步的按钮：

xpath2 = '/html/body/table/tbody/tr/td[3]/div'
destination_page_link = wait_for_element(driver, xpath2, 'XPATH')

希望这会有所帮助！