如何使 Selenium 只单击一个按钮而没有别的？不一致的点击

Question

我的目标：收集用户在可汗学院完成的项目数量。

为此，我需要解析个人资料用户页面。但我需要点击show more 来查看用户所做的所有项目，然后将它们刮掉。

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException,StaleElementReferenceException
from bs4 import BeautifulSoup

# here is one example of a user
driver = webdriver.Chrome()
driver.get('https://www.khanacademy.org/profile/trekcelt/projects')

# to infinite click on show more button until there is none
while True:
    try:
        showmore_project=WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME,'showMore_17tx5ln')))
        showmore_project.click()
    except TimeoutException:
        break
    except StaleElementReferenceException:
        break

# parsing the profile
soup=BeautifulSoup(driver.page_source,'html.parser')

# get a list of all the projects
project=soup.find_all(class_='title_1usue9n')

# get the number of projects
print(len(project))

此代码为print(len(project)) 返回0。这不正常，因为当您手动检查 https://www.khanacademy.org/profile/trekcelt/projects 时，您会看到那里的项目数量肯定不是 0。

奇怪的事情：首先，你可以看到（使用 webdriver）这段代码运行良好，然后 selenium 点击了 show more button 以外的其他东西，它点击了项目的链接之一，因此发生了变化这就是我们得到0的原因。

我不明白如何更正我的代码，所以 selenium 只点击右键，没有别的。

Answer 1

查看以下实现以获得所需的行为。脚本运行时，仔细查看滚动条以查看进度。

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup

with webdriver.Chrome() as driver:
    wait = WebDriverWait(driver,10)
    driver.get('https://www.khanacademy.org/profile/trekcelt/projects')

    while True:
        try:
            showmore = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR,'[class^="showMore"] > a')))
            driver.execute_script("arguments[0].click();",showmore)
        except Exception:
            break

    soup = BeautifulSoup(driver.page_source,'html.parser')
    project = soup.find_all(class_='title_1usue9n')
    print(len(project))

另一种方法是：

while True:
    try:
        showmore = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR,'[class^="showMore"] > a')))
        showmore.location_once_scrolled_into_view
        showmore.click()
        wait.until(EC.invisibility_of_element_located((By.CSS_SELECTOR,'[class^="spinnerContainer"] > img[class^="loadingSpinner"]')))
    except Exception:
        break

此时输出：

381

Answer 2

我已修改接受的答案以提高脚本的性能。在代码中评论如何实现它

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException, StaleElementReferenceException
from bs4 import BeautifulSoup
import time
start_time = time.time()
# here is one example of a user
with webdriver.Chrome() as driver:
    driver.get('https://www.khanacademy.org/profile/trekcelt/projects')
    # This code will wait until the first Show More is displayed (After page loaded)
    showmore_project = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME,
                                                                                       'showMore_17tx5ln')))
    showmore_project.click()
    # to infinite click on show more button until there is none
    while True:
        try:
            # We will retrieve and click until we do not find the element
            # NoSuchElementException will be raised when we reach the button. This will save the wait time of 10 sec
            showmore_project= driver.find_element_by_css_selector('.showMore_17tx5ln [role="button"]')
            # Using a JS to send the click will avoid Selenium to through an exception where the click would not be
            # performed on the right element.
            driver.execute_script("arguments[0].click();", showmore_project)
        except StaleElementReferenceException:
            continue
        except NoSuchElementException:
            break

    # parsing the profile
    soup=BeautifulSoup(driver.page_source,'html.parser')

# get a list of all the projects
project=soup.find_all(class_='title_1usue9n')

# get the number of projects
print(len(project))
print(time.time() - start_time)

执行时间1：14.343502759933472
执行时间2：13.955228090286255

希望对您有所帮助！