如何使用 Selenium 和 python 从 Tradingview 网站单击“加载更多”按钮？答案

【问题标题】：How do I click on the 'Load More' button from the Tradingview website using Selenium and python?如何使用 Selenium 和 python 从 Tradingview 网站单击“加载更多”按钮？
【发布时间】：2020-11-27 06:15:18
【问题描述】：

我正在尝试从 tradingview 网站查询所有行。我只需要股票代码和名称。我无法单击页面底部的“加载更多”按钮来加载所有行。有没有人可以解决这个问题？

这是我为获取股票代码和名称而编写的代码（在“加载更多”按钮之前一直有效）。

from bs4 import BeautifulSoup

URL = 'https://www.tradingview.com/markets/stocks-usa/sectorandindustry-industry/biotechnology/'
page = requests.get(URL)

soup = BeautifulSoup(page.content, 'html.parser')`

for tr in soup.find_all('tr'):
    #tds = tr.find_all('td')
    ticker = tr.find('a', class_='tv-screener__symbol')
    stock_name = tr.find('span', class_='tv-screener__description')
    if None in (ticker, stock_name):
        continue
    print(ticker.text.strip())
    print(stock_name.text.strip())
    print("\n\n")

这是我尝试为“加载更多”按钮编写但没有成功的 selenium 代码 -

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
#options.headless = True
options.add_argument("--window-size=1920,1200")

driver = webdriver.Chrome(options=options, executable_path=DRIVER_PATH)
driver.get("https://www.tradingview.com/markets/stocks-usa/sectorandindustry-industry/biotechnology/")
loadMoreButton = driver.find_elements_by_xpath("//span[@class='tv-load-more__btn']")
loadMoreButton.click()
driver.quit()

这是我得到的错误：

AttributeError                            Traceback (most recent call last)
<ipython-input-80-cf801ef16cdd> in <module>
      9 driver.get("https://www.tradingview.com/markets/stocks-usa/sectorandindustry-industry/biotechnology/")
     10 loadMoreButton = driver.find_elements_by_xpath("//div[@class='tv-load-more tv-load-more--screener js-screener-load-more']")
---> 11 loadMoreButton.click()
     12 driver.quit()

AttributeError: 'list' object has no attribute 'click'

【问题讨论】：

标签： python selenium selenium-webdriver web-scraping beautifulsoup

【解决方案1】：

问题是find_elements_by_xpath 这会返回一个列表。请参阅下面使用find_element_by_xpath。

from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
#options.headless = True
options.add_argument("--window-size=1920,1200")

driver = webdriver.Chrome(options=options, executable_path=DRIVER_PATH)
driver.get("https://www.tradingview.com/markets/stocks-usa/sectorandindustry-industry/biotechnology/")
loadMoreButton = driver.find_element_by_xpath("//span[@class='tv-load-more__btn']")
loadMoreButton.click()
driver.quit()

我建议使用By、WebDriverWait、expected_conditions 和element_to_be_clickable：

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.options import Options

options = Options()
#options.headless = True
options.add_argument("--window-size=1920,1200")

driver = webdriver.Chrome(options=options, executable_path=DRIVER_PATH)
driver.get("https://www.tradingview.com/markets/stocks-usa/sectorandindustry-industry/biotechnology/")
loadMoreButton = WebDriverWait(driver, 3).until(EC.element_to_be_clickable((By.XPATH, "//span[@class='tv-load-more__btn']")))
loadMoreButton.click()
driver.quit()

【讨论】：

【解决方案2】：

编写这段代码只是为了点击加载更多按钮直到整个页面加载完毕。此外，我总是更喜欢使用 XPath 而不是 class 名称来查找元素。

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver import ActionChains
from selenium.webdriver.common.by import By
import time

driver = webdriver.Chrome()
wait = WebDriverWait(driver, 5)
action = ActionChains(driver)

driver.get("https://www.tradingview.com/markets/stocks-usa/sectorandindustry-industry/biotechnology/")
# Used to get a click count
i = 0

while True:
    try:
        # Given some timeout so that data will be loaded in the page
        time.sleep(2)
        Load_More = wait.until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Load More']")))
        action.move_to_element(Load_More).click().perform()
        i += 1
        print(f"Clicked {i} time.")
    except:
        print("Reached End of the Page")
        break

【讨论】：