如何在网络抓取期间修复 python 中的页面循环？答案

【问题标题】：How to fix a page loop in python during webscraping?如何在网络抓取期间修复 python 中的页面循环？
【发布时间】：2019-12-24 17:04:49
【问题描述】：

我正在尝试遍历每一页，但是一旦到达页面末尾，它就会跳过所需的行。页面因链接而异。所以我需要一个网页数量的动态解决方案。这是一个工作示例，因此结果将显示在运行中。 stackoverflow 需要我添加更多细节

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from time import sleep
driver=webdriver.Chrome()
driver.maximize_window()
driver.get("https://www.oddsportal.com")
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.LINK_TEXT,"BASKETBALL"))).click()
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.LINK_TEXT,"Europe"))).click()
WebDriverWait(driver,15).until(EC.element_to_be_clickable((By.LINK_TEXT,"Euroleague"))).click()
WebDriverWait(driver,10).until(EC.element_to_be_clickable((By.LINK_TEXT,"RESULTS"))).click()

allyears=WebDriverWait(driver,20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"div.main-menu2.main-menu-gray >ul.main-filter a[href^='/basketball/europe/euroleague']")))
allelements=WebDriverWait(driver,15).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"td.name.table-participant >a[href^='/basketball/europe/euroleague/']")))
max_page= 10
scores=[]
games=[]
#Get the all year text of link in a list.
alltext=[ele.text for ele in allyears]
allyearslink=[ele.get_attribute('href') for ele in allyears]
for link in allyearslink:
    driver.get(link)

    url = driver.current_url
    print(url)
    for j in range(1, max_page + 1):
        current_page = url + '#/page' + str(j)
        driver.get(current_page)
        print(current_page)
        for i in range(3):
            allelements = WebDriverWait(driver, 15).until(EC.visibility_of_all_elements_located(
                (By.CSS_SELECTOR, "td.name.table-participant >a[href^='/basketball/europe/euroleague']")))
            print(allelements[i].text)

            scores.append(allelements[i].text)
            games.append(allelements[i].text)

            driver.execute_script("arguments[0].click();", allelements[i])

            sleep(2)
            elem1 = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.LINK_TEXT, "AH"))).click()
            sleep(2)
            # .date
            date_ofGame = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".date")))
            print(date_ofGame.text)
            elem2 = driver.find_element_by_id("odds-data-table")
            scores.append(date_ofGame.text)
            scores.append(elem2.text)
            driver.back()
            sleep(2)
            driver.back()

results:
https://www.oddsportal.com/basketball/europe/euroleague/results/
Lyon-Villeurbanne - Alba Berlin
Friday, 20 Dec 2019, 13:45
Valencia - Khimki M.
Thursday, 21 Nov 2019, 14:00
Olimpia Milano - Fenerbahce
Friday, 25 Oct 2019, 13:45
https://www.oddsportal.com/basketball/europe/euroleague-2018-2019/results/
https://www.oddsportal.com/basketball/europe/euroleague-2017-2018/results/
https://www.oddsportal.com/basketball/europe/euroleague-2016-2017/results/
https://www.oddsportal.com/basketball/europe/euroleague-2015-2016/results/
https://www.oddsportal.com/basketball/europe/euroleague-2014-2015/results/
https://www.oddsportal.com/basketball/europe/euroleague-2013-2014/results/
https://www.oddsportal.com/basketball/europe/euroleague-2012-2013/results/
https://www.oddsportal.com/basketball/europe/euroleague-2011-2012/results/
etc....

想要的结果：

 https://www.oddsportal.com/basketball/europe/euroleague/results/
        Lyon-Villeurbanne - Alba Berlin
        Friday, 20 Dec 2019, 13:45
        Valencia - Khimki M.
        Thursday, 21 Nov 2019, 14:00
        Olimpia Milano - Fenerbahce
        Friday, 25 Oct 2019, 13:45
    https://www.oddsportal.com/basketball/europe/euroleague-2018-2019/results/
        Lyon-Villeurbanne - Alba Berlin
        Friday, 20 Dec 2019, 13:45
        Valencia - Khimki M.
        Thursday, 21 Nov 2019, 14:00
        Olimpia Milano - Fenerbahce
        Friday, 25 Oct 2019, 13:45
    https://www.oddsportal.com/basketball/europe/euroleague-2016-2017/results/
        Lyon-Villeurbanne - Alba Berlin
        Friday, 20 Dec 2019, 13:45
        Valencia - Khimki M.
        Thursday, 21 Nov 2019, 14:00
        Olimpia Milano - Fenerbahce
        Friday, 25 Oct 2019, 13:45

【问题讨论】：

不要吞下异常
@CoreyGoldberg 我应该把它放在哪里？还是有更好的方法？
扩展@CoreyGoldberg 评论。无论如何，吞下异常对您没有帮助。了解这一点的最佳方式是MemoryError。如果发生这样的错误，接下来可能会发生以下两种情况之一，要么您的程序崩溃，要么您删除对象引用，以便将它们从内存中丢弃。你可以吞下错误，就像你正在做的那样，但你的程序会崩溃，此时不会有任何迹象表明原因（指示是被吞下的异常）。因此，仅捕获您实际上正在处理的异常，并考虑重新引发该异常。
@PedroRodrigues 我正在吞下错误，因为我需要页面在最后停止并从明年开始重置。当它进入下一年时，页面循环不会重置。我同意这可能不是最好的方法。
然后在应该完成的地方捕获一个特定的错误，而不是基本异常。因为你正在处理MemoryError 和其他所有事情都以同样的方式。可能发生的实际错误消失在虚空中。这类似于使用生成器，您应该调用next() 直到引发StopIteration；或使用 for 循环来为您执行此操作。

标签： python selenium for-loop web-scraping

【解决方案1】：

您可以通过如下替换循环来获得所需的输出。

for link in allyearslink:
    driver.get(link)
    url = driver.current_url
    print(url)
    # click on the last page button
    driver.find_element_by_xpath("(//div[@id='pagination']//span)[last()]").click()
    time.sleep(3) # we can handle this better
    max_page = int(driver.find_element_by_class_name('active-page').text)

    ##################### This is where I believe my problem is at ######################
    for j in range(1, max_page + 1):
        current_page = url + '#/page/' + str(j)
        driver.get(current_page)

        for i in range(3):
            allelements = WebDriverWait(driver, 15).until(EC.visibility_of_all_elements_located(
                (By.CSS_SELECTOR, "td.name.table-participant >a[href^='/basketball/europe/euroleague']")))
            print(allelements[i].text)

            scores.append(allelements[i].text)
            games.append(allelements[i].text)

            driver.execute_script("arguments[0].click();", allelements[i])

            time.sleep(2)
            elem1 = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.LINK_TEXT, "AH"))).click()
            time.sleep(2)
            # .date
            date_ofGame = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".date")))
            print(date_ofGame.text)
            elem2 = driver.find_element_by_id("odds-data-table")
            scores.append(date_ofGame.text)
            scores.append(elem2.text)
            driver.back()
            time.sleep(2)
            driver.back()

您收到错误的原因是/ 在td.name.table-participant >a[href^='/basketball/europe/euroleague/'] 的末尾。

这是示例输出：

【讨论】：

我在页面上出错了，有没有办法通过标签将最大页面修复为所需页面的长度？我更改了上面的代码以显示我正在运行的内容
是否要获取基于年份的页数并遍历所有页面并从每个页面中获取前 3 条记录？
for i in range(len(allelements)):.... 是的，通过所有年份和所有可用页面浏览所有游戏。我只是使用范围三进行测试。
好的。让我根据您的说明更新答案。
然后从亚洲让分盘中抓取数据以放入数据框中，但表格中缺少数据。所以在追加列出它的问题时。