使用 Python 和 Selenium 进行 Web 抓取：寻找加快代码速度的方法（现在非常缓慢且效率低下）

Question

作为 Python 和 Selenium 的初学者，我尝试编写自己的脚本来从博彩网站 (Winamax) 抓取数据并将其保存到数据集中。我的脚本使用 Selenium 模块的 find_elements_by_tag_name('div') 函数来查找所有带有 div 标签名称的元素。在我的情况下，搜索 div 而不是 scan（如第一次推荐，请参阅 this discussion）更方便，因为结果列表的一个元素收集了我需要的表的所有信息。我基本上需要比赛的日期（在元素的开头，例如 MERCREDI 6 JANVIER）、球队（例如 Brest-Nice）、三个投注每场比赛的费率和比赛的时间。这是表格一部分的打印屏幕：

这是我想要获取的数据框： 这种方法的缺点是有很多div标签，这使得我的代码很慢而且效率很低。根据我下面的代码，有什么方法可以加快我的代码，同时保持相同的结果字符串列表？

我在我的代码中包含了一些print() 以查看所有带有div 标签的元素。没有打印的模型大约需要 2 分钟，打印的模型大约需要 4 分钟。

from selenium import webdriver
import re
import time

start = time.time()

#Load script from website
option = webdriver.ChromeOptions()
option.add_argument('--headless')
option.binary_location = r'C:/Program Files (x86)/Google/Chrome/Application/chrome.exe'

end1 = time.time()

browser = webdriver.Chrome(executable_path=r'C:/webdrivers/chromedriver.exe', options=option)
browser.get(r"https://www.winamax.fr/paris-sportifs/sports/1/7/4")

end2 = time.time()

#Select the part of the script where data are stored
span_tags = browser.find_elements_by_tag_name('div')

end25 = time.time()

tags = []
for span_tag in span_tags:
    tags.append(span_tag.text)
    print(span_tag.text)
    print("----------------------------------------------------------------")
print(len(tags))

end3 = time.time()

#Select the element of the list that describe the data we need
pattern = re.compile(r'(?:LUNDI|MARDI|MERCREDI|JEUDI|VENDREDI|SAMEDI|DIMANCHE)')
my_list5 = list(filter(pattern.match,tags))

#Print the indices of the selected element
print([i for i, j in enumerate(tags) if pattern.match(j)])

print("-------------------------------------------------------------------------------------------------------")

#Print the selected element
for m in my_list5:
    print(m)
    print("----------------------------------------------------------------")

#Select the first element, as it gathers all the necessary information
string=my_list5[0]
string2=string.splitlines()

#Print the executing times between the different tasks
print(end1-start)
print(end2-end1)
print(end25-end2)
print(end3-end25)

Answer 1

        import datetime

        from selenium import webdriver
        from selenium.webdriver.chrome.options import Options
        from selenium.webdriver.common.by import By
        from selenium.webdriver.support.wait import WebDriverWait
        from selenium.webdriver.support import expected_conditions as EC

        url = 'https://www.winamax.fr/paris-sportifs/sports/1/7/4'

        options = Options()
        options.add_argument('--no-sandbox')
        options.add_argument('--ignore-certificate-errors')

        driver = webdriver.Chrome(executable_path='venv/chromedriver-Darwin', options=options)

        driver.get(url)
        start = datetime.datetime.now()
    
        # get all divs containing the day and teams
        wait = WebDriverWait(driver, 10)
        teams_elements = wait.until(EC.presence_of_all_elements_located((By.XPATH, './/div[@class=\'ReactVirtualized__Grid__innerScrollContainer\']/div')))
    
        # declare the current day
        current_day = ''
        try:
            for elem in teams_elements:
                if len(elem.find_elements_by_css_selector('.sc-plgXW.lfShEr')) == 1:
                    print('============================================================================')
                    # if the current div is actually a day displayed, update the variable and use it until the next one is found
                    current_day = elem.find_element_by_css_selector('.sc-plgXW.lfShEr').text
                else:
                    # get playing teams text
                    playing_teams = elem.find_element_by_css_selector('.sc-prPLn.bKrOhT').text
    
                    # get playing hour text
                    playing_hour = elem.find_element_by_css_selector('.sc-qQlyE.jlXdeO').text
    
                    # get the betting odds 
                    odds = elem.find_elements_by_css_selector('.sc-pKLCU.WflrK button')
    
                    odds_string = ''
                    # for each odd print the odd value (1,2 or N) and the actual odd
                    for odd_elem in odds:
                        odd_number = odd_elem.find_element_by_css_selector('.sc-qPIWj.cQreJJ').text
                        odd_value = str(odd_elem.text).split('\n')[0]
                        odds_string += f'{odd_number} - {odd_value}; '
    
                    print(f'Day: {current_day} \n'
                          f'Teams playing: {playing_teams} \n'
                          f'Time: {playing_hour} \n'
                          f'Odds: {odds_string}'
                          f'\n')
        except:
            pass
    
        end = datetime.datetime.now()
        print(f'It took: {(end-start).seconds} seconds')

        driver.quit()

尝试将 cmets 添加到所有内容中，希望它更清晰。

这应该显示如下数据：

============================================================================
Day: MERCREDI 6 JANVIER 
Teams playing: Brest - Nice 
Time: 20:00 
Odds: 1 - 2,15; N - 3,65; 2 - 3,10; 

Day: MERCREDI 6 JANVIER 
Teams playing: Lorient - Monaco 
Time: 20:00 
Odds: 1 - 4,40; N - 4,10; 2 - 1,75; 

#the rest of them here

============================================================================
Day: SAMEDI 9 JANVIER 
Teams playing: Paris SG - Brest 
Time: 22:00 
Odds: 1 - 1,16; N - 8,25; 2 - 16; 

在时间方面，最后一个print() 说：

It took: 3 seconds

注意： 它没有考虑PARIS SUR LA COMPÉTITION 部分，但是如果您也需要它，也应该很容易更新代码以获取它。