【发布时间】：2023-03-05 19:04:01
【问题描述】：
我目前有一个 selenium 函数，它执行以下代码摘要：
def (list):
FOR LOOP in list: # Page A (initial), Contains 12
requests,bs4 grabs element coordinates.
[f''string transforms into CSS selector]. # this is the list and loops through this
selenium.driver opens, detect and selects that element
FOR LOOP in [f'string...']: # Page B:, Contains 1
Driver.current url, used to prepare new elements to be detected
requests,bs4 grabs element coordinates. # this is list and loops through this
f''string transforms into CSS selector.
selenium.driver opens, detect and selects that element
download beginds.
sleep for .5 sec
driver goes back to previous page.
现在，我的问题是，在可预测的迭代中，特别是当 for 循环 B 位于列表中的 6/12 元素上时，它会崩溃并出现以下错误代码：
'//OBJECT//' is not clickable at point (591, 797). Other element would receive the click: <div style="position: relative" class="cookie-consent-inner">...</div>
(Session info: MicrosoftEdge=...)
Stacktrace:
Backtrace:
...
现在我这样做没有任何问题，但我希望它会继续到 PAGE B 7/12 等等，因为它确实有 Driver.back()。而是应用程序停止。
我尝试使用 try and except: PASS 来封装整个内容，以捕获此错误。但是，它从页面 A 开始，仍然错过了其余部分。
我想要一种方法，我可以在某处以某种方式执行“继续”声明，但我才刚刚开始学习，而且我的想法已经用完了。您可以在原始代码中看到我尝试执行 FOR IF: ERROR 语句以希望通过，但这似乎是语法错误。请参阅下面的原始代码：
import concurrent.futures
import os
import time
import requests
import re
import selenium.common.exceptions
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.common.by import By
import multiprocessing
edge_driver = 'C:selenium\\webdriver\\edge'
os.environ['PATH'] += edge_driver
web_links = {'digital archive': 'https://digital.nmla.metoffice.gov.uk/SO_1118bfbb-f2c9-476f-aa07-eb58b6db5ce6/', }
def scraping_bot(css_selector):
# First stage: Years
print('FIRST STAGE INITIATED....')
driver = webdriver.Edge()
driver.get(web_links.get('digital archive'))
year_args = (By.CSS_SELECTOR, f'a[href="{css_selector}"]')
driver.find_element(*year_args).click()
# Second Stage: Months
print('SECOND STAGE INITIATED....')
sTWO_url = driver.current_url
sTWO_site = requests.get(sTWO_url)
sTWO_web_objects = BeautifulSoup(sTWO_site.text, 'lxml')
monthly_placeholders = sTWO_web_objects.find(name='div', attrs={'class': 'twelve columns last results'})
months = monthly_placeholders.find_all(name='h5')
month_css_selector = {}
for month_href_tags in months:
month_tag = f'{month_href_tags.get_text()}'
month_hrefs = re.findall(regex, str(month_href_tags))
for month_href in month_hrefs:
month_css_selector.update({month_tag: month_href})
for v, y in zip(month_css_selector.values(), month_css_selector.keys()):
print(v) ##############################
month_args = (By.CSS_SELECTOR, f'a[href="{v}/"]')
driver.find_element(*month_args).click()
# Third Stage: Download
print(f'THIRD STAGE INITIATED for: {y}: {v}')
sTWO_url = driver.current_url
download_site = requests.get(sTWO_url)
content = BeautifulSoup(download_site.text, 'lxml')
nav_controls = content.find_all('nav')
download_button = [nav_controls.find(attrs={'title': 'download'}) for nav_controls in nav_controls]
download_regex = r'(?<=href=\").{1,}(?=\" title)'
for button in download_button:
if button is not None:
print(button) ##############################
downl = re.findall(download_regex, str(button))
if len(downl) == 1:
for downl_button in downl:
download_args = (By.CSS_SELECTOR, f'a[href="{downl_button}"]')
driver.find_element(*download_args).click()
time.sleep(2)
print(f'THIRD STAGE DOWNLOAD COMPLETE: {y}; {v}')
##### END OF TREE HERE ####
driver.back() # goes back to Second Stage and so on
else:
print(f'Your download button matches exceeds 1: {len(downl)}')
if selenium.common.exceptions.ElementClickInterceptedException:
continue
if __name__ == '__main__':
sONE_url = requests.get(web_links.get('digital archive'))
sONE_web_objects = BeautifulSoup(sONE_url.text, 'lxml')
year_placeholder = sONE_web_objects.find(name='div', attrs={'class': 'sixteen columns results-and-filters'})
years = year_placeholder.find_all(name='div', attrs={'class': ['one_sixth grey_block new-secondary-background result-item',
'one_sixth grey_block new-secondary-background result-item last']}) # don't skip, needed for titles.
unit = [years.find('h5') for years in years]
regex = r'(?<=href=\").{1,}(?=\/")' # lookaround = PositiveLookBehind...PositiveLookAhead
year_css_selector = []
titles = [years.get('title') for years in years]
for year_href_tags, year_tag in zip(unit, titles): # href_tag -> bs4 component
hrefs = re.findall(regex, str(year_href_tags.get_text)) # href_tag.get_text -> method that enables str.
for year_href in hrefs:
year_css_selector.append(f'{year_href}/')
for i in year_css_selector:
scraping_bot(i)
因此，我希望我的预期输出能够通过或继续跳过这个错误的网页，我可以在其中手动下载。
【问题讨论】：
-
您的“原始代码”中有一些缩进错误。我想确保我理解：你有两个嵌套循环。外循环开始 `for v, y in zip(month_css_selector.values()...` 内循环开始
for button in download_button:。您是否在内循环中遇到异常并且您想继续内循环的下一次迭代循环？如果您修复缩进并使用 cmets 插入try/catch来确定您想要恢复的位置，这会有所帮助。此外，多处理也不合适，因为每个驱动程序都已经是一个进程。多线程最终将是一个更好的选择。 -
但是使用多线程池很棘手，因为您将不必要地创建和销毁太多驱动程序进程而不是重用它们（否则，使用池而不是单个
Thread实例有什么意义?) 或者你会想出一种方法来重用 N 个线程池来处理 M > N 的 M 个任务，但让 N 个驱动程序保持打开状态。准备好后请参阅this。 -
@Booboo，据我所知，缩进是完美的。我使用了 try and except 之前，但这会停止应用程序，并在 if_name==main FOR 循环上开始，并且错过了完成“内部”和“外部”循环。我相信我的问题现在很复杂而且很长。所以我会听取你关于多线程的建议，以及答案
标签： python selenium beautifulsoup multiprocessing