【问题标题】:Script fails to produce descriptions from a webpage when hardcoded delay is not in place当硬编码延迟未到位时,脚本无法从网页生成描述
【发布时间】:2022-01-08 05:15:47
【问题描述】:

我正在尝试从webpage 获取结果的标题和描述。单击标题时会显示描述。以下脚本仅在我在单击后定义硬编码延迟时才有效。但是,我希望摆脱脚本中的硬编码延迟。

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

link = 'https://innovation.ised-isde.canada.ca/s/list-liste?language=en_CA&token=a0B5W000000WsFSUA0'

driver = webdriver.Chrome()
wait = WebDriverWait(driver,10)
driver.get(link)

for i,item in enumerate(wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,".advanced-results .h4")))):
    driver.execute_script("arguments[0].click();",item)
    title = item.text
    time.sleep(2)
    desc = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,"p.program-dov-description")))[i].text
    print(title,desc)
    print()

driver.quit()

如何在不使用硬编码延迟的情况下从该网页获取标题和描述?

【问题讨论】:

  • 可见性是在元素文本显示时。

标签: python python-3.x selenium selenium-webdriver web-scraping


【解决方案1】:

这应该可以。问题是我不知道如何使用 css_selector 获取多个元素,所以我信任我的老朋友 xpath,所以我获取了结果,而没有按要求使用 time.sleep。你的这行代码我收到一个错误,指出由于[i],该元素不可下标,但我的 xpath 有效。

desc = wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,"p.program-dov-description")))[i].text
    print(title,desc)

这是重构后的代码:

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager

link = 'https://innovation.ised-isde.canada.ca/s/list-liste?language=en_CA&token=a0B5W000000WsFSUA0'

driver = webdriver.Chrome(ChromeDriverManager().install())
wait = WebDriverWait(driver,10)
driver.get(link)

for i,item in enumerate(wait.until(EC.presence_of_all_elements_located((By.CSS_SELECTOR,".advanced-results .h4")))):
    # print(i, item)
    driver.execute_script("arguments[0].click();",item)
    title = item.text
    desc_text = "//*[@class='program-dov-description']"
    desc = wait.until(EC.visibility_of_element_located((By.XPATH,'(' + desc_text + ')' + '[' + str(i+1) + ']' ))).text
    print(title,desc)
    print()

driver.quit()

输出(粘贴部分输出以节省空间)

Income support for sick or self-isolating workers Get up to $500 per week for six weeks if you are employed or self-employed and need to take time off because you’re sick or need to self-isolate due to COVID-19 or have an underlying medical condition that puts you at higher risks of getting COVID-19.

Income support to care for dependent family members Get up to $500 per week for up to 44 weeks, per household, if you haven’t been able to work for at least 50% of your normal work week because you are caring for children under 12 years old or other dependent family members because of COVID-19.

Support for businesses to avoid layoffs If your employees are eligible for Employment Insurance (EI) benefits, and your business is suffering a downturn due to COVID-19, you can apply for a work-sharing agreement. The agreement would allow your employees to work a temporarily reduced work week while receiving EI benefits.

【讨论】:

  • 虽然我不喜欢这种可怕的结构'(' + desc_text + ')' + '[' + str(i+1) + ']',但我发现它工作得很好。
  • 我同意@SMITH 我倾向于谨慎使用它,这个案例就是其中之一。
  • f"(//*[@class='program-dov-description'])[{i}]" 你可以用 f 字符串 {i} 代替。
【解决方案2】:
wait=WebDriverWait(driver,30)                                 
driver.get('https://innovation.ised-isde.canada.ca/s/list-liste?language=en_CA&token=a0B5W000000WsFSUA0')
elements=wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"div.cIC_DraggableList.cIC_Dovs > div")))
for x in elements:
    x.click()
    time.sleep(1)
title=[x.get_attribute("data-dovtitle") for x in wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"div.row.mh-0 button:nth-child(2)")))]
desc=[x.text for x in wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"p.program-dov-description")))]
    
print(list(zip(title,desc)))

所以只需点击并打开每一个并收集它们的标题和描述。

进口:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC

输出:

[('Canada Recovery Sickness Benefit', 'Get up to $500 per week for six weeks if you are employed or self-employed and need to take time off because you’re sick or need to self-isolate due to COVID-19 or have an underlying medical condition that puts you at higher risks of getting COVID-19.'), ('Canada Recovery Caregiver Benefit', 'Get up to $500 per week for up to 44 weeks, per household, if you haven’t been able to work for at least 50% of your normal work week because you are caring for children under 12 years old or other dependent family members because of COVID-19.'), ('Work-Sharing program — COVID-19', 'If your employees are eligible for Employment Insurance (EI) benefits, and your business is suffering a downturn due to COVID-19, you can apply for a work-sharing agreement. The agreement would allow your employees to work a temporarily reduced work week while receiving EI benefits.'), ('Workplace screening of employees', 'Get free rapid tests to help identify employees who may be infected with COVID-19 but have no symptoms. Rapid tests are safe, easy to administer and provide results in 15 minutes.'), ('Canada Emergency Rent Subsidy (CERS)', 'If your business has been impacted by COVID-19, you could get a retroactive rent subsidy that covers up to 65% of eligible expenses to a maximum of $75,000 per business location, and $300,000 in total for all locations, with additional amounts for temporary closures.'), ('BDC Co-Lending Program (BCAP)', 'If your business has been affected by COVID-19, you could get up to $12.5 million to support your operational cash flow, co-financed by BDC and your primary lending institution.'), ('Essential Services Contingency Reserve', 'If you’re an essential business and can’t get personal protective equipment for your workers through other means, you can purchase from Canada’s reserve on a cost-recovery basis.'), ('Accelerate HER PEI', "Take part in a 12-week pilot program (virtually or in person) that includes one-on-one mentorship, peer support, market research guidance and more. Learn how to grow your business whether you're in the early stage of development or taking your business in a new direction."), ('Indigenous Community Business Fund — Stream 2', 'Get a non-repayable contribution to cover eligible operating costs if your Indigenous business has experienced a significant reduction in revenues due to COVID-19.'), ('COVID-19: Waiving tariffs on certain medical goods', 'If your business imported certain medical items into Canada on or after May 5, 2020, including PPE (masks, gloves, garments, sterilization products), you will not have to pay tariffs on those items.'), ('Mortgage Payment Deferral', 'If you have experienced financial hardship due to COVID-19, you may be eligible for deferred payments on your insured home mortgage, for a specified amount of time, through your lender.'), ('COVID-19 Support: Futurpreneur Canada', 'If you received a start-up loan from Futurpreneur prior to April 1, 2020, your payments will be covered from March 25 through to September 24, 2020. If you do not qualify for the Canadian Emergency Business Account, you can top up your loan to a maximum of $10,000, interest-free, with one year of deferred repayment, to help cover operating costs during this challenging period.'), ('COVID-19: ExploreIP', 'Search this database for available patents related to COVID-19 for licensing or collaboration opportunities with public sector patent holders. Filter your search results by technology, expertise, or location.'), ('Community Futures Network of Canada', 'If your business is located in a rural community, and doesn’t qualify for the federal wage subsidy or other supports, you may be eligible for financial assistance through your Community Futures Network Canada.'), ('Regional development agencies: COVID-19 support', 'If the COVID-19 pandemic is affecting your operations, contact your local Regional Development Agency office to get a review of your situation and guidance on getting funding, support and/or flexible arrangements, on a case-by-case basis.'), ('Smart Renewables and Electrification Pathways Program', 'Get a repayable contribution of up to $50 million for a project that involves renewable energy, an emerging technology, or grid modernization, and covers up to 10%, 30% and 50% of the project costs for each of these 3 streams, respectively.'), ('COVID-19 and Your Business newsletter', 'Subscribe to receive timely information from the federal government on keeping your workers safe, receiving your vaccine and finding the support you need.'), ('PPE Supply Hub', 'Whether you’re selling personal protective equipment (PPE) during the COVID-19 pandemic, or looking for PPE to protect your workers, you can use this hub that connects buyers and sellers.'), ('BDC Capital Bridge Financing Program', 'If you operate a high-growth company that has been specifically impacted by COVID-19, you could get matched financing of at least $500,000 in the form of a convertible note if you are already backed by a qualified venture capital firm and have raised at least $500,000 in external capital.'), ('Temporary Wage Subsidy for Employers', 'Get a wage subsidy of up to $25,000 to cover 10% of the remuneration you paid from March 18, 2020 to June 19, 2020, up to $1,375 per eligible employee.'), ('Innovation Canada — Innovation Advisors', "If you're developing a new product, technology or process, an Innovation Advisor can provide you with free, one-on-one assistance to help you find the government supports that are right for your business. Our advisors are former entrepreneurs with established networks in both private and public sectors, and can help put you in touch with specialists in finance, technical advice and foreign market expertise, to help your business grow from here. Innovation Advisors help companies that are incorporated for profit in Canada. Is this you?")]

【讨论】:

  • time.sleep(1) 仍然在您建议的脚本中。此外,我试图同时点击和解析。
猜你喜欢
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
相关资源
最近更新 更多