使用 Python Selenium 抓取文本：无法找到真正存在的元素

Question

我试图从以下页面源中抓取文本：

我使用 selenium 和 python scrape“Diese Termine stehen zu ...”。

到目前为止我尝试了什么？

1. 使用xpath查找元素并使用绝对位置：

availability = driver.find_elements_by_xpath("//*[@id='booking-content']/div[2]/div[4]/div/div[2]/div/div/div/div[1]/div/div/span")

1. 使用类名：

elements = driver.find_elements_by_class_name("dl-text dl-text-body dl-text-regular dl-text-s dl-text-color-inherit")

1. 使用 CSS 选择器：

使用以下关键字：.booking-message .dl-text

availability = driver.find_element_by_css_selector('.booking-message .dl-text')

以上所有方法均无效。通过第 3 步，我确信它应该可以工作，因为从屏幕截图中可以看出，我可以在 Chrome 中使用相同的关键字找到元素。但仍然没有运气。

错误信息是：

Traceback (most recent call last):
  File "/Users/GunardiLin/Desktop/Codes/Tracker.py", line 18, in <module>
    availability = driver.find_element_by_css_selector('.booking-message .dl-text')
  File "/Users/GunardiLin/opt/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 598, in find_element_by_css_selector
    return self.find_element(by=By.CSS_SELECTOR, value=css_selector)
  File "/Users/GunardiLin/opt/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element
    'value': value})['value']
  File "/Users/GunardiLin/opt/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/Users/GunardiLin/opt/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".booking-message .dl-text"}
  (Session info: chrome=90.0.4430.212)

我知道另一个有同样问题的帖子： Python with selenium: unable to locate element which really exist

这就是我检查网站是否使用“iframe”的原因。 我通过搜索“iframe-tags”来检查它，就像在屏幕截图中一样。搜索结果为0，表示没有找到。

有人可以指点如何抓取文本吗？ 我更喜欢使用 css 选择器（选项 3）并且不喜欢使用选项 1（xpath + 绝对位置）。但目前我会感谢任何解决方案。

提前谢谢你:-)

更新：

到目前为止我的代码：

import os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import Select

url = r"https://www.doctolib.de/gemeinschaftspraxis/muenchen/fuchs-hierl?practitioner_id=any&speciality_id=5593&utm_campaign=website-button&utm_source=fuchs-hierl-website-button&utm_medium=referral&utm_content=custom&utm_term=fuchs-hierl"

chrome_options = Options()
chrome_options.add_argument('--headless')
driver = webdriver.Chrome(executable_path="/Applications/chromedriver", options=chrome_options)
driver.get(url)
print('*** Title:', driver.title)
# print(driver.page_source.encode("utf-8"))
dropdown_besuchgrund = driver.find_element_by_id("booking_motive")
select_besuchgrund = Select(dropdown_besuchgrund)
# print(dir(select_besuchgrund))
select_besuchgrund.select_by_visible_text("Erste Impfung Covid-19 (BioNTech-Pfizer)")
# availability = driver.find_elements_by_xpath("//*[@id='booking-content']/div[2]/div[4]/div/div[2]/div/div/div/div[1]/div/div/span")
#elements = driver.find_elements_by_class_name("dl-text dl-text-body dl-text-regular dl-text-s dl-text-color-inherit")
# availability = driver.find_element_by_css_selector('.booking-message .dl-text')
availability = driver.find_element_by_xpath(".//div[contains(@class,'booking-message')]/span")
print("***")
print(availability.text)
# for elem in elements:
#     print ("***", elem.text)
#     if elem.text == "Diese Termine stehen zu einem späteren Zeitpunkt wieder für eine Online-Buchung zur Verfügung. ":
#         print("*** Ausgebucht")
driver.quit()

@itronic1990 22.05.2021 07:45：我已经检查了您的建议：

driver.find_element_by_xpath(".//div[contains(@class,'booking-message')]/span").text

正如您在上面看到的，chrome 可以使用您的过滤器找到文本。但是如果我运行代码，它就找不到它。我的测试代码：

import os
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
url = r"https://www.doctolib.de/gemeinschaftspraxis/muenchen/fuchs-hierl"
chrome_options = Options()
chrome_options.add_argument('--headless')
driver = webdriver.Chrome(executable_path="/Applications/chromedriver", options=chrome_options)
driver.get(url)
element_text = driver.find_element_by_xpath(".//div[contains(@class,'booking-message')]/span").text
print(element_text)
driver.quit()

错误信息：

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":".//div[contains(@class,'booking-message')]/span"}
  (Session info: headless chrome=90.0.4430.212)

我不明白为什么？谢谢你的建议。

Answer 1

您已经在 xpath 和类名中使用了 find_elements。对吗？

试试这个

driver.find_element_by_xpath(".//div[contains(@class,'booking-message')]/span").text

Answer 2

为什么要打扰 Selenium？直接从源中获取数据：

import requests

url = 'https://www.doctolib.de/availabilities.json'
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'}
payload = {
'start_date': '2021-05-21',
'visit_motive_ids': '2820334',
'agenda_ids': '466608',
'insurance_sector': 'public',
'practice_ids': '25230',
'limit': '4'}

jsonData = requests.get(url, headers=headers, params=payload).json()

输出：

print(jsonData['message'])
Diese Termine stehen zu einem späteren Zeitpunkt wieder für eine Online-Buchung zur Verfügung. 

我不熟悉德语，否则我可以提高效率。基本上使用practice_id 输入它并从每个练习中获取数据。

import requests
from bs4 import BeautifulSoup
from datetime import datetime

# Get location practice_ids
url = 'https://www.doctolib.de/allgemeinmedizin/81667-muenchen'
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'}

practice_ids_list = []
for page in range(1,100):
    payload = {'page':page}

    response = requests.get(url, headers=headers, params=payload)
    if response.status_code == 404:
        break
    
    else:
        print('Page: %s' %page)
        soup = BeautifulSoup(response.text, 'html.parser')
        divs = soup.find_all('div',{'class':'dl-search-result'})
        
        for div in divs:
            practice_id = div['id'].split('-')[-1]
            practice_ids_list.append(practice_id)

today = datetime.today().strftime('%Y-%m-%d')

url = 'https://www.doctolib.de/availabilities.json'
for practice_id in practice_ids_list:
    payload = {
    'start_date': today,
    'visit_motive_ids': '2820334',
    'agenda_ids': '466606',
    'insurance_sector': 'public',
    'practice_ids': '%s' %practice_id,
    'limit': '15'}
    
    jsonData = requests.get(url, headers=headers, params=payload).json()
    
   
    if jsonData['total'] == 0 and 'next_slot' not in jsonData.keys():
        #print('\t', jsonData['message'],'\n')
        print(practice_id)
    else:
        # Get Clinic Details
        clinic_url = 'https://www.doctolib.de/search_results/%s.json' %practice_id
        clinic_jsonData = requests.get(clinic_url, headers=headers).json()
        clinic_name = clinic_jsonData['search_result']['name_with_title']
        address = clinic_jsonData['search_result']['address']
        city = clinic_jsonData['search_result']['city']
        zipcode = clinic_jsonData['search_result']['zipcode']
        print('%s\n%s %s %s' %(clinic_name, address, city, zipcode))
        
        payload.update({'start_date':jsonData['next_slot']})
        jsonData = requests.get(url, headers=headers, params=payload).json()
        print('\n\t','*'*50,'\nThe follow dates are available:')
        for each_date in jsonData['availabilities']:
            if len(each_date['slots']) > 0:
                print('\t\t',each_date['date'])