Selenium/Webscrape 这个字段

Question

我的代码运行良好并打印所有行的标题但带有下拉菜单的行。

例如，如果单击，第 4 行会有一个下拉菜单。我实现了一个 try，理论上它会启动下拉菜单，然后拉出标题。

但是当我执行 click() 并尝试打印时，对于带有这些下拉菜单的行，它们没有打印。

预期输出 - 打印所有标题，包括下拉列表中的标题。

from selenium import webdriver
from bs4 import BeautifulSoup
import time
driver = webdriver.Chrome()
driver.get('https://cslide.ctimeetingtech.com/esmo2021/attendee/confcal/session/list')
time.sleep(4)
page_source = driver.page_source
soup = BeautifulSoup(page_source,'html.parser')

productlist=soup.find_all('div',class_='card item-container session')
for property in productlist:
    sessiontitle=property.find('h4',class_='session-title card-title').text
    print(sessiontitle)
    try:
        ifDropdown=driver.find_elements_by_class_name('item-expand-action expand')
        ifDropdown.click()
        time.sleep(4)
        newTitle=driver.find_element_by_class_name('card-title').text
        print(newTitle)
    except:
        newTitle='none'

Answer 1

import requests
from bs4 import BeautifulSoup
import pandas as pd


def get_soup(content):
    return BeautifulSoup(content, 'lxml')


def my_filter(req, content):
    try:
        r = req.get(content['href'])
        soup = get_soup(r.text)
        return [x.text for x in soup.select('.card-title')[1:]]
    except TypeError:
        return 'N/A'


def main(url):
    with requests.Session() as req:
        for page in range(1, 2):
            print(f"Extracting Page# {page}\n")
            params = {
                "p": page
            }
            r = req.get(url, params=params)
            soup = get_soup(r.text)
            goal = {x.select_one('.session-title').text: my_filter(
                req, x.select_one('.item-expand-action')) for x in soup.select('.card')}
        df = pd.DataFrame(goal.items(), columns=['Title', 'Menu'])
        print(df)


main('https://cslide.ctimeetingtech.com/esmo2021/attendee/confcal/session/list')

输出：

                                                Title                                               Menu
0                      Educational sessions on-demand                                                N/A
1                          Special Symposia on-demand                                                N/A
2                Multidisciplinary sessions on-demand                                                N/A
3   Illumina - Diagnosing Non-Small Cell Lung Canc...  [Illumina gives an update on their IVD road ma...
4   MSD - Homologous Recombination Deficiency: BRC...  [Welcome and Introductions, Homologous Recombi...
5   Servier - The clinical value of IDH inhibition...  [Isocitric dehydrogenase: an actionable geneti...
6   AstraZeneca - Redefining Breast Cancer – Biolo...  [Welcome and Opening, Redefining Breast Cancer...
7   ITM Isotopen Technologien München AG - A Globa...  [Welcome & Introduction, Changes in the Incide...
8   MSD - The Role of Biomarkers in Patient Manage...  [Welcome and Introductions, The Role of Pd-L1 ...
9   AstraZeneca - Re-evaluating the role of gBRCA ...  [Welcome and introduction, What do we know abo...
10  Novartis - Unmet needs in oncogene-driven NSCL...  [Welcome and introduction, Unmet needs in onco...
11                                    Opening session                                                N/A