协助 selenium web 抓取

Question

我正在努力挖掘音乐家获得的荣誉勋章的细节！所以here 是指向音乐家列表的链接。然后，当您单击每个视图的视图时，它有一个整洁的小表，其中包含 RANK、COMPANY、DATE OF ISSUE 等。我想从该表中抓取数据。

from selenium import webdriver

driver = webdriver.Chrome("/Users/ashkij/Desktop/chromedriver")
driver.get("http://www.cmohs.org/search-results.php?q=&x=40&y=9&rank=Musician&organization=&division=&company=&conflict=")
person = driver.find_elements_by_class_name("div.floatElement.recipientView")
details = driver.find_elements_by_class_name("div.detailDatacol1") 
for i in details:
    print(i.text)

Answer 1

首先收集所有视图hrefs，然后访问每个视图并收集目标数据。

from selenium import webdriver

driver = webdriver.Chrome()
driver.get('http://www.cmohs.org/search-results.php?q=&x=40&y=9&rank=Musician&organization=&division=&company=&conflict=')

# get hrefs
view_links = driver.find_elements_by_class_name('recipientView')
for index, view in enumerate(view_links):
    html = view.get_attribute('innerHTML')
    href = html.split('"')[1]

    view_links[index] = href

# visit each href and get data
for href in view_links:
    driver.get(href)

    detail_data = driver.find_elements_by_class_name('dataBar')
    for detail in detail_data:
        print(detail.text)