Web 抓取由 Javascript 函数创建的表

Question

我正在尝试在下面的链接中抓取研究表

https://clinicaltrials.gov/ct2/results?cond=COVID&term=&cntry=&state=&city=&dist=

表格使用 Javascript 动态创建内容。 我尝试使用 selenium，但间歇性地得到 StaleElementException。 请帮助我。

我想检索表中的所有行并将它们存储在本地数据库中。 这是我在硒中尝试过的

 import selenium.webdriver as webdriver
 url = 'https://clinicaltrials.gov/ct2/results?cond=COVID&term=&cntry=&state=&city=&dist='
 driver=webdriver.Firefox()
 #driver.implicitly_wait(30)
 driver.get(url)
 data = []
 for tr in driver.find_elements_by_xpath('//table[@id="theDataTable"]//tbody//tr'):
  tds = tr.find_elements_by_tag_name('td')
   if tds:
    for td in tds:
     print(td.text)
     if td.text not in data:
      data.append(td.text)          
 driver.quit()
 print('*********************************************************************')
 print(data)

我将存储在数据库中的“数据”变量中的更多数据。

我是 selenium 和 Web Scraping 的新手，此外，我想单击“研究标题”列中的每个链接并从该页面中提取每项研究的数据。

我想要避免/处理过时元素异常或 Selenium webdriver 替代方案的建议。 提前致谢！

Answer 1

我尝试了以下代码，所有数据都正确存储。可以试试吗？

代码

driver.get("https://clinicaltrials.gov/ct2/results?cond=COVID&term=&cntry=&state=&city=&dist=")


time.sleep(5)

array = []

flag = True
next_counter = 0

time.sleep(4)

select = Select(driver.find_element_by_name('theDataTable_length'))
select.select_by_value('100')

time.sleep(5)

while flag == True:
    if next_counter == 13:
        print("Stoped")
    else:
        item = driver.find_elements_by_tag_name("tbody")[1]
        rows = item.find_elements_by_tag_name('tr')

        for x in range(len(rows)):
            for i in range(7):
                array.insert(x, rows[x].find_elements_by_tag_name('td')[i].text)
                print(rows[x].find_elements_by_tag_name('td')[i].text)

        time.sleep(5)
    next = driver.find_element_by_id('theDataTable_next')
    next.click()
    next_counter = next_counter + 1


    time.sleep(7)

输出

1

Not yet recruiting
NEW
Indirect Endovenous Systemic Ozone for New Coronavirus Disease (COVID19) in Non-intubated Patients
COVID
Other: Systemic indirect endovenous ozone therapy
SEOT
Valencia, Spain
2

Recruiting
NEW
Prediction of Clinical Course in COVID19 Patients
COVID 19
Other: CT-Scan
Chu Saint-Etienne
Saint-Étienne, France
3

Not yet recruiting
NEW
Risks of COVID19 in the Pregnant Population
COVID19
Other: Biospecimen collection
Mayo Clinic in Rochester
Rochester, Minnesota, United States
4

Recruiting
NEW
Saved From COVID-19
COVID
Drug: Chloroquine
Drug: Placebo oral tablet
Columbia University Irving Medical Center/NYP
New York, New York, United States
5

Recruiting
NEW
Efficacy of Convalescent Plasma Therapy in Severely Sick COVID-19 Patients
COVID
Drug: Convalescent Plasma Transfusion
Other: Supportive Care
Drug: Random Donor Plasma
Maulana Azad medical College
New Delhi, Delhi, India
Institute of Liver and Biliary Sciences
New Delhi, Delhi, India
6

Not yet recruiting
NEW
A Real-life Experience on Treatment of Patients With COVID 19
COVID
Drug: Chloroquine
Drug: Favipiravir
Drug: Nitazoxanide
(and 3 more...)
Tanta university hospital
Tanta, Egypt
7

Recruiting
International COVID19 Clinical Evaluation Registry,
COVID 19
Combination Product: Observational (registry)
Hospital Lclinico San Carlos
Madrid, Spain
8

Completed
NEW
AiM COVID for Covid 19 Tracking and Prediction
COVID 19
Other: No Intervention
Aarogyam (UK)
Leicester, United Kingdom
9

Recruiting
NEW
Establishing a COVID-19 Prospective Cohort for Identification of Secondary HLH
COVID

Department of nephrology, Klinikum rechts der Isar
München, Bavaria, Germany
10

Recruiting
NEW
Max Ivermectin- COVID 19 Study Versus Standard of Care Treatment for COVID 19 Cases. A Pilot Study
COVID
Drug: Ivermectin
Max Super Speciality hospital, Saket (A unit of Devki Devi Foundation)
New Delhi, Delhi, India

我的代码正在执行以下逻辑步骤：

• 首先，我选择查看 100 个结果而不是 10 个结果的选项，以便节省一些时间来检索数据。
• 其次，我阅读了页面的所有结果（100），当我完成后，我去点击下一页符号。然后我有一个 sleep 命令等待 4 秒（你可以用更好的方式来做，但我这样做是为了给你一些快速的东西 - 你必须插入 waitUntilElementIsVisible 概念）
• 点击下一页按钮后，我再次保存结果（100）
• 此功能将一直运行，直到标志变为 False。当 next_counter 为 14（超过 13 是最大值）时，它将为假。数字 13 实际上是 1300（结果）除以 100（每页的最大结果数）所以 1300/100 = 13。所以我们有 13 页。

编辑和传输数据是您可以管理的事情，不需要 Selenium 知识或有关网络自动化的事情。这是一个 100% 的 Python 概念。