通过 javascript 加载的抓取表答案

【问题标题】：Scrape table loaded through javascript通过 javascript 加载的抓取表
【发布时间】：2020-01-21 00:55:20
【问题描述】：

我正在尝试抓取通过 javascript 生成的表格，但我很挣扎。我的代码是：

from selenium import webdriver
driver = webdriver.Firefox(executable_path=r'path\to\geckodriver')


# driver = webdriver.Firefox()
url = 'https://af.ktnlandscapes.com/'
driver.get(url)

table = driver.find_element_by_xpath('//*[@id="list-view"]')
table_html = table.get_attribute('innerHTML')

df = read_html(table_html)
print (df)

driver.close()

我收到错误：

ValueError: No tables found

【问题讨论】：

标签： python selenium html-table

【解决方案1】：

最可能的情况是在您尝试从表格中读取 HTML 之前表格尚未完全加载。为确保表格已加载，您可以使用以下代码：

# get table -- first wait for table and all rows to fully load
WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.XPATH, "//*[@id='list-view']/tbody/tr")))

table = driver.find_element_by_xpath("//*[@id='list-view']")

你也可以只使用普通的 Selenium 来刮桌子——如果这不是你的首选策略，这里不需要使用 read_html。以下是一些抓取表值的示例代码：

driver = webdriver.Chrome();

driver.get("https://af.ktnlandscapes.com/")

# get table -- first wait for table to fully load
WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.XPATH, "//*[@id='list-view']/tbody/tr")))
table = driver.find_element_by_xpath("//*[@id='list-view']")

# get rows
rows = table.find_elements_by_xpath("tbody/tr")

# iterate rows and get cells
for row in rows:

    # get cells
    cells = row.find_elements_by_xpath("td")

    # print cell contents
    for cell in cells:
        print(cell.text)

    # print newline to separate rows
    print('\n')

driver.close()
driver.quit()

上面的代码示例获取所有表格行和单元格，在嵌套的for 中循环遍历它们，并打印每个单元格的值。您可以对其进行修改以满足您的程序需求。

输出：

Networks
> Industry and research networks (networks)
A Network of Integrated Technolog...
Food and Drink, Plants...
 South West


Research Capabilities
> Livestock and Aquaculture Research Capabili...
Aberystwyth University (Institute...
Livestock and Aquaculture
 Wales


Research & Training Programmes
> Advanced Training Partnerships
Advanced Training Partnership for...
Livestock and Aquaculture
 Wales


Research Capabilities
> Livestock and Aquaculture Research Capabili...
Agri-Food and Biosciences Institu...
Livestock and Aquaculture
 Northern Ireland


Research Capabilities
> Plants and Crops Research Capabilities
Agri-Food and Biosciences Institu...
Plants and Crops
 Northern Ireland


Networks
> Industry Networks
Agri-Tech East
Livestock and Aquacult...
 East of England


Underpinning organisations
> Quality assurance bodies
Agricultural Industries Confedera...
Livestock and Aquacult...
 East of England


Underpinning organisations
> Trade associations
Agricultural Industries Confedera...
Livestock and Aquacult...
 East of England


Funders
> Levy bodies (funders)
Agriculture and Horticulture Deve...
Livestock and Aquacult...
 West Midlands


Underpinning organisations
> Levy bodies (underpinning)
Agriculture and Horticulture Deve...
Livestock and Aquacult...
 West Midlands


Innovation centres
> Centres for Agricultural Innovation
Agriculutral Engineering and Prec...
Livestock and Aquacult...
 Scotland


Research & Training Programmes
> Advanced Training Partnerships
AgriFood Advanced Training Partne...
AgriFood ATP, Plants a...
 East Midlands


Innovation centres
> Centres for Agricultural Innovation
Agrimetrics
Livestock and Aquacult...
 East of England


Networks
> Industry and research networks (networks)
Agrisearch
Livestock and Aquacult...
 Northern Ireland


Funders
> Research funding bodies (funders)
Agrisearch
Livestock and Aquacult...
 Northern Ireland


Networks
> Industry and research networks (networks)
Anaerobic Digestion Network
Livestock and Aquacult...
 South East


Key sector organisations
> UK Government organisations (key sector)
Animal and Plant Health Agency
Livestock and Aquacult...
 Nationwide


Research & Training Programmes
> Research and Technology Clubs
Animal Health Research Club
ARC, Livestock and Aqu...
 Nationwide


Underpinning organisations
> Professional organisations
Association of Applied Biologists
Livestock and Aquacult...
 West Midlands


Research Capabilities
> Livestock and Aquaculture Research Capabili...
Bangor University (College of Nat...
Livestock and Aquaculture
 Wales

【讨论】：