R - 网页抓取

Question

首先非常感谢您的帮助和考虑，

我想从这个页面中提取一些东西http://ieeexplore.ieee.org/document/6875970/keywords 一旦你成为网站的一员，我就有兴趣在网络上抓取相关的信息

• IEEE 关键字

• INSPEC：受控索引

• INSPEC：非受控索引

  install.packages("rvest") 图书馆（rvest） 关键字

  关键字 %>% 一个% html_text(a)

但它不起作用！

你能帮帮我吗？

非常感谢！

Answer 1

# if you get an error while importing, just use install.packages('jsonlite') and /or install.packages('stringr')
library(jsonlite) 
library(stringr) 

# the ieee doc url we're interested in
url <- 'http://ieeexplore.ieee.org/document/6875970/keywords'

# read the document as text, no HTML parsing at all
doc <- readLines(url)

# after inspecting it, we notice it's built with angular-js,
# and the data we need to extract is defined as a single javascript variable.

# so, we first find the id of the line which defines 
# the javascript variable containing the tags
idx <- which(!is.na(str_match(doc, 'global.document.metadata=')))

# get the line ;-)
line <- doc[idx]

# since it's a javascript variable, we need to massage it
# a little to be able to read it as json.
# step 1: remove the "var global.document.metadata=" part (everything before the actual json)
line <- str_replace(line, '^[^{]*', '')

# step 2: remove the trailing ';' symbol
line <- str_replace(line, ';$', '')

# now we can parse the json data
df <- fromJSON(line)

# and get the information we need
df$keywords[df$keywords$type == 'IEEE Keywords',]$kwd[[1]]

df$keywords[df$keywords$type == 'INSPEC: Controlled Indexing',]$kwd[[1]]

df$keywords[df$keywords$type == 'INSPEC: Non-Controlled Indexing',]$kwd[[1]]

样本输出：

[1] "CTC incident datasets"                             
[2] "proactive spatiotemporal resource allocation"      
[3] "predictive visual analytics"                  
[4] "community policing"                               
[5] "law enforcement"