R rvest 无法获取 html_node

Question

我有一些使用rvest 包从网络上抓取我需要的数据的经验，但是我遇到了这个页面的问题：

https://www.nytimes.com/interactive/2020/us/covid-college-cases-tracker.html

如果您向下滚动一点，您会看到所有学校所在的部分。

我想要学校、案例和位置数据。我应该注意到有人在NYT GitHub 上询问将其发布为 csv 并且他们recommended that the data is all in the page and can just be pulled from there. 因此，我认为从这个页面上抓取是可以的。

但我无法让它工作。假设我只想从第一所学校的简单选择器开始。我使用检查器来查找 xpath。

我没有得到任何结果：

library(rvest)

URL <- "https://www.nytimes.com/interactive/2020/us/covid-college-cases-tracker.html"
pg <- read_html(URL)

# xpath copied from inspector
xpath_first_school <- '//*[@id="school100663"]'

node_first_school <- html_node(pg, xpath = xpath_first_school)

> node_first_school
{xml_missing}
<NA>

我收到{xml_missing}。

不知不觉我还有很多事情要做来概括这一点并为所有人收集数据 学校，但是对于网络抓取，我通常尝试从简单而具体的开始，然后扩大范围。但即使我的简单测试也不起作用。有什么想法吗？

Answer 1

设置 Rselenium 可能需要一些时间。首先你必须下载 chromedriver (https://chromedriver.chromium.org/)，选择你当前 chrome 最接近的版本。然后解压到你的 R 工作目录。

我尝试使用一个名为 decapitated 的包，它可以抓取 javascript 呈现的网站，但因为该网站包含“显示更多”，需要在显示所有数据之前进行物理点击，所以我必须先使用 Rselenium 来“点击”它获取页面源然后使用rvest进行解析

代码：

library(rvest)
library(tidyverse)
library(RSelenium)

url <- "https://www.nytimes.com/interactive/2020/us/covid-college-cases-tracker.html"

driver <- rsDriver(browser = c("chrome"), chromever = "85.0.4183.87", port = 560L)
remote_driver <- driver[["client"]] 
remote_driver$navigate(url)

showmore <- remote_driver$findElement(using = "xpath", value = "//*[@id=\"showall\"]/p")
showmore$clickElement()

test <- remote_driver$getPageSource()

school <- read_html(test[[1]]) %>%
  html_nodes(xpath = "//*[contains(@id, \"school\")]/div[2]/h2") %>%
  html_text() %>%
  as.tibble()

case <- read_html(test[[1]]) %>%
  html_nodes(xpath = "//*[contains(@id, \"school\")]/div[3]/p") %>%
  html_text() %>%
  as.tibble() 

location <- read_html(test[[1]]) %>%
  html_nodes(xpath = "//*[contains(@id, \"school\")]/div[4]/p") %>%
  html_text() %>%
  as.tibble() 

combined_table <- bind_cols(school,case = case[2:nrow(case),],location = location[2:nrow(location),]) 
names(combined_table) <- c("school", "case", "location")

combined_table %>% view()

输出：

# A tibble: 913 x 3
   school                                      case  location              
   <chr>                                       <chr> <chr>                 
 1 University of Alabama at Birmingham*        972   Birmingham, Ala.      
 2 University of North Carolina at Chapel Hill 835   Chapel Hill, N.C.     
 3 University of Central Florida               727   Orlando, Fla.         
 4 University of Alabama                       568   Tuscaloosa, Ala.      
 5 Auburn University                           557   Auburn, Ala.          
 6 North Carolina State University             509   Raleigh, N.C.         
 7 University of Georgia                       504   Athens, Ga.           
 8 Texas A&M University                        500   College Station, Texas
 9 University of Texas at Austin               483   Austin, Texas         
10 University of Notre Dame                    473   Notre Dame, Ind.      
# ... with 903 more rows

希望这对你有用！

Answer 2

所以我将在这里提供一个违反a very important rule described here 的答案，通常是一个丑陋的解决方案。但它是一个解决方案，让我们避免使用 Selenium。

要在此使用html_nodes，我们需要启动需要 Selenium 的 JS 操作。 @KWN 的解决方案似乎可以在他们的机器上运行，但我无法让 chromedriver 在我的机器上运行。我可以使用带有 Firefox 或 Chrome 的 Docker 获得几乎，但无法获得结果。所以我会先检查那个解决方案。如果失败了，试一试。差不多，这个网站有我需要的数据以 JSON 的形式公开。所以我提取了网站的文本，我使用正则表达式来隔离 JSON，然后 jsonlite 进行解析。

library(jsonlite)
library(rvest)
library(tidyverse)

url <- "https://www.nytimes.com/interactive/2020/us/covid-college-cases-tracker.html"

html_res <- read_html(url)

# get text
text_res <- html_res %>% 
  html_text(trim = TRUE)

# find the area of interest
# find the area of interest
data1 <- str_extract_all(text_res, "(?<=var NYTG_schools = ).*(?=;)")[[1]]

# get json into data frame
json_res <- fromJSON(data1)

# did it work?
glimpse(json_res)

Rows: 1,515
Columns: 16
$ ipeds_id    <chr> "100663", "199120", "132903", "100751"...
$ nytname     <chr> "University of Alabama at Birmingham",...
$ shortname   <chr> "U.A.B.", "North Carolina", "Central F...
$ city        <chr> "Birmingham", "Chapel Hill", "Orlando"...
$ state       <chr> "Ala.", "N.C.", "Fla.", "Ala.", "Ala."...
$ county      <chr> "Jefferson", "Orange", "Orange", "Tusc...
$ fips        <chr> "01073", "37135", "12095", "01125", "0...
$ lat         <dbl> 33.50199, 35.90491, 28.60258, 33.21402...
$ long        <dbl> -86.80644, -79.04691, -81.20223, -87.5...
$ logo        <chr> "https://static01.nyt.com/newsgraphics...
$ infected    <int> 972, 835, 727, 568, 557, 509, 504, 500...
$ death       <int> 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0,...
$ dateline    <chr> "n", "n", "n", "n", "n", "n", "n", "n"...
$ ranking     <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12,...
$ medicalnote <chr> "y", NA, NA, NA, NA, NA, NA, NA, NA, N...
$ coord       <list> [<847052.5, -406444.3>, <1508445.93, ...