通过 Google Playstore 在 R 中进行网页抓取

Question

我想从 google play store 中抓取我想要的几个应用程序评论的数据。

1. 名称字段

2. 他们得了多少星

3. 评论他们写的

This is the snap of the senerio

#Loading the rvest package
library('rvest')

#Specifying the url for desired website to be scrapped
url <- 'https://play.google.com/store/apps/details?id=com.phonegap.rxpal&hl=en_IN'

#Reading the HTML code from the website
webpage <- read_html(url)

#Using CSS gradient_Selector to scrap the name section
Name_data_html <- html_nodes(webpage,'.kx8XBd .X43Kjb')

#Converting the Name data to text
Name_data <- html_text(Name_data_html)

#Look at the Name
head(Name_data)

但结果是

> head(Name_data)

character(0)

后来我尝试发现更多我发现 Name_data_html 有

> Name_data_html
{xml_nodeset (0)}

我是网络抓取的新手，有什么可以帮我解决这个问题的！

Answer 1

您应该使用 Xpaths 来选择网页上的对象：

#Loading the rvest package
library('rvest')
#Specifying the url for desired website to be scrapped
url <- 'https://play.google.com/store/apps/details?id=com.phonegap.rxpal&hl=en_IN'
#Reading the HTML code from the website
webpage <- read_html(url)
# Using Xpath
Name_data_html <- webpage %>% html_nodes(xpath='/html/body/div[1]/div[4]/c-wiz/div/div[2]/div/div[1]/div/c-wiz[1]/c-wiz[1]/div/div[2]/div/div[1]/c-wiz[1]/h1/span')
#Converting the Name data to text
Name_data <- html_text(Name_data_html)
#Look at the Name
head(Name_data)

看看如何获​​取这张图片中的路径：

Answer 2

在分析了您的代码和您发布的 URL 的源页面后，我认为您无法废弃任何内容的原因是因为内容是动态生成的，因此 rvest 无法正确处理。

这是我的解决方案：

#Loading the rvest package
library(rvest)
library(magrittr) # for the '%>%' pipe symbols
library(RSelenium) # to get the loaded html of 

#Specifying the url for desired website to be scrapped
url <- 'https://play.google.com/store/apps/details?id=com.phonegap.rxpal&hl=en_IN'

# starting local RSelenium (this is the only way to start RSelenium that is working for me atm)
selCommand <- wdman::selenium(jvmargs = c("-Dwebdriver.chrome.verboseLogging=true"), retcommand = TRUE)
shell(selCommand, wait = FALSE, minimized = TRUE)
remDr <- remoteDriver(port = 4567L, browserName = "chrome")
remDr$open()

# go to website
remDr$navigate(url)

# get page source and save it as an html object with rvest
html_obj <- remDr$getPageSource(header = TRUE)[[1]] %>% read_html()

# 1) name field (assuming that with 'name' you refer to the name of the reviewer)
names <- html_obj %>% html_nodes(".kx8XBd .X43Kjb") %>% html_text()

# 2) How much star they got 
stars <- html_obj %>% html_nodes(".kx8XBd .nt2C1d [role='img']") %>% html_attr("aria-label")

# 3) review they wrote
reviews <- html_obj %>% html_nodes(".UD7Dzf") %>% html_text()

# create the df with all the info
review_data <- data.frame(names = names, stars = stars, reviews = reviews, stringsAsFactors = F)

在我的解决方案中，我使用的是 RSelenium，它能够像导航到它一样加载网页（而不是像 rvest 那样仅仅下载它）。这样，所有动态生成的内容都会被加载，并且在加载时，您现在可以使用 rvest 检索它并废弃它。

如果您对我的解决方案有任何疑问，请告诉我！

希望对您有所帮助！