投资 |将数据抓取为长格式答案

【问题标题】：rvest | Webscraping data into a long format投资 |将数据抓取为长格式
【发布时间】：2015-10-17 08:08:44
【问题描述】：

在进行网络抓取时，我遇到了以下问题，我认为可能有更好的解决方案：

拥有这些数据：

dat <- data.frame(query = c("Washington, USA", "Frankfurt, Germany"))

               query
1    Washington, USA
2 Frankfurt, Germany

我想查询例如Google Maps Api 并返回格式化的地址。可能有多种格式。结果应该如下：

               query         formatted_address
1    Washington, USA       Washington, DC, USA
2    Washington, USA       Washington, UT, USA
3    Washington, USA Washington, VA 22747, USA
4    Washington, USA Washington, IA 52353, USA
5    Washington, USA Washington, GA 30673, USA
6    Washington, USA Washington, PA 15301, USA
7 Frankfurt, Germany        Frankfurt, Germany

我现在要做的是：

require(RCurl)
require(rvest)
require(magrittr)

build_url <- function(x, base_url = "https://maps.googleapis.com/maps/api/geocode/xml?address="){
  paste0(base_url, RCurl::curlEscape(x))
}

l <- lapply(dat$query, function(q){
  formatted_address <- q %>% build_url %>% read_xml %>% xml_nodes("formatted_address") %>% xml_text
  data.frame(query = q, formatted_address)
})

do.call(rbind, l) # This can be done via data.table::rbindlist as well

有没有更好的解决方案？也许更多data.table或dplyr风格？

【问题讨论】：

请包含library/require 调用以使您的代码可重现
当然。刚刚添加了require 语句
除了stringsAsFactors=FALSE 上的data.frame 创建之外，您已经完美地优化了这个IMO。我建议在lappl 中添加sleep，并确保将调用次数限制为2500 或更少的IIRC（usage limits 信息）。

标签： r google-maps data.table dplyr transformation

【解决方案1】：

我已经编写了包 googleway 以使用有效的 API 密钥访问 google maps API（因此，如果您的数据超过 2,500 项，您可以为 API 密钥付费）。

要获取地址详细信息，请使用google_geocode()

library(googleway)

key <- "your_api_key"

dat <- data.frame(query = c("Washington, USA", "Frankfurt, Germany"))

## To get all the data:
res <- apply(dat, 1, function(x){
  google_geocode(address = x["query"],
                 key = key)  ## use simplify = F to return JSON
})

## to access the 'formatted address' part, see
res[[1]]$results$formatted_address
# [1] "Washington, DC, USA"       "Washington, UT, USA"       "Washington, VA 22747, USA" "Washington, IA 52353, USA"
# [5] "Washington, GA 30673, USA" "Washington, PA 15301, USA"

## so to get everything as a list
lapply(res, function(x){
  x$results$formatted_address
})

# [[1]]
# [1] "Washington, DC, USA"       "Washington, UT, USA"       "Washington, VA 22747, USA" "Washington, IA 52353, USA"
# [5] "Washington, GA 30673, USA" "Washington, PA 15301, USA"
# 
# [[2]]
# [1] "Frankfurt, Germany"

## and to put back onto your original data.frame:
lst <- lapply(1:length(res), function(x){
  data.frame(query = dat[x, "query"],
             formatted_address = res[[x]]$results$formatted_address)
})

data.table::rbindlist(lst)
#                 query         formatted_address
# 1:    Washington, USA       Washington, DC, USA
# 2:    Washington, USA       Washington, UT, USA
# 3:    Washington, USA Washington, VA 22747, USA
# 4:    Washington, USA Washington, IA 52353, USA
# 5:    Washington, USA Washington, GA 30673, USA
# 6:    Washington, USA Washington, PA 15301, USA
# 7: Frankfurt, Germany        Frankfurt, Germany

【讨论】：