【发布时间】：2020-06-04 02:31:30
【问题描述】：
我正在尝试使用 rvest 构建数据库。由于我有很多数据要下载，我尝试编写几个函数来中断抓取过程并在我离开的地方重新启动它。然而，虽然这些功能或多或少地工作，但每当我手动中断它们时，我都会失去输出。有谁知道一个解决方案可以让我在不丢失循环正在构建的数据框的情况下停止该功能？我很乐意提供任何建议！
我尝试从以下网址抓取数据的一些网址：
to_do <- c("https://jobs.51job.com/shenzhen-nsq/116924235.html?s=01&t=0",
"https://jobs.51job.com/shenzhen-nsq/116923692.html?s=01&t=0",
"https://jobs.51job.com/shenzhen-nsq/116923628.html?s=01&t=0",
"https://jobs.51job.com/shenzhen-nsq/116923578.html?s=01&t=0",
"https://jobs.51job.com/shenzhen-nsq/116920896.html?s=01&t=0")
我为下载创建的函数：
# In order to initiate the dowload
dl_data_start <- function(to_do){
output <- tibble()
i = 1
while (to_do[i] %in% to_do) {
page <- read_html(to_do[i])
position <- page %>%
html_nodes(.,'h1') %>%
html_text(.)
resume <- page %>%
html_nodes(.,'.ltype') %>%
html_text(.)
job_offer <- page %>%
html_nodes(.,'.job_msg') %>%
html_text(.)
eps <- page %>%
html_nodes(.,'.com_msg') %>%
html_text(.)
eps_status <- page %>%
html_nodes(.,'.at:nth-child(1)') %>%
html_text(.)
eps_description <- page %>%
html_nodes(.,'.tmsg') %>%
html_text(.)
employees <- page %>%
html_nodes(.,'.at:nth-child(2)') %>%
html_text(.)
category <- page %>%
html_nodes(.,'.at:nth-child(3)') %>%
html_text(.)
salary <- page %>%
html_nodes(.,'.cn strong') %>%
html_text(.)
url <- to_do[i]
id <- i
current <- tibble(position,resume,job_offer,eps,eps_description,eps_status,
employees,category,salary,url,id)
output <- bind_rows(output,current)
print(output[i,])
i = i + 1
}
return(output)
}
# function in order to continue the download where I left it
dl_data_continue <- function(to_do,df,done){
i = (match(tail(done,n=1),to_do) + 1)
while (to_do[i] %in% to_do) {
page <- read_html(to_do[i])
position <- page %>%
html_nodes(.,'h1') %>%
html_text(.)
resume <- page %>%
html_nodes(.,'.ltype') %>%
html_text(.)
job_offer <- page %>%
html_nodes(.,'.job_msg') %>%
html_text(.)
eps <- page %>%
html_nodes(.,'.com_msg') %>%
html_text(.)
eps_status <- page %>%
html_nodes(.,'.at:nth-child(1)') %>%
html_text(.)
eps_description <- page %>%
html_nodes(.,'.tmsg') %>%
html_text(.)
employees <- page %>%
html_nodes(.,'.at:nth-child(2)') %>%
html_text(.)
category <- page %>%
html_nodes(.,'.at:nth-child(3)') %>%
html_text(.)
salary <- page %>%
html_nodes(.,'.cn strong') %>%
html_text(.)
url <- to_do[i]
id <- i
current <- tibble(position,resume,job_offer,eps,eps_description,eps_status,
employees,category,salary,url,id)
df <- bind_rows(df,current)
print(df[i,])
i = i + 1
}
return(df)
}
我遇到的问题是，每当我中断循环或发生错误时，我都会丢失所有数据。有人可以解决这个问题吗？我尝试了一些方法，例如安全或 tryCatch，但我无法理解这里出了什么问题。 非常感谢。
编辑： 我还用 tryCatch 做了一些尝试。使用下面的函数，代码在遇到问题（例如 HTTP 404 错误）时不再中断。但是当出现错误时，循环会一直卡在有问题的迭代中，所以我必须用错了。
dl_data_continue_2 <- function(to_do,df,done){
i = (match(tail(done,n=1),to_do) + 1)
while (to_do[i] %in% to_do) {
tryCatch(
{expr =
page <- read_html(to_do[i])
position <- page %>%
html_nodes(.,'h1') %>%
html_text(.)
resume <- page %>%
html_nodes(.,'.ltype') %>%
html_text(.)
job_offer <- page %>%
html_nodes(.,'.job_msg') %>%
html_text(.)
eps <- page %>%
html_nodes(.,'.com_msg') %>%
html_text(.)
eps_status <- page %>%
html_nodes(.,'.at:nth-child(1)') %>%
html_text(.)
eps_description <- page %>%
html_nodes(.,'.tmsg') %>%
html_text(.)
employees <- page %>%
html_nodes(.,'.at:nth-child(2)') %>%
html_text(.)
category <- page %>%
html_nodes(.,'.at:nth-child(3)') %>%
html_text(.)
salary <- page %>%
html_nodes(.,'.cn strong') %>%
html_text(.)
url <- to_do[i]
id <- i
current <- tibble(position,resume,job_offer,eps,eps_description,eps_status,
employees,category,salary,url,id)
df <- bind_rows(df,current)
print(df[i,])
i = i + 1},
error = function(e){
message("* Caught an error on itertion ")
print(e)
i = i + 1
}
)
}
out
}
安全使用，我基本试过了
library(purrr)
dl_safely <- safely(dl_data_continue)
【问题讨论】：
-
请在
purrr::safely或tryCatch中包含您的尝试。 -
谢谢！我现在已经添加了。
-
您是否考虑过将元素添加到列表中，并且仅在所有内容完成后也意味着从函数返回列表，然后才在
do.call(rbind, your_list)而不是在您的函数中执行df <- bind_rows(df,current)，这将通常在原地覆盖对象。注意：这只是初步分析。