在 is(object, Cl) 中：获取行 R 时出错

Question

我有一个 MySQL 表，我正尝试使用 RMySQL 访问 R。

有 1690004 行应该从

返回

dbGetQuery(con, "SELECT * FROM tablename WHERE export_date ='2015-01-29'")

很遗憾，我收到以下警告消息：

In is(object, Cl) : error while fetching row
In dbGetQuery(con, "SELECT * FROM tablename WHERE export_date ='2015-01-29'",  : pending rows

并且只接收约 400K 行。

如果我使用 dbSendQuery 将查询分成几个“提取”，则在收到大约 400K 行后开始出现警告消息。

任何帮助将不胜感激。

Answer 1

所以，这似乎是由于我的托管服务提供商强加了 60 秒的超时（该死的 Arvixe！）。我通过“分页/分块”输出解决了这个问题。因为我的数据有一个自动递增的主键，所以返回的每一行都是有序的，允许我在每次迭代后获取接下来的 X 行。

为了获得 160 万行，我执行了以下操作：

library(RMySQL)
con <- MySQLConnect() # mysql connection function
day <- '2015-01-29' # date of interest
numofids <- 50000 # number of rows to include in each 'chunk'
count <- dbGetQuery(con, paste0("SELECT COUNT(*) as count FROM tablename WHERE export_date = '",day,"'"))$count # get the number of rows returned from the table.
dbDisconnect(con)
ns <- seq(1, count, numofids) # get sequence of rows to work over
tosave <- data.frame() # data frame to bind results to
# iterate through table to get data in 50k row chunks
for(nextseries in ns){ # for each row
  print(nextseries) # print the row it's on
  con <- MySQLConnect()
  d1 <- dbGetQuery(con, paste0("SELECT * FROM tablename WHERE export_date = '",day,"' LIMIT ", nextseries,",",numofids)) # extract data in chunks of 50k rows
  dbDisconnect(con)
  # bind data to tosave dataframe. (the ifelse is avoid an error when it tries to rbind d1 to an empty dataframe on the first pass).
  if(nrow(tosave)>0){
      tosave <- rbind(tosave, d1)
  }else{
      tosave <- d1
  }
}