使用 R 从 URL 下载多个文件不起作用答案

【问题标题】：Downloading multiple files from URL using R not working使用 R 从 URL 下载多个文件不起作用
【发布时间】：2019-11-19 11:59:51
【问题描述】：

我正在尝试从多个 URL 循环下载多个 NetCDF (.nc) 格式文件。但是，当我尝试打开文件时，它们似乎已损坏。

您将在下面找到我的代码。我尝试了不同的方法，例如使用 download.file 或 system。

这是我需要下载的文件示例： http://thredds.met.no/thredds/catalog/metusers/senorge2/seNorge2/provisional_archive/PREC1d/gridded_dataset/201601/catalog.html

但我需要下载数百个文件，因为每个文件代表一天。

到目前为止，这是我的代码：

year = c("2016","2017")
mon = c("01","02")
day = c("01","02","03","04","05","06","07","08","09","10",
        "11","12","13","14","15","16","17","18","19","20",
        "21","22","23","24","25","26","27","28","29","30","31")
for (y in year){
  for (m in mon){
    for (d in day){
      download.file(paste("http://thredds.met.no/thredds/fileServer/metusers/senorge2/seNorge2/provisional_archive/",
                          "PREC1d/gridded_dataset/",y,m,"/seNorge_v2_0_PREC1d_grid_",y,m,d,"_",y,m,d,".nc",sep=""),
                    destfile=paste("seNorge_v2_0_PREC1d_grid_",y,m,d,"_",y,m,d,".nc",sep=""),method="curl",mode="wb")

      #try(system(paste("wget ",paste("http://thredds.met.no/thredds/fileServer/metusers/senorge2/seNorge2/provisional_archive/",
      #                               "PREC1d/gridded_dataset/",y,m,"/seNorge_v2_0_PREC1d_grid_",y,m,d,"_",y,m,d,".nc",sep=""),sep=""), 
      #           intern = TRUE, ignore.stderr = TRUE, wait=TRUE))
    }
  }
}

感谢任何帮助。

谢谢！

最好的，

米歇尔

【问题讨论】：

标签： r bash url download wget

【解决方案1】：

当我尝试你的代码时，我得到了一些文件503 Service Temporarily Unavailable。要在这种情况下重试下载，请添加 --retry-on-http-error=503。 Maye 还添加了--random-wait。我将method 从curl 更改为wget 并删除了mode="wb"，因为manula 说不用于方法“wget”和“curl”。希望以下内容能解决您的问题。

year = c("2016","2017")
mon = c("01","02")
day = c("01","02","03","04","05","06","07","08","09","10",
        "11","12","13","14","15","16","17","18","19","20",
        "21","22","23","24","25","26","27","28","29","30","31")
for (y in year){
  for (m in mon){
    for (d in day){
      download.file(paste("http://thredds.met.no/thredds/fileServer/metusers/senorge2/seNorge2/provisional_archive/",
                          "PREC1d/gridded_dataset/",y,m,"/seNorge_v2_0_PREC1d_grid_",y,m,d,"_",y,m,d,".nc",sep=""),
                    destfile=paste("seNorge_v2_0_PREC1d_grid_",y,m,d,"_",y,m,d,".nc",sep=""),method="wget",extra="--random-wait --retry-on-http-error=503")
    }
  }
}

【讨论】：

【解决方案2】：

您说文件“已损坏”是什么意思？您如何尝试读取 nc 文件？

您的代码似乎可以正常工作，我可以阅读下载的文件。您可以使用 R 中的 raster 包来读取文件。还请确保您已安装 ncdf4 软件包。

library(raster)
r = raster('seNorge_v2_0_PREC1d_grid_20160101_20160101.nc')

【讨论】：

我用CDO、ncview、R等读取了nc文件。上面@GKi给出的解决方案解决了这个问题。之前发生的事情是文件“出现”了，但它们实际上是空的 - 好像它们没有完全下载一样。