如何在服务器身份验证后使用 RCurl *下载大型二进制文件

Question

我最初询问this question 关于使用httr 包执行此任务，但我认为使用httr 是不可能的。所以我重写了我的代码以使用 RCurl 代替 - 但我仍然在可能与 writefunction 相关的东西上绊倒......但我真的不明白为什么。

您应该能够使用 32 位版本的 R 来重现我的工作，因此如果您将任何内容读入 RAM，就会达到内存限制。我需要一个直接下载到硬盘的解决方案。

首先，此代码工作 -- 压缩文件被适当地保存到磁盘。

library(RCurl)
filename <- tempfile()
f <- CFILE(filename, "wb")
url <- "http://www2.census.gov/acs2011_5yr/pums/csv_pus.zip"
curlPerform(url = url, writedata = f@ref)
close(f)
# 2.1 GB file successfully written to disk

现在这里有一些无效的RCurl 代码。正如the previous question 中所述，要准确复制它需要在ipums 上创建一个提取。

your.email <- "email@address.com"
your.password <- "password"
extract.path <- "https://usa.ipums.org/usa-action/downloads/extract_files/some_file.csv.gz"

library(RCurl)

values <- 
    list(
        "login[email]" = your.email , 
        "login[password]" = your.password , 
        "login[is_for_login]" = 1
    )

curl = getCurlHandle()

curlSetOpt(
    cookiejar = 'cookies.txt', 
    followlocation = TRUE, 
    autoreferer = TRUE, 
    ssl.verifypeer = FALSE,
    curl = curl
)

params <- 
    list(
        "login[email]" = your.email , 
        "login[password]" = your.password , 
        "login[is_for_login]" = 1
    )

html <- postForm("https://usa.ipums.org/usa-action/users/validate_login", .params = params, curl = curl)
dl <- getURL( "https://usa.ipums.org/usa-action/extract_requests/download" , curl = curl)

现在我已登录，尝试与上述相同的命令，但使用 curl 对象来保留 cookie。

filename <- tempfile()
f <- CFILE(filename, mode = "wb")

这行换行--

curlPerform(url = extract.path, writedata = f@ref, curl = curl)
close(f)

# the error is:
Error in curlPerform(url = extract.path, writedata = f@ref, curl = curl) : 
  embedded nul in string: [[binary jibberish here]]

我上一篇文章的答案将我推荐给this c-level writefunction 答案，但我不知道如何重新创建 curl_writer C 程序（在 Windows 上？）..

dyn.load("curl_writer.so")
writer <- getNativeSymbolInfo("writer", PACKAGE="curl_writer")$address
curlPerform(URL=url, writefunction=writer)

..或者为什么它甚至是必要的，因为这个问题顶部的五行代码没有像getNativeSymbolInfo 这样疯狂的工作。我只是不明白为什么传入那个额外的 curl 对象来存储身份验证/cookie 并告诉它不要验证 SSL 会导致原本可以工作的代码..中断？

Answer 1

现在可以使用httr 包来实现。谢谢哈德利！

https://github.com/hadley/httr/issues/44

Answer 2

1. 从this link创建一个名为curl_writer.c的文件并将其保存到C:\<folder where you save your R files>

   #include <stdio.h>
   
   /**
    * Original code just sent some message to stderr
    */
   size_t writer(void *buffer, size_t size, size_t nmemb, void *stream) {
       fwrite(buffer,size,nmemb,(FILE *)stream);
       return size * nmemb;
   }
   

2. 打开命令窗口，进入保存curl_writer.c的文件夹并运行R编译器

   c:> cd "C:\<folder where you save your R files>"
   c:> R CMD SHLIB -o curl_writer.dll curl_writer.c
   

3. 打开 R 并运行您的脚本

   C:> R
   
   your.email <- "email@address.com"
   your.password <- "password"
   extract.path <- "https://usa.ipums.org/usa-action/downloads/extract_files/some_file.csv.gz"
   
   library(RCurl)
   
   values <- 
       list(
           "login[email]" = your.email , 
           "login[password]" = your.password , 
           "login[is_for_login]" = 1
       )
   
   curl = getCurlHandle()
   
   curlSetOpt(
       cookiejar = 'cookies.txt', 
       followlocation = TRUE, 
       autoreferer = TRUE, 
       ssl.verifypeer = FALSE,
       curl = curl
   )
   
   params <- 
       list(
           "login[email]" = your.email , 
           "login[password]" = your.password , 
           "login[is_for_login]" = 1
       )
   
   html <- postForm("https://usa.ipums.org/usa-action/users/validate_login", .params = params, curl = curl)
   dl <- getURL( "https://usa.ipums.org/usa-action/extract_requests/download" , curl = curl)
   
   # Load the DLL you created
   # "writer" is the name of the function
   # "curl_writer" is the name of the dll
   dyn.load("curl_writer.dll")
   writer <- getNativeSymbolInfo("writer", PACKAGE="curl_writer")$address
   
   # Note that "URL" parameter is upper case, in your code it is lowercase
   # I'm not sure if that has something to do
   # "writer" is the symbol defined above
   f <- CFILE(filename <- tempfile(), "wb")
   curlPerform(URL=url, writedata=f@ref, writefunction=writer, curl=curl)
   close(f)