无法在 R 中下载文件 - 状态 503答案

【问题标题】：Cannot download file in R - status 503无法在 R 中下载文件 - 状态 503
【发布时间】：2018-05-23 13:22:52
【问题描述】：

我正在尝试下载文件：

> URL <- "https://www.bitmarket.pl/graphs/BTCPLN/90m.json"
> download.file(URL, destfile = "res.json", method = "curl")
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4676    0  4676    0     0  56930          0 --:--:-- --:--:-- --:--:-- 57024

但它返回 503 状态。整体输出：

<!DOCTYPE HTML>
<html lang="en-US">
<head>
  <meta charset="UTF-8" />
  <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
  <meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
  <meta name="robots" content="noindex, nofollow" />
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1" />
  <title>Just a moment...</title>
  <style type="text/css">
    html, body {width: 100%; height: 100%; margin: 0; padding: 0;}
    body {background-color: #ffffff; font-family: Helvetica, Arial, sans-serif; font-size: 100%;}
    h1 {font-size: 1.5em; color: #404040; text-align: center;}
    p {font-size: 1em; color: #404040; text-align: center; margin: 10px 0 0 0;}
    #spinner {margin: 0 auto 30px auto; display: block;}
    .attribution {margin-top: 20px;}
    @-webkit-keyframes bubbles { 33%: { -webkit-transform: translateY(10px); transform: translateY(10px); } 66% { -webkit-transform: translateY(-10px); transform: translateY(-10px); } 100% { -webkit-transform: translateY(0); transform: translateY(0); } }
    @keyframes bubbles { 33%: { -webkit-transform: translateY(10px); transform: translateY(10px); } 66% { -webkit-transform: translateY(-10px); transform: translateY(-10px); } 100% { -webkit-transform: translateY(0); transform: translateY(0); } }
    .bubbles { background-color: #404040; width:15px; height: 15px; margin:2px; border-radius:100%; -webkit-animation:bubbles 0.6s 0.07s infinite ease-in-out; animation:bubbles 0.6s 0.07s infinite ease-in-out; -webkit-animation-fill-mode:both; animation-fill-mode:both; display:inline-block; }
  </style>

    <script type="text/javascript">
  //<![CDATA[
  (function(){
    var a = function() {try{return !!window.addEventListener} catch(e) {return !1} },
    b = function(b, c) {a() ? document.addEventListener("DOMContentLoaded", b, c) : document.attachEvent("onreadystatechange", b)};
    b(function(){
      var a = document.getElementById('cf-content');a.style.display = 'block';
      setTimeout(function(){
        var s,t,o,p,b,r,e,a,k,i,n,g,f, eoQNdpG={"GwwAAtfX":+((+!![]+[])+(+!![]))};
        t = document.createElement('div');
        t.innerHTML="<a href='/'>x</a>";
        t = t.firstChild.href;r = t.match(/https?:\/\//)[0];
        t = t.substr(r.length); t = t.substr(0,t.length-1);
        a = document.getElementById('jschl-answer');
        f = document.getElementById('challenge-form');
        ;eoQNdpG.GwwAAtfX+=+((!+[]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]));eoQNdpG.GwwAAtfX*=+((!+[]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]));eoQNdpG.GwwAAtfX-=+((!+[]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]));eoQNdpG.GwwAAtfX-=+((+!![]+[])+(+[]));eoQNdpG.GwwAAtfX-=+((+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]));eoQNdpG.GwwAAtfX+=+((!+[]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]));eoQNdpG.GwwAAtfX+=+((!+[]+!![]+!![]+!![]+[])+(!+[]+!![]+!![]+!![]+!![]+!![]+!![]+!![]+!![]));eoQNdpG.GwwAAtfX*=+((!+[]+!![]+!![]+[])+(!+[]+!![]+!![]));a.value = parseInt(eoQNdpG.GwwAAtfX, 10) + t.length; '; 121'
        f.action += location.hash;
        f.submit();
      }, 4000);
    }, false);
  })();
  //]]>
</script>


</head>
<body>
  <table width="100%" height="100%" cellpadding="20">
    <tr>
      <td align="center" valign="middle">
          <div class="cf-browser-verification cf-im-under-attack">
  <noscript><h1 data-translate="turn_on_js" style="color:#bd2426;">Please turn JavaScript on and reload the page.</h1></noscript>
  <div id="cf-content" style="display:none">

    <div>
      <div class="bubbles"></div>
      <div class="bubbles"></div>
      <div class="bubbles"></div>
    </div>
    <h1><span data-translate="checking_browser">Checking your browser before accessing</span> bitmarket.pl.</h1>

    <p data-translate="process_is_automatic">This process is automatic. Your browser will redirect to your requested content shortly.</p>
    <p data-translate="allow_5_secs">Please allow up to 5 seconds&hellip;</p>
  </div>

  <form id="challenge-form" action="/cdn-cgi/l/chk_jschl" method="get">
    <input type="hidden" name="jschl_vc" value="51a7cb71596dbf54fdd307c1e65de941"/>
    <input type="hidden" name="pass" value="1512824604.589-Uwtm9TfzWe"/>
    <input type="hidden" id="jschl-answer" name="jschl_answer"/>
  </form>
</div>


          <div class="attribution">
            <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=iuam" target="_blank" style="font-size: 12px;">DDoS protection by Cloudflare</a>
            <br>
            Ray ID: 3ca829f9aed06afb
          </div>
      </td>

    </tr>
  </table>
</body>
</html>

wget 也不行：

--2017-12-09 14:01:29--  https://www.bitmarket.pl/graphs/BTCPLN/90m.json
Resolving www.bitmarket.pl... 104.20.67.184, 104.20.68.184
Connecting to www.bitmarket.pl|104.20.67.184|:443... connected.
HTTP request sent, awaiting response... 503 Service Temporarily Unavailable
2017-12-09 14:01:29 ERROR 503: Service Temporarily Unavailable.

但是当您访问此链接时：https://www.bitmarket.pl/graphs/BTCPLN/90m.json 您的网络浏览器将返回正确的 json 文件。任何想法为什么它不起作用？

【问题讨论】：

标签： r curl download wget http-status-code-503

【解决方案1】：

这是因为该页面正在使用 DDoS 保护服务。在第一次加载时，页面本身会在 5 秒后执行由 JavaScript 发起的重定向以获取最终内容，因此该过程会因 wget/curl 等不解释 JavaScript 的工具而失败。如果您认为这样做是合理的，那么一种选择是使用例如 phantomjs 并提供自定义脚本（例如，save.js）：

var system = require('system');
var page = require('webpage').create();

page.userAgent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5';

page.open(system.args[1], function(){
    setTimeout(function(){
        console.log(page.evaluate(function(){
            //gets the JSON from the first <pre> element rendered on the page
            return document.getElementsByTagName('pre')[0].textContent;
        }));
        phantom.exit();
    }, 6000); //waits 6 seconds for the page to reload
});

然后用它代替wget作为：

phantomjs save.js https://www.bitmarket.pl/graphs/BTCPLN/90m.json

【讨论】：

【解决方案2】：

无需离开 R。我们可以为此使用 V8 包并制作一个特殊的 GET 函数：

#' Work around cloudflare anti-DDoS protection
#' 
#' SUPER FRAGILE AS IT NEEDS TO BE MODIFIED WHENEVER CLOUDFLARE CHANGES THE CHALLENGE CODE
#' 
#' @param cf_url the URL you want
#' @param ... other params passed to all `httr::GET`` calls (headers, verbose, etc)
#' @return `httr::response object``
cf_GET <- function(cf_url, ...) {

  require(urltools)
  require(stringi)
  require(rvest)
  library(httr)
  require(V8)

  c(
    ua_macos_chrome = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36",
    ua_ios_safari = "Mozilla/5.0 (iPad; CPU OS 10_2 like Mac OS X) AppleWebKit/602.3.12 (KHTML, like Gecko) Version/10.0 Mobile/14C92 Safari/602.1",
    ua_win7_firefox = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0"
  ) -> agents

  # use a valid browser user-agent but don't always use the same one
  (cf_agent <- unname(sample(agents, 1)))

  httr::GET(
    url = cf_url,
    httr::user_agent(cf_agent),
    ...
  ) -> res

  # sometimes you get lucky and the page comes back
  if (!httr::status_code(res) == 503) return(res) # return now if no cf redirect

  # get the page
  cf_pg <- httr::content(res, as="parsed")

  # get form/form variables we'll need later
  (jschl_vc <- html_attr(html_node(cf_pg, "input[name='jschl_vc']"), "value"))
  (pass <- html_attr(html_node(cf_pg, "input[name='pass']"), "value"))
  (action <- html_attr(html_node(cf_pg, "form[id='challenge-form']"), "action"))

  # get the page as just lines of text
  cf_code <- httr::content(res, as="text")
  writeLines(cf_code, "/tmp/a.html")
  cf_code <- stri_split_lines(cf_code)[[1]]

  # find the javascript
  decl <- cf_code[which(stri_detect_fixed(cf_code, "s,t,o,p,b"))]
  (init_line <- stri_match_first_regex(decl, "s,t,o,p,b[[:alpha:], ]+ (.*$)")[,2])
  (var_name <- stri_match_first_regex(init_line, "([[:alnum:]]+)")[,2])
  (exec_line <- cf_code[which(stri_detect_fixed(cf_code, var_name))[2]])

  # tweak and execute the javascript
  ctx <- v8()
  ctx$eval(sprintf("var a = {}; t = '%s';%s\n%s", domain(cf_url), decl, exec_line))
  (ctx$get("a.value"))

  # this lying but you can wait 10s
  message("Waiting 5 seconds...")
  Sys.sleep(10)

  # solve the DDoS challenge and make the request
  httr::GET(
    url = sprintf("%s://%s/%s", scheme(cf_url), domain(cf_url), action),
    httr::user_agent(cf_agent),
    httr::add_headers(
      `Referer` = cf_url
    ),
    query = list(
      `jschl-answer` = ctx$get("a.value"),
      jschl_vc = jschl_vc,
      pass = pass
    ),
    ...
  ) -> res

  res

}

而且，它有效：

res <- cf_GET("https://www.bitmarket.pl/graphs/BTCPLN/90m.json")

str(content(res, as="parsed"))
## List of 90
##  $ :List of 6
##   ..$ time : int 1512906360
##   ..$ open : chr "48303.78770000"
##   ..$ high : chr "48303.78770000"
##   ..$ low  : chr "48303.78770000"
##   ..$ close: chr "48303.78770000"
##   ..$ vol  : chr "0.13550275"
##  $ :List of 6
##   ..$ time : int 1512906420
##   ..$ open : chr "48303.78770000"
##   ..$ high : chr "48303.78770000"
##   ..$ low  : chr "48000.10000000"
##   ..$ close: chr "48000.10000000"
##   ..$ vol  : chr "1.12078334"
## ...

更新：

我把它包裹在一个包裹里：

devtools::install_github("hrbrmstr/cfhttr")
library(cfhttr)

res <- cf_GET("https://www.bitmarket.pl/graphs/BTCPLN/90m.json")

（相同的输出）

【讨论】：

令人印象深刻，看起来 R 确实是终极工具！ :)
我已经实现了@ewcz 解决方案，但这个看起来也很棒。我也需要检查一下；）