curl :: curl_fetch_memory（url，handle = handle）中的错误：发送失败：连接已重置（RStudio.cloud）

Question

我想从此网页获取 id_product 和 id_parent。 昨天，我可以得到结果，但是当我今天再次尝试时，我收到了错误消息。无论如何，我是从 rstudio.cloud 做的。

url <-  paste("https://www.tokopedia.com/zhafranseafood/cumi-asin-1kg-per-pack")

    headers = c('User-Agent' = 'Mozilla/5.0')
    doc <- read_html(httr::GET(url, httr::add_headers(.headers=headers)))%>%
          html_text()
    id_product <- str_match_all(doc,'product_id\\s+=\\s+(\\d+);')[[1]][,2]
    id_parent <- str_match_all(doc,'parent_id\\s+=\\s+(\\d+);')[[1]][,2]

    id_product
    id_parent

Error in curl::curl_fetch_memory(url, handle = handle) : 
  Send failure: Connection was reset

我一直在寻找可能的解释，但仍然无济于事。

Answer 1

服务器需要额外的标头

library(httr)
library(stringr)
library(magrittr)

headers = c(
  'User-Agent' = 'Mozilla/5.0',
  'Accept' = 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3'
)

doc <- read_html(httr::GET(url = 'https://www.tokopedia.com/zhafranseafood/cumi-asin-1kg-per-pack', httr::add_headers(.headers=headers)))%>%
       html_text()

id_product <- str_match_all(doc,'product_id\\s+=\\s+(\\d+);')[[1]][,2]
id_parent <- str_match_all(doc,'parent_id\\s+=\\s+(\\d+);')[[1]][,2]

id_product
id_parent