R：带有隐藏字段的 Web 抓取 .aspx 表单，“未知字段名称”错误

Question

这两天，我一直在琢磨如何填写表格并提交以从https://www.igb.illinois.gov/VideoReports.aspx 下载.csv 文件。不幸的是，我似乎无法破解它。完全披露：我是一个新手网络爬虫。我可以进行基本的抓取，但这对我来说是新的领域。我希望最终能编写一个程序，将所有机构的月度收入报告拉回到 2009 年 9 月。

似乎主要问题与表单的布局方式有关。我似乎无法弄清楚如何指定要填写以请求 .csv 文件的字段。我一直在使用rvest 和RHTMLForms。我在 chrome 开发工具中找到了表单，可以看到我需要的一切。我似乎无法深入到我需要去哪里提交查询。

到目前为止，我已经做到了：

library('rvest')
library('RHTMLForms')

igb <- "https://www.igb.illinois.gov/VideoReports.aspx"
igb_html <- read_html(igb)

igbForm <- html_form(igb_html)
igbForm

问题似乎从这里开始。 “表单”只有一个元素，它包括隐藏的输入。我要查询的字段接近尾声。看起来是这样的……

[[1]]
<form> 'aspnetForm' (POST VideoReports.aspx)
  <input hidden> '__VIEWSTATE': /wEPDwUKMTU1MTExNzA3NQ9kFgJmD2QWAgIDD2QWAgIBD2QWBAIBD2QWEgIDDw8WAh4EVGV4dAUOU2VwdGVtYmVyIDIwMTJkZAIFDw8WAh8ABQ1GZWJydWFyeSAyMDIwZGQCFQ9kFgICAw8QZBAVAg5TdW1tYXJ5IHJlcG9ydA1EZXRhaWwgcmVwb3J0FQIOU3VtbWFyeSByZXBvcnQNRGV0YWlsIHJlcG9ydBQrAwJnZ2RkAhcPZBYCAgMPEA8WBh4ORGF0YVZhbHVlRmllbGQFA0tleR4NRGF0YVRleHRGaWVsZAUFVmFsdWUeC18 ....[TRUNCATE]

最后，我得到了我想查询的内容......

  <input radio> 'ctl00$MainPlaceHolder$SearchType': TypeStatewide
  <input radio> 'ctl00$MainPlaceHolder$SearchType': TypeMuni
  <input radio> 'ctl00$MainPlaceHolder$SearchType': TypeEst
  <select> 'ctl00$MainPlaceHolder$SearchStateType' [1/2]
  <select> 'ctl00$MainPlaceHolder$SearchMunicipality' [0/1069]
  <select> 'ctl00$MainPlaceHolder$SearchEstablishment' [0/10182]
  <input text> 'ctl00$MainPlaceHolder$SearchLicenseNumber': 
  <select> 'ctl00$MainPlaceHolder$SearchStartMonth' [1/12]
  <select> 'ctl00$MainPlaceHolder$SearchStartYear' [1/9]
  <select> 'ctl00$MainPlaceHolder$SearchEndMonth' [1/12]
  <select> 'ctl00$MainPlaceHolder$SearchEndYear' [1/9]
  <input radio> 'ctl00$MainPlaceHolder$ViewType': ViewPDF
  <input radio> 'ctl00$MainPlaceHolder$ViewType': ViewCSV

我使用以下内容来达到我需要的效果......

igb_form <- getHTMLFormDescription(igb_html)
igb_form[[1]]

... 以及此代码来定位每个字段和值。比如……

igb_form_att <- igb_form[[1]]
igb_form_att$elements[[9]]

...从下拉菜单中显示开始月份字段和值...

ctl00$MainPlaceHolder$SearchStartMonth: [ February ]  January, February, March, April, May, June, July, August, September, October, November, December

我以为这样就可以了。所以我跑了以下......

igb_fill <- set_values(igb_html,
                      'ctl00$MainPlaceHolder$SearchType' = 'TypeEst',
                      'ctl00$MainPlaceHolder$SearchEstablishment'='All Establishments',
                      'ctl00$MainPlaceHolder$SearchEstablishment' ='',
                      'ctl00$MainPlaceHolder$SearchStartMonth'='September',
                      'ctl00$MainPlaceHolder$SearchStartYear'='2009',
                      'ctl00$MainPlaceHolder$SearchEndMonth' ='February',
                      'ctl00$MainPlaceHolder$SearchEndYear'='2020',
                      'ctl00$MainPlaceHolder$ViewType'='ViewCSV')

submit_form(session=igb_html, form=igb_fill, POST(igb))

但收到此错误...

Error: Unknown field names: ctl00$MainPlaceHolder$SearchType, ctl00$MainPlaceHolder$SearchEstablishment, ctl00$MainPlaceHolder$SearchStartMonth, ctl00$MainPlaceHolder$SearchStartYear, ctl00$MainPlaceHolder$SearchEndMonth, ctl00$MainPlaceHolder$SearchEndYear, ctl00$MainPlaceHolder$ViewType
Traceback:

1. set_values(igb_form, `ctl00$MainPlaceHolder$SearchType` = "TypeEst", 
 .     `ctl00$MainPlaceHolder$SearchEstablishment` = "All Establishments", 
 .     `ctl00$MainPlaceHolder$SearchEstablishment` = "", `ctl00$MainPlaceHolder$SearchStartMonth` = "September", 
 .     `ctl00$MainPlaceHolder$SearchStartYear` = "2009", `ctl00$MainPlaceHolder$SearchEndMonth` = "February", 
 .     `ctl00$MainPlaceHolder$SearchEndYear` = "2020", `ctl00$MainPlaceHolder$ViewType` = "ViewCSV")
2. stop("Unknown field names: ", paste(no_match, collapse = ", "), 
 .     call. = FALSE)

对于这个冗长的问题，我深表歉意，但我对此进行了很多探讨，似乎无法找到可以帮助我到达我需要去的地方的答案。也许我在我的头上。但我会很感激任何帮助！ （我也很确定提交代码是错误的，但之后我可以解决这个问题。）

Answer 1

您的代码存在一些问题：

• set_values(...) 函数采用一种形式，而不是整个 html，所以我在那里将 igb_html 替换为 igb_form。
• submit_form(...) 函数采用html_session，因此我将read_html(igb) 替换为html_session(igb)。

以下代码应该可以工作：

library(rvest)

igb <- "https://www.igb.illinois.gov/VideoReports.aspx"
igb_html <- html_session(igb)

igb_form <- html_form(igb_html)[[1]]

igb_fill <- set_values(igb_form,
                       'ctl00$MainPlaceHolder$SearchType' = 'TypeEst',
                       'ctl00$MainPlaceHolder$SearchEstablishment'='All Establishments',
                       'ctl00$MainPlaceHolder$SearchEstablishment' ='',
                       'ctl00$MainPlaceHolder$SearchStartMonth'='September',
                       'ctl00$MainPlaceHolder$SearchStartYear'='2009',
                       'ctl00$MainPlaceHolder$SearchEndMonth' ='February',
                       'ctl00$MainPlaceHolder$SearchEndYear'='2020',
                       'ctl00$MainPlaceHolder$ViewType'='ViewCSV')

igb_html <- submit_form(igb_html, igb_fill, submit = "ctl00$MainPlaceHolder$ButtonSearch")

igb_html