当有多行跨列时在 RVest 中抓取表答案

【问题标题】：Scraping table in RVest when there are multiple rows that span columns当有多行跨列时在 RVest 中抓取表
【发布时间】：2023-02-11 06:25:00
【问题描述】：

我正在尝试抓取以下网页：https://rptsvr1.tea.texas.gov/cgi/sas/broker?_service=marykay&_program=sfadhoc.budget_report_2022.sas&_service=appserv&_debug=0&who_box=&who_list=031901

最初，我尝试了这段代码：

library(rvest)
library(tidyverse)
webpage = read_html("https://rptsvr1.tea.texas.gov/cgi/sas/broker?_service=marykay&_program=sfadhoc.budget_report_2022.sas&_service=appserv&_debug=0&who_box=&who_list=101912")
tables <- html_nodes(webpage, "table") %>%
  html_table()
budget = tables[[2]]

然后我意识到结果很乱，因为 html_table() 无法读取行跨越多列的表。我可以在抓取数据帧后清理它，但我想知道是否有一种方法可以从一开始就避免这个问题。

看过类似问题的答案，都涉及到只有表头行跨多列，或者只有第一列跨多行的情况。在这种情况下，该表由多个压缩在一起的表组成，因此整个表中都有标题。是否有解决方案可以处理整个表中的行跨越列？

【问题讨论】：

这只是为了好玩吗？有一个下载为 Excel 选项。我假设您想最终得到一个类似于平面文件格式的数据帧？

标签： r web-scraping rvest

【解决方案1】：

一种方法如下：

确定您的最终格式。我选择了一种平面文件格式，它需要将一些标题值（具有合并单元格的标题值）拆分为单独的新列，并在以后重复这些值
收集并循环所有数据行
在循环中根据一组规则处理行
在循环中生成输出行并写入大小列表
删除空条目列表

转换为 DataFrame 并添加所需的任何附加信息

我选择应用的规则基于查看每一行的第一列，如下所示（伪代码）：

if (first column className contains "linecontent") {
  if (if first column has br and span child elements) {
    split colmn text, after trimming, on line break
    assign split to red and measure columns (re-use as filldown)
  } else {
    assign measure the trimmed column text
  }
} else {
  we are on a non-header row so grab the $ and % values by matching on class "data"
}

回复:

library(tidyverse)
library(rvest)
library(httr2)

page <- request("https://rptsvr1.tea.texas.gov/cgi/sas/broker?_service=marykay&_program=sfadhoc.budget_report_2022.sas&_service=appserv&_debug=0&who_box=&who_list=031901") %>%
  req_headers(
    "user-agent" = "Mozilla/5.0",
    "accept" = "text/html",
    "connection" = "keep-alive"
  ) %>%
  req_perform() %>%
  resp_body_html()

rows <- page %>% html_elements(".table tbody tr")

row_data <- vector("list", length(rows))
row <- 1

for (row_node in rows) {
  first_column_node <- row_node %>% html_element("td")
  first_column_node_classname <- html_attr(first_column_node, "class")
  is_new_column <- if_else(grepl("linecontent", first_column_node_classname), 1, 0)
  column_needs_split <- if_else((length(html_elements(first_column_node, "br")) > 0 &
    !is.na(html_element(first_column_node, "span"))), 1, 0)
  node_text <- first_column_node %>% html_text2()
  data <- NULL

  if (is_new_column) {
    if (column_needs_split) {
      new_values <- trimws(str_split_1(trimws(node_text), "\n"))
      red <- new_values[[1]]
      measure <- new_values[[2]]
    } else {
      measure <- trimws(node_text)
    }
  } else {
    data <- row_node %>%
      html_elements(".data") %>%
      html_text2()
  }

  if (!is.null(data)) {
    row_data[[row]] <- c(c(red, measure), data)
    # print(c(c(red, measure), data))
    row <- row + 1
  }
}



row_data <- discard(row_data, is.null)
df <- do.call(rbind, row_data) %>% as.data.frame()

colnames(df) <- c(
  "red", "category", "measure",
  "gen_fund", "gen_fund_perc", "gen_fund_per_student",
  "all_fund", "all_fund_perc", "all_fund_per_student"
)

additional_info <- str_match_all(
  gsub("\n", "", summary_info),
  "(\d{4}\s-\s\d{4}).*Totals for (.*?)\sISD.*?\((\d{6})\)"
)


df$year <- additional_info[[1]][, 2]
df$district <- additional_info[[1]][, 3]
df$isd_code <- additional_info[[1]][, 4]

head(df)

示例输出：

> head(df)
       red          category                                           measure     gen_fund gen_fund_perc
1 Revenues Operating Revenue Local Property Tax from M&O (excluding recapture)  $70,019,020        15.03%
2 Revenues Operating Revenue                             State Operating Funds $333,999,269        71.68%
3 Revenues Operating Revenue                                     Federal Funds  $59,326,937        12.73%
4 Revenues Operating Revenue                                       Other Local   $2,644,317         0.57%
5 Revenues Operating Revenue                           Total Operating Revenue $465,989,543       100.00%
6 Revenues     Other Revenue                       Local Property Tax from I&S           $0         0.00%
  gen_fund_per_student     all_fund all_fund_perc all_fund_per_student        year    district isd_code
1               $1,823  $70,019,020        15.03%               $1,823 2021 - 2022 BROWNSVILLE   031901
2               $8,695 $333,999,269        71.68%               $8,695 2021 - 2022 BROWNSVILLE   031901
3               $1,544  $59,326,937        12.73%               $1,544 2021 - 2022 BROWNSVILLE   031901
4                  $69   $2,644,317         0.57%                  $69 2021 - 2022 BROWNSVILLE   031901
5              $12,131 $465,989,543       100.00%              $12,131 2021 - 2022 BROWNSVILLE   031901
6                   $0   $9,212,992        17.23%                 $240 2021 - 2022 BROWNSVILLE   031901

【讨论】：