如何从 R 中的部分非结构化 txt 文件中提取表？答案

【问题标题】：How to extract tables from partly unstructured txt files in R?如何从 R 中的部分非结构化 txt 文件中提取表？
【发布时间】：2020-06-10 12:25:25
【问题描述】：

我有一个 txt 文件的 URL 列表。 txt 文件的结构使得有些部分是纯文本，有些部分是表格。我想提取表并将它们导出到数据框。下面是一个 URL 示例：

https://www.sec.gov/Archives/edgar/data/1000275/0001140361-13-007449.txt

txt 文件的结构使得表格以<TABLE> 开头并以</TABLE> 结尾。我想合并所有表格。我曾尝试使用 read.delim，但我不知道如何仅将其用于表格。以下是预期输出的示例。对于如何继续我的项目的任何指导，我将不胜感激。

Example of current df:
+----+--------------------------------------------------------------------------+
| ID |                                   URL                                    |
+----+--------------------------------------------------------------------------+
|  1 | https://www.sec.gov/Archives/edgar/data/1000097/0000919574-13-001835.txt |
|  2 | https://www.sec.gov/Archives/edgar/data/1000275/0001140361-13-007449.txt |
|  3 | https://www.sec.gov/Archives/edgar/data/1000742/0000898432-13-000218.txt |
+----+--------------------------------------------------------------------------+

Example of txt file from url:

text text text
text text text
text text text

<TABLE>
+---------------------+----------------+-----------+--------------+-----------------+--------+----------+-----------------------+-------------+------------------+
|   NAME OF ISSUER    | TITLE OF CLASS |   CUSIP   | VALUE (x1000 | SHRS OR PRN AMT | SH/PRN | PUT/CALL | INVESTMENT DISCRETION | OTHER MNGRS | VOTING AUTHORITY |
+---------------------+----------------+-----------+--------------+-----------------+--------+----------+-----------------------+-------------+------------------+
| ABBVIE INC          | COM            | 00287Y109 |        1,547 |          45,300 | SHS    |          | Shared-Defined        | 1/2/3       |           45,300 |
| ABERCROMBIE & FITCH | CL A           | 002896207 |        4,797 |         100,000 | SHS    |          | Shared-Defined        | 1/2/3       |          100,000 |
+---------------------+----------------+-----------+--------------+-----------------+--------+----------+-----------------------+-------------+------------------+
</TABLE>

<TABLE>
+---------------------+----------------+-----------+--------------+-----------------+--------+----------+-----------------------+-------------+------------------+
|   NAME OF ISSUER    | TITLE OF CLASS |   CUSIP   | VALUE (x1000 | SHRS OR PRN AMT | SH/PRN | PUT/CALL | INVESTMENT DISCRETION | OTHER MNGRS | VOTING AUTHORITY |
+---------------------+----------------+-----------+--------------+-----------------+--------+----------+-----------------------+-------------+------------------+
| ABBVIE INC          | COM            | 00287Y109 |        1,547 |          45,300 | SHS    |          | Shared-Defined        | 1/2/3       |           45,300 |
| ABERCROMBIE & FITCH | CL A           | 002896207 |        4,797 |         100,000 | SHS    |          | Shared-Defined        | 1/2/3       |          100,000 |
+---------------------+----------------+-----------+--------------+-----------------+--------+----------+-----------------------+-------------+------------------+
</TABLE>



Expected output:
+----+----------------+----------------+-------+--------------+-----------------+--------+----------+-----------------------+-------------+------------------+
| ID | NAME OF ISSUER | TITLE OF CLASS | CUSIP | VALUE (x1000 | SHRS OR PRN AMT | SH/PRN | PUT/CALL | INVESTMENT DISCRETION | OTHER MNGRS | VOTING AUTHORITY |
+----+----------------+----------------+-------+--------------+-----------------+--------+----------+-----------------------+-------------+------------------+
|  1 | x              | x              | x     | x            | x               | x      | x        | x                     | x           | x                |
|  1 | x              | x              | x     | x            | x               | x      | x        | x                     | x           | x                |
|  1 | x              | x              | x     | x            | x               | x      | x        | x                     | x           | x                |
|  2 | x              | x              | x     | x            | x               | x      | x        | x                     | x           | x                |
|  2 | x              | x              | x     | x            | x               | x      | x        | x                     | x           | x                |
|  2 | x              | x              | x     | x            | x               | x      | x        | x                     | x           | x                |
+----+----------------+----------------+-------+--------------+-----------------+--------+----------+-----------------------+-------------+------------------+

【问题讨论】：

嗯，第一步是定位<TABLE> 和</TABLE> 之间的文本块。你是怎么做到的？然后你需要解析每个块中的单元格定义。给我们一些可以合作的东西！
不幸的是，我也被困在那个部分。我在网上查看并尝试了几种方法，包括fread、read.pattern 和Readlines，但我无法让它们按预期工作。

标签： r dataframe text-extraction readr

【解决方案1】：

这是一个粗略的解决方案。

# Read the text files from the web
fileContents <- readr::read_file("https://www.sec.gov/Archives/edgar/data/1000275/0001140361-13-007449.txt")
# Extract the tables.  The regex isn't quite right, as it includes the starting <TABLE>
# and ending </TABLE> tags, but more complicated regexes failed.  Regex isn't my
# strong point, and I can handle the extra work
tables <- stringr::str_extract_all(
            fileContents, 
            stringr::regex("(?s)<TABLE>(.*?)</TABLE>", 
              multiline=TRUE, 
              dotall=TRUE
            )
          )

# Function to process a single tibble
toTibble <- function(y) {
  lines <- str_split_fixed(y, "\n", n=Inf)
  colStarts <- c()
  colEnds <- c()
  # Scroll through to final table header
  for (i in 1:(length(lines)-1)) { # Final line is '</TABLE>' because of initial regex
    # Could probably to this with regexes, but my head is hurting
    if (any(!is.na(stringr::str_locate(lines[i], "<\\w>")))) {
      # Define column widths based on locations of type markers.  THIS IS AN ASSUMPTION
      colStarts <- stringr::str_locate_all(lines[i], "<\\w>")[[1]][,"start"]
      for (i in 1:(length(colStarts)-1)) colEnds[i] <- colStarts[i+1] -1
      colEnds[length(colStarts)] <- stringr::str_length(lines[i])
      lines <- lines[(i+1):(length(lines)-1)]
      data <- dplyr::bind_rows(
                lapply(
                  lines,                   # For each data line  
                  function(line) 
                    tibble::enframe(       # Split in to columns and convert to a tibble of name/value pairs
                      stringr::str_trim(
                        stringr::str_sub(
                          line, 
                          colStarts, 
                          colEnds
                        )
                      )
                    ) %>%                  # Convert from name/value pairs to columns
                    tidyr::pivot_wider(
                      values_from="value", 
                      names_from="name", 
                      names_prefix="Column"
                    )
                  )
                )
      # Finished
      return(data)
    }
  }
}

处理单个文件：

firstTable <- toTibble(tables[[1]][[1]])
firstTable

给予

# A tibble: 59 x 12
   Column1  Column2       Column3   Column4 Column5 Column6 Column7 Column8 Column9 Column10 Column11 Column12
   <chr>    <chr>         <chr>     <chr>   <chr>   <chr>   <chr>   <chr>   <chr>   <chr>    <chr>    <chr>   
 1 AAR CORP COM           000361105 190     10158   SH      ""      DEFINED "2"     10158    0        0       
 2 AAR CORP COM           000361105 15      803     SH      ""      DEFINED "3"     0        0        803     
 3 AAR CORP COM           000361105 37      2000    SH      ""      DEFINED "5"     2000     0        0       
 4 AAR CORP COM           000361105 78      4200    SH      ""      DEFINED ""      4200     0        0       
 5 ABB LTD  SPONSORED ADR 000375204 2164    104112  SH      ""      DEFINED "3"     257      0        103855  
 6 ABB LTD  SPONSORED ADR 000375204 10774   518215  SH      ""      DEFINED "5"     518215   0        0       
 7 ABB LTD  SPONSORED ADR 000375204 64      3100    SH      ""      DEFINED "7"     0        3100     0       
 8 ABB LTD  SPONSORED ADR 000375204 1044    50200   SH      ""      DEFINED "8"     50200    0        0       
 9 ABB LTD  SPONSORED ADR 000375204 9       410     SH      ""      DEFINED "9"     410      0        0       
10 ABB LTD  SPONSORED ADR 000375204 103     4958    SH      ""      DEFINED "15"    4958     0        0       
# … with 49 more rows

在我的系统上

system.time({firstTable <- toTibble(tables[[1]][[1]])})
   user  system elapsed 
  0.843   0.004   0.849

所以在一秒钟内处理一张表。

length(tables[[1]])
[1] 299

文件中只有不到 300 个表，因此将所有表绑定到一个 tibble 中

alldata <- bind_rows(lapply(tables[[1]], function(t) toTibble(t)))

大约需要五分钟来提取表格，再加上一点时间来将它们绑定在一起。 [此代码未经测试或计时。]

一旦您处于此阶段，您就可以尝试使用列类型和名称来获得您想要的内容。这应该很简单。

【讨论】：