使用 R 中的 Data.Table 或 Rcpp 用字符串快速替换 NA

Question

我有一个大表：10M 行乘 33 列，其中 28 列有一些 NA 值。这些 NA 值需要使用locf() 进行修补。我阅读了一些关于这个主题的主题（efficiently locf by groups in a single R data.table 和na.locf and inverse.rle in Rcpp）。但是，这些线程是关于替换数字向量的。我对Rcpp不太熟悉，所以我不知道如何更改他们的代码以适应字符串——我的数据都是字符串。

这是我的示例数据：

输入数据

Sample_File = structure(list(SO = c(112, 112, 112, 112, 113, 113, 113, 113), 
    Product.ID = c("AB123", "CD234", "DE345", "EF456", "FG456", 
    "GH567", "HI678", "IJ789"), Name = c(NA, NA, NA, "Human Being", 
    NA, "Lion", NA, "Bird"), Family = c(NA, NA, NA, "Homo Sapiens", 
    NA, NA, NA, "Passeridae"), SL1_Continent = c("Asia", NA, 
    "Asia", "Asia", NA, NA, NA, "Australia"), SL2_Country = c("China", 
    "China", NA, NA, NA, NA, NA, "Australia"), SL3_Direction = c("East", 
    NA, "East", "East", NA, NA, NA, "West"), Expiration_FY = c(2021, 
    NA, 2018, NA, 2012, 2012, NA, 2012), Flag = c("Y", NA, "N", 
    "N", NA, NA, NA, "TBD"), Insured = c("No", NA, NA, NA, NA, 
    NA, NA, "Yes"), Revenue = c(0, 478227.44, 0, 0, 0, 0, 125550.4, 
    44314.51), Quantity = c(1000, 100, 100, 4, 6, 6, 4, 6)), .Names = c("SO", 
"Product.ID", "Name", "Family", "SL1_Continent", "SL2_Country", 
"SL3_Direction", "Expiration_FY", "Flag", "Insured", "Revenue", 
"Quantity"), row.names = c(NA, 8L), class = "data.frame")

这是我使用data.table的代码：

data.table::setDT(Sample_File)
cols <- c("Name","Family","SL1_Continent","SL2_Country","SL3_Direction","Expiration_FY","Flag","Insured")
Sample_File[, (cols):=lapply(.SD, function(x){na.locf(x,fromLast = TRUE,na.rm=TRUE)}), by = SO, .SDcols = cols]

预期输出：

Output = structure(list(SO = c(112, 112, 112, 112, 113, 113, 113, 113), 
    Product.ID = c("AB123", "CD234", "DE345", "EF456", "FG456", 
    "GH567", "HI678", "IJ789"), Name = c("Human Being", "Human Being", 
    "Human Being", "Human Being", "Lion", "Lion", "Bird", "Bird"
    ), Family = c("Homo Sapiens", "Homo Sapiens", "Homo Sapiens", 
    "Homo Sapiens", "Passeridae", "Passeridae", "Passeridae", 
    "Passeridae"), SL1_Continent = c("Asia", "Asia", "Asia", 
    "Asia", "Australia", "Australia", "Australia", "Australia"
    ), SL2_Country = c("China", "China", "China", "China", "Australia", 
    "Australia", "Australia", "Australia"), SL3_Direction = c("East", 
    "East", "East", "East", "West", "West", "West", "West"), 
    Expiration_FY = c(2021, 2018, 2018, 2021, 2012, 2012, 2012, 
    2012), Flag = c("Y", "N", "N", "N", "TBD", "TBD", "TBD", 
    "TBD"), Insured = c("No", "No", "No", "No", "Yes", "Yes", 
    "Yes", "Yes"), Revenue = c(0, 478227.44, 0, 0, 0, 0, 125550.4, 
    44314.51), Quantity = c(1000, 100, 100, 4, 6, 6, 4, 6)), .Names = c("SO", 
"Product.ID", "Name", "Family", "SL1_Continent", "SL2_Country", 
"SL3_Direction", "Expiration_FY", "Flag", "Insured", "Revenue", 
"Quantity"), row.names = c(NA, -8L), class = "data.frame")

虽然执行上述代码只需要几分之一秒，但在我的原始数据集中处理一列需要大约 10 分钟，这意味着即使使用 data.table 也需要大约 280 分钟来处理 28 列。

我假设我并没有真正利用上面data.table 的力量。我不太确定。对于加速 na.locf() 函数的任何帮助，我将由衷地感谢。

有没有更有效的方法来替换上面的NA？

Answer 1

为了这个例子的目的，我简化了这个问题，但我想它很容易概括。下面的代码使用 C++11 语法在 Rcpp 中定义了locppf 函数：

#include <Rcpp.h>
using namespace Rcpp;

// [[Rcpp::plugins(cpp11)]]

using Map = std::unordered_map<double, int> ;
using Pair = Map::value_type ;

// [[Rcpp::export]]
CharacterVector locppf(NumericVector g, CharacterVector s) {
  auto n = g.size() ;
  CharacterVector out = clone(s) ;

  Map map ;
  for(int i=n-1; i>=0; i--){
    double value = g[i] ;
    auto it = map.find( value ) ;

    if( it == map.end() ){
      map.insert( Pair(value, i) ) ;
    } else {
      // if the current value is NA, replace it with the data at correct idx  
      auto current = s[i] ;
      if( CharacterVector::is_na( current ) ){
        out[i] = s[ it->second ] ;
      } else {
        it->second = i ;
      } 
    }
  }
  return out ;
}

这个想法是定义一个地图来跟踪我们上次在组中看到不是NA 的东西的索引。我使用std::unordered_map<double, int> 作为地图，因为您的示例还使用了数字组的向量。

让我们打破相关的掘金：

if( it == map.end() ){
  map.insert( Pair(value, i) ) ;
} 

这里我们检查地图是否已经看到当前值，如果没有，我们保留当前索引。

      auto current = s[i] ;
      if( CharacterVector::is_na( current ) ){
        out[i] = s[ it->second ] ;
      } else {
        it->second = i ;
      } 

这里我们用CharacterVector::is_na检查当前值是否为NA。

如果是，我们用我们之前保留的索引中的值填充结果向量。

如果没有，我们会更改地图为该组记住的索引。

现在让我们给自己一些数据：

library("zoo")
library("dplyr")
library("data.table")

with_holes <- function(x, p = .2){
  n <- length(x)
  x[ sample(n, n*p) ] <- NA
  x
}

n <- 1e6
x <- sample( as.numeric(1:100), n, replace= TRUE )
y <- with_holes( sample( letters, n, replace = TRUE) )
d <- data_frame( x = x, y = y )

并使用各种选项测量时间：

将 dplyr 语法与 group_by、mutate 和 na.locf 一起使用

> system.time( d %>% group_by(x) %>% mutate( y = na.locf(y, fromLast = TRUE, na.rm = FALSE) ) )
  user  system elapsed 
  0.173   0.023   0.198 

将data.table 语法与na.locf 一起使用。我不保证这是最好的data.table 方式来做到这一点。

> d2 <- as.data.table(d)
> system.time( d2[ , y := na.locf(y, fromLast = TRUE, na.rm = FALSE) , x ]  )
  user  system elapsed 
  0.159   0.030   0.188 

没有自定义locppf函数：

> system.time( locppf(d$x, d$y) )
  user  system elapsed 
  0.028   0.001   0.028