在 r 中比 gsub 更快的方法答案

【问题标题】：Faster approach than gsub in r在 r 中比 gsub 更快的方法
【发布时间】：2015-05-30 04:50:41
【问题描述】：

我正在尝试找出是否有比 R 中的 gsub 矢量化函数更快的方法。我有以下带有一些“句子”（sent$words）的数据框，然后我有单词从这些句子中删除（存储在 wordsForRemoving 变量中）。

sent <- data.frame(words = 
                     c("just right size and i love this notebook", "benefits great laptop",
                       "wouldnt bad notebook", "very good quality", "bad orgtop but great",
                       "great improvement for that bad product but overall is not good", 
                       "notebook is not good but i love batterytop"), 
                   user = c(1,2,3,4,5,6,7),
                   stringsAsFactors=F)

wordsForRemoving <- c("great","improvement","love","great improvement","very good","good",
                      "right", "very","benefits", "extra","benefit","top","extraordinarily",
                      "extraordinary", "super","benefits super","good","benefits great",
                      "wouldnt bad")

那我要为时间消耗计算创建“大数据”模拟...

df.expanded <- as.data.frame(replicate(1000000,sent$words))
library(zoo)
sent <- coredata(sent)[rep(seq(nrow(sent)),1000000),]
rownames(sent) <- NULL

使用以下 gsub 方法从 sent$words 中删除单词 (wordsForRemoving) 需要 72.87 秒。我知道，这不是很好的模拟，但在现实中，我使用超过 3.000 个单词的字典来处理 300.000 个句子，整个处理需要超过 1.5 小时。

pattern <- paste0("\\b(?:", paste(wordsForRemoving, collapse = "|"), ")\\b ?")
res <- gsub(pattern, "", sent$words)

#  user  system elapsed 
# 72.87    0.05   73.79

请，任何人都可以帮助我为我的任务编写更快的方法。非常感谢任何帮助或建议。非常感谢转发。

【问题讨论】：

通过使用stringi::stri_replace_all_regex(sent$words, pattern, "")，您将获得一些改进（在我尝试过的示例中为 60%，其复制量减少了）

标签： regex r

【解决方案1】：

我构建了两个标记器函数，但有一个区别，第一个函数使用 gsub，第二个函数使用来自 stringr package 的 str_replace_all。
这是第一个功能：

tokenize_gsub <- function(df){

    require(lexicon)
    require(dplyr)
    require(tidyr)
    require(tidytext)
    myStopWords <- c(
        "ø",
        "øthe",
        "iii"
    )

    profanity <- c(
        profanity_alvarez,
        profanity_arr_bad,
        profanity_banned,
        profanity_racist,
        profanity_zac_anger
    ) %>%
        unique()

    df %>%
        mutate(text = gsub(x = text, pattern = "[0-9]+|[[:punct:]]|\\(.*\\)", replacement = "")) %>%
        unnest_tokens(word, text) %>%
        anti_join(stop_words, by = "word") %>%
        anti_join(tibble(word = profanity), by = "word") %>%
        anti_join(tibble(word = myStopWords), by = "word")

}

这是第二个功能：

tokenize_stringr <- function(df){

    require(stringr)
    require(lexicon)
    require(dplyr)
    require(tidyr)
    require(tidytext)

    myStopWords <- c(
        "ø",
        "øthe",
        "iii"
    )

    profanity <- c(
        profanity_alvarez,
        profanity_arr_bad,
        profanity_banned,
        profanity_racist,
        profanity_zac_anger
    ) %>%
        unique()

    df %>%
        mutate(text = str_replace_all(text, "[0-9]+|[[:punct:]]|\\(.*\\)", "")) %>%
        unnest_tokens(word, text) %>%
        anti_join(stop_words, by = "word") %>%
        anti_join(tibble(word = profanity), by = "word") %>%
        anti_join(tibble(word = myStopWords), by = "word")

}

然后，我使用了一个基准函数来将性能与包含 4,269,678 个社交媒体帖子（推特、博客等）的数据集进行比较

library(microbenchmark)
mc <- microbenchmark(
    gsubOption = tokenize_gsub(englishPosts),
    stringrOption = tokenize_stringr(englishPosts)
)

mc

这是输出：

Unit: seconds
          expr      min       lq     mean   median       uq      max neval cld
    gsubOption 161.4945 175.3040 211.6979 197.5054 240.6451 376.2927   100   b
 stringrOption 101.4138 117.0748 142.9605 132.4253 159.6291 328.1517   100  a

结论：在上述条件下，str_replace_all 函数比 gsub 选项快得多。

【讨论】：

【解决方案2】：

这不是一个真正的答案，因为我没有找到任何总是更快的方法。显然，这取决于您的文本/矢量的长度。短文本gsub 表现最快。对于较长的文本或矢量，有时 gsub 和 perl=TRUE 有时 stri_replace_all_regex 执行速度最快。

这里有一些测试代码可以试用：

library(stringi)
text = "(a1,\"something (f fdd71)\");(b2,\"something else (a fa171)\");(b4,\"something else (a fa171)\")"
# text = paste(rep(text, 5), collapse = ",")
# text = rep(text, 100)
nchar(text)

a = gsub(pattern = "[()]", replacement = "", x = text)
b = gsub(pattern = "[()]", replacement = "", x = text, perl=T)
c = stri_replace_all_regex(str = text, pattern = "[()]", replacement = "")
d = stri_replace(str = text, regex = "[()]", replacement = "", mode="all")

identical(a,b); identical(a,c); identical(a,d)

library(microbenchmark)
mc <- microbenchmark(
  gsub = gsub(pattern = "[()]", replacement = "", x = text),
  gsub_perl = gsub(pattern = "[()]", replacement = "", x = text, perl=T),
  stringi_all = stri_replace_all_regex(str = text, pattern = "[()]", replacement = ""),
  stringi = stri_replace(str = text, regex = "[()]", replacement = "", mode="all")
)
mc

Unit: microseconds
        expr    min      lq     mean  median     uq     max neval  cld
        gsub 10.868 11.7740 13.47869 13.5840 14.490  31.394   100 a   
   gsub_perl 79.690 80.2945 82.58225 82.4070 83.312 137.043   100    d
 stringi_all 14.188 14.7920 15.58558 15.5460 16.301  17.509   100  b  
     stringi 36.828 38.0350 39.90904 38.7895 39.543 129.194   100   c

【讨论】：

【解决方案3】：

正如 Jason 所说，stringi 对你来说是个不错的选择..

以下是stringi的表现

system.time(res <- gsub(pattern, "", sent$words))
   user  system elapsed 
 66.229   0.000  66.199 

library(stringi)
system.time(stri_replace_all_regex(sent$words, pattern, ""))
   user  system elapsed 
 21.246   0.320  21.552

更新（感谢阿伦）

system.time(res <- gsub(pattern, "", sent$words, perl = TRUE))
   user  system elapsed 
 12.290   0.000  12.281

【讨论】：

尝试使用基本 R gsub 和 perl=TRUE。