使用 tm 包删除停用词（Gsub 错误）答案

【问题标题】：Remove Stop Words Using tm package (Gsub Error)使用 tm 包删除停用词（Gsub 错误）
【发布时间】：2016-07-20 17:52:28
【问题描述】：

我正在尝试删除我从语料库中创建的停用词列表。我不确定发生了什么，因为我已经从停用词列表中删除了所有特殊字符，并且已经完成了语料库上的文本清理。任何帮助将不胜感激。代码和错误消息如下。此处列出了带有用户定义的停止词的 csv： Stop Words

    myCorpus <- Corpus(VectorSource(c("blank", "blank", "blank", "blank", "blank", "blank", "blank", 
"blank", "blank", "blank", "blank", "blank", "blank", "<br />Key skills:<br />Octopus Deploy, MS Build, PowerShell, Azure, NuGet, CI / CD concepts, release management<br /><br /> * Minimum 5 years plus relevant experience in Application Development lifecycle, Automation and Release and Configuration Management<br /> * Considerable experience in the following disciplines - TFS (Team Foundation Server), DevOps, Continuous Delivery, Release Engineering, Application Architect, Database Architect, Information Modeling, Service Oriented Architecture (SOA), Quality Assurance, Branch Management, Network setup and troubleshooting, Server setup, configuration, maintenance and patching<br /> * Solid understanding of Software Development Life Cycle, Test Driven Development, Continuous Integration and Continuous Delivery<br /> * Solid understanding and experience working with high availability and high performance, multi-data center systems and hybrid cloud environments.<br /> * Proficient with Agile methodologies and working closely within small teams and vendors<br /> * Knowledge of Deployment and configuration automation platforms<br /> * Extensive PowerShell experience<br /> * Extensive knowledge of Windows based systems including hardware, software and .NET applications<br /> * Strong ability to troubleshoot complex issues ranging from system resources to application stack traces<br /><br />REQUIRED SKILLS:<br />Bachelor's degree & 5-10 years of relevant work experience.", 
    "blank")))

for (j in seq(myCorpus)) {
  myCorpus[[j]] <- gsub("<.*>", " ", myCorpus[[j]])
  myCorpus[[j]] <- gsub("\\b[[:alnum:]]{20,}\\b", " ", myCorpus[[j]], perl=T)
  myCorpus[[j]] <- gsub("[[:punct:]]", " ", myCorpus[[j]])
}

#Clean Corpus
myCorpus <- tm_map(myCorpus, PlainTextDocument)
myCorpus <- tm_map(myCorpus, content_transformer(tolower))
myCorpus <- tm_map(myCorpus, removePunctuation)
myCorpus <- tm_map(myCorpus, removeNumbers)
myCorpus <- tm_map(myCorpus, stripWhitespace)

#User defined stop word
manualStopwords <- read.csv("r_stop.csv", header = TRUE)
myStopwords <- paste(manualStopwords[,1])
myStopwords <- str_replace_all(myStopwords, "[[:punct:]]", "")
myStopwords <- gsub("\\+", "plus", myStopwords)
myStopwords <- gsub("\\$", "dollars", myStopwords)

myCorpus <- tm_map(myCorpus, removeWords, myStopwords)

第一个错误

gsub(sprintf("(*UCP)\b(%s)\b", paste(sort(words, reduction = TRUE)) 中的错误：无效的正则表达式 '(*UCP)\b(zimmermann|yrs|yr|youve|.....其余的停用词

其他错误

另外：警告信息：在 gsub(sprintf("(*UCP)\b(%s)\b", paste(sort(words, reduction = TRUE), : PCRE 模式编译错误 '正则表达式太大' 在''

【问题讨论】：

请提供reproducible example，以便其他人可以帮助您。
我拥有的停用词列表约为 4000 个单词，我相信这就是问题所在。我认为关键在于附加错误。 '' 似乎是让 gsub 关闭的原因。我无法在此处发布整个数据集。
@Stewpants 您可以通过缩小导致错误的过程部分并使用可重现的数据来制作可重现的示例。
有了停用词的 csv 文件，代码应该是可重现的。

标签： r gsub tm stop-words

【解决方案1】：

我能够将停用词分解成更小的桶，然后代码运行。可能是内存有问题。

chunk <- 500
n <- length(myStopwords)
r <- rep(1:ceiling(n/chunk),each=chunk)[1:n]
d <- split(myStopwords,r)

for (i in 1:length(d)) {
  myCorpus <- tm_map(myCorpus, removeWords, c(paste(d[[i]])))
}

【讨论】：