修改 cSplit_e 函数以考虑多个值

Question

我了解“splitstackshape”中的“cSplit_e”可用于将一列下的多个值转换为具有二进制值的分隔列。我正在处理一个用于计算 tf-idf 的文本问题，并且不必在列下拥有所有唯一值。例如，

docname   ftype                        doc_text
    1      mw               hello, hi, how, are, you, hello
    2      gw                       hi,yo,man
    3      mw                     woha,yo, yoman

DPUT(df)

   structure(list(docname = 1:3, ftype = c("mw", "gw", "mw"), doc_text = structure(1:3, .Label = c("hello, hi, how, are, you, hello", 
"hi,yo,man", "woha,yo, yoman"), class = "factor")), .Names = c("docname", 
"ftype", "doc_text"), class = "data.frame", row.names = c(NA, 
-3L))

对于上面的例子，如果我们考虑 doc-1，那么当 "hello" 出现两次时，cSplit_e 会将 doc_text 转换为 5 个单独的列，其值为 "1"。有没有办法修改这个函数来解释重复值？

本质上，这就是我想要实现的目标：给定一个数据框

docname ftype doc_text 1 兆瓦你好，你好，你好，你好，你好 2 gw 嗨，哟，伙计 3 me woha,yo,yoman

我想根据用“，”分隔的列值将 doc_text 转换为多列，并获取它们各自的频率。所以结果应该是

docname ftype are hello hi how man woha yo yoman you
     1   mw    1     2  1   1   0    0  0     0   1
     2   gw    0     0  1   0   1    0  1     0   0
     3   mw    0     0  0   0   0    1  1     1   0

如果有人知道如何使用“splitstackshape”或其他方式完成此操作，我将不胜感激。最终目的是计算 tf-idf。

谢谢。

Answer 1

我们可以用'doc_text'分割后的mtabulate来做到这一点

library(qdapTools)
cbind(df[1], mtabulate(strsplit(as.character(df$doc_text), ",\\s*")))
#   docname are hello hi how man woha yo yoman you
#1       1   1     2  1   1   0    0  0     0   1
#2       2   0     0  1   0   1    0  1     0   0
#3       3   0     0  0   0   0    1  1     1   0

---

或者另一个选项是tidyverse

library(tidyverse)
separate_rows(df, doc_text) %>% #split to long format
           group_by(docname, doc_text) %>% #group by variables
           tally() %>% #get the frequency
           spread(doc_text, n, fill=0) #reshape to wide

---

或者按照@Frank 的建议

library(splitstackshape)
cSplit(df, "doc_text", ",", "long")[, dcast(.SD, docname ~ doc_text)]

Answer 2

带一点text-mining:

docs <- gsub('[[:punct:]]+', ' ', as.character(df$doc_text))
library(tm)
corpus <- Corpus(VectorSource(docs))

# compute Term Frequencies
as.matrix(DocumentTermMatrix(corpus, control = list(wordLengths=c(2,Inf))))
#     Terms
#Docs are hello hi how man woha yo yoman you
#   1   1     2  1   1   0    0  0     0   1
#   2   0     0  1   0   1    0  1     0   0
#   3   0     0  0   0   0    1  1     1   0

# compute Tf-Idf scores
as.matrix(DocumentTermMatrix(corpus, control = list(wordLengths=c(2,Inf), weighting=weightTfIdf)))
#         Terms
#Docs       are     hello         hi       how       man      woha        yo     yoman`       you
#   1 0.2641604 0.5283208 0.09749375 0.2641604 0.0000000 0.0000000 0.0000000 0.0000000 0.2641604
#   2 0.0000000 0.0000000 0.19498750 0.0000000 0.5283208 0.0000000 0.1949875 0.0000000 0.0000000
#   3 0.0000000 0.0000000 0.00000000 0.0000000 0.0000000 0.5283208 0.1949875 0.5283208 0.0000000