r 文本挖掘：查找字符模式的频率

Question

我试图在大型数据集中查找字符模式（单词部分）的频率。

例如，我在 csv 文件中有以下列表：

• 苹果草莓酸橙
• 苹果酸橙
• 菠萝芒果番石榴
• 猕猴桃
• 葡萄柚
• 混合浆果
• 奇异果菠萝
• 石灰混合浆果

有没有办法找到所有字符组合的频率？喜欢：

• 苹果莓
• 番石榴
• 苹果草莓
• 猕猴桃
• 葡萄柚
• 稻草
• 应用程序
• ap
• 假发
• 内存
• 去

更新：这是我在数据中查找所有长度为 3 的字符模式的频率的方法：

threecombo  <- do.call(paste0,expand.grid(rep(list(c('a', 'b', 'c', 'd','e','f','g','h','i','j','k','l','m','n','o','p','q','r','s','t','u','v','w','x','y','z')), 3)))

threecompare<-sapply(threecombo, function(x) length(grep(x, myData)))

代码按我想要的方式工作，我想重复上述步骤以获得更长的字符长度（4、5、6 等），但运行需要一段时间。有没有更好的方法来做到这一点？

Answer 1

由于您可能正在从一组包含非水果词的文本中寻找水果风味的组合，因此我编写了一些与您示例中的文档类似的文档。我使用 quanteda 包构建了一个文档术语矩阵，然后基于包含水果词的 ngram 进行过滤。

docs <- c("One flavor is apple strawberry lime.", 
          "Another flavor is apple grape lime.", 
          "Pineapple mango guava is our newest flavor.",
          "There is also kiwi guava and grape apple.", 
          "Mixed berry was introduced last year.", 
          "Did you like kiwi guava pineapple?",
          "Try the lime mixed berry.")
flavorwords <- c("apple", "guava", "berry", "kiwi", "guava", "grape")

require(quanteda)
# form a document-feature matrix ignoring common stopwords + "like"
# for ngrams, bigrams, trigrams
fruitDfm <- dfm(docs, ngrams = 1:3, ignoredFeatures = c("like", "also", stopwords("english")))
## Creating a dfm from a character vector ...
##    ... lowercasing
##    ... tokenizing
##    ... indexing documents: 7 documents
##    ... indexing features: 90 feature types
##    ... removed 47 features, from 176 supplied (glob) feature types
##    ... complete. 
##    ... created a 7 x 40 sparse dfm
## Elapsed time: 0.01 seconds.
# select only those features containing flavorwords as regular expression
fruitDfm <- selectFeatures(fruitDfm, flavorwords, valuetype = "regex")
## kept 22 features, from 5 supplied (regex) feature types
# show the features
topfeatures(fruitDfm, nfeature(fruitDfm))
##                apple                 guava                 grape             pineapple                  kiwi 
##                    3                     3                     2                     2                     2 
##           kiwi_guava                 berry           mixed_berry            strawberry      apple_strawberry 
##                    2                     2                     2                     1                     1 
##      strawberry_lime apple_strawberry_lime           apple_grape            grape_lime      apple_grape_lime 
##                    1                     1                     1                     1                     1 
##      pineapple_mango           mango_guava pineapple_mango_guava           grape_apple       guava_pineapple 
##                    1                     1                     1                     1                     1 
## kiwi_guava_pineapple      lime_mixed_berry 
##                    1                     1 

添加：

如果您希望将未用空格分隔的术语与文档匹配，您可以使用空字符串连接器形成 ngram，并按如下方式进行匹配。

flavorwordsConcat <- c("applestrawberrylime", "applegrapelime", "pineapplemangoguava",
                       "kiwiguava", "grapeapple", "mixedberry", "kiwiguavapineapple",
                       "limemixedberry")

fruitDfm <- dfm(docs, ngrams = 1:3, concatenator = "")
fruitDfm <- fruitDfm[, features(fruitDfm) %in% flavorwordsConcat]
fruitDfm
# Document-feature matrix of: 7 documents, 8 features.
# 7 x 8 sparse Matrix of class "dfmSparse"
#        features
# docs  applestrawberrylime applegrapelime pineapplemangoguava kiwiguava grapeapple mixedberry kiwiguavapineapple limemixedberry
# text1                   1              0                   0         0          0          0                  0              0
# text2                   0              1                   0         0          0          0                  0              0
# text3                   0              0                   1         0          0          0                  0              0
# text4                   0              0                   0         1          1          0                  0              0
# text5                   0              0                   0         0          0          1                  0              0
# text6                   0              0                   0         1          0          0                  1              0
# text7                   0              0                   0         0          0          1                  0              1

如果您的文本包含连接的风味词，那么您可以使用 将一元 dfm 匹配到单个水果词的所有三元排列

unigramFlavorWords <- c("apple", "guava", "grape", "pineapple", "kiwi")
head(unlist(combinat::permn(unigramFlavorWords, paste, collapse = "")))
[1] "appleguavagrapepineapplekiwi" "appleguavagrapekiwipineapple" "appleguavakiwigrapepineapple" 
[4] "applekiwiguavagrapepineapple" "kiwiappleguavagrapepineapple" "kiwiappleguavapineapplegrape"

Answer 2

您最初的问题是 grep / grepl 的一项简单任务，我看到您已将这部分答案纳入您修改后的问题中。

docs <- c('applestrawberrylime', 'applegrapelime', 'pineapplemangoguava',
          'kiwiguava', 'grapeapple', 'mixedberry', 'kiwiguavapineapple',
          'limemixedberry')

patterns <-  c('appleberry', 'guava', 'applestrawberry', 'kiwiguava', 
               'grapeapple', 'grape', 'app', 'ap', 'wig', 'mem', 'go')

# how often does each pattern occur in the set of docs?
sapply(patterns, function(x) sum(grepl(x, docs)))

如果您想检查每个可能的模式，您可以搜索每个字母组合（正如您在上面开始所做的那样），但这显然是很长的路要走。

一种策略是只计算实际发生的每种模式的频率。每个字符长度为n 的文档有1 个可能的长度为n 的模式，2 个长度为n - 1 的模式，以此类推。您可以提取其中的每一个，然后计算它们。

all_patterns <- lapply(docs, function(x) {

    # individual chars in this doc
    chars <- unlist(strsplit(x, ''))

    # unique possible sequence lengths
    seqs <- sapply(1:nchar(x), seq)

    # each sequence in each position
    sapply(seqs, function(y) {
      start_pos <- 0:(nchar(x) - max(y))
      sapply(start_pos, function(z) paste(chars[z + y], collapse=''))
    })
})

unq_patterns <- unique(unlist(all_patterns))

# how often does each unique pattern occur in the set of docs?
occur <- sapply(unq_patterns, function(x) sum(grepl(x, docs)))

# top 25 most frequent patterns
sort(occur, decreasing = T)[1:25]     

# e     i     a     l     p     r     m    ap    pp    pl    le   app   ppl   
# 7     7     6     6     5     5     5     5     5     5     5     5     5
# ple  appl  pple apple     g     w     b     y  ra    be    er    rr 
#   5     5     5     5     5     3     3     3   3     3     3     3 

这工作起来很快，但随着文档语料库的增长，您可能会陷入困境（即使在这个简单的示例中，也有 625 个独特的模式）。可以对所有 s/lapply 调用使用并行处理，但仍然...