在R中对数组中的tf-idf分数进行排名

Question

我写了如下函数来判断一个文档的tf-idf：

确定 tf

tf <- function(specific_word, text){
 count = 0
 list = unlist(strsplit(text, " "))

 for(word in (list)){
  if(word == specific_word){
   count = count + 1
   }
  }
 hit_rate <- count/length(list)
 return(hit_rate)
}

确定 idf 值

idf <- function(specific_word, texts){

  times_a_word_appears <- 0
  total_number_of_documents <- length(texts)

  for(document in texts){
    list = strsplit(document, " ")
    list = unlist(list)

    for(word in list){
      if(word == specific_word){
        times_a_word_appears = times_a_word_appears + 1
        break
      }
    }

  }
  times_a_word_appears = times_a_word_appears + 1

  idf = log(total_number_of_documents/ times_a_word_appears)
  return(idf)
 }

最后——确定 tf-idf

tfidf <- function(specific_word, text, texts){

  x = tf(specific_word, text)
  y = idf(specific_word, texts)
  z = x * y

   print(paste0("The tf-idf value is: ", z))
}

我现在可以使用它来确定这些文档的 fe 的 tf-idf 值：

document1 = c("films is a 2000 made-for-TV horror movie directed by Richard Clabaugh. The film features several cult favorite actors, including William Zabka of The Karate Kid fame, Wil Wheaton, Casper Van Dien, Jenny McCarthy, Keith Coogan, Robert Englund (best known for his role as Freddy Krueger in the
A Nightmare on Elm Street series of films), Dana Barron, David Bowe, and Sean Whalen. The film concerns a genetically engineered snake, a python, that escapes and unleashes itself on a small town. It includes the classic final girl scenario evident in films like Friday the 13th. It was filmed in Los Angeles,
California and Malibu, California. Python was followed by two sequels: Python II (2002) and Boa vs. Python (2004), both also made-for-TV films")

document2 = c("Python, from the Greek word, is a genus of nonvenomous pythons[2] found in Africa and Asia. Currently, 7 species are recognised.[2] A member of this genus, P. reticulatus, is among the longest snakes known.")

document3 = c("The Colt Python is a .357 Magnum caliber revolver formerly manufactured by Colt's Manufacturing Company of Hartford, Connecticut. It is sometimes referred to as a Combat Magnum It was first introduced in 1955, the same year as Smith & Wesson's M29 .44 Magnum. The now discontinued
Colt Python targeted the premium revolver market segment. Some firearm collectors and writers such as Jeff Cooper, Ian V. Hogg, Chuck Hawks, Leroy Thompson, Renee Smeets and Martin Dougherty have described the Python as the finest production revolver ever made")

texts = c(document1, document2, document3)

并在document1中找到“films”的tf-idf值

word = "films"
relevant_text = document1
tfidf(word, relevant_text, texts) 

但是我现在想要的是遍历所有文档中的所有单词，以确定文档中评分最高的单词。

所以对于文档 1 有点像：

words = unlist(unique(strsplit(document1, " ")))

for(word in words){
  tfidf(word, document1, texts)
  }

但是这些值应该存储在一个数组中并进行排序。在python中有点像这样：

scores = {word: tfidf(word, document1, texts) for word in document1.words}
sorted_words = sorted(scores.items(), key=lambda x: x[1], reverse=True)

关于如何在 R 中最有效地完成此操作有什么想法吗？

Answer 1

我建议您查看CRAN Task View: Natural Language Processing 上列出的软件包。有几个包可用于创建文档术语矩阵的整个过程，包括规范化或 tfidf 加权。他们的小插曲还展示了许多下游任务，例如回归模型或分类聚类、主题建模等。

下面我使用了其中一个包，即text2vec，来解决您创建 tfidf 加权文档术语矩阵的任务。

希望对你有帮助。

library(text2vec)

#with your docuemnts...
#texts = c(document1, document2, document3)

#create iterator to split texts into tokens
iterator <- itoken(texts, 
                  preprocessor = tolower, 
                  tokenizer = word_tokenizer, 
                  progressbar = FALSE)

#create the vocabulary of tokens
vocabulary <- create_vocabulary(iterator)

#combine tokens into a document term matrix
#this will be a sparse matrix (see the package "Matrix" for details)
#you might need to convert your dtm objects with as.matrix() to "normal" matrices 
#depending on your downstream task (although most packages accept sparse matrices)
#note, that when converting with as.matrix(), you will loose the memory of advantage of sparse matrices
dtm <- create_dtm(iterator, vocab_vectorizer(vocabulary))
str(dtm)
# Formal class 'dgCMatrix' [package "Matrix"] with 6 slots
# ..@ i       : int [1:192] 1 2 2 0 2 2 0 1 2 0 ...
# ..@ p       : int [1:173] 0 1 2 3 4 5 6 7 8 9 ...
# ..@ Dim     : int [1:2] 3 172
# ..@ Dimnames:List of 2
# .. ..$ : chr [1:3] "1" "2" "3"
# .. ..$ : chr [1:172] "recognised" "premium" "finest" "william" ...
# ..@ x       : num [1:192] 1 1 1 1 1 1 1 1 1 1 ...
# ..@ factors : list()

#set up basic tfidf model
tfidf <- TfIdf$new()

#apply model to your dtm
dtm_tfidf <-  fit_transform(dtm, tfidf)

str(dtm_tfidf)
# Formal class 'dgCMatrix' [package "Matrix"] with 6 slots
# ..@ i       : int [1:192] 1 2 2 0 2 2 0 1 2 0 ...
# ..@ p       : int [1:173] 0 1 2 3 4 5 6 7 8 9 ...
# ..@ Dim     : int [1:2] 3 172
# ..@ Dimnames:List of 2
# .. ..$ : chr [1:3] "1" "2" "3"
# .. ..$ : chr [1:172] "recognised" "premium" "finest" "william" ...
# ..@ x       : num [1:192] 0.01126 0.00461 0.00461 0.00322 0.00461 ...
# ..@ factors : list()