有没有办法在python中找到带有TF-IDF的句子权重答案

【问题标题】：is there any way to find weitage of sentence with TF-IDF in python有没有办法在python中找到带有TF-IDF的句子权重
【发布时间】：2019-12-10 21:16:11
【问题描述】：

我有一份清单

x=["hello there","hello world","my name is john"]

我已经完成了 TF-IDF 的矢量化

这是 TF-idf 的输出

  from sklearn.feature_extraction.text import TfidfVectorizer
  corpus = [
         "hello there","hello world","my name is john", ]
  vectorizer = TfidfVectorizer()

  X = vectorizer.fit_transform(corpus)

  X.toarray()



array([[0.60534851, 0.        , 0.        , 0.        , 0.        ,
      0.79596054, 0.        ],
     [0.60534851, 0.        , 0.        , 0.        , 0.        ,
      0.        , 0.79596054],
     [0.        , 0.5       , 0.5       , 0.5       , 0.5       ,
      0.        , 0.        ]])

我们可以找到每个句子的权重（与所有文档进行比较）吗？

如果是，那么如何？

【问题讨论】：

你能分享完整的代码并解释一下 TF-idf 的想法是什么吗？
是的，做到了，看看@Anteino
好的，等我安装这些库并弄清楚 TF-idf 是如何工作的。
你能解释一下每句话的权重是什么意思吗？你的意思是那个句子中所有单词的附加权重吗？还是您想在另一个文档中找到这些完整的句子？
@Anteino 我想做类似Page rankinglike 之类的事情（句子1 很有可能是这样的）

标签： python machine-learning scikit-learn

【解决方案1】：

我相信使用 TF-idf 你只能计算单个单词在一个句子（或相关的文档）中的权重，这意味着你不能用它来计算句子在其他句子或文档中的权重。

但是，我从this 页面了解了 TF-idf 的工作原理。您可以通过将它们更改为您特别需要的功能来“滥用”它们提供的功能。请允许我演示一下：

import math

corpus = ["hello there", "hello world"]

file = open("your_document.txt", "r")
text = file.read()
file.close()

def computeTF(sentences, document):
    dict = {i: 0 for i in sentences}
    filelen = len(text.split(' ')) - 1

    for s in sentences:
        #   Since we're counting a whole sentence (containing >= 1 words) we need to count
        #   that whole sentence as a single word.
        sLength = len(s.split(' '))
        dict[s] = document.count(s)
        #   When you know the amount of occurences of the specific sentence s in the
        #   document, you can recalculate the amount of words in that document (considering
        #   s as a single word.
        filelen = filelen - dict[s] * (sLength - 1)

    for s in sentences:
        #   Since only after the previous we know the amount of words in the document, we
        #   need a separate loop to calculate the actual weights of each word.
        dict[s] = dict[s] / filelen

    return dict

def computeIDF(dict, sentences):
    idfDict = {s: dict[s] for s in sentences}
    N = len(dict)

    for s in sentences:
        if(idfDict[s] > 0):
            idfDict[s] = math.log10(N)
        else:
            idfDict[s] = 0

    return idfDict

dict = computeTF(corpus, text)
idfDict = computeIDF(dict, corpus)

for s in corpus:
    print("Sentence: {}, TF: {}, TF-idf: {}".format(s, dict[s], idfDict[s]))

此代码示例仅查看单个文本文件，但您可以轻松扩展它以查看多个文本文件。

【讨论】：