使用 word2vec 提取段落的主要特征

Question

我刚刚掌握了 Google 的 word2vec 模型，对这个概念还很陌生。我正在尝试使用以下方法提取段落的主要特征。

from gensim.models.keyedvectors import KeyedVectors
model = KeyedVectors.load_word2vec_format('../../usr/myProject/word2vec/GoogleNews-vectors-negative300.bin', binary=True)

...

for para in paragraph_array:
    para_name = "para_"+ file_name + '{0}'
    sentence_array = d[para_name.format(number_of_paragraphs)] = []

    # Split Paragraph on basis of '.' or ? or !.
    for l in re.split(r"\.|\?|\!", para):
        # Split line into list using space.
        sentence_array.append(l)
        #sentence_array.append(l.split(" "))

     print (model.wv.most_similar(positive=para, topn = 1))

但出现以下错误，表明检查的段落不是词汇表中的单词。

KeyError: 'word \'加纳共和国是西非的一个国家。它西与科特迪瓦（也称为科特迪瓦）接壤，北与布基纳法索接壤，东与多哥接壤，南与几内亚湾接壤。 “加纳”一词的意思是“战士王”，杰克逊，约翰 G. 非洲文明简介，2001。第 201 页。并且是“几内亚”（通过法语 Guinoye）这个名称的来源，用于指代西非海岸（如在几内亚湾）。\'不在词汇表中'

现在我知道most_similar() 函数需要一个数组。但我想知道如何将其翻译为使用 word2vec 模型提取显示段落主要概念的一个主要特征或单词。

修改

我修改了上面的代码，将 word_array 传递给 most_similar() 方法，我得到了以下错误。

Traceback（最近一次调用最后一次）： 文件“/home/manuelanayantarajeyaraj/PycharmProjects/ChatbotWord2Vec/new_approach.py​​”，第 108 行，在 打印（model.wv.most_similar（正=word_array，topn=1）） 文件“/home/manuelanayantarajeyaraj/usr/myProject/my_project/lib/python3.5/site-packages/gensim/models/keyedvectors.py”，第 361 行，在 most_similar 对于单词，正负权重： ValueError：要解压的值太多（预期为 2）

修改后的实现

for sentence in sentence_array:
    if sentence:
        for w in re.split(r"\.|\?|\!|\@|\#|\$|\%|\^|\&|\*|\(|\)|\-",   sentence):
            split_word = w.split(" ")
            if split_word:
                word_array.append(split_word)
print(model.wv.most_similar(positive=word_array, topn=1))

非常感谢您提供这方面的任何建议。

Answer 1

我重新编写了整个代码，添加了检查点，以避免将空字符串存储到从段落、句子到单词的每个级别的对象中。

工作版

for file_name in files:
    file_identifier = file_name
    file_array = file_dictionary[file_identifier] =[]
    #file_array = file_dictionary[file_name.format((file_count))] = []
    file_path = directory_path+'/'+file_name

    with open(file_path) as f:
        #Level 2 Intents : Each file's main intent (One for each file)
        first_line = f.readline()
        print ()
        print("Level 2 Intent for ", c, " : ", first_line)

        #Level 3 Intents : Each paragraph's main intent (one for each para)

        paragraph_count = 0

        data = f.read()
        splat = data.split("\n")
        paragraph_array = []

        for number, paragraph in enumerate(splat, 1):
            paragraph_identifier = file_name + "_paragraph_" + str(paragraph_count)
            #print(paragraph_identifier)
            paragraph_array = paragraph_dictionary[paragraph_identifier.format(paragraph_count)] = []
            if paragraph :
                paragraph_array.append(paragraph)
            paragraph_count += 1
            if len(paragraph_array) >0 :
                file_array.append(paragraph_array)

            # Level 4 Intents : Each sentence's main intent (one for each sentence)

            sentence_count = 0
            sentence_array = []

            for sentence in paragraph_array:
                for line in re.split(r"\.|\?|\!", sentence):
                    sentence_identifier = paragraph_identifier + "_sentence_" + str(sentence_count)
                    sentence_array = sentence_dictionary[sentence_identifier.format(sentence_count)] = []
                    if line :
                        sentence_array.append(line)
                        sentence_count += 1

                    # Level 5 Intents : Each word with a certain level of prominance (one for each prominant word)

                    word_count = 0
                    word_array = []

                    for words in sentence_array:
                        for word in re.split(r" ", words):
                            word_identifier = sentence_identifier + "_word_" + str(word_count)
                            word_array = word_dictionary[word_identifier.format(word_count)] = []

                            if word :
                                word_array.append(word)
                                word_count += 1

访问字典项的代码

#Accessing any paragraph array can be done as follows
print (paragraph_dictionary['S08_set4_a5.txt.clean_paragraph_4'])

#Accessing any sentence corresponding to a paragraph
print (sentence_dictionary['S08_set4_a5.txt.clean_paragraph_4_sentence_1'])

#Accessing any word corresponding to a sentence
print (word_dictionary['S08_set4_a5.txt.clean_paragraph_4_sentence_1_word_3'])

输出

['Celsius was born in Uppsala in Sweden. He was professor of astronomy at Uppsala University from 1730 to 1744, but traveled from 1732 to 1735 visiting notable observatories in Germany, Italy and France.']
[' He was professor of astronomy at Uppsala University from 1730 to 1744, but traveled from 1732 to 1735 visiting notable observatories in Germany, Italy and France']
['of']

Answer 2

您的错误表明您正在查找整个字符串 ('The Republic of Ghana is a country in West Africa. It borders Côte d\'Ivoire (also known as Ivory Coast) to the west, Burkina Faso to the north, Togo to the east, and the Gulf of Guinea to the south. The word "Ghana" means "Warrior King", Jackson, John G. Introduction to African Civilizations, 2001. Page 201. and was the source of the name "Guinea" (via French Guinoye) used to refer to the West African coast (as in Gulf of Guinea).')，就好像它是一个单词一样，并且该单词不存在。

most_similar() 方法可以获取一个正例列表，但您必须将该字符串标记为可能在词向量集中的词。 （这可能需要同时打破空格和标点符号以匹配 Google 为准备该词向量集所做的任何事情。）

在这种情况下，most_similar() 会将所有给定单词的向量平均在一起，并返回接近该平均值的其他单词。

这是否真正抓住了文本的“主要概念”尚不清楚。虽然词向量可能在识别文本的概念方面证明是有用的，但这不是它们的主要或唯一功能，也不是自动的。您可能希望将一组词过滤到那些以其他方式唯一的词——例如总体上不太常见的词，或者在某些依赖于语料库的度量（如 TF/IDF）中具有影响力的词。