Python中的词义消歧算法[重复]

Question

我正在开发一个简单的 NLP 项目，我正在寻找，给定一个文本和一个单词，在文本中找到该单词最可能的含义。

在 Python 中是否有任何 wsd 算法的实现？目前还不清楚 NLTK 中是否有一些东西可以帮助我。即使是像 Lesk 算法这样幼稚的实现，我也会很高兴。

我读过类似的问题，如Word sense disambiguation in NLTK Python，但它们只提供了对 NLTK 书籍的参考，这不是 WSD 问题。

Answer 1

简而言之： https://github.com/alvations/pywsd

长篇大论： 用于 WSD 的技术无穷无尽，从需要大量 GPU 能力的令人兴奋的机器技术到简单地使用 wordnet 中的信息甚至只是使用频率，请参阅 http://dl.acm.org/citation.cfm?id=1459355 .

让我们从允许可选词干的简单 lesk 算法开始，参见http://en.wikipedia.org/wiki/Lesk_algorithm：

from nltk.corpus import wordnet as wn
from nltk.stem import PorterStemmer
from itertools import chain

bank_sents = ['I went to the bank to deposit my money',
'The river bank was full of dead fishes']

plant_sents = ['The workers at the industrial plant were overworked',
'The plant was no longer bearing flowers']

ps = PorterStemmer()

def lesk(context_sentence, ambiguous_word, pos=None, stem=True, hyperhypo=True):
    max_overlaps = 0; lesk_sense = None
    context_sentence = context_sentence.split()
    for ss in wn.synsets(ambiguous_word):
        # If POS is specified.
        if pos and ss.pos is not pos:
            continue

        lesk_dictionary = []

        # Includes definition.
        lesk_dictionary+= ss.definition.split()
        # Includes lemma_names.
        lesk_dictionary+= ss.lemma_names

        # Optional: includes lemma_names of hypernyms and hyponyms.
        if hyperhypo == True:
            lesk_dictionary+= list(chain(*[i.lemma_names for i in ss.hypernyms()+ss.hyponyms()]))       

        if stem == True: # Matching exact words causes sparsity, so lets match stems.
            lesk_dictionary = [ps.stem(i) for i in lesk_dictionary]
            context_sentence = [ps.stem(i) for i in context_sentence] 

        overlaps = set(lesk_dictionary).intersection(context_sentence)

        if len(overlaps) > max_overlaps:
            lesk_sense = ss
            max_overlaps = len(overlaps)
    return lesk_sense

print "Context:", bank_sents[0]
answer = lesk(bank_sents[0],'bank')
print "Sense:", answer
print "Definition:",answer.definition
print

print "Context:", bank_sents[1]
answer = lesk(bank_sents[1],'bank','n')
print "Sense:", answer
print "Definition:",answer.definition
print

print "Context:", plant_sents[0]
answer = lesk(plant_sents[0],'plant','n', True)
print "Sense:", answer
print "Definition:",answer.definition
print

除了类似 lesk 的算法，人们尝试了不同的相似性度量，这是一个很好但过时但仍然有用的调查：http://acl.ldc.upenn.edu/P/P97/P97-1008.pdf‎

Answer 2

您可以尝试使用 NLTK 中包含的 WordNet 获得每个单词的第一感觉，使用以下短代码：

from nltk.corpus import wordnet as wn

def get_first_sense(word, pos=None):
    if pos:
        synsets = wn.synsets(word,pos)
    else:
        synsets = wn.synsets(word)
    return synsets[0]

best_synset = get_first_sense('bank')
print '%s: %s' % (best_synset.name, best_synset.definition)
best_synset = get_first_sense('set','n')
print '%s: %s' % (best_synset.name, best_synset.definition)
best_synset = get_first_sense('set','v')
print '%s: %s' % (best_synset.name, best_synset.definition)

将打印：

bank.n.01：坡地（尤其是水体旁边的斜坡） set.n.01：一组同类的东西，它们属于一起并被如此使用 put.v.01：放到某个地方或抽象位置

这很有效，令人惊讶的是，因为第一种感觉经常支配其他感觉。

Answer 3

对于 Python 中的 WSD，您可以尝试在 NLTK 或 Gensim 库中使用 Wordnet 绑定。构建模块就在那里，但开发完整的算法可能要靠你了。

例如，使用 Wordnet，您可以实现简化的 Lesk 算法，如 Wikipedia entry 中所述。