NLP 生成集合

Question

我正在动手做，预期的输出是

[('fans', 3), ('car', 3), ('disciplines', 1)]

['跑车'，'体育迷']

我的代码如下。我能够获得第一个预期输出，但无法正确获得第二个输出。谁能帮我看看这里出了什么问题

    from nltk.tokenize import RegexpTokenizer
    text='Thirty-five sports disciplines and four cultural activities will be offered during seven days of competitions. He skated with charisma, changing from one gear to another, from one direction to another, faster than a sports car. Armchair sports fans settling down to watch the Olympic Games could be for the high jump if they do not pay their TV licence fee. Such invitationals will attract more viewership for sports fans by sparking interest among sports fans. She barely noticed a flashy sports car almost run them over, until Eddie lunged forward and grabbed her body away. And he flatters the mother and she kind of gets prissy and he talks her into going for a ride in the sports car.'
    word='sports'
    tokenizedword = nltk.tokenize.regexp_tokenize(text, pattern = '\w*', gaps = False)
    #Step 2
    tokenizedwords = [x.lower() for x in tokenizedword if x != '']

    tokenizedwordsbigram=list(nltk.bigrams(tokenizedwords))
    stop_words = set(stopwords.words('english')) 
    filteredwords = []
    for x in tokenizedwordsbigram:
       if x not in stop_words:
          filteredwords.append(x)
     
    tokenizednonstopwordsbigram = nltk.ConditionalFreqDist(filteredwords)  
    print(tokenizednonstopwordsbigram[word].most_common(3))
    gen_text=nltk.Text(tokenizedwords)
    print(gen_text.collocations())

Answer 1

替换

print(gen_text.collocations())

与

print(gen_text.collocation_list())

你的程序可以正常运行

Answer 2

我通过添加所需的导入 nltk import 和 from nltk.corpus import stopwords 运行代码并得到以下输出。

import nltk
from nltk.corpus import stopwords
from nltk.tokenize import RegexpTokenizer

# use to find bigrams, which are pairs of words

text = \
    'Thirty-five sports disciplines and four cultural activities will be offered during seven days of competitions. He skated with charisma, changing from one gear to another, from one direction to another, faster than a sports car. Armchair sports fans settling down to watch the Olympic Games could be for the high jump if they do not pay their TV licence fee. Such invitationals will attract more viewership for sports fans by sparking interest among sports fans. She barely noticed a flashy sports car almost run them over, until Eddie lunged forward and grabbed her body away. And he flatters the mother and she kind of gets prissy and he talks her into going for a ride in the sports car.'
word = 'sports'
tokenizedword = nltk.tokenize.regexp_tokenize(text, pattern='\w*',
        gaps=False)

# Step 2
tokenizedwords = [x.lower() for x in tokenizedword if x != '']

tokenizedwordsbigram = list(nltk.bigrams(tokenizedwords))
stop_words = set(stopwords.words('english'))
filteredwords = []

for x in tokenizedwordsbigram:
    if x not in stop_words:
        filteredwords.append(x)

tokenizednonstopwordsbigram = nltk.ConditionalFreqDist(filteredwords)
print tokenizednonstopwordsbigram[word].most_common(3)

gen_text = nltk.Text(tokenizedwords)
print gen_text.collocations()

这是输出：

[('car', 3), ('fans', 3), ('disciplines', 1)]
sports car; sports fans
None

Answer 3

gen_text = nltk.Text(tokenizedwords).collocation_list()

b=[i[0]+" "+i[1] for i in gen_text]

返回 b

您将输出为：

['跑车'，'体育迷']

Answer 4

from nltk.corpus import stopwords

def performBigramsAndCollocations(textcontent, word):
    stop_words=set(stopwords.words('english'))
    pattern =r'\w+'
    tokenizewords=nltk.regexp_tokenize(textcontent,pattern)
    tokenizewords=[word.lower() for word in tokenizewords]
    tokenizewordsbiagrams=nltk.bigrams(tokenizewords)
    tokenizednonstopwordbigrams=[(w1,w2) for w1,w2 in tokenizewordsbiagrams if w1 not in stop_words and w2 not in stop_words]
    cfd_bigrams=nltk.ConditionalFreqDist(tokenizednonstopwordbigrams)
    cfd_bigrams=cfd_bigrams[word]
    mostfrequentwordafter=cfd_bigrams.most_common(3)
    collocationwords=nltk.Text(tokenizewords)
    collocationwords=collocationwords.collocation_list()
    collocationwords=[i[0]+" "+i[1] for i in collocationwords]
    
    return mostfrequentwordafter,collocationwords

你可以试试这个。它对我有用！