【发布时间】：2020-05-28 12:04:41
【问题描述】：
我写了下面的代码：为了在上面实现 word2vec，现在我正在测试嵌入 w2v_model.wv['car_NOUN'] 但我得到如下错误：“word 'car_NOUN'不在词汇表中” 但我确定单词 car_NOUN 在词汇表中，有什么问题？ 有人可以帮我吗？
关于代码：我使用 Use spacy 将推文中的单词限制为实词，即名词、动词和形容词。将单词转换为小写并添加带有下划线的 POS。 E.g.:love_VERB .then 我想在新列表上实现 word2vec 但我想出了那个错误
love_VERB old-fashioneds_NOUN
KeyError Traceback (most recent call last)
<ipython-input-145-f6fb9c62175c> in <module>()
----> 1 w2v_model.wv['car_NOUN']
2 frames
/usr/local/lib/python3.6/dist-packages/gensim/models/keyedvectors.py in word_vec(self, word, use_norm)
450 return result
451 else:
--> 452 raise KeyError("word '%s' not in vocabulary" % word)
453
454 def get_vector(self, word):
KeyError: "word 'car_NOUN' not in vocabulary"
! pip install wget
import wget
url = 'https://raw.githubusercontent.com/dirkhovy/NLPclass/master/data/reviews.full.tsv.zip'
wget.download(url, 'reviews.full.tsv.zip')
from zipfile import ZipFile
with ZipFile('reviews.full.tsv.zip', 'r') as zf:
zf.extractall()
import pandas as pd
df = pd.read_csv('reviews.full.tsv', sep='\t', nrows=100000) # nrows , max amount of rows
documents = df.text.values.tolist()
print(documents[:4])
import spacy
nlp = spacy.load('en_core_web_sm') #you can use other methods
# excluded tags
included_tags = {"NOUN", "VERB", "ADJ"}
#document = [line.strip() for line in open('moby_dick.txt', encoding='utf8').readlines()]
sentences = documents[:103] #first 10 sentences
new_sentences = []
for sentence in sentences:
new_sentence = []
for token in nlp(sentence):
if token.pos_ in included_tags:
new_sentence.append(token.text.lower()+'_'+token.pos_)
new_sentences.append(" ".join(new_sentence))
def convert(new_sentences):
return ' '.join(new_sentences).split()
x=convert(new_sentences)
from gensim.models import Word2Vec
from gensim.models.word2vec import FAST_VERSION
# initialize model
w2v_model = Word2Vec(size=100,
window=15,
sample=0.0001,
iter=200,
negative=5,
min_count=100,
workers=-1,
hs=0
)
w2v_model.build_vocab(x)
w2v_model.train(x,
total_examples=w2v_model.corpus_count,
epochs=w2v_model.epochs)
w2v_model.wv['car_NOUN']
【问题讨论】：
-
w2v_model.wv.vocab.keys()显示什么？ -
它显示 dict_keys(['p', 'r', 'i', 'c', 'e', 's', '_', 'N', 'O', ' U',...])
-
啊哈！看看
x- 这是你想要的吗？ -
我是python新手，不知道写对不对，我觉得x有问题
-
x 的输出是这个，我想要这个是的 ['prices_NOUN', 'change_VERB', 'want_VERB', 'research_VERB', 'price_NOUN', 'many_ADJ', 'different_ADJ', ' sites_NOUN'，'found_VERB'，'cheaper_ADJ'，'cars_NOUN'，'don_VERB'，'t_NOUN'，'lot_NOUN'，'time_NOUN'，'research_VERB'，'price_NOUN'，'site_NOUN'，'top_ADJ'，'e_NOUN' ]