Python的单词列表列表：

Question

有一个长长的 cmets 列表（说是 50 个），比如这个：

“这是我们这次旅行最大的失望。餐厅有 收到了一些非常好的评论，所以我们的期望很高。这 即使餐厅不是很满，服务也很慢。我有 家常沙拉，它可能来自美国的任何一个嘶嘶声。 keshi yesa，虽然好吃让我想起了烧烤拉 鸡。这家餐厅被高估了”。

我想使用 python 创建一个保留句子标记化的单词列表。

删除停用词后，我想要所有 50 个 cmets 的结果，其中保留了句子标记，并且将单词标记保留到每个标记化的句子中。最后我希望结果类似于：

list(c("disappointment", "trip"), 
     c("restaurant", "received", "good", "reviews", "expectations", "high"), 
     c("service", "slow", "even", "though", "restaurant", "full"),
     c("house", "salad", "come", "us"), 
     c("although", "tasty", "reminded", "pulled"), 
     "restaurant")  

我怎么能在 python 中做到这一点？在这种情况下，R 是一个不错的选择吗？非常感谢您的帮助。

Answer 1

如果您不想手动创建停用词列表，我建议您使用 python 中的 nltk 库。它还处理句子拆分（而不是在每个句点上拆分）。解析您的句子的示例可能如下所示：

import nltk
stop_words = set(nltk.corpus.stopwords.words('english'))
text = "this was the biggest disappointment of our trip. the restaurant had received some very good reviews, so our expectations were high. the service was slow even though the restaurant was not very full. I had the house salad which could have come out of any sizzler in the us. the keshi yena, although tasty reminded me of barbequed pulled chicken. this restaurant is very overrated"
sentence_detector = nltk.data.load('tokenizers/punkt/english.pickle')
sentences = sentence_detector.tokenize(text.strip())
results = []
for sentence in sentences:
    tokens = nltk.word_tokenize(sentence)
    words = [t.lower() for t in tokens if t.isalnum()]
    not_stop_words = tuple([w for w in words if w not in stop_words])
    results.append(not_stop_words)
print results

但是，请注意，这不会提供与您的问题中列出的完全相同的输出，而是如下所示：

[('biggest', 'disappointment', 'trip'), ('restaurant', 'received', 'good', 'reviews', 'expectations', 'high'), ('service', 'slow', 'even', 'though', 'restaurant', 'full'), ('house', 'salad', 'could', 'come', 'sizzler', 'us'), ('keshi', 'yena', 'although', 'tasty', 'reminded', 'barbequed', 'pulled', 'chicken'), ('restaurant', 'overrated')]

如果输出需要看起来相同，您可能需要手动添加一些停用词。

Answer 2

不确定您是否需要 R，但根据您的要求，我认为它也可以以纯 Python 的方式完成。

您基本上想要一个列表，其中包含每个句子的重要单词（不是停用词）的小列表。

所以你可以做类似的事情

input_reviews = """
this was the biggest disappointment of our trip. the restaurant had received some very good reviews, so our expectations were high. 
the service was slow even though the restaurant was not very full. I had the house salad which could have come out of any sizzler in the us. 
the keshi yena, although tasty reminded me of barbequed pulled chicken. this restaurant is very overrated.
"""

# load your stop words list here
stop_words_list = ['this', 'was', 'the', 'of', 'our', 'biggest', 'had', 'some', 'very', 'so', 'were', 'not']


def main():
    sentences = input_reviews.split('.')
    sentence_list = []
    for sentence in sentences:
        inner_list = []
        words_in_sentence = sentence.split(' ')
        for word in words_in_sentence:
            stripped_word = str(word).lstrip('\n')
            if stripped_word and stripped_word not in stop_words_list:
                # this is a good word
                inner_list.append(stripped_word)

        if inner_list:
            sentence_list.append(inner_list)

    print(sentence_list)



if __name__ == '__main__':
    main()

在我这边，这个输出

[['disappointment', 'trip'], ['restaurant', 'received', 'good', 'reviews,', 'expectations', 'high'], ['service', 'slow', 'even', 'though', 'restaurant', 'full'], ['I', 'house', 'salad', 'which', 'could', 'have', 'come', 'out', 'any', 'sizzler', 'in', 'us'], ['keshi', 'yena,', 'although', 'tasty', 'reminded', 'me', 'barbequed', 'pulled', 'chicken'], ['restaurant', 'is', 'overrated']]

Answer 3

这是一种方法。您可能需要根据您的应用程序初始化stop_words。我假设stop_words 是小写的：因此，在原始句子上使用lower() 进行比较。 sentences.lower().split('.') 给出句子。 s.split() 给出每个句子中的单词列表。

stokens = [list(filter(lambda x: x not in stop_words, s.split())) for s in sentences.lower().split('.')]

您可能想知道为什么我们使用filter 和lambda。另一种方法是这样，但这会给出一个平面列表，因此不适合：

stokens = [word for s in sentences.lower().split('.') for word in s.split() if word not in stop_words]

filter 是一个函数式编程结构。它可以帮助我们处理整个列表，在这种情况下，通过使用 lambda 语法的匿名函数。