仅过滤单词而不是标点符号

Question

我正在尝试浏览正面 ('pos') 和负面 ('neg') 列表。这里的问题是下面的脚本也包括了空格和标点符号，我不希望这样。我只想从这些列表中提取最常见的 30 个单词。知道如何做到这一点吗？

#for counting frequently occurrence of negative and positive words.

from collections import Counter

count1 = Counter(" ".join(data[data['sentiment']=='pos']["text"]).split()).most_common(30)
data1 = pd.DataFrame.from_dict(count1)
data1 = data1.rename(columns={0: "words of positive", 1 : "count"})
count2 = Counter(" ".join(data[data['sentiment']=='neg']["text"]).split()).most_common(30)
data2 = pd.DataFrame.from_dict(count2)
data2 = data2.rename(columns={0: "words of negative", 1 : "count_"})

编辑

上面的实现有正确的想法，但是我希望提取有意义的单词而不是逻辑连接词，例如 (the, and a it that) 以及标点符号

Answer 1

您可以替换.split()

data[data['sentiment']=='pos']["text"]).split()

带有regex 函数

import re

def return_words(string):
    return re.findall(r'\b\S+\b', string.lower())

return_words(data[data['sentiment']=='pos']["text"]))

所以把它放在你的代码中：

from collections import Counter
import re

def return_words(string):
    return re.findall(r'\b\S+\b', string.lower())

count1 = Counter(" ".join(return_words(data[data['sentiment']=='pos']["text"]))).most_common(30)
data1 = pd.DataFrame.from_dict(count1)
data1 = data1.rename(columns={0: "words of positive", 1 : "count"})
count2 = Counter(" ".join(retrn_words(data[data['sentiment']=='neg']["text"]))).most_common(30)
data2 = pd.DataFrame.from_dict(count2)
data2 = data2.rename(columns={0: "words of negative", 1 : "count_"})