【发布时间】:2020-05-21 15:57:47
【问题描述】:
只有当我的列的每一行中的单词不在停用词中且不在字符串中 标点符号时,我才想选择单词。
这是我在标记和删除停用词后的数据,我还想在删除停用词的同时删除标点符号。见第二个 usf 后面有逗号。我想到了if word not in (stopwords,string.punctuation),因为它是not in stopwords and not in string.punctuation,我从here 看到它,但它导致无法删除停用词和标点符号。如何解决这个问题?
data['text'].head(5)
Out[38]:
0 ['ve, searching, right, words, thank, breather...
1 [free, entry, 2, wkly, comp, win, fa, cup, fin...
2 [nah, n't, think, goes, usf, ,, lives, around,...
3 [even, brother, like, speak, ., treat, like, a...
4 [date, sunday, !, !]
Name: text, dtype: object
import pandas as pd
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
import string
data = pd.read_csv(r"D:/python projects/read_files/SMSSpamCollection.tsv",
sep='\t', header=None)
data.columns = ['label','text']
stopwords = set(stopwords.words('english'))
def process(df):
data = word_tokenize(df.lower())
data = [word for word in data if word not in (stopwords,string.punctuation)]
return data
data['text'] = data['text'].apply(process)
【问题讨论】:
-
写
(stopwords, string.punctuation)时你期望的结果是什么? -
你已经创建了一个包含两个元素的元组,并且只有这两个元素。
stopwords in (stopwords, string.punctuation)将返回True,例如 -
@Anwarvic 我希望它能像
data = [word for word in data if word not in stopwords and word not in string.punctuation]一样工作 -
@G.Anderson 是的,我认为如果单词不是
stopwords而不是string.punctuation,它会返回满足的单词 -
你应该知道
string.punctuation是一个字符串。word not in string.punctuation不会返回True除非word是标点符号...这是你想要的吗?
标签: python if-statement list-comprehension