文本类型数据框的word-tokenize列

Question

我的数据框如下：

我想使用 word-tokenize 并提取句子的特征，以将它们分类到不同的类别。

我的代码：

import pandas as pd
from nltk.tokenize import word_tokenize


cov = pd.read_csv("F:/kipro/ml/dataset.csv", 
              names = ["Complaint", "type"])
print(cov)
cov['tokenized_text'] = cov.apply(lambda 
              row:word_tokenize(cov['Complaint']), axis=1) 
print(cov['tokenized_text'])   
wd=[]
all_words=(list(cov['tokenied_text'])
for w in all_words:
    wd.append(w.lower())
wd=nltk.FreqDist(wd)
word_feature=(list(wd.keys()))[:3000]
def find_feature(cov):
    stmt=set(cov)
    features={}
    for w in word_feature:
        features[w]=w in words
    return features
std=cov.type.unique()
featureset=[(find_feature(cov.Complaint),std) for (Complaint,type) in cov]

但是当我使用 word tokenize 时，我在行出现以下错误

cov['tokenized_text'] = cov.apply(lambda row: word_tokenize(cov['Complaint']), axis=1)

在 self._lang_vars.period_context_re().finditer(text) 中匹配： TypeError: ('expected string or bytes-like object', '发生在索引 0')

Answer 1

改变

cov['tokenized_text'] = cov.apply(lambda row:word_tokenize(cov['Complaint']), axis=1) 到

cov['tokenized_text'] = cov.Complaint.apply(lambda row: word_tokenize(row))

将序列上的操作应用于数据框。

>>> cov
                                           Complaint
0  The agent was not able to open because the bot...
1                                    Sample sentence
2  The agent was not able to open because the bot...
3  The agent was not able to open because the bot...
>>> cov['tokenized'] = cov.Complaint.apply(lambda x: word_tokenize(x))
>>> cov
                                           Complaint  \
0  The agent was not able to open because the bot...   
1                                    Sample sentence   
2  The agent was not able to open because the bot...   
3  The agent was not able to open because the bot...   

                                           tokenized  
0  [The, agent, was, not, able, to, open, because...  
1                              [Sample, sentence]  
2  [The, agent, was, not, able, to, open, because...  
3  [The, agent, was, not, able, to, open, because...  
>>> 

如果您想使用计数等文本功能，我建议您使用 sklearn(CountVectorizer 或 TfIdfVectorizer)