在熊猫数据框中使用 RegexpTokenizer答案

【问题标题】：Use RegexpTokenizer in a pandas dataframe在熊猫数据框中使用 RegexpTokenizer
【发布时间】：2019-07-15 20:51:27
【问题描述】：

我正在尝试在数据框的列中应用 RegexpTokenizer。

数据框：

    all_cols
0   who is your hero and why
1   what do you do to relax
2   can't stop to eat
4   how many hours of sleep do you get a night
5   describe the last time you were relax

脚本：

import re
import nltk
import pandas as pd
from nltk import RegexpTokenizer

#tokenization of data and suppression of None (NA)
df['all_cols'].dropna(inplace=True)

tokenizer = RegexpTokenizer("[\w']+")
df['all_cols'] = df['all_cols'].apply(tokenizer)

错误：

TypeError: 'RegexpTokenizer' 对象不可调用

但我不明白。当我使用另一种 nltk 标记化模式 word_tokenize 时，效果很好......

【问题讨论】：

标签： python pandas nltk

【解决方案1】：

请注意，当调用RegexpTokenizer 时，您只是创建了一个带有一组参数的类的实例（调用它的__init__ 方法）。为了使用指定的模式实际标记数据框列，您必须调用其RegexpTokenizer.tokenize 方法：

tokenizer = RegexpTokenizer("[\w']+")
df['all_cols'] = df['all_cols'].map(tokenizer.tokenize)

       all_cols
0  [who, is, your, hero, and, why]
1   [what, do, you, do, to, relax]
...

【讨论】：

【解决方案2】：

首先要删除缺失值，必须使用 DataFrame.dropna 并指定列名，然后使用 tokenizer.tokenize 方法，因为您的解决方案不会删除缺失值：

df = pd.DataFrame({'all_cols':['who is your hero and why',
                               'what do you do to relax', 
                               "can't stop to eat", np.nan]})
print (df)
                   all_cols
0  who is your hero and why
1   what do you do to relax
2         can't stop to eat
3                       NaN

#solution remove missing values from Series, not rows from df
df['all_cols'].dropna(inplace=True)
print (df)
                   all_cols
0  who is your hero and why
1   what do you do to relax
2         can't stop to eat
3                       NaN

#solution correct remove rows by missing values
df.dropna(subset=['all_cols'], inplace=True)
print (df)
                   all_cols
0  who is your hero and why
1   what do you do to relax
2         can't stop to eat

tokenizer = RegexpTokenizer("[\w']+")
df['all_cols'] = df['all_cols'].apply(tokenizer.tokenize)
print (df)
                          all_cols
0  [who, is, your, hero, and, why]
1   [what, do, you, do, to, relax]
2           [can't, stop, to, eat]

【讨论】：