groupby 并应用于两个数据帧

Question

我有一个包含 3 列的 pandas 数据框：key1, key2, document。所有三列都是文本字段，大小为document，范围从50 个字符到5000 个字符。我根据每个(key1, key2) 的文档集的最小频率确定了一个词汇表，我为其使用scikit-learn CountVectorizer 并设置min_df。我可以使用df.groupby[['key1','key2']]['document'].apply(vocab).reset_index() 来做到这一点，其中vocab 是一个函数，我在其中计算并返回作为集合的词汇表（如上定义）。

现在，我想使用这些词汇表（每个key1, key2 一组）来过滤相应的文档，以便每个文档只包含其词汇表中的单词。对于这部分我能得到任何帮助，我将不胜感激。

样本数据

Input

key1 | key2 | document
 aa  | bb   | He went home that evening. Then he had soup for dinner.
 aa  | bb   | We want to sit down and eat dinner
 cc  | mm   | Sometimes people eat in a restaurant
 aa  | bb   | The culinary skills of that chef are terrible.  Let us not go there.
 cc  | mm   | People go home after dinner and try to sleep.


Vocabulary - not using counts for the purpose of this example

key1 | key2 | vocab
 aa  | bb   | {went, evening, sit, down, culinary, chef, dinner}
 cc  | mm   | {people, restaurant, home, dinner, sleep}

Result - only use words from corresponding vocab in document

key1 | key2 | document
 aa  | bb   | went evening dinner
 aa  | bb   | sit down dinner
 cc  | mm   | people restaurant
 aa  | bb   | culinary chef
 cc  | mm   | people home dinner sleep

Answer 1

您可以使用第一个merge 将列vocab 添加到第一个DataFrame：

import re

df = df.groupby[['key1','key2']]['document'].apply(vocab).reset_index()
df = pd.merge(df1, df2, on=['key1','key2'], how='left')

#another theoretical solution
#df['vocab'] = df.groupby[['key1','key2']]['document'].transform(vocab)

然后通过findall提取所有单词，re.I用于忽略大小写，最后删除列vocab：

df['document'] = df['document'].str.findall('\w+', flags=re.I)

最后获取sets之间的交集并通过str.join转换为字符串：

df['document'] = df.apply(lambda x: set(x['document']) & x['vocab'], axis=1).str.join(' ')
df = df.drop('vocab', axis=1)
print (df)
  key1 key2                  document
0   aa   bb       evening went dinner
1   aa   bb           sit down dinner
2   cc   mm         restaurant people
3   aa   bb             chef culinary
4   cc   mm  home people sleep dinner