pandas 按组值过滤行

Question

这是我正在练习的数据

import pandas as pd
df = pd.read_csv("https://raw.githubusercontent.com/mwaskom/seaborn-data/master/tips.csv")

我想按分组值过滤单个行。我知道我可以执行以下操作来过滤组

df.groupby("day").filter(lambda x: x['total_bill'].mean() > 20).day.unique()

找出哪些天的平均账单大于 20 美元。这是有效的，因为 groupby.filter 需要一个函数来应用于每个应该返回 True 或 False 的子帧。但是，如果我想找到 total_bill 的值大于当天的 total_bill 的每一餐（行）怎么办。例如，如果一行有 total_bill 和 22 并且是在星期日，那么应该保留它，因为星期日的 total_bill 平均值是 21.41。

这是我的尝试：

df.groupby('day').apply(lambda x: x['total_bill'] > x['total_bill'].mean())

但是，这会产生如下所示的内容（前几行）

day    
Fri  90     True
     91     True
     92    False
     93    False
     94     True
Name: total_bill, dtype: bool

这与数据框的顺序不同，所以我不能只取布尔列并使用它来索引数据。

所以现在我执行以下操作：

grouped = (df
           .groupby('day')
           .apply(lambda x: x['total_bill'] > x['total_bill'].mean())
           .reset_index())

index_bill = (grouped
             .loc[grouped.total_bill == True, 'level_1'].values)
df.loc[index_bill]

这给了我想要的结果......必须有一个更简单的方法，对吧？请让我知道是否有适当的方法来做到这一点。如果没有，至少有一种方法可以将这两个步骤合二为一吗？我可以进行 groupby，但不确定如何在不将分组对象存储为变量然后引用它的情况下获取值。谢谢！

Answer 1

我认为最好的方法是使用带有groupby 和transfrom 的布尔索引。首先，您按天分组以查找当天的平均值，然后使用转换将该平均值应用于每一行，然后将该平均值与当天的实际 total_billed 进行比较，然后使用该布尔系列通过布尔索引过滤您的数据框。

df[df.groupby('day')['total_bill'].transform('mean') < df['total_bill']]

输出：

     total_bill   tip     sex smoker   day    time  size
3         23.68  3.31    Male     No   Sun  Dinner     2
4         24.59  3.61  Female     No   Sun  Dinner     4
5         25.29  4.71    Male     No   Sun  Dinner     4
7         26.88  3.12    Male     No   Sun  Dinner     4
11        35.26  5.00  Female     No   Sun  Dinner     4
15        21.58  3.92    Male     No   Sun  Dinner     2
19        20.65  3.35    Male     No   Sat  Dinner     3
23        39.42  7.58    Male     No   Sat  Dinner     4
28        21.70  4.30    Male     No   Sat  Dinner     2
33        20.69  2.45  Female     No   Sat  Dinner     4
35        24.06  3.60    Male     No   Sat  Dinner     3
39        31.27  5.00    Male     No   Sat  Dinner     3
44        30.40  5.60    Male     No   Sun  Dinner     4
46        22.23  5.00    Male     No   Sun  Dinner     2
47        32.40  6.00    Male     No   Sun  Dinner     4
48        28.55  2.05    Male     No   Sun  Dinner     3
52        34.81  5.20  Female     No   Sun  Dinner     4
54        25.56  4.34    Male     No   Sun  Dinner     4
56        38.01  3.00    Male    Yes   Sat  Dinner     4
57        26.41  1.50  Female     No   Sat  Dinner     2
59        48.27  6.73    Male     No   Sat  Dinner     4
72        26.86  3.14  Female    Yes   Sat  Dinner     2
73        25.28  5.00  Female    Yes   Sat  Dinner     2
77        27.20  4.00    Male     No  Thur   Lunch     4
78        22.76  3.00    Male     No  Thur   Lunch     2
80        19.44  3.00    Male    Yes  Thur   Lunch     2
83        32.68  5.00    Male    Yes  Thur   Lunch     2
85        34.83  5.17  Female     No  Thur   Lunch     4
87        18.28  4.00    Male     No  Thur   Lunch     2
88        24.71  5.85    Male     No  Thur   Lunch     2
..          ...   ...     ...    ...   ...     ...   ...
180       34.65  3.68    Male    Yes   Sun  Dinner     4
181       23.33  5.65    Male    Yes   Sun  Dinner     2
182       45.35  3.50    Male    Yes   Sun  Dinner     3
183       23.17  6.50    Male    Yes   Sun  Dinner     4
184       40.55  3.00    Male    Yes   Sun  Dinner     2
187       30.46  2.00    Male    Yes   Sun  Dinner     5
189       23.10  4.00    Male    Yes   Sun  Dinner     3
191       19.81  4.19  Female    Yes  Thur   Lunch     2
192       28.44  2.56    Male    Yes  Thur   Lunch     2
197       43.11  5.00  Female    Yes  Thur   Lunch     4
200       18.71  4.00    Male    Yes  Thur   Lunch     3
204       20.53  4.00    Male    Yes  Thur   Lunch     4
206       26.59  3.41    Male    Yes   Sat  Dinner     3
207       38.73  3.00    Male    Yes   Sat  Dinner     4
208       24.27  2.03    Male    Yes   Sat  Dinner     2
210       30.06  2.00    Male    Yes   Sat  Dinner     3
211       25.89  5.16    Male    Yes   Sat  Dinner     4
212       48.33  9.00    Male     No   Sat  Dinner     4
214       28.17  6.50  Female    Yes   Sat  Dinner     3
216       28.15  3.00    Male    Yes   Sat  Dinner     5
219       30.14  3.09  Female    Yes   Sat  Dinner     4
227       20.45  3.00    Male     No   Sat  Dinner     4
229       22.12  2.88  Female    Yes   Sat  Dinner     2
230       24.01  2.00    Male    Yes   Sat  Dinner     4
237       32.83  1.17    Male    Yes   Sat  Dinner     2
238       35.83  4.67  Female     No   Sat  Dinner     3
239       29.03  5.92    Male     No   Sat  Dinner     3
240       27.18  2.00  Female    Yes   Sat  Dinner     2
241       22.67  2.00    Male    Yes   Sat  Dinner     2
243       18.78  3.00  Female     No  Thur  Dinner     2

[97 rows x 7 columns]

Answer 2

对于相同的输出需要transform 以返回相同的Series 或mean 每组，然后按boolean indexing 过滤并最后添加sort_values：

a=df[df['total_bill'] > df.groupby('day')['total_bill'].transform('mean')].sort_values('day')
print (a.head(20))
     total_bill   tip     sex smoker  day    time  size
90        28.97  3.00    Male    Yes  Fri  Dinner     2
91        22.49  3.50    Male     No  Fri  Dinner     2
94        22.75  3.25  Female     No  Fri  Dinner     2
95        40.17  4.73    Male    Yes  Fri  Dinner     4
96        27.28  4.00    Male    Yes  Fri  Dinner     2
98        21.01  3.00    Male    Yes  Fri  Dinner     2
102       44.30  2.50  Female    Yes  Sat  Dinner     3
206       26.59  3.41    Male    Yes  Sat  Dinner     3
229       22.12  2.88  Female    Yes  Sat  Dinner     2
227       20.45  3.00    Male     No  Sat  Dinner     4
219       30.14  3.09  Female    Yes  Sat  Dinner     4
237       32.83  1.17    Male    Yes  Sat  Dinner     2
103       22.42  3.48  Female    Yes  Sat  Dinner     2
106       20.49  4.06    Male    Yes  Sat  Dinner     2
107       25.21  4.29    Male    Yes  Sat  Dinner     2
216       28.15  3.00    Male    Yes  Sat  Dinner     5
214       28.17  6.50  Female    Yes  Sat  Dinner     3
241       22.67  2.00    Male    Yes  Sat  Dinner     2
212       48.33  9.00    Male     No  Sat  Dinner     4
211       25.89  5.16    Male    Yes  Sat  Dinner     4

编辑：

对于days 的正确排序，可以使用ordered categorical：

cats = ['Mon','Tue','Wed','Thur','Fri','Sat','Sun']
df['day'] = pd.Categorical(df['day'], categories=cats, ordered=True)
means = df.groupby('day')['total_bill'].transform('mean')
df1 = df[df['total_bill'] > means].sort_values('day')
print (df1.head(20))
     total_bill   tip     sex smoker   day   time  size
129       22.82  2.18    Male     No  Thur  Lunch     3
80        19.44  3.00    Male    Yes  Thur  Lunch     2
83        32.68  5.00    Male    Yes  Thur  Lunch     2
85        34.83  5.17  Female     No  Thur  Lunch     4
87        18.28  4.00    Male     No  Thur  Lunch     2
88        24.71  5.85    Male     No  Thur  Lunch     2
89        21.16  3.00    Male     No  Thur  Lunch     2
119       24.08  2.92  Female     No  Thur  Lunch     4
125       29.80  4.20  Female     No  Thur  Lunch     6
130       19.08  1.50    Male     No  Thur  Lunch     2
78        22.76  3.00    Male     No  Thur  Lunch     2
131       20.27  2.83  Female     No  Thur  Lunch     2
141       34.30  6.70    Male     No  Thur  Lunch     6
142       41.19  5.00    Male     No  Thur  Lunch     5
143       27.05  5.00  Female     No  Thur  Lunch     6
146       18.64  1.36  Female     No  Thur  Lunch     3
191       19.81  4.19  Female    Yes  Thur  Lunch     2
192       28.44  2.56    Male    Yes  Thur  Lunch     2
197       43.11  5.00  Female    Yes  Thur  Lunch     4
200       18.71  4.00    Male    Yes  Thur  Lunch     3