如何使用 pandas 优雅地删除已识别的记录？

Question

我有一个如下所示的 pandas 数据框

df = pd.DataFrame({'subject_id': [101,102,103,201,202],
                  'test_id':[21,21,np.nan,24,25],
                  'test_name':['A','B',np.nan,'D','E'],
                  'invalid_condition':[0,0,0,0,1]})

我想根据以下条件识别数据中的问题并删除它们

虽然我可以使用下面的代码识别它们，但我不确定如何删除它们

subject_with_no_test_info = len(df.groupby('subject_id').filter(lambda x: x['test_id'].count() == 0))*100/len(df)
test_id_diff_names = len(df.groupby('test_id').filter(lambda x: x['test_name'].nunique() > 1))*100/len(df)
invalid_condition = len(df[df['invalid_condition']==1])*100/len(df)
data_inconsistencies_df = pd.DataFrame([[subject_with_no_test_info,test_id_diff_names,invalid_condition]],columns = ['subject_with_no_test_info','test_id_diff_names','invalid_condition'])

这给了我如下的输出

但现在我想在 data_incosistencies_df 的每一列下删除那些对20%, 40% and 20% 有贡献的记录？

有什么优雅有效的方法可以从数据框中删除这些记录？

Answer 1

试试：

d1=df.groupby('subject_id').filter(lambda x: x['test_id'].count() == 0)
d2=df.groupby('test_id').filter(lambda x: x['test_name'].nunique() > 1)
d3=df[df['invalid_condition']==1]
#your conditions

data_inconsistencies_df = pd.DataFrame([[(len(d1)*100/len(df)),(len(d2)*100/len(df)),(len(d3)*100/len(df))]],columns = ['subject_with_no_test_info','test_id_diff_names','invalid_condition'])
#created dataframe to show percentages

d2=d2.drop_duplicates('test_id',keep='last')
          #^your sub condition
          #(We don't drop  101, because we keep = first of duplicate test ids)
to_drop=pd.concat([d1,d2,d3]).index
#concatinating 3 dataframes to grab the index of the rows which are going to drop

最后：

df=df.drop(to_drop)
#dropping those indexes

df 的输出：

  subject_id    test_id     test_name   invalid_condition
0   101         21.0            A           0
3   201         24.0            D           0

data_inconsistencies_df的输出：

  subject_with_no_test_info     test_id_diff_names  invalid_condition
0          20.0                         40.0            20.0