以日期值为条件计算平均值和差异答案

【问题标题】：Calculating mean and diff conditional to date values以日期值为条件计算平均值和差异
【发布时间】：2020-11-17 16:02:23
【问题描述】：

我有以下数据框，其中给定的作业work_id 由学生s_id 在日期work_date 执行，相对分数为score。对于每个学生，日期按降序排列。

df = pd.DataFrame(columns=['work_id', 's_id', 'score','work_date'],
...                   data =[['a3', 'p01', np.nan,'2020-05-01'],
...                          ['a2', 'p01',10,'2020-06-10'],
...                          ['a1','p01', 5, '2020-06-15'],
...                          ['a5','p02', 5, '2019-10-10'],
...                          ['a7','p02', 11, '2020-03-01'],
...                          ['a6','p02', np.nan, '2020-04-01'],
...                          ['a4','p02', 4, '2020-06-20'],
...                          ])

>>> df
  work_id s_id  score   work_date
0      a3  p01    NaN  2020-05-01
1      a2  p01   10.0  2020-06-10
2      a1  p01    5.0  2020-06-15
3      a5  p02    5.0  2019-10-10
4      a7  p02   11.0  2020-03-01
5      a6  p02    NaN  2020-04-01
6      a4  p02    4.0  2020-06-20

我想添加两列：mean_score 和 diff_score。 mean_score 列应显示每个学生获得的平均分数，其中计算的平均值包括之前作业中获得的所有分数。 diff_score 列应包含当前分数与前一个分数之间的差异（不是 NaN）。因此，最终的数据帧必须如下所示：

work_id s_id  score   work_date  mean_score  diff_score
0      a3  p01    9.0  2020-05-01         NaN         NaN
1      a2  p01   10.0  2020-06-10    10.00000         NaN
2      a1  p01    5.0  2020-06-15    7.500000        -5.0
3      a5  p02    5.0  2019-10-10    5.000000         NaN
4      a7  p02   11.0  2020-03-01    8.000000         6.0
5      a6  p02    NaN  2020-04-01         NaN         NaN
6      a4  p02    4.0  2020-06-20    6.666667        -7.0

我可以通过定义以下两个函数（负责处理可能存在的 NaN 条目）并使用 apply/lambda 以一种繁琐的方式来实现这一点：

def calculate_mean(workid):
    date = df[df.work_id == workid].work_date.iloc[0]
    sid = df[df.work_id == workid].s_id.iloc[0]
    if df[(df.work_id==workid) & (df.s_id==sid) & (df.work_date == date)].score.notnull().item():
        mean = df[(df.s_id == sid) & (df.work_date <= date)].score.mean()
    else:
        mean = np.nan
    return mean

def calculate_diff(workid):
    date = df[df.work_id == workid].work_date.iloc[0]
    sid = df[df.work_id == workid].s_id.iloc[0]
    try:
        if df[(df.s_id==sid) & (df.work_date == date)].score.notnull().item():
            delta = df[(df.s_id == sid) & (df.work_date <= date) & (df.score.notnull())].score.diff().iloc[-1]
        else:
            delta = np.nan
    except:
        delta = np.nan
    return delta 

df['mean_score'] = df['work_id'].apply(lambda x: calculate_mean(x) )
df['diff_score'] = df['work_id'].apply(lambda x: calculate_diff(x) )

我需要一种更有效的方法（也许使用 groupby），因为这种方法在大型数据帧上非常慢。

【问题讨论】：

标签： python pandas dataframe diff mean

【解决方案1】：

IIUC，将pandas.DataFrame.groupby 与expanding.mean 和diff 一起使用：

g = df.groupby("s_id")["score"]
s1 = g.apply(lambda x: x.dropna().expanding().mean())
s2 = g.apply(lambda x: x.dropna().diff())

df["mean_score"] = s1.reset_index(level=0, drop=True)
df["diff_score"] = s2.reset_index(level=0, drop=True)
print(df)

或者做一个函数：

def mean_and_diff(series):
    s = series.dropna()
    d = {"mean_score": s.expanding().mean(), "diff_score": s.diff()}
    return pd.DataFrame(d)
tmp = df.groupby("s_id")["score"].apply(mean_and_diff).reset_index(level=0, drop=True)
df[["mean_score", "diff_score"]] = tmp[["mean_score", "diff_score"]]

输出：

  work_id s_id  score   work_date  mean_score  diff_score
0      a3  p01    NaN  2020-05-01         NaN         NaN
1      a2  p01   10.0  2020-06-10   10.000000         NaN
2      a1  p01    5.0  2020-06-15    7.500000        -5.0
3      a5  p02    5.0  2019-10-10    5.000000         NaN
4      a7  p02   11.0  2020-03-01    8.000000         6.0
5      a6  p02    NaN  2020-04-01         NaN         NaN
6      a4  p02    4.0  2020-06-20    6.666667        -7.0

【讨论】：