使用 60,000 行数据框、groupby 和 apply 时更快的 pandas 解决方案答案

【问题标题】：faster pandas solution when working with 60,000 row dataframe, groupby and apply使用 60,000 行数据框、groupby 和 apply 时更快的 pandas 解决方案
【发布时间】：2022-01-26 01:31:04
【问题描述】：

我有当前比赛的比赛和主队结果的数据集

match_date  home    away    home_result   
2021-11-22  team1   team2   Win
2021-11-22  team3   team4   Win 
2021-11-23  team1   team8   Lose
2021-11-23  team6   team7   Win
2021-11-25  team1   team2   Win 
2021-11-25  team3   team8   Lose 
2021-11-25  team1   team5   Lose 
2021-11-25  team6   team5   Win 
2021-11-28  team3   team1   Lose 
2021-11-29  team1   team5   Win 
2021-11-29  team6   team9   Win

我有一个代码来创建新列，我可以在其中放置当前比赛之前每个主队的先前结果：

df['home_team_previous_results'] = (
    df.groupby('home')
    .apply(
        lambda x: pd.Series(
            [
                [
                    tuple([row[col] for col in ['home_result', 'match_date']])
                    for _, row in x.iloc[0:i].iterrows()
                ] or np.nan
                for i in range(len(x))
            ],
        index=x.index)
    ).droplevel(0)
)

这是输出：

match_date  home    away    home_result   home_team_previous_results
2021-11-22  team1   team2   Win           NaN
2021-11-22  team3   team4   Win           NaN
2021-11-23  team1   team8   Lose          [("Win","2021-11-22")]  
2021-11-23  team6   team7   Win           NaN 
2021-11-25  team1   team2   Win           [("Win","2021-11-22"), ("Lose","2021-11-23")]
2021-11-25  team3   team8   Lose          [("Win","2021-11-22")]
2021-11-25  team1   team5   Lose          [("Win","2021-11-22"), ("Lose","2021-11-23"), ("Win","2021-11-25")]
2021-11-25  team6   team5   Win           [("Win","2021-11-23")]
2021-11-28  team3   team1   Lose          [("Win","2021-11-22"), ("Lose","2021-11-25")]
2021-11-29  team1   team5   Win           [("Win","2021-11-22"), ("Lose","2021-11-23"), ("Win","2021-11-25"), ("Lose","2021-11-25")]
2021-11-29  team6   team9   Win           [("Win","2021-11-23"), ("Win","2021-11-25")]

问题是代码需要很长时间才能处理大型数据集（超过 60,000 行），我需要它运行得非常非常快。知道如何让它更快或更好的版本吗？

【问题讨论】：

不要使用iterrows，而是找到一种矢量方法来解决您的任务
您可以将自定义函数（如果需要，可以使用附加参数）应用于行/列。 Maybe this can help

标签： python pandas numpy

【解决方案1】：

你可以做鸡蛋tuple 然后cumsum

df['new'] = df[['match_date','home_result']].agg(tuple,1).groupby(df['home']).apply(lambda x : x.cumsum().shift())
0                                                   NaN
1                                                   NaN
2                                     (2021-11-22, Win)
3                                                   NaN
4                   (2021-11-22, Win, 2021-11-23, Lose)
5                                     (2021-11-22, Win)
6     (2021-11-22, Win, 2021-11-23, Lose, 2021-11-25...
7                                     (2021-11-23, Win)
8                   (2021-11-22, Win, 2021-11-25, Lose)
9     (2021-11-22, Win, 2021-11-23, Lose, 2021-11-25...
10                   (2021-11-23, Win, 2021-11-25, Win)
dtype: object

【讨论】：