在 groupby 对象内的列列表上生成滚动计算的更快方法答案

【问题标题】：Faster Way to Generate Rolling Calculations on a list of columns within a groupby object在 groupby 对象内的列列表上生成滚动计算的更快方法
【发布时间】：2018-07-29 14:01:46
【问题描述】：

我创建了这个函数来计算我的 df 中一系列壮举的滚动统计数据。此功能按预期工作，但在我的 df 上运行大约需要 30 分钟，它有大约 100 万行。在 python/pandas 中是否有更快的方法来做到这一点？

def add_rolling_vars(df, feats, amounts, group):
#creates rolling stats for a list of feats(columns) over a list of amounts[12,48](window sizes)
#grouped by a group like $gvkey or $sector
orig_feats = feats.copy()
new_feats= []
for amount in amounts:
    for name in feats:
        df[group+'_'+name+f'_{amount}_sma'] = df.groupby(group)[name].rolling(amount,1).mean().values
        df[group+'_'+name+f'_{amount}_std'] = df.groupby(group)[name].rolling(amount,1).std().values
        df[group+'_'+name+f'_{amount}_min'] = df.groupby(group)[name].rolling(amount,1).min().values
        df[group+'_'+name+f'_{amount}_max'] = df.groupby(group)[name].rolling(amount,1).max().values
        df[group+'_'+name+f'_{amount}_med'] = df.groupby(group)[name].rolling(amount,1).median().values
        df[group+'_'+name+f'_{amount}_25Q'] = df.groupby(group)[name].rolling(amount,1).quantile(.25).values
        df[group+'_'+name+f'_{amount}_75Q'] = df.groupby(group)[name].rolling(amount,1).quantile(.75).values

作为一个例子，我还创建了这个函数，它在大约 1 分钟内在同一个数据集上运行。显然，它是不同的，因为它不必遍历行窗口，但我仍然可以传递专长列表而不是专长中的名称，然后使用列表理解命名方案将整个转换后的输出添加到我的数据框：

def add_cat_stats(df,feats,group):
    #feats is a list of continuous feats to compute the monthly stats of       
    df[[group+'_'+name+'_avg' for name in feats]] = df.groupby([group,'Date'])[feats].transform('mean')
    df[[group+'_'+name+'_std' for name in feats]] = df.groupby([group,'Date'])[feats].transform('std')
    df[[group+'_'+name+'_min' for name in feats]] = df.groupby([group,'Date'])[feats].transform('min')
    df[[group+'_'+name+'_max' for name in feats]] = df.groupby([group,'Date'])[feats].transform('max')
    df[[group+'_'+name+'_med' for name in feats]] = df.groupby([group,'Date'])[feats].transform('median')

更新

len(数量) = 2

len(feats)= 16

【问题讨论】：

标签： python python-3.x performance pandas cython

【解决方案1】：

我无法让@John Zwinck 代码工作，但它确实让我想到了重新格式化代码，将时间从 30 分钟缩短到 4 分钟 45 秒，这很棒！进一步降低它会很好，但这是一个可行的解决方案：

def add_rolling_vars(df, feats, amounts, group):
    for amount in amounts:
        grouped = df.groupby(group)[feats].rolling(amount,1)
        prefix = ['_'.join([group, name, str(amount)]) for name in feats]
        df[[pre+'_sma' for pre in prefix]] = grouped.mean().reset_index(0,drop=True)
        df[[pre+'_std' for pre in prefix]] = grouped.std().reset_index(0,drop=True)
        df[[pre+'_min' for pre in prefix]] = grouped.min().reset_index(0,drop=True)
        df[[pre+'_max' for pre in prefix]] = grouped.max().reset_index(0,drop=True)
        df[[pre+'_med' for pre in prefix]] = grouped.median().reset_index(0,drop=True)
        df[[pre+'_25Q' for pre in prefix]] = grouped.quantile(.25).reset_index(0,drop=True)
        df[[pre+'_75Q' for pre in prefix]] = grouped.quantile(.75).reset_index(0,drop=True)

【讨论】：