使 Pandas 中的聚合更高效

Question

在这个问题中我有两个数据框我想向loan_df 添加一个列，该列在recharge_df 中聚合。因此，对于每笔贷款，我想在贷款日期之前获得借款人的平均充值（在这种情况下是 90 天前）。然后我会将这个新列添加到loan_df。我下面的代码有效，但速度很慢。关于如何使其超级高效的任何想法？

def mean_rec_func(msisdn,date,advance_id,window, name):
"""Returns mean recharges within a specified number of days prior to loan being taken
Keyword Arguments:
msisdn -- APF_MSISDN for loan (this is like customer ID)
date -- APF_DATE on which loan taken
advance_id -- APF_ADVANCE_ID for loan
window -- number of days to look back(int)
name -- name of the newly computed stat
"""
mean_rec = recharge_df.loc[(recharge_df['APF_MSISDN'] == msisdn) &
                          (recharge_df['APF_DATE']<date)
                          & (recharge_df['APF_DATE']>=date - datetime.timedelta(days = window))
                          ]['APF_AMOUNT'].mean()
return pd.Series([advance_id,msisdn,mean_rec], index=['APF_ADVANCE_ID', 'APF_MSISDN', name])
# Mean recharge over last 90 days
mean_recharge_90 = loan_df.apply(lambda row: mean_rec_func(row['APF_MSISDN'], row['APF_DATE'],
                                                         row['APF_ADVANCE_ID'],
                                                         window = 90,
                                                         name ="MEAN_RECHARGE_90"), axis = 1)

编辑：

Answer 1

考虑一个 SQL 解决方案，因为您的逻辑将转换为具有相关聚合子查询的以下查询（诚然，这也是一种昂贵的查询类型，因为聚合针对每个外部查询行运行，类似于 pandas apply 循环）。

SELECT l.*, 
       (SELECT AVG([APF_AMOUNT]) FROM recharge_df r
        WHERE r.[APF_DATE] >= date(l.[APF_DATE], '-90 day') 
          AND r.[APF_DATE] < l.[APF_DATE]
          AND r.[APF_MSISDN] = l.[APF_MSISDN]) AS mean_recharge_90
FROM loan_df l

在 pandas 中，您可以使用 pandasql 模块，该模块将内存中的实例运行到 SQLite：

from pandasql import sqldf

pysqldf = lambda q: sqldf(q, globals())

sql = """SELECT l.*, 
            (SELECT AVG([APF_AMOUNT]) FROM recharge_df r
             WHERE r.[APF_DATE] >= date(l.[APF_DATE], '-90 day') 
               AND r.[APF_DATE] < l.[APF_DATE]
               AND r.[APF_MSISDN] = l.[APF_MSISDN]) AS mean_recharge_90
         FROM loan_df l"""

output_df = pysqldf(q)

以下是在pandasql 的后台运行的扩展版本，与 SQLAlchemy 和 pandas 的导入/导出调用接口：read_sql 和 to_sql。

from sqlalchemy import create_engine

# IN-MEMORY DATABASE (NO PATH SPECIFIED)
engine = create_engine('sqlite://')

# EXPORT DATAFRAMES
recharge_df.to_sql("recharge_tbl", con=engine, if_exists='replace')
loan_df.to_sql("loan_tbl", con=engine, if_exists='replace')

sql = """SELECT l.*, 
            (SELECT AVG([APF_AMOUNT]) FROM recharge_tbl r
             WHERE r.[APF_DATE] >= date(l.[APF_DATE], '-90 day') 
               AND r.[APF_DATE] < l.[APF_DATE]
               AND r.[APF_MSISDN] = l.[APF_MSISDN]) AS mean_recharge_90
         FROM loan_tbl l"""

# IMPORT QUERY RESULT
output_df = pd.read_sql(strSQL, engine)

# IN-MEMORY DATABASE DESTROYED
engine.dispose()