熊猫重采样和后填充的有效方法

Question

如下例所示，目标是将多索引时间序列重新采样为特定长度和频率。我已经达到了这个目标，但不幸的是通过.apply 电话。 apply 无限期地减慢我的代码速度。

除了多处理之外，还有更有效的方法吗？

import pandas as pd

#---------------------------------------------------------
#                  The input Data
#---------------------------------------------------------

input = pd.DataFrame([
    
    ['E', '2020-03-07', '2020-03-04', 0.3],
    ['E', '2020-03-07', '2020-03-06', 0.1],
    
    ['D', '2020-03-09', '2020-03-05', 0.5],
    ['D', '2020-03-09', '2020-03-06', 0.6],
    ],
columns=['id','y_date', 'x_date','a'])
input['x_date'] = pd.to_datetime(input['x_date'])
input['y_date'] = pd.to_datetime(input['y_date'])

#---------------------------------------------------------
# some paramters for the function groupwise_asfreq
#---------------------------------------------------------

input_t_dim = 6 # the desiered length of the back padded timeseries
missing_value = -1 # the value to fill for missing values


#---------------------------------------------------------
#         the resampling and passing function
#---------------------------------------------------------

def groupwise_asfreq(group):
    
    # resample the available data into the desiered interval e.g. 12h
    freqenced = group.resample('12h', ).mean() # force a result with mean()
    
    # take the resampled data and reindex them with a constucted date_range
    padded=freqenced.reindex(pd.date_range(end=freqenced.index.max(),freq='12h',periods=input_t_dim, name='x_date'),fill_value=missing_value)
    
    return padded


#---------------------------------------------------------
#                 the "convinient" apply
#---------------------------------------------------------
# use the unfortunate apply    
output = input.set_index('x_date').groupby(['id','y_date']).apply(groupwise_asfreq)

# fill the remaining missing values
output = output.fillna(missing_value)

#---------------------------------------------------------
#                  Resulting DataFrame
#---------------------------------------------------------
                                    a
id  y_date      x_date  
D   2020-03-09  2020-03-03 12:00:00 -1.0
                2020-03-04 00:00:00 -1.0
                2020-03-04 12:00:00 -1.0
                2020-03-05 00:00:00 0.5
                2020-03-05 12:00:00 -1.0
                2020-03-06 00:00:00 0.6
E   2020-03-07  2020-03-03 12:00:00 -1.0
                2020-03-04 00:00:00 0.3
                2020-03-04 12:00:00 -1.0
                2020-03-05 00:00:00 -1.0
                2020-03-05 12:00:00 -1.0
                2020-03-06 00:00:00 0.1```

Answer 1

一个选项是使用pyjanitor 中的complete 函数，这是一种暴露缺失行的便捷方法：

# pip install pyjanitor
import pandas as pd
import janitor

df = pd.DataFrame([
    
    ['E', '2020-03-07', '2020-03-04', 0.3],
    ['E', '2020-03-07', '2020-03-06', 0.1],
    
    ['D', '2020-03-09', '2020-03-05', 0.5],
    ['D', '2020-03-09', '2020-03-06', 0.6],
    ],
columns=['id','y_date', 'x_date','a'])
df['x_date'] = pd.to_datetime(df['x_date'])
df['y_date'] = pd.to_datetime(df['y_date'])

input_t_dim = 6
missing_value = -1

# build a dictionary for `X_date`, with new values:
# the end date will be the max date for the group
# generated for every 12 hours, with 6 periods
dates = {'x_date' : lambda df: pd.date_range(end = df.max(), 
                                            freq='12H', 
                                            periods = input_t_dim)
          }

# run the complete function
df.complete('y_date', dates, by='id', sort = True).fillna({'a' : missing_value})

   id     y_date              x_date    a
0   D 2020-03-09 2020-03-03 12:00:00 -1.0
1   D 2020-03-09 2020-03-04 00:00:00 -1.0
2   D 2020-03-09 2020-03-04 12:00:00 -1.0
3   D 2020-03-09 2020-03-05 00:00:00  0.5
4   D 2020-03-09 2020-03-05 12:00:00 -1.0
5   D 2020-03-09 2020-03-06 00:00:00  0.6
6   E 2020-03-07 2020-03-03 12:00:00 -1.0
7   E 2020-03-07 2020-03-04 00:00:00  0.3
8   E 2020-03-07 2020-03-04 12:00:00 -1.0
9   E 2020-03-07 2020-03-05 00:00:00 -1.0
10  E 2020-03-07 2020-03-05 12:00:00 -1.0
11  E 2020-03-07 2020-03-06 00:00:00  0.1

你可以在 Pandas 中运行它并忽略来自 pyjanitor 的辅助函数：

(df
.set_index('x_date')
.groupby('id')
.apply(lambda df: df.reindex(pd.date_range(end = df.index.max(), 
                                           freq = '12H', 
                                           periods = input_t_dim, 
                                           name = 'x_date')
                             )
         )
.fillna({'a':missing_value})
.assign(y_date = lambda df: df.y_date.bfill())
.drop(columns='id')
.reset_index()
)
 
   id              x_date     y_date    a
0   D 2020-03-03 12:00:00 2020-03-09 -1.0
1   D 2020-03-04 00:00:00 2020-03-09 -1.0
2   D 2020-03-04 12:00:00 2020-03-09 -1.0
3   D 2020-03-05 00:00:00 2020-03-09  0.5
4   D 2020-03-05 12:00:00 2020-03-09 -1.0
5   D 2020-03-06 00:00:00 2020-03-09  0.6
6   E 2020-03-03 12:00:00 2020-03-07 -1.0
7   E 2020-03-04 00:00:00 2020-03-07  0.3
8   E 2020-03-04 12:00:00 2020-03-07 -1.0
9   E 2020-03-05 00:00:00 2020-03-07 -1.0
10  E 2020-03-05 12:00:00 2020-03-07 -1.0
11  E 2020-03-06 00:00:00 2020-03-07  0.1

测试这两个选项，看看它的速度如何。