矢量化实现从熊猫数据框中的单行创建多行

Question

对于输入表中的每一行，我需要通过按月分隔日期范围来生成多行。 （请参考下面的示例输出）。

有一种简单的迭代方法可以逐行转换，但在大型数据帧上速度很慢。

谁能建议一种矢量化方法，例如使用 apply()、map() 等来实现目标？

输出表是一个新表。

输入：

ID, START_DATE, END_DATE
1, 2010-12-08, 2011-03-01
2, 2010-12-10, 2011-01-12
3, 2010-12-16, 2011-03-07

输出：

ID, START_DATE, END_DATE, NUMBER_DAYS, ACTION_DATE
1, 2010-12-08, 2010-12-31, 23, 201012
1, 2010-12-08, 2011-01-31, 54, 201101
1, 2010-12-08, 2011-02-28, 82, 201102
1, 2010-12-08, 2011-03-01, 83, 201103
2, 2010-12-10, 2010-12-31, 21, 201012
2, 2010-12-10, 2011-01-12, 33, 201101
3, 2010-12-16, 2010-12-31, 15, 201012
4, 2010-12-16, 2011-01-31, 46, 201101
5, 2010-12-16, 2011-02-28, 74, 201102
6, 2010-12-16, 2011-03-07, 81, 201103

Answer 1

我认为你可以使用：

import pandas as pd

df = pd.DataFrame({'ID': {0: 1, 1: 2, 2: 3}, 
'END_DATE': {0: pd.Timestamp('2011-03-01 00:00:00'),
             1: pd.Timestamp('2011-01-12 00:00:00'), 
             2: pd.Timestamp('2011-03-07 00:00:00')}, 
'START_DATE': {0: pd.Timestamp('2010-12-08 00:00:00'), 
               1: pd.Timestamp('2010-12-10 00:00:00'), 
               2: pd.Timestamp('2010-12-16 00:00:00')}}, 
columns=['ID','START_DATE', 'END_DATE'])

print df
   ID START_DATE   END_DATE
0   1 2010-12-08 2011-03-01
1   2 2010-12-10 2011-01-12
2   3 2010-12-16 2011-03-07

#if multiple columns, you can filter them by subset
#df = df[['ID','START_DATE', 'END_DATE']]

#stack columns START_DATE and END_DATE
df1 = df.set_index('ID')
        .stack()
        .reset_index(level=1, drop=True)
        .to_frame()
        .rename(columns={0:'Date'})
#print df1

#resample and fill missing data 
df1 = df1.groupby(df1.index).apply(lambda x: x.set_index('Date').resample('M').asfreq())
         .reset_index()
print df1

   ID       Date
0   1 2010-12-31
1   1 2011-01-31
2   1 2011-02-28
3   1 2011-03-31
4   2 2010-12-31
5   2 2011-01-31
6   3 2010-12-31
7   3 2011-01-31
8   3 2011-02-28
9   3 2011-03-31

Month 的最后一天有问题，因为resample 添加了Month 的最后一天，所以先创建period 列，然后再创建merge 它们。通过combine_first 添加列Date 的缺失值，并通过bfill 添加列START_DATE 的缺失值。

df['period'] = df.END_DATE.dt.to_period('M')
df1['period'] = df1.Date.dt.to_period('M')

df2 = pd.merge(df1, df, on=['ID','period'], how='left')

df2['END_DATE'] = df2.END_DATE.combine_first(df2.Date)
df2['START_DATE'] = df2.START_DATE.bfill()
df2 = df2.drop(['Date','period'], axis=1)

最后按与dt.days和dt.strftime的区别添加新列：

df2['NUMBER_DAYS'] = (df2.END_DATE - df2.START_DATE).dt.days
df2['ACTION_DATE'] = df2.END_DATE.dt.strftime('%Y%m')

print df2
   ID START_DATE   END_DATE  NUMBER_DAYS ACTION_DATE
0   1 2010-12-08 2010-12-31           23      201012
1   1 2010-12-08 2011-01-31           54      201101
2   1 2010-12-08 2011-02-28           82      201102
3   1 2010-12-08 2011-03-01           83      201103
4   2 2010-12-10 2010-12-31           21      201012
5   2 2010-12-10 2011-01-12           33      201101
6   3 2010-12-16 2010-12-31           15      201012
7   3 2010-12-16 2011-01-31           46      201101
8   3 2010-12-16 2011-02-28           74      201102
9   3 2010-12-16 2011-03-07           81      201103

Answer 2

你也可以试试这个。使用 Pandas 的 date_range 函数和 DataFrame 应用概念。

在您的输出中，对于 3 之后的 ID，您提到了 4、5、6。我认为应该是 3。请检查。

import pandas as pd
from datetime import datetime

l_ret_df = pd.DataFrame(columns=('ID', 'START_DATE', 'END_DATE', 'NUMBER_DAYS', 'ACTION_DATE'))

def generate_ts_df(p_row):
    l_id = p_row['ID']
    l_start = p_row['START_DATE']
    l_start_date = datetime.strptime(l_start,'%Y-%m-%d')
    l_end = p_row['END_DATE']
    l_end_date = datetime.strptime(l_end,'%Y-%m-%d')
    l_df = pd.date_range(start=l_start,end=l_end,freq='M',closed=None)
    global l_ret_df

    for e in l_df:
        l_ret_df = l_ret_df.append(pd.DataFrame([[l_id,l_start,e.date(),(e.date()-l_start_date.date()).days,e.strftime('%Y%m')]],columns=('ID', 'START_DATE', 'END_DATE', 'NUMBER_DAYS', 'ACTION_DATE')))
    l_ret_df = l_ret_df.append(pd.DataFrame([[l_id,l_start,l_end,(l_end_date.date()-l_start_date.date()).days,l_end_date.strftime('%Y%m')]],columns=('ID', 'START_DATE', 'END_DATE', 'NUMBER_DAYS', 'ACTION_DATE')))
    return 1

if __name__ == "__main__":
    l_ts_base = pd.DataFrame([[1, '2010-12-08', '2011-03-01'],
                            [2, '2010-12-10', '2011-01-12'],
                            [3, '2010-12-16', '2011-03-07']], columns=('ID', 'START_DATE', 'END_DATE'))

    l_ts_base.apply(generate_ts_df, axis=1)
    print l_ret_df

输出

   ID  START_DATE    END_DATE  NUMBER_DAYS ACTION_DATE
0   1  2010-12-08  2010-12-31           23      201012
0   1  2010-12-08  2011-01-31           54      201101
0   1  2010-12-08  2011-02-28           82      201102
0   1  2010-12-08  2011-03-01           83      201103
0   2  2010-12-10  2010-12-31           21      201012
0   2  2010-12-10  2011-01-12           33      201101
0   3  2010-12-16  2010-12-31           15      201012
0   3  2010-12-16  2011-01-31           46      201101
0   3  2010-12-16  2011-02-28           74      201102
0   3  2010-12-16  2011-03-07           81      201103