我提出以下解决方案。具有相同customerID的客户不会出现在训练和测试中； aslo 客户按其活动划分 - 即具有相同贷款数量的用户中大致相等的部分将被置于训练和测试中。
出于演示目的，我扩展了数据样本：
d = {'loan_date': ['20170101','20170701','20170301','20170415','20170515','20170905', '20170814', '20170819', '20170304'],
'customerID': [111,111,222,333,444,222,111,444,555],
'loanID': ['aaa','fff','ccc','ddd','bbb','eee', 'kkk', 'zzz', 'yyy'],
'loan_duration' : [6,3,12,5,12, 3, 17, 4, 6],
'gender':['F','F','M','F','M','M', 'F', 'M','F'],
'loan_amount': [20000,10000,30000,10000,40000,20000,30000,30000,40000],
'default':[0,1,0,0,1,0,1,1,0]}
df = pd.DataFrame(data=d)
代码：
from sklearn.model_selection import train_test_split
def group_customers_by_activity(df):
value_count = df.customerID.value_counts().reset_index()
df_by_customer = df.set_index('customerID')
df_s = [df_by_customer.loc[value_count[value_count.customerID == count]['index']] for count in value_count.customerID.unique()]
return df_s
- 此函数将 df 按customerID 活动（具有相同customerID 的条目数）拆分。
此函数的示例输出：
group_customers_by_activity(df)
Out:
[ loan_date loanID loan_duration gender loan_amount default
customerID
111 20170101 aaa 6 F 20000 0
111 20170701 fff 3 F 10000 1
111 20170814 kkk 17 F 30000 1,
loan_date loanID loan_duration gender loan_amount default
customerID
222 20170301 ccc 12 M 30000 0
222 20170905 eee 3 M 20000 0
444 20170515 bbb 12 M 40000 1
444 20170819 zzz 4 M 30000 1,
loan_date loanID loan_duration gender loan_amount default
customerID
333 20170415 ddd 5 F 10000 0
555 20170304 yyy 6 F 40000 0]
- 拥有 1、2、3 笔贷款等的用户组。
此函数以用户上火车或测试的方式拆分组：
def split_group(df_group, train_size=0.8):
customers = df_group.index.unique()
train_customers, test_customers = train_test_split(customers, train_size=train_size)
train_df, test_df = df_group.loc[train_customers], df_group.loc[test_customers]
return train_df, test_df
split_group(df_s[2])
Out:
( loan_date loanID loan_duration gender loan_amount default
customerID
444 20170515 bbb 12 M 40000 1
444 20170819 zzz 4 M 30000 1,
loan_date loanID loan_duration gender loan_amount default
customerID
222 20170301 ccc 12 M 30000 0
222 20170905 eee 3 M 20000 0)
剩下的就是将此应用于所有“客户活动”组：
def get_sized_splits(df_s, train_size):
train_splits, test_splits = zip(*[split_group(df_group, train_size) for df_group in df_s])
return train_splits, test_splits
df_s = group_customers_by_activity(df)
train_splits, test_splits = get_sized_splits(df_s, 0.8)
train_splits, test_splits
Out:
((Empty DataFrame
Columns: [loan_date, loanID, loan_duration, gender, loan_amount, default]
Index: [],
loan_date loanID loan_duration gender loan_amount default
customerID
444 20170515 bbb 12 M 40000 1
444 20170819 zzz 4 M 30000 1,
loan_date loanID loan_duration gender loan_amount default
customerID
333 20170415 ddd 5 F 10000 0),
( loan_date loanID loan_duration gender loan_amount default
customerID
111 20170101 aaa 6 F 20000 0
111 20170701 fff 3 F 10000 1
111 20170814 kkk 17 F 30000 1,
loan_date loanID loan_duration gender loan_amount default
customerID
222 20170301 ccc 12 M 30000 0
222 20170905 eee 3 M 20000 0,
loan_date loanID loan_duration gender loan_amount default
customerID
555 20170304 yyy 6 F 40000 0))
不要害怕 emty DataFrame，它很快就会被连接起来。 split 函数定义如下：
def split(df, train_size):
df_s = group_customers_by_activity(df)
train_splits, test_splits = get_sized_splits(df_s, train_size=train_size)
return pd.concat(train_splits), pd.concat(test_splits)
split(df, 0.8)
Out[106]:
( loan_date loanID loan_duration gender loan_amount default
customerID
444 20170515 bbb 12 M 40000 1
444 20170819 zzz 4 M 30000 1
555 20170304 yyy 6 F 40000 0,
loan_date loanID loan_duration gender loan_amount default
customerID
111 20170101 aaa 6 F 20000 0
111 20170701 fff 3 F 10000 1
111 20170814 kkk 17 F 30000 1
222 20170301 ccc 12 M 30000 0
222 20170905 eee 3 M 20000 0
333 20170415 ddd 5 F 10000 0)
- 因此，customerID 被放置在训练数据或测试数据中。由于输入数据的大小，我猜想这样一个奇怪的狭缝（火车 > 测试）。
如果您不需要按“customerID 活动”进行分组，则可以省略它，只需使用split_group 即可达到目标。