确保 pandas 中的每个组都有相同的产品

Question

我有一个这样的数据集：

date	company_id	product_id	sales
2020-11-30	001	1	200
2020-11-30	002	2	150
2020-12-30	001	1	50
2020-12-30	001	2	400
2020-12-30	002	3	300
2020-12-30	001	4	350
2021-01-30	002	1	600
2021-01-30	001	2	400
2021-01-30	002	4	100
2021-01-30	001	5	50
2021-01-30	002	6	35
2021-01-30	001	7	400

我需要通过 company_id 计算出今天和前一天的销售额之间的差异，这可以通过将其转换为 pandas 数据框、按 company_id 分组并进行轮班来轻松解决。但是，如果销售额为 0，则不报告产品，这意味着当我换班时，我得到 0 来表示变化，而不是增加/减少。

我尝试了下面的方法，它基本上是通过 company_id 获取所有产品的列表，如果没有报告他们用 0 添加它们，但这似乎不能正常工作，并且需要永远运行。

company_ids= set(df['company_id'].tolist())
for company_id company_ids:
    full_prod_list= set(df[df['company_id'] == company_id]['product_id'].tolist())
    for date in set(df[df['company_id'] == company_id]['date'].tolist()):
        date_prod_list =  set(df[(df['product_id'] == product_id) & (df['date'] == date)]['product_id'].tolist())
        diff = [x for x in full_prod_list if x not in date_prod_list]
        for i in diff:
            df.loc[len(df)]=[date, company_id, i, 0]

例如，11 月应如下所示：

date	company_id	product_id	sales
2020-11-30	001	1	200
2020-11-30	001	2	0
2020-11-30	001	4	0
2020-11-30	001	5	0
2020-11-30	001	7	0
2020-11-30	002	1	0
2020-11-30	002	2	150
2020-11-30	002	3	0
2020-11-30	002	4	0
2020-11-30	002	6	0

谢谢

Answer 1

我提供的代码可以运行，但是运行时间很长（69k 行的数据集超过 30m）

将最后一个 for 循环转换为使用新数据帧追加的时间可缩短至 30 秒。

company_ids= set(df['company_id'].tolist())
for company_id company_ids:
    full_prod_list= set(df[df['company_id'] == company_id]['product_id'].tolist())
    for date in set(df[df['company_id'] == company_id]['date'].tolist()):
        date_prod_list =  set(df[(df['product_id'] == product_id) & (df['date'] == date)]['product_id'].tolist())
        diff = [x for x in full_prod_list if x not in date_prod_list]
        insert = [[date] * len(diff), [company_id] * len(diff), diff, [0] * len(diff)]
        append_df = pd.DataFrame (insert).transpose()
        append_df.columns = df.columns
        df = df.append(append_df) 

Answer 2

您可以尝试以下方法：

pd_catType = pd.CategoricalDtype(categories=list(set(df['product_id'])), ordered=True)
ci_catType = pd.CategoricalDtype(categories=list(set(df['company_id'])), ordered=True)
df['product_id'] = df['product_id'].astype(pd_catType)
df['company_id'] = df['company_id'].astype(ci_catType)
df.set_index(['date','company_id', 'product_id'], inplace= True)
cats = pd.MultiIndex.from_product(df.index.levels, names=df.index.names)
df = df.reindex(cats).reset_index()
df['sales'] = df['sales'].fillna(0)

对于您提供的示例，结果如下：

date    company_id  product_id  sales
0   2020-11-30  002 1   0.0
1   2020-11-30  002 2   150.0
2   2020-11-30  002 3   0.0
3   2020-11-30  002 4   0.0
4   2020-11-30  002 5   0.0
5   2020-11-30  002 6   0.0
6   2020-11-30  002 7   0.0
7   2020-11-30  001 1   200.0
8   2020-11-30  001 2   0.0
9   2020-11-30  001 3   0.0
10  2020-11-30  001 4   0.0
11  2020-11-30  001 5   0.0
12  2020-11-30  001 6   0.0
13  2020-11-30  001 7   0.0
14  2020-12-30  002 1   0.0
15  2020-12-30  002 2   0.0
16  2020-12-30  002 3   300.0
17  2020-12-30  002 4   0.0
18  2020-12-30  002 5   0.0
19  2020-12-30  002 6   0.0
20  2020-12-30  002 7   0.0
21  2020-12-30  001 1   50.0
22  2020-12-30  001 2   400.0
23  2020-12-30  001 3   0.0
24  2020-12-30  001 4   350.0
25  2020-12-30  001 5   0.0
26  2020-12-30  001 6   0.0
27  2020-12-30  001 7   0.0
28  2021-01-30  002 1   600.0
29  2021-01-30  002 2   0.0
30  2021-01-30  002 3   0.0
31  2021-01-30  002 4   100.0
32  2021-01-30  002 5   0.0
33  2021-01-30  002 6   35.0
34  2021-01-30  002 7   0.0
35  2021-01-30  001 1   0.0
36  2021-01-30  001 2   400.0
37  2021-01-30  001 3   0.0
38  2021-01-30  001 4   0.0
39  2021-01-30  001 5   50.0
40  2021-01-30  001 6   0.0
41  2021-01-30  001 7   400.0