使用熊猫的宽到长数据集

Question

有很多类似标题的问题，但我无法解决我的数据集遇到的问题。

数据集：

ID   Country Type Region Gender IA01_Raw  IA01_Class1  IA01_Class2 IA02_Raw IA02_Class1 IA02_Class2 QA_Include QA_Comments

SC1  France  A    Europe Male   4         8            1            J         4            1           yes       N/A
SC2  France  A    Europe Female 2         7            2            Q         6            4           yes       N/A
SC3  France  B    Europe Male   3         7            2            K         8            2           yes       N/A
SC4  France  A    Europe Male   4         8            2            A         2            1           yes       N/A
SC5  France  B    Europe Male   1         7            1            F         1            3           yes       N/A
ID6  France  A    Europe Male   2         8            1            R         3            7           yes       N/A
ID7  France  B    Europe Male   2         8            1            Q         4            6           yes       N/A
UC8  France  B    Europe Male   4         8            2            P         4            2           yes       N/A

所需输出：

ID   Country Type Region Gender IA Raw Class1 Class2 QA_Include QA_Comments

SC1  France  A    Europe Male   01 K   8      1      yes        N/A
SC1  France  A    Europe Male   01 L   8      1      yes       N/A
SC1  France  A    Europe Male   01 P   8      1      yes       N/A
SC1  France  A    Europe Male   02 Q   8      1      yes       N/A
SC1  France  A    Europe Male   02 R   8      1      yes       N/A
SC1  France  A    Europe Male   02 T   8      1      yes       N/A
SC1  France  A    Europe Male   03 G   8      1      yes       N/A
SC1  France  A    Europe Male   03 R   8      1      yes       N/A
SC1  France  A    Europe Male   03 G   8      1      yes       N/A
SC1  France  A    Europe Male   04 K   8      1      yes       N/A
SC1  France  A    Europe Male   04 A   8      1      yes       N/A
SC1  France  A    Europe Male   04 P   8      1      yes       N/A
SC1  France  A    Europe Male   05 R   8      1      yes       N/A
....

---

在数据集中，我的列名称为 IA[X]_NAME 其中 X = 1..9 和 NAME = Raw, Class1 和 Class2。

我要做的只是转置这些列，使其看起来像必需输出中显示的表格，即IA将显示X 值，就像这样 raw 和 classes 将显示它们的透视值。

所以为了实现它，我将列切片为：

idVars = list(excel_df_final.columns[0:40]) + list(excel_df_final.columns[472:527]) #These contain columns like ID, Country, Type etc
valueVars = excel_df_final.columns[41:472].tolist() #All the IA_ columns

我不知道这一步是否必要，但这给了我完美的列切片，但是当我将它放入 melt 时，它无法正常工作。我几乎尝试了其他问题中可用的所有方法。

pd.melt(excel_df_final, id_vars=idVars,value_vars=valueVars)

我也试过这个：

excel_df_final.set_index(idVars)[41:472].unstack()

但是没有用，这里是从宽到长的实现，它也没有用：

pd.wide_to_long(excel_df_final, stubnames = ['IA', 'Raw', 'Class1', 'Class2'], i=idVars, j=valueVars)

我得到的从宽到长的错误是：

ValueError: 操作数不能与形状一起广播 (95,) (431,)

由于我的数据集实际上有 526 列，所以我将它们分成两个列表，其中一个包含 95 个列名，即 i，其余 431 个是我需要在如示例数据集中所示的行。

Answer 1

这将使您入门。本质是使用set_index，列转换为MultiIndex，然后stack。可能存在更好的解决方案，但我会这样做，因为它是您输出的一个简单步骤。

# Set the index with columns that we don't want to "transpose"
df2 = df.set_index([
   'ID', 'Country', 'Type', 'Region', 'Gender', 'QA_Include', 'QA_Comments'])
# Convert headers to MultiIndex -- this is so we can melt IA values
df2.columns = pd.MultiIndex.from_tuples(map(tuple, df2.columns.str.split('_')))
# Call stack to replicate data, then reset the index
out =  df2.stack(level=0).reset_index().rename({'level_7': 'IA'}, axis=1)

out

     ID Country Type  Region  Gender QA_Include  QA_Comments    IA  Class1  Class2 Raw
0   SC1  France    A  Europe    Male        yes          NaN  IA01       8       1   4
1   SC1  France    A  Europe    Male        yes          NaN  IA02       4       1   J
2   SC2  France    A  Europe  Female        yes          NaN  IA01       7       2   2
3   SC2  France    A  Europe  Female        yes          NaN  IA02       6       4   Q
4   SC3  France    B  Europe    Male        yes          NaN  IA01       7       2   3
5   SC3  France    B  Europe    Male        yes          NaN  IA02       8       2   K
6   SC4  France    A  Europe    Male        yes          NaN  IA01       8       2   4
7   SC4  France    A  Europe    Male        yes          NaN  IA02       2       1   A
8   SC5  France    B  Europe    Male        yes          NaN  IA01       7       1   1
9   SC5  France    B  Europe    Male        yes          NaN  IA02       1       3   F
10  ID6  France    A  Europe    Male        yes          NaN  IA01       8       1   2
11  ID6  France    A  Europe    Male        yes          NaN  IA02       3       7   R
12  ID7  France    B  Europe    Male        yes          NaN  IA01       8       1   2
13  ID7  France    B  Europe    Male        yes          NaN  IA02       4       6   Q
14  UC8  France    B  Europe    Male        yes          NaN  IA01       8       2   4
15  UC8  France    B  Europe    Male        yes          NaN  IA02       4       2   P

Answer 2

你可以使用pd.lreshape

pd.lreshape(df.assign(IA01=['01']*len(df), IA02=['02']*len(df),IA09=['09']*len(df)), 
            {'IA': ['IA01', 'IA02','IA09'],
             'Raw': ['IA01_Raw','IA02_Raw','IA09_Raw'], 
             'Class1': ['IA01_Class1','IA02_Class1','IA09_Class1'], 
             'Class2': ['IA01_Class2', 'IA02_Class2','IA09_Class2']
             })


edit : 

pd.lreshape(df.assign(IA01=['01']*len(df), IA02=['02']*len(df),IA09=['09']*len(df)), 
            {'IA': ['IA01', 'IA02','IA09'],
             'Raw': ['IA01_Raw_baseline','IA02_Raw_midline','IA09_Raw_whatever'], 
             'Class1': ['IA01_Class1_baseline','IA02_Class1_midline','IA09_Class1_whatever'], 
             'Class2': ['IA01_Class2_baseline', 'IA02_Class2_midline','IA09_Class2_whatever']
             })

编辑：只需将输出的Raw/Class1/Class2 列中的输入中的column names 添加到字典内的列表中即可

这方面的文档不可用。使用help(pd.lreshape) 或参考here

输出：

    Country Gender  ID  QA_Comments QA_Include  Region  Type    IA  Raw Class1  Class2
0   France  Male    SC1 NaN         yes         Europe  A       01  4   8       1
1   France  Female  SC2 NaN         yes         Europe  A       01  2   7       2
2   France  Male    SC3 NaN         yes         Europe  B       01  3   7       2
3   France  Male    SC4 NaN         yes         Europe  A       01  4   8       2
4   France  Male    SC5 NaN         yes         Europe  B       01  1   7       1
5   France  Male    ID6 NaN         yes         Europe  A       01  2   8       1
6   France  Male    ID7 NaN         yes         Europe  B       01  2   8       1
7   France  Male    UC8 NaN         yes         Europe  B       01  4   8       2
8   France  Male    SC1 NaN         yes         Europe  A       02  J   4       1
9   France  Female  SC2 NaN         yes         Europe  A       02  Q   6       4
10  France  Male    SC3 NaN         yes         Europe  B       02  K   8       2
11  France  Male    SC4 NaN         yes         Europe  A       02  A   2       1
12  France  Male    SC5 NaN         yes         Europe  B       02  F   1       3
13  France  Male    ID6 NaN         yes         Europe  A       02  R   3       7
14  France  Male    ID7 NaN         yes         Europe  B       02  Q   4       6
15  France  Male    UC8 NaN         yes         Europe  B       02  P   4       2
16  France  Male    SC1 NaN         yes         Europe  A       09  W   6       3
17  France  Female  SC2 NaN         yes         Europe  A       09  X   5       2
18  France  Male    SC3 NaN         yes         Europe  B       09  Y   5       5
19  France  Male    SC4 NaN         yes         Europe  A       09  P   5       2
20  France  Male    SC5 NaN         yes         Europe  B       09  T   5       2
21  France  Male    ID6 NaN         yes         Europe  A       09  I   5       2
22  France  Male    ID7 NaN         yes         Europe  B       09  A   8       2
23  France  Male    UC8 NaN         yes         Europe  B       09  K   7       5