如何合并具有不同列索引级别的两个 Pandas 数据框？

Question

我想连接两个具有相同索引但列级别不同的数据帧。一个数据框有层次索引，另一个没有。

print df1

              A_1               A_2               A_3                .....
              Value_V  Value_y  Value_V  Value_y  Value_V  Value_y

instance200   50       0        6500     1        50       0
instance201   100      0        6400     1        50       0

另一个：

print df2

              PV         Estimate

instance200   2002313    1231233
instance201   2134124    1124724

结果应该是这样的：

             PV        Estimate   A_1               A_2               A_3                .....
                                  Value_V  Value_y  Value_V  Value_y  Value_V  Value_y

instance200  2002313   1231233    50       0        6500     1        50       0
instance201  2134124   1124724    100      0        6400     1        50       0

但是帧上的合并或连接会给我一个带有一维列索引的df：

             PV        Estimate   (A_1,Value_V) (A_1,Value_y) (A_2,Value_V) (A_2,Value_y)  .....


instance200  2002313   1231233    50             0             6500         1
instance201  2134124   1124724    100            0             6400         1 

如何保持 df1 的层次索引？

Answer 1

也许使用好的 ole 赋值：

df3 = df1.copy()
df3[df2.columns] = df2

产量

                A_1             A_2             A_3               PV Estimate
            Value_V Value_y Value_V Value_y Value_V Value_y                  
instance200      50       0    6500       1      50       0  2002313  1231233
instance201     100       0    6400       1      50       0  2134124  1124724

Answer 2

您可以通过使 df2 与 df1 具有相同数量的级别来做到这一点：

In [11]: df1
Out[11]:
                A_1             A_2             A_3
            Value_V Value_y Value_V Value_y Value_V Value_y
instance200      50       0    6500       1      50       0
instance201     100       0    6400       1      50       0

In [12]: df2
Out[12]:
                  PV  Estimate
instance200  2002313   1231233
instance201  2134124   1124724

In [13]: df2.columns = pd.MultiIndex.from_arrays([df2.columns, [None] * len(df2.columns)])

In [14]: df2
Out[14]:
                  PV Estimate
                 NaN      NaN
instance200  2002313  1231233
instance201  2134124  1124724

现在您可以在不破坏列名的情况下进行连接：

In [15]: pd.concat([df1, df2], axis=1)
Out[15]:
                A_1             A_2             A_3               PV Estimate
            Value_V Value_y Value_V Value_y Value_V Value_y      NaN      NaN
instance200      50       0    6500       1      50       0  2002313  1231233
instance201     100       0    6400       1      50       0  2134124  1124724

注意：首先要让 df2 列使用 pd.concat([df2, df1], axis=1)。

---

也就是说，我不确定我是否能想到一个用例，将它们作为单独的 DataFrame 保存实际上可能是一个更简单的解决方案......！

Answer 3

我为pandas.concat 函数制作了一个包装器，它接受具有不相等数量级别的数据帧。

从下方添加空级别。优点是它允许使用df_cols.c 访问系列（在下面的df_cols 中），并且在打印时清楚地表明'c' 不是('CC', 'one') 的子级别。

def concat(dfs, axis=0, *args, **kwargs):   
    """
    Wrapper for `pandas.concat'; concatenate pandas objects even if they have 
    unequal number of levels on concatenation axis.
    
    Levels containing empty strings are added from below (when concatenating along
    columns) or right (when concateniting along rows) to match the maximum number 
    found in the dataframes.
    
    Parameters
    ----------
    dfs : Iterable
        Dataframes that must be concatenated.
    axis : int, optional
        Axis along which concatenation must take place. The default is 0.

    Returns
    -------
    pd.DataFrame
        Concatenated Dataframe.
    
    Notes
    -----
    Any arguments and kwarguments are passed onto the `pandas.concat` function.
    
    See also
    --------
    pandas.concat
    """
    def index(df):
        return df.columns if axis==1 else df.index
    
    def add_levels(df):
        need = want - index(df).nlevels
        if need > 0:
            df = pd.concat([df], keys=[('',)*need], axis=axis) # prepend empty levels
            for i in range(want-need): # move empty levels to bottom
                df = df.swaplevel(i, i+need, axis=axis) 
        return df
    
    want = np.max([index(df).nlevels for df in dfs])    
    dfs = [add_levels(df) for df in dfs]
    return pd.concat(dfs, axis=axis, *args, **kwargs)

希望这对某人有所帮助。

测试：

df1

   AA      BB      CC    
  one     one     one    
    a   b   a   b   a   b
0  91  63   2  59  26  93
1  34   4  73  55  16  66
2   2   6   9  15  51  95

df2

    c   d   e
0  68  49  69
1  35  53  71
2  68  75  54


df3

       c   d   e
i  x  27  83  53
   y  54  51   9
   z  41   1  24
ii x  44  76  54
   y  76  85  21
   z  83  82   6


df_cols = concat([df1, df2], axis=1)

df_cols

   AA      BB      CC       c   d   e
  one     one     one                
    a   b   a   b   a   b            
0  91  63   2  59  26  93  68  49  69
1  34   4  73  55  16  66  35  53  71
2   2   6   9  15  51  95  68  75  54


df_rows = concat([df2, df3])

df_rows

       c   d   e
0     68  49  69
1     35  53  71
2     68  75  54
i  x  27  83  53
   y  54  51   9
   z  41   1  24
ii x  44  76  54
   y  76  85  21
   z  83  82   6

Answer 4

更新（20201 年 1 月） 为此我构建了一个函数如下：

def concat( df1, df2 ):

  """
  Function concatenates two dataframes df1 snd df2 even if the two datafames
  have different number of hierarchical columns levels.

  In the case of one dataframe having more hierarchical columns levels than the
  other, blank string will be added to the upper hierarchical columns levels
  """

  nLevels1 = df1.columns.nlevels
  nLevels2 = df2.columns.nlevels
  diff     = nLevels2 - nLevels1

  mLevels  = max(nLevels1, nLevels2)

  if nLevels1 == nLevels2:
    # if the same simply concat as normal
    return pd.concat( [df1, df2 ], axis = 1 )

  elif nLevels1 < nLevels2:
    # if there is a difference expand smaller dataframe with black strings, then concat

    df_temp = df1.copy()
    new_cols  = [[""] * len( df1.columns )] * np.abs(diff)

    new_cols = join_lists( df1.columns, new_cols)
    df_temp.columns = new_cols

    concatonated = pd.concat( [df_temp, df2 ], axis = 1 )
    return concatonated

  elif nLevels1 > nLevels2:
    # same as above but for the other way around

    df_temp = df2.copy()


    new_cols = [[""] * len( df2.columns )] * np.abs(diff)
    new_cols = join_lists( df2.columns, new_cols)

    new_cols.append( df2.columns.to_list() )

    df_temp.columns = new_cols

    concatonated = pd.concat( [df1, df_temp ], axis = 1)

    return concatonated

现在如果我们提供数据框

gender  f  m
            
n       2  1
y       2  2

和

gender        f                         m             
age         old        young          old        young
location london paris london paris london paris london
                                                      
n             1     0      1     0      0     1      0
y             0     1      0     1      1     0      1

我们得到

             f                         m                   
            old        young          old        young      
         london paris london paris london paris london  f  m
                                                            
n             1     0      1     0      0     1      0  2  1
y             0     1      0     1      1     0      1  2  2

请注意，将来加入性别类别可能会很好，这样它们就处于同一级别，但这主要是为了加入具有完全不同列的数据框。