带条件的左外连接

Question

我想合并两个数据框（how=left），但不仅在索引上，而且在条件上。

例如假设两个数据框

      C1 C2 
  A = I  3  
      K  2  
      L  5

      C1 C2 C3
  B = I  5  T
      I  0  U
      K  1  X     
      L  7  Z

现在我想在 A.C2 > B.C2 的条件下使用索引 C1 将外部连接表 A 与 B 分开。也就是说，最终的结果应该是这样的

       A.C1  A.C2 B.C2 B.C3
A<-B = I        1    0    U
       K        2    1    X
       L        5    Null Null

P.S.：如果你想自己测试一下：

import pandas as pd

df_A = pd.DataFrame([], columns={'C 1', 'C2'})
df_A['C 1'] = ['I', 'K', 'L']
df_A['C2'] = [3, 2, 5]

df_B = pd.DataFrame([], columns={'C1', 'C2', 'C3'})
df_B['C1'] = ['I', 'I', 'K', 'L']
df_B['C2'] = [5, 0, 2, 7]
df_B['C3'] = ['T', 'U', 'X', 'Z']

Answer 1

快速而肮脏的解决方案是简单地加入 C1 列，然后为 C2_1 > C2_2 的所有行将 NULL 或 NaN 放入 C3。

Answer 2

方法：使用pandasql库直接SQL查询到pandas。 reference

import pandas as pd

df_A = pd.DataFrame([], columns={'C1', 'C2'})
df_A['C1'] = ['I', 'K']
df_A['C2'] = [3, 2]

df_B = pd.DataFrame([], columns={'C1', 'C2', 'C3'})
df_B['C1'] = ['I', 'I', 'K']
df_B['C2'] = [5, 0, 2]
df_B['C3'] = ['T', 'U', 'X']

在我看来，您为在 (A.C1 = B.C1) 上执行外连接指定的条件不会产生预期的结果。我需要做GROUP BY A.C1，以便在加入后删除 A.C1 中具有相同值的重复行。

import pandasql as ps

q = """
SELECT A.C1 as 'A.C1', 
       A.C2 as 'A.C2', 
       B.C2 as 'B.C2', 
       B.C3 as 'B.C3' 
    FROM df_A AS A    
    LEFT OUTER JOIN df_B AS B 
    --ON A.C1 = B.C1 AND A.C2 = B.C2
    WHERE A.C2 > B.C2
    GROUP BY A.C1
"""
print(ps.sqldf(q, locals()))

输出

  A.C1  A.C2  B.C2 B.C3
0    I     3     2    X
1    K     2     0    U

其他参考

1. https://www.zentut.com/sql-tutorial/sql-outer-join/
2. Executing an SQL query over a pandas dataset
3. How to do a conditional join in python Pandas?
4. https://github.com/pandas-dev/pandas/issues/7480
5. https://medium.com/jbennetcodes/how-to-rewrite-your-sql-queries-in-pandas-and-more-149d341fc53e

Answer 3

我找到了一个非熊猫原生解决方案：

import pandas as pd
from pandasql import sqldf
pysqldf = lambda q: sqldf(q, globals())

df_A = pd.DataFrame([], columns={'C1', 'C2'})
df_A['C1'] = ['I', 'K', 'L']
df_A['C2'] = [3, 2, 5]
cols = df_A.columns
cols = cols.map(lambda x: x.replace(' ', '_'))
df_A.columns = cols

df_B = pd.DataFrame([], columns={'C1', 'C2', 'C3'})
df_B['C1'] = ['I', 'I', 'K', 'L']
df_B['C2'] = [5, 0, 2, 7]
df_B['C3'] = ['T', 'U', 'X', 'Z']


# df_merge = pd.merge(left=df_A, right=df_B, how='left', on='C1')

df_sql = pysqldf("""
select *
from df_A t_1
left join df_B t_2 on t_1.C1 = t_2.C1 and t_1.C2 >= t_2.C2
;
""")

但是，对于大表，pandasql 的性能较差。

输出：

   C2 C1    C3   C2    C1
0   3  I     U  0.0     I
1   2  K     X  2.0     K
2   5  L  None  NaN  None