SQL到等效的熊猫 - 合并列为空的列答案

【问题标题】：SQL to equivalent pandas - Merge on columns where column is nullSQL到等效的熊猫 - 合并列为空的列
【发布时间】：2022-08-17 02:30:55
【问题描述】：

我提出了这个新问题，因为我不确定用户的请求和措辞是否匹配：pandas left join where right is null on multiple columns

这个 SQL 的等效 pandas 代码是什么？在上下文中，我们正在从 table_y 中的列中查找相对于几列不在 table_x 中的条目。

SELECT
   table_x.column,
   table_x.column2,
   table_x.column3,
   table_y.column,
   table_y.column2,
   table_y.column3,
FROM table_x
LEFT JOIN table_y
   ON table_x.column = table_y.column
   ON table_x.column2 = table_y.column2
WHERE
   table_y.column2 is NULL

是这个吗？

columns_join = [\'column\', \'column2\']
data_y = data_y.set_index(columns_join)
data_x = data_x.set_index(columns_join)

data_diff = pandas.concat([data_x, data_y]).drop_duplicates(keep=False) # any row not in both

# Select the diff representative from each dataset - in case datasets are too large
x1 = data_x[data_x.index.isin(data_diff.index)]
x2 = data_y[data_y.index.isin(data_diff.index)]

# Perform an outer join with the joined indices from each set,
# then remove the entries only contributed from table_x 
data_compare = x1.merge(x2, how = \'outer\', indicator=True, left_index=True, right_index=True)

data_compare_final = (
    data_compare
    .query(\'_merge == left_join\')
    .drop(\'_merge\', axis=1)
)

我不认为这是等效的，因为我们仅从 table_x 中删除了不在基于多列的连接中的条目。我认为我们必须继续将该列与 table_y 进行比较。

data_compare = data_compare.reset_index().set_index(\'column2\')
data_y = data_y.reset_index().set_index(\'column2\')
mask_column2 = data_y.index.isin(data_compare.index)
result = data_y[~mask_column2]

请提供具有预期输出的示例数据框
这有一些奇怪的边缘情况。今天下午晚些时候我可以回答我自己的问题。

标签： mysql pandas

【解决方案1】：

如果没有测试数据，很难确定这会有所帮助，但您可以尝试：

# Only if columns to join on in the right dataframe have the same name as columns in left
table_y[['col_join_1', 'col_join_2']] = table_y[['column', 'column2']] # Else this is not needed

# Merge left (LEFT JOIN)
table_merged = table_x.merge(
    table_y,
    how='left',
    left_on=['column', 'column2'],
    right_on=['col_join_1', 'col_join_2'],
    suffixes=['_x', '_y']
)

# Filter dataframe
table_merged = table_merged.loc[
    table_merged.column2_y.isna(),
    ['column_x', 'column2_x', 'column3_x', 'column_y', 'column2_y', 'column3_y']
]

【讨论】：

【解决方案2】：

我找到了相当于将索引设置为连接列、联合表、删除重复项以及在联合的贡献之间执行交叉连接的等价物。从那里，可以选择

left_only 对于这个等效的 SQL

SELECT
   table_x.*,
   table_y.*
FROM table_x
LEFT JOIN table_y
   ON table_x.column = table_y.column
   ON table_x.column2 = table_y.column2
WHERE
   table_y.column2 is NULL

right_only 对于这个等效的 SQL

SELECT
   table_x.*,
   table_y.*
FROM table_y
LEFT JOIN table_x
   ON table_y.column = table_x.column
   ON table_y.column2 = table_x.column2
WHERE
   table_x.column2 is NULL

def create_dataframe_joined_diffs(dataframe_prod, dataframe_new, columns_join):
    """
    Set the indices to the columns_key
    Concat the dataframes and remove duplicates
    Select the diff representative from each dataset
    Reset the indices and perform an outer join

    Pseudo-SQL:

    SELECT
        UNIQUE(*)
    FROM dataframe_prod
    OUTER JOIN dataframe_new
        ON columns_join
    """
    data_new = dataframe_new.set_index(columns_join)
    data_prod = dataframe_prod.set_index(columns_join)

    # Get any row not in both (may be removing too many)
    data_diff = pandas.concat([data_prod, data_new]).drop_duplicates(keep=False) # any row not in both
    # Select the diff representative from each dataset
    x1 = data_prod[data_prod.index.isin(data_diff.index)]
    x2 = data_new[data_new.index.isin(data_diff.index)]

    # Perform an outer join and keep the joined indices from each set
    # Sort the columns to make them easier to compare
    data_compare = x1.merge(x2, how = 'outer', indicator=True, left_index=True, right_index=True).sort_index(axis=1)

    return data_compare

mask_left = dataframe_compare['_merge'] == 'left_only'
mask_right = dataframe_compare['_merge'] == 'right_only'

【讨论】：