雪花中的合并查询给出了不正确的数据计数

Question

我正在下载两个文件并将它们添加到临时表的暂存区，然后将它们合并到主表。 但我的日期计数不正确：

    def load_snowflake_table(self, directory):
        try:
            self.snowflake_client.run("ALTER SESSION SET TIMEZONE = 'UTC';")
            self.stage_name = f"{self.sf_temp_table}_{self.snowflake_client.generate_random_string()}"
            create_stage = f"CREATE TEMPORARY STAGE {self.stage_name} COMMENT = 'TEMPORARY STAGE FOR {self.sf_temp_table} DATA LOAD'"
            self.snowflake_client.run(create_stage)
            logger.info("Temporary stage created.")

            self.snowflake_client.run(f"put file://{directory}/* @{self.stage_name} PARALLEL=4")
            logger.info(f"Successfully uploaded all files to staging area.")

            self.snowflake_client.run(
                f"COPY INTO {self.sf_temp_table}  FROM @{self.stage_name} PURGE = TRUE MATCH_BY_COLUMN_NAME = "
                f"'CASE_INSENSITIVE'  FILE_FORMAT "
                f"= (TYPE = 'AVRO')")
            logger.info(f"Successfully copied the files data temporary table {self.sf_temp_table}")

            sf_query = f"SELECT COUNT(*) from {self.sf_temp_table}"
            logger.info(sf_query)
            sf_count = self.snowflake_client.run(sf_query).fetchall()
            sf_count = sf_count[0][0]
            print("=====================")
            print(sf_count)

            merge_query = self.form_merge_query(self.COLUMNS)

            logger.info(f"Executing merge query: {merge_query}")
            self.snowflake_client.run(merge_query)

            logger.info("Truncating temporary table")
            self.snowflake_client.run(f"TRUNCATE TABLE IF EXISTS {self.sf_temp_table}")

        except Exception as e:
            logger.error("Error Loading to Snowflake{e}".format(e=e))
            raise e
        finally:
            self.snowflake_client.run(f"DROP STAGE IF EXISTS {self.stage_name}")
            logger.info(f"Dropped temporary stage {self.stage_name}")

第一个文件中的计数为 135839，临时表中的计数相同，第二个文件中的计数为 135687，临时表中的计数相同。因此，决赛桌的人数应该是 271,526，但事实证明这是不正确的。

所有记录在主键组合上都是唯一的。

我的合并查询是：

MERGE
INTO
MERGETEST
USING 
(SELECT $1 CHANNELGROUPING, $2 CLIENTID, $3 CUSTOMDIMENSIONS, $4 DATE, $5 DEVICE, $6 FULLVISITORID,
$7 GEONETWORK, $8 SOCIALENGAGEMENTTYPE, $9 TOTALS,
$10 TRAFFICSOURCE, $11 VISITID, $12 VISITNUMBER, $13 VISITSTARTTIME, $14 APPINFO, $15 CONTENTGROUP,      
$16 HITS_CUSTOMDIMENSIONS, $17 CUSTOMMETRICS,$18 CUSTOMVARIABLES, $19 DATASOURCE, $20 ECOMMERCEACTION,
$21 EVENTINFO, $22 EXCEPTIONINFO, $23 EXPERIMENT, $24 HITNUMBER
FROM T_SESSIONS)
as TEST_SESSIONS
ON MERGETEST.FULLVISITORID = TEST_SESSIONS.FULLVISITORID and MERGETEST.VISITID = TEST_SESSIONS.VISITID and
MERGETEST.VISITSTARTTIME = TEST_SESSIONS.VISITSTARTTIME and MERGETEST.HITNUMBER = TEST_SESSIONS.HITNUMBER
WHEN
NOT
MATCHED and 
THEN
INSERT(CHANNELGROUPING,CLIENTID,CUSTOMDIMENSIONS,DATE,DEVICE,FULLVISITORID,GEONETWORK,SOCIALENGAGEMENTTYPE,TOTALS,
       TRAFFICSOURCE,VISITID,VISITNUMBER,VISITSTARTTIME,APPINFO,CONTENTGROUP,HITS_CUSTOMDIMENSIONS,CUSTOMMETRICS,
       CUSTOMVARIABLES,DATASOURCE,ECOMMERCEACTION,EVENTINFO,EXCEPTIONINFO,EXPERIMENT,HITNUMBER)
VALUES
(TEST_SESSIONS.CHANNELGROUPING,TEST_SESSIONS.CLIENTID,TEST_SESSIONS.CUSTOMDIMENSIONS,TEST_SESSIONS.DATE,
TEST_SESSIONS.DEVICE,TEST_SESSIONS.FULLVISITORID,TEST_SESSIONS.GEONETWORK,TEST_SESSIONS.SOCIALENGAGEMENTTYPE,
TEST_SESSIONS.TOTALS,TEST_SESSIONS.TRAFFICSOURCE,TEST_SESSIONS.VISITID,TEST_SESSIONS.VISITNUMBER,
TEST_SESSIONS.VISITSTARTTIME,TEST_SESSIONS.APPINFO,TEST_SESSIONS.CONTENTGROUP,TEST_SESSIONS.HITS_CUSTOMDIMENSIONS,
TEST_SESSIONS.CUSTOMMETRICS,TEST_SESSIONS.CUSTOMVARIABLES,TEST_SESSIONS.DATASOURCE,TEST_SESSIONS.ECOMMERCEACTION,
TEST_SESSIONS.EVENTINFO,TEST_SESSIONS.EXCEPTIONINFO,TEST_SESSIONS.EXPERIMENT,TEST_SESSIONS.HITNUMBER);

我所有的行在 FULLVISITORID、VISITID、VISITSTARTTIME 和 HITNUMBER 组合上都是唯一的。但我仍然没有得到正确的行数。

直到临时表我得到了预期的结果。方法或我的合并查询有问题吗？

Answer 1

文件/表一：135,839 文件/表二：135,687 预期：135,839 + 135,687 = 271,526

当他们在两个表之间共享相同的键时，他们的合并版本必须是总和的假设可能不是这样。

示例场景（两行 + 一行）

(FULLVISITORID, VISITID, VISITSTARTTIME, HITNUMBER, ...)
1, 1, 2021-01-01, 1, ...
2, 2, 2021-01-01, 2, ...

其他表：

(FULLVISITORID, VISITID, VISITSTARTTIME, HITNUMBER, ...)
1, 1, 2021-01-01, 1, ...

在执行 INSERT 的合并后，仅输出为 2 行而不是 3 行。

(FULLVISITORID, VISITID, VISITSTARTTIME, HITNUMBER, ...)
1, 1, 2021-01-01, 1, ...
2, 2, 2021-01-01, 2, ...

---

要找到重叠可以使用INTERSECT：

SELECT $6 FULLVISITORID, $11 VISITID, $13 VISITSTARTTIME, $24 HITNUMBER 
FROM T_SESSIONS
INTERSECT
SELECT FULLVISITORID, VISITID, VISITSTARTTIME, HITNUMBER 
FROM MERGETEST;