大查询 STRING_AGG() 多个字段产生错误

Question

我的查询如下所示

 SELECT
    DISTINCT id,
    STRING_AGG(DISTINCT column1) AS mobile,
    STRING_AGG(DISTINCT country) AS country,
    STRING_AGG(DISTINCT language) AS language,
    STRING_AGG(DISTINCT address) AS address,
    STRING_AGG(DISTINCT model) AS model,
    STRING_AGG(DISTINCT car) AS car,
    STRING_AGG(DISTINCT class) AS class,
    home_email,
    buisness_email,
    MAX(timestamp) AS timestamp
  FROM
    user
  GROUP BY
    id,
    home_email,
    buisness_email

当我在 bigquery 中对我的 2TB 表运行此查询并设置查询设置以将输出导出到表时会产生错误 注意：当我在 500 GB 数据大小上运行它时，它工作正常

查询执行期间超出资源：您的项目或组织超出了可用于 shuffle 操作的最大磁盘和内存限制。考虑在此作业中配置更多槽、减少查询并发或使用更高效的逻辑。

请问我该如何解决呢？

此外，在此之后，我需要在此表联合上与另一个 1TB 表运行相同的查询

Answer 1

我可以执行一个命题（但我不知道它是否有效，我没有足够大的数据集。所以，如果没有，我将删除这个“answer-try”）

这个想法是将 agg 拆分为不同的子查询。然后合并所有这些子部分。

WITH agg_mobile AS (
 SELECT
    DISTINCT id,
    STRING_AGG(DISTINCT column1) AS mobile,
    home_email,
    buisness_email,
    MAX(timestamp) AS timestamp
  FROM
    user
  GROUP BY
    id,
    home_email,
    buisness_email
),
agg_country AS (
 SELECT
    DISTINCT id,
    STRING_AGG(DISTINCT country) AS country,
  FROM
    user
  GROUP BY
    id,
    home_email,
    buisness_email
)
....
SELECT agg_mobile.*,
       agg_country.country, 
       ....
FROM agg_mobile 
  LEFT JOIN agg_country ON agg_mobile.id = agg_country.id 
  LEFT JOIN .....