具有分组+时间戳列的最新数据的物化视图

Question

我在 Bigquery 中对特征（或属性）进行建模。这是模型的示例

uid         string, link to owner id
uuid        string, unique among all rows
trait_name  string, name of the trait
trait_value string
added_at    timestamp, when the trait was added

我正在尝试构建一个物化视图，该视图包含每个 uid 的每个特征的最新特征。我可以通过这个查询得到结果：

WITH traits AS (
  SELECT 'u1' uid, 'uu1' uuid, 't1' trait_name, 't1v1' trait_value, timestamp("2021-10-01 10:00:00") as added_at UNION ALL
  SELECT 'u1' uid, 'uu2' uuid, 't1' trait_name, 't1v2' trait_value, timestamp("2021-10-02 10:00:00") as added_at UNION ALL
  SELECT 'u1' uid, 'uu3' uuid, 't2' trait_name, 't2v1' trait_value, timestamp("2021-10-03 10:00:00") as added_at UNION ALL
  SELECT 'u1' uid, 'uu4' uuid, 't2' trait_name, 't2v2' trait_value, timestamp("2021-10-04 10:00:00") as added_at UNION ALL
  SELECT 'u2' uid, 'uu5' uuid, 't1' trait_name, 't1v1' trait_value, timestamp("2021-10-05 10:00:00") as added_at UNION ALL
  SELECT 'u2' uid, 'uu6' uuid, 't1' trait_name, 't1v2' trait_value, timestamp("2021-10-06 10:00:00") as added_at UNION ALL
  SELECT 'u2' uid, 'uu7' uuid, 't2' trait_name, 't2v1' trait_value, timestamp("2021-10-07 10:00:00") as added_at UNION ALL
  SELECT 'u2' uid, 'uu8' uuid, 't2' trait_name, 't2v2' trait_value, timestamp("2021-10-08 10:00:00") as added_at
)
SELECT * FROM (
  SELECT *,
  MAX(added_at) OVER (PARTITION BY uid, trait_name) as latest_added_at FROM traits
) WHERE latest_added_at = added_at

#  Row  uid uuid    trait_name  trait_value added_at                 latest_added_at    
#  1    u1  uu2     t1          t1v2        2021-10-02 10:00:00 UTC  2021-10-02 10:00:00 UTC
#  2    u1  uu4     t2          t2v2        2021-10-04 10:00:00 UTC  2021-10-04 10:00:00 UTC
#  3    u2  uu6     t1          t1v2        2021-10-06 10:00:00 UTC  2021-10-06 10:00:00 UTC
#  4    u2  uu8     t2          t2v2        2021-10-08 10:00:00 UTC  2021-10-08 10:00:00 UTC

但我不能将它用于物化视图，因为它们不支持它：

物化视图不支持分析函数或 WITH OFFSET。

我也尝试过使用连接

SELECT * FROM traits t
JOIN (
    SELECT
    uid,
    trait_name,
    MAX(added_at) AS max_added_at 
    FROM traits GROUP BY uid, trait_name
) grouped
ON t.uid = grouped.uid 
AND t.trait_name = grouped.trait_name
AND t.added_at = grouped.max_added_at

但它们也不支持

物化视图查询不能多次引用同一个表。多次看到表格特征。

有没有办法将其作为物化视图来实现？

Answer 1

物化视图有一些限制，可能无法实现（有关更多信息，请参阅this 公共文档）。

因此，目前看来您想要实现的目标并未得到完全支持，并且使用 ARRAY_AGG 似乎是最接近的，缺点之一是结果是数组类型，而不是值本身。

我设法用这个查询检索了预期的结果：

CREATE MATERIALIZED VIEW test_traits.mv_sample_traits as
SELECT uid,
       ARRAY_AGG(uuid IGNORE NULLS ORDER BY added_at DESC, uuid DESC, trait_value DESC LIMIT 1) as uuid,
       trait_name,
       ARRAY_AGG(trait_value IGNORE NULLS ORDER BY added_at DESC, uuid DESC, trait_value DESC LIMIT 1) as trait_value,
       ARRAY_AGG(added_at IGNORE NULLS ORDER BY added_at DESC, uuid DESC, trait_value DESC LIMIT 1) as added_at,
       MAX(added_at) AS latest_added_at,
FROM test_traits.traits
GROUP BY uid,trait_name


/*
SELECT * FROM test_traits.mv_sample_traits;
*/

请注意，代码仅供参考，您需要运行测试以验证它是否适合您的用例，不能保证这会按预期工作，因此我不推荐将其用于生产。

Answer 2

我按照谷歌文档https://cloud.google.com/bigquery/docs/materialized-views 为您的要求创建了物化视图。

我将数据插入到如下表格中

我运行下面的查询来创建一个物化视图

CREATE MATERIALIZED VIEW  `graphical-reach-285218.test.traits1`
as select max(added_at) as latest_added_at,uid,trait_name from `graphical-reach-285218.test.traits`
group by uid, trait_name;

物化视图创建如下

在查询物化视图时，以下是我得到的数据