bigquery：依靠子查询的联合

Question

我有两个表（light 和 enhanced），它们的架构略有不同。第二个表 (enhanced) 有额外的可为空字段 t。 我想根据两个字段（d,p）从每个组的第一个和第二个表中获取行数

为了实现这一点，我使用带星号的子查询联合

select
  d
  , p
  ,count(1) - count(t) light_cnt
  ,count(t) enhanced_cnt
from 
  (select * from light)
  , (select * from enhanced)
group by
  d, p

但是这个查询返回错误的计数（大约是两倍） 只有当我按两个字段分组时才会发生这种情况。单场效果很好。我发现当我将联合包装在另一个子查询中时它可以正常工作

select
  d
  , p
  ,count(1) - count(t) light_cnt
  ,count(t) enhanced_cnt
from
  (select * from 
      (select * from light)
      , (select * from enhanced)
  )
group by
  d, p

这是一个错误还是我做错了什么？

---

编辑：

我在没有group by 的情况下重现了相同的损坏行为，只是使用了where

select count(1) from enhanced where p = 124

返回 292

select count(1) from light where p = 124

返回 12512

select count(1)
from (select * from light), (select * from enhanced)
where p = 124

返回正确的 12804，而

select count(1), count(t)
from (select * from light), (select * from enhanced)
where p = 124

返回 24527, 501... 很奇怪。似乎是一个错误。

解决方法：

select count(1), count(t)
from (select * from (select * from light), (select * from enhanced))
where p = 124

返回 12804、292。正确。

light 和 enhanced 两个表都具有从 avro 继承的复杂模式。有记录和重复字段。为简单起见，字段p 和t 在上面的选择中采用缩写形式。实数是p -> record.record.record.id（叶子是整数）和t -> record.time（叶子是整数）。路径 p 和 t 中的记录均不可重复。都可以为空。

Answer 1

请参阅Difference in statistics from Google Analytics Report and BigQuery Data in Hive table，您可能会遇到类似的问题，因为您正在展平来自 2 个重复列的数据：这会产生 n*m 个结果。

证明这一点的查询：

SELECT col, x FROM (
  SELECT "wrong" col, SUM(totals.pageviews) x
  FROM (FLATTEN ([google.com:analytics-bigquery:LondonCycleHelmet.ga_sessions_20130910], hits))
), (
  SELECT "correct" col, SUM(totals.pageviews) x
  FROM [google.com:analytics-bigquery:LondonCycleHelmet.ga_sessions_20130910]
)

col     x    
wrong   2262     
correct 249 

准确显示您正在经历的问题的查询。所有结果数字都应该相同，但它们不同 - 结果取决于 BigQuery 做出的 FLATTEN() 选择。如果您想获得正确的数字，请在计数之前明确说明 BigQuery 应如何展平表格：

SELECT a, b, c    FROM (
SELECT COUNT(1)  a
FROM (SELECT * FROM [google.com:analytics-bigquery:LondonCycleHelmet.ga_sessions_20130910] ), (SELECT * FROM [google.com:analytics-bigquery:LondonCycleHelmet.ga_sessions_20130910] )
),(
SELECT COUNT(customDimensions.index) b
FROM (SELECT * FROM [google.com:analytics-bigquery:LondonCycleHelmet.ga_sessions_20130910] ), (SELECT * FROM [google.com:analytics-bigquery:LondonCycleHelmet.ga_sessions_20130910] )
),(
SELECT COUNT(hits.hitNumber) c
FROM (SELECT * FROM [google.com:analytics-bigquery:LondonCycleHelmet.ga_sessions_20130910] ), (SELECT * FROM [google.com:analytics-bigquery:LondonCycleHelmet.ga_sessions_20130910] )
),(
SELECT COUNT(1)  a, COUNT(customDimensions.index) b, COUNT(hits.hitNumber) c
FROM (SELECT * FROM [google.com:analytics-bigquery:LondonCycleHelmet.ga_sessions_20130910] ), (SELECT * FROM [google.com:analytics-bigquery:LondonCycleHelmet.ga_sessions_20130910] )
),(
SELECT COUNT(1)  a, COUNT(customDimensions.index) b
FROM (SELECT * FROM [google.com:analytics-bigquery:LondonCycleHelmet.ga_sessions_20130910] ), (SELECT * FROM [google.com:analytics-bigquery:LondonCycleHelmet.ga_sessions_20130910] )
),(
SELECT COUNT(1)  a, COUNT(hits.hitNumber) c
FROM (SELECT * FROM [google.com:analytics-bigquery:LondonCycleHelmet.ga_sessions_20130910] ), (SELECT * FROM [google.com:analytics-bigquery:LondonCycleHelmet.ga_sessions_20130910] )
),(
SELECT COUNT(customDimensions.index) b, COUNT(hits.hitNumber) c
FROM (SELECT * FROM [google.com:analytics-bigquery:LondonCycleHelmet.ga_sessions_20130910] ), (SELECT * FROM [google.com:analytics-bigquery:LondonCycleHelmet.ga_sessions_20130910] )
),(
SELECT COUNT(1) a, COUNT(customDimensions.index) b, COUNT(hits.hitNumber) c
FROM [google.com:analytics-bigquery:LondonCycleHelmet.ga_sessions_20130910], [google.com:analytics-bigquery:LondonCycleHelmet.ga_sessions_20130910])

Row a   b   c    
1   126          
2       510      
3           724  
4   126 102 766  
5   726 510      
6   126     724  
7       102 766  
8   126 102 724