从表中提取 metadata_fields 以及从 bigquery 表中提取数据列答案

【问题标题】：Extract metadata_fields from table as well as data columns from bigquery table从表中提取 metadata_fields 以及从 bigquery 表中提取数据列
【发布时间】：2019-09-20 10:19:22
【问题描述】：

我必须查询一个 bigquery 表，其中要求获取以下字段：

a) 表元数据列（2 列）

b) 数据列（3 列）

例如：

table_id ,   creation_time , col_id1 , col2_id2, col3_id3
tbl_20180424, 1524641477022,  1, 2, 3
tbl_20180524, 1524647897022,  11, 12, 13

我尝试了以下查询，但它不起作用：

SELECT
table_id,
creation_time,
year,
Week,
id1,
id2,
id3
FROM (
SELECT  
table_id,
TIMESTAMP_MILLIS(creation_time) AS creation_time,
EXTRACT(YEAR FROM CAST(TIMESTAMP_MILLIS(creation_time) as DATE)) Year,
EXTRACT(WEEK FROM CAST(TIMESTAMP_MILLIS(creation_time) as DATE)) Week
FROM `project.dataset.__TABLES_SUMMARY__`
where table_id like 'tbl_%' )

【问题讨论】：

标签： google-bigquery

【解决方案1】：

以下是 BigQuery 标准 SQL

#standardSQL
WITH data AS (
  SELECT CONCAT('tbl_', _TABLE_SUFFIX) AS table_id, col_id1, col_id2, col_id3
  FROM `project.dataset.tbl_*`
), metadata AS (
  SELECT 
    table_id,
    TIMESTAMP_MILLIS(creation_time) AS creation_time,
    EXTRACT(YEAR FROM CAST(TIMESTAMP_MILLIS(creation_time) AS DATE)) Year,
    EXTRACT(WEEK FROM CAST(TIMESTAMP_MILLIS(creation_time) AS DATE)) Week
  FROM `project.dataset.__TABLES_SUMMARY__`
  WHERE table_id LIKE 'tbl_%'
)
SELECT *
FROM metadata
JOIN data
USING(table_id)

如果适用于您问题中的简化示例 - 结果将是

Row table_id        creation_time               Year    Week    col_id1 col_id2 col_id3  
1   tbl_20180424    2019-09-20 19:21:38.600 UTC 2019    37      1       2       3    
2   tbl_20180524    2019-09-20 19:22:00.676 UTC 2019    37      11      12      13

【讨论】：

我已经以不同的格式编写了上述连接查询，但它给我带来了性能问题，因为它在 3 分钟内运行，而您的查询在 23 秒内运行。你能帮我理解我的查询出现性能问题的原因吗？
如果没有看到您的查询，我该怎么做？ :o)
select year , Week, id1 , COUNT(DISTINCT id2) as col4 from (SELECT table_id, creation_time, year, Week, id1, id2, id3 FROM project.dataset.<tables>_* , ( SELECT table_id, TIMESTAMP_MILLIS(creation_time ) AS creation_time, EXTRACT(YEAR FROM CAST(TIMESTAMP_MILLIS(creation_time) as DATE)) Year, EXTRACT(WEEK FROM CAST(TIMESTAMP_MILLIS(creation_time) as DATE)) Week FROM project.dataset.__TABLES_SUMMARY__ where table_id like 'tbl_%' )) where col1 = 'some value' GROUP by year , week , id1 ORDER by col4 DESC)
@Mikhail 你能解释一下这两个查询的区别吗
哪两个查询？我的答案中只看到一个查询

【解决方案2】：

在您的查询中，您尝试选择 id1、id2、id3，但它们不在您的子查询中。此外，请考虑查看INFORMATION_SCHEMA，如下所示：

with tables as (select table_name, creation_time from dataset.INFORMATION_SCHEMA.TABLES),
    columns as (select table_name, column_name, ordinal_position from dataset.INFORMATION_SCHEMA.COLUMNS)
select
  table_name,
  creation_time,
  EXTRACT(YEAR from creation_time) as Year,
  EXTRACT(WEEK from creation_time) as Week,
  column_name,
  ordinal_position
from tables
inner join columns using(table_name)
order by table_name, ordinal_position

此查询提供数据集中所有表/列的未透视结果集。

【讨论】：