BigQuery BI Engine：如何选择合适的预留大小？答案

【问题标题】：BigQuery BI Engine: how to choose a good reservation size?BigQuery BI Engine：如何选择合适的预留大小？
【发布时间】：2022-12-09 20:53:38
【问题描述】：

我正在启用 BI Engine 来加快我的查询速度并为我在欧盟地区的项目节省成本。
设置预留大小的最佳选择是什么？ 1GB、2GB、4GB？
我如何做出这个决定？

【问题讨论】：

标签： google-cloud-platform google-bigquery google-bi-engine

【解决方案1】：

下面是一个 SQL 脚本，它按处理的 GB 数量对查询进行分组，因此第一行是每个查询处理的 0 到 1 GB，第二行是处理的 1 到 2 GB，等等。
然后对于每一行，它显示处理的金额、计费金额以及相关成本和节省的成本。

这应该可以帮助您了解您的成本所在、您有多少个特定规模的查询以及您是否可以增加或降低您的预留规模。

请注意，BI 引擎只能加速某些 SELECT QUERY 而不能加速 MERGE、INSERT、CREATE 等语句。还有更多的例外。因此，为了公平比较，我排除了这些类型的查询，以便更好地了解节省的规模。另见：https://cloud.google.com/bigquery/docs/bi-engine-intro#bi-engine-use-cases

DECLARE QUERY_COST_PER_TB NUMERIC DEFAULT 5.00; -- current cost in dollars of processing 1 TB of data in BQ

with possible_bi_engine_jobs_incl_parent_jobs as (
    select 
        creation_time, 
        bi_engine_statistics,
        cache_hit,
        total_bytes_processed / power(1024, 3) GB_processed,
        floor(total_bytes_processed / power(1024, 3)) GB_processed_floor,
        total_bytes_billed / power(1024, 3) GB_billed, 
        total_bytes_processed / power(1024, 4) * QUERY_COST_PER_TB expected_cost_in_euros,
        total_bytes_billed / power(1024, 4) * QUERY_COST_PER_TB actual_cost_in_euros,
        query,
        job_id,
        parent_job_id,
        user_email,
        job_type,
        statement_type,
    from `my_project_id.region-eu.INFORMATION_SCHEMA.JOBS`
    where 1=1
        and creation_time >= '2022-12-08'
        and creation_time <  '2022-12-09'
        and cache_hit = false  -- bi engine will not be improving on queries that are already cache hits
        and total_bytes_processed is not null  -- if there's no bytes processed, then ignore the job
        and statement_type = 'SELECT'  -- statement types such as MERGE, CREATE, UPDATE cannot be run by bi engine, only SELECT statements
        and job_type = 'QUERY'  -- LOAD jobs etc. cannot be run by bi engine, only QUERY jobs 
        and upper(query) like '%FROM%'  -- query should contain FROM, otherwise it will not be run by bi engine
        and upper(query) not like '%INFORMATION_SCHEMA.%'  -- metadata queries can not be run by bi engine
),
-- to prevent double counting of total_bytes_processed and total_bytes_billed
parent_job_ids_to_ignore as (
    select distinct parent_job_id
    from possible_bi_engine_jobs_incl_parent_jobs
    where parent_job_id is not null
),
possible_bi_engine_jobs_excl_parent_jobs as (
    select *
    from possible_bi_engine_jobs_incl_parent_jobs
    where job_id not in (select parent_job_id from parent_job_ids_to_ignore) -- to prevent double counting of total_bytes_processed and total_bytes_billed
)
select 
    GB_processed_floor, -- all queries which processed less GB than the floor value
    count(1) query_count, 
    sum(case when bi_engine_statistics.bi_engine_mode in ('FULL', 'PARTIAL') then 1 else 0 end) bi_engine_enabled,
    sum(case when bi_engine_statistics.bi_engine_mode in ('DISABLED') or bi_engine_statistics.bi_engine_mode IS NULL then 1 else 0 end) bi_engine_disabled,
    round(sum(GB_processed), 1) GB_processed, 
    round(sum(GB_billed), 1) GB_billed,
    round(sum(expected_cost_in_euros), 2) expected_cost_in_euros,
    round(sum(actual_cost_in_euros), 2) actual_cost_in_euros,
    round(sum(expected_cost_in_euros) - sum(actual_cost_in_euros), 2) saved_cost 
from possible_bi_engine_jobs_excl_parent_jobs
group by GB_processed_floor
order by GB_processed_floor
;

这将生成以下按查询大小分组的成本节省表：

有关 BI Engine 节省的其他有用链接：

【讨论】：