Bigquery 分区日期 Python答案

【问题标题】：Bigquery Partition Date PythonBigquery 分区日期 Python
【发布时间】：2020-04-01 18:11:45
【问题描述】：

我想在 bigquery 中按分区日期写另一个表。但我找不到怎么做。我使用 Python 和谷歌云库。我想使用标准 SQL 创建一个表。但是我得到一个错误。

错误：google.api_core.exceptions.BadRequest: 400 GET https://bigquery.googleapis.com/bigquery/v2/projects/astute-baton-272707/queries/f4b9dadb-1390-4260-bb0e-fb525aff662c?maxResults=0&location=US: 列定义列表中的列数与 [2:72] 时查询生成的列数不匹配

如果有其他解决方案，请告诉我。每天插入到项目的下一阶段。

我可能从一开始就做错了。我不确定。

谢谢。

client = bigquery.Client()

sql = """
CREATE OR REPLACE TABLE zzzzz.xxxxx.yyyyy (visitStartTime_ts INT64,date TIMESTAMP,hitsTime_ts INT64,appId STRING,fullVisitorId STRING,cUserId STRING,eventCategory STRING,eventLabel STRING,player_type STRING,PLAY_SESSION_ID STRING,CHANNEL_ID STRING,CONTENT_EPG_ID STRING,OFF_SET STRING)
 PARTITION BY date 
 OPTIONS (
   description="weather stations with precipitation, partitioned by day"
 ) AS 
select  
FORMAT_TIMESTAMP("%Y-%m-%d %H:%M:%S", TIMESTAMP_SECONDS(SAFE_CAST(visitStartTime AS INT64)), "Turkey") AS visitStartTime_ts,
date
,FORMAT_TIMESTAMP("%Y-%m-%d %H:%M:%S", TIMESTAMP_SECONDS(SAFE_CAST(visitStartTime+(h.time/1000) AS INT64)), "Turkey") AS hitsTime_ts
,h.appInfo.appId as appId
,fullVisitorId
,(SELECT  value FROM h.customDimensions where index=1) as cUserId
,h.eventInfo.eventCategory as eventCategory
,h.eventInfo.eventAction as eventAction
,h.eventInfo.eventLabel as eventLabel
,REPLACE(SPLIT(h.eventInfo.eventCategory,'/{')[OFFSET(1)],'}','') as player_type
,SPLIT(h.eventInfo.eventLabel,'|')[OFFSET(0)] as PLAY_SESSION_ID
,SPLIT(h.eventInfo.eventLabel,'|')[OFFSET(1)] as CHANNEL_ID
,SPLIT(h.eventInfo.eventLabel,'|')[OFFSET(2)] as CONTENT_EPG_ID
,SPLIT(h.eventInfo.eventLabel,'|')[OFFSET(3)] as OFF_SET 
FROM `zzzzz.yyyyyy.xxxxxx*` a,
UNNEST(hits) AS h
where 
1=1 
and SPLIT(SPLIT(h.eventInfo.eventCategory,'/{')[OFFSET(0)],'/')[OFFSET(0)] like 'player' 
and  _TABLE_SUFFIX BETWEEN FORMAT_DATE('%Y%m%d',DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY))
AND FORMAT_DATE('%Y%m%d',DATE_SUB(CURRENT_DATE(), INTERVAL 1 DAY))
AND (BYTE_LENGTH(h.eventInfo.eventCategory) - BYTE_LENGTH(REPLACE(h.eventInfo.eventCategory,'/{','')))/2 + 1 = 2
AND h.eventInfo.eventAction='heartBeat' 
"""

job = client.query(sql)  # API request.
job.result() 


query_job.result()  # Waits for the query to finish
print('Query results loaded to table {}'.format(table_ref.path))

【问题讨论】：

标签： python sql oracle google-bigquery cloud

【解决方案1】：

这里提出的问题的快速解决方案：创建表时，如果存在数据来自哪里的查询，则不需要声明它的架构。现在数据和声明的模式之间存在冲突。所以删除一个。

而不是开始查询：

CREATE OR REPLACE TABLE zzzzz.xxxxx.yyyyy (visitStartTime_ts INT64,date TIMESTAMP,hitsTime_ts INT64,appId STRING,fullVisitorId STRING,cUserId STRING,eventCategory STRING,eventLabel STRING,player_type STRING,PLAY_SESSION_ID STRING,CHANNEL_ID STRING,CONTENT_EPG_ID STRING,OFF_SET STRING)
PARTITION BY date

开始查询：

CREATE OR REPLACE TABLE zzzzz.xxxxx.yyyyy 
PARTITION BY date

【讨论】：