如何从 Kinesis Analytics (SQL) 中格式化为字符串的 json 中选择数据

Question

我有一个以这种格式提供数据的 kinesis 数据流：

created_at：时间戳 有效载荷：varchar(6000)

payload 元素的简化示例

{
    "version": 2.0,
    "data": {
        "whatever": "someString",
        "observations": [{
            "obs_id": 1,
            "locaiton": {
                "lat": 10.000,
                "lng": 20.000
            }
        }, {
            "obs_id": 2,
            "locaiton": {
                "lat": 10.0001,
                "lng": 20.0001
            }
        }]
    }
}

在实时中，payload 列中的数组data.observations 的长度通常在 0 到 200 个元素之间。

我正在尝试扩展payload 中的数据，并为其中的每个元素创建一个新行。我对这个示例的预期结果应该是具有以下结构的数据流：

created_at 时间戳，-- 从根 obs_id 整数， -- 来自 data.observations 内部 location_lat: integer, -- 来自 data.observations.location location_lng: integer, -- 来自 data.observations.location 版本：来自根的整数

这就是我现在的位置，它正在工作（但不提取 json）

-- CREATE OR REPLACE STREAM for cleaned up referrer
CREATE OR REPLACE STREAM "DESTINATION_SQL_STREAM" (
    "created_at" TIMESTAMP,
    "version" Integer
    );

CREATE OR REPLACE PUMP "myPUMP" AS 
   INSERT INTO "DESTINATION_SQL_STREAM"
      SELECT STREAM 
         "created_at", 
         "version"
      FROM "SOURCE_SQL_STREAM_001";

但是，如果我尝试这样做，它会中断：

-- CREATE OR REPLACE STREAM for cleaned up referrer
CREATE OR REPLACE STREAM "DESTINATION_SQL_STREAM" (
    "created_at" TIMESTAMP,
    "version" Integer,
    "obs_id" integer 
    );

CREATE OR REPLACE PUMP "myPUMP" AS 
   INSERT INTO "DESTINATION_SQL_STREAM"
      SELECT STREAM 
         "created_at", 
         "version",
         "data"."observations"."obs_id" as obs_id
      FROM "SOURCE_SQL_STREAM_001";

错误是：table data not found

任何帮助都非常感谢！

编辑：我现在尝试了这个：

-- CREATE OR REPLACE STREAM for cleaned up referrer
CREATE OR REPLACE STREAM "DESTINATION_SQL_STREAM" (
    "version" Integer
    , "whatever" varchar(10)
);

CREATE OR REPLACE PUMP "myPUMP" AS 
   INSERT INTO "DESTINATION_SQL_STREAM"
      SELECT STREAM 
        "version"
        , json_extract("data", "$.whatever") AS whatever,
      FROM "SOURCE_SQL_STREAM_001";

我得到了错误：

org.eigenbase.sql.parser.SqlParseException: Encountered "FROM" at line 10, column 7. Was expecting one of: "*" ... <IDENTIFIER> ... <QUOTED_IDENTIFIER> ... <UNICODE_QUOTED_IDENTIFIER> ... "+" ... "-" ... <UNSIGNED_INTEGER_LITERAL> ... <DECIMAL_NUMERIC_LITERAL> ... <APPROX_NUMERIC_LITERAL> ... <BINARY_STRING_LITERAL> ... <PREFIXED_STRING_LITERAL> ... <QUOTED_STRING> ... <UNICODE_STRING_LITERAL> ... "TRUE" ... "FALSE" ... "UNKNOWN" ... "NULL" ... <LBRACE_D> ... <LBRACE_T> ... <LBRACE_TS> ... "DATE" ... "TIME" ... "TIMESTAMP" ... "INTERVAL" ... "?" ... "CAST" ... "DATEDIFF" ... "EXTRACT" ... "POSITION" ... "CONVERT" ... "TRANSLATE" ... "OVERLAY" ... "FLOOR" ... "CEIL" ... "CEILING" ... "STEP" ... "TUMBLE_WINDOW" ... "SUBSTRING" ... "TRIM" ... "FIRST_VALUE" ... "LAST_VALUE" ... "LAG" ... "NTH_VALUE" ... <LBRACE_FN> ... "MULTISET" ... "SPECIFIC" ... "ABS" ... "ANY" ... "AVG" ... "CARDINALITY" ... "CHAR_LENGTH" ... "CHARACTER_LENGTH" ... "COALESCE" ... "COLLECT" ... "CUME_DIST" ... "COUNT" ... "CURRENT_DATE" ... "CURRENT_TIME" ... "CURRENT_TIMESTAMP" ... "DENSE_RANK" ... "ELEMENT" ... "EVERY" ... "EXP_AVG" ... "EXP" ... "FUSION" ... "INITCAP" ... "LN" ... "LOCALTIME" ... "LOCALTIMESTAMP" ... "LOWER" ... "MAX" ... "MIN" ... "MOD" ... "NULLIF" ... "OCTET_LENGTH" ... "PERCENT_RANK" ... "POWER" ... "RANK" ... "ROW_NUMBER" ... "SQRT" ... "STDDEV" ... "STDDEV_POP" ... "STDDEV_SAMP" ... "SUM" ... "UPPER" ... "VAR_POP" ... "VAR_SAMP" ... "CURRENT_CATALOG" ... "CURRENT_DEFAULT_TRANSFORM_GROUP" ... "CURRENT_PATH" ... "ROWNUM" ... "CURRENT_ROLE" ... "CURRENT_SCHEMA" ... "CURRENT_USER" ... "SESSION_USER" ... "SYSTEM_USER" ... "USER" ... "NEW" ... "CASE" ... "PERIOD" ... "TSDIFF" ... "CURSOR" ... "ROW" ... "NOT" ... "EXISTS" ... "(" ...

Answer 1

根据https://docs.aws.amazon.com/athena/latest/ug/extracting-data-from-JSON.html，

您可以为此使用json_extract。

如下图

select data from vendor_meraki_data_raw
limit 5 
),

jsondata as(

select
  json_extract(data, '$.data') as fulldata
 from dataset
)

select
  json_extract(fulldata, '$.apMac') as apMac
from jsondata```