如何将JSON字符串列行转换为可查询表

Question

因此，我已将整个集合从 Firestore 导出到 BigQuery，以对其执行某些查询。

在我的 BigQuery 控制台中填充数据后，现在我可以像这样查询整个集合

SELECT * FROM `myapp-1a602.firestore_orders.orders_raw_changelog` LIMIT 1000

现在，这条语句抛出了我的不同列，但我要查找的是数据列，在我的数据列中是每个文档 JSON，但是是 json 格式，我需要查询所有这些值。

现在，这是一行的数据

{
    "cart": [{
        "qty": 1,
        "description": "Sprite 1 L",
        "productName": "Sprite 1 Liter",
        "price": 1.99,
        "productId": 9
    }],
    "storeName": "My awesome shop",
    "status": 5,
    "timestamp": {
        "_seconds": 1590713204,
        "_nanoseconds": 916000000
    }
}

这个数据在 data 列中，所以如果我这样做

SELECT data FROM `myapp-1a602.firestore_orders.orders_raw_changelog` LIMIT 1000

我将获取每个文档的所有 json 值，但我不知道如何查询这些值，假设我想知道所有具有 status 5 和 shopName My awesome shop 的订单，现在，我需要用这个 json 做点什么来把它转换成一个表吗？我需要在 json 本身中执行查询吗？

如何查询这个 json 输出？

谢谢

Answer 1

我需要用这个 json 做点什么来把它转换成一个表吗？我需要在 json 本身中执行查询吗？

以下是 BigQuery 标准 SQL

#standardSQL
SELECT * EXCEPT(data, cart_item), 
  JSON_EXTRACT(data, '$.status') AS status, 
  JSON_EXTRACT(data, '$.storeName') AS storeName,
  JSON_EXTRACT(cart_item, '$.qty') AS qty,
  JSON_EXTRACT(cart_item, '$.description') AS description,
  JSON_EXTRACT(cart_item, '$.productName') AS productName,
  JSON_EXTRACT(cart_item, '$.price') AS price,
  JSON_EXTRACT(cart_item, '$.productId') AS productId
FROM `project.dataset.table`,
UNNEST(JSON_EXTRACT_ARRAY(data, '$.cart')) cart_item   

如果应用到您的问题中的样本数据，如下例所示

#standardSQL
WITH `project.dataset.table` AS (
  SELECT 1 order_id, '''
{
    "cart": [{
        "qty": 1,
        "description": "Sprite 1 L",
        "productName": "Sprite 1 Liter",
        "price": 1.99,
        "productId": 9
    },{
        "qty": 2,
        "description": "Fanta 1 L",
        "productName": "Fanta 1 Liter",
        "price": 1.99,
        "productId": 10
    }],
    "storeName": "My awesome shop",
    "status": 5,
    "timestamp": {
        "_seconds": 1590713204,
        "_nanoseconds": 916000000
    }
}  
'''  data 
)
SELECT * EXCEPT(data, cart_item), 
  JSON_EXTRACT(data, '$.status') AS status, 
  JSON_EXTRACT(data, '$.storeName') AS storeName,
  JSON_EXTRACT(cart_item, '$.qty') AS qty,
  JSON_EXTRACT(cart_item, '$.description') AS description,
  JSON_EXTRACT(cart_item, '$.productName') AS productName,
  JSON_EXTRACT(cart_item, '$.price') AS price,
  JSON_EXTRACT(cart_item, '$.productId') AS productId
FROM `project.dataset.table`,
UNNEST(JSON_EXTRACT_ARRAY(data, '$.cart')) cart_item   

结果是

Row order_id    status  storeName           qty     description     productName         price   productId    
1   1           5       "My awesome shop"   1       "Sprite 1 L"    "Sprite 1 Liter"    1.99    9    
2   1           5       "My awesome shop"   2       "Fanta 1 L"     "Fanta 1 Liter"     1.99    10   

Answer 2

您可以使用 json 函数，例如

CrEATE Table products (id Integer,attribs_json JSON );

INSERT INTO products VALUES (1,'{
    "cart": [{
        "qty": 1,
        "description": "Sprite 1 L",
        "productName": "Sprite 1 Liter",
        "price": 1.99,
        "productId": 9
    }],
    "storeName": "My awesome shop",
    "status": 5,
    "timestamp": {
        "_seconds": 1590713204,
        "_nanoseconds": 916000000
    }
}');

select * from products where attribs_json->"$.status" 
= 5 AND attribs_json->"$.storeName" 
= 'My awesome shop';

编号 | attribs_json -: | :------------------------------------------------ -------------------------------------------------- -------------------------------------------------- -------------------------------------------------- -------------------------- 1 | {"cart": [{"qty": 1, "price": 1.99, "productId": 9, "description": "Sprite 1 L", "productName": "Sprite 1 Liter"}], "status" : 5, "storeName": "My awesome shop", "timestamp": {"_seconds": 1590713204, "_nanoseconds": 916000000}}

db小提琴here

select attribs_json->"$.storeName",attribs_json->"$.status",attribs_json->"$.cart[0].qty" from products where attribs_json->"$.status" 
= 5 AND attribs_json->"$.storeName" 
= 'My awesome shop';

attribs_json->"$.storeName" | attribs_json->"$.status" | attribs_json->"$.cart[0].qty" :---------------------------- | :------------------------ | :---------------------------- “我的好店” | 5 | 1

db小提琴here

还有JSON_EXTRACT 用于mysql 5.7 及更高版本。

最后只有文字，所以你也可以使用REGEXP or RLIKE

要将 jaso 再次转移到行，您可以使用 JSON_TABLE

Answer 3

您必须做的是从 json 数据中提取值： 选择 ....... WHERE data->'$.storeName'= "My awesome shop" and data->'$.status' = 5

从 'cart' 或 'timestamp' 键中提取将为您提供一个 Json 对象，该对象需要进一步提取以获取数据。 我希望它会帮助你 您可能想查看 MySql 文档 (https://dev.mysql.com/doc/refman/8.0/en/json.html) 或 https://www.mysqltutorial.org/mysql-json/。

Answer 4

您可以在 WHERE 子句中使用 UNNEST 来访问购物车的列，并在 WHERE 子句中使用 JSON_EXTRACT 函数来过滤所需的行。您需要注意访问 json 根目录或数组 cart； json_data 和 cart_items 在下面的示例中（顺便说一下，在您的示例中 shopName 不存在，但 storeName 存在）。

WITH
  `myapp-1a602.firestore_orders.orders_raw_changelog` AS (
  SELECT
    '{"cart": [{"qty": 1,"description": "Sprite 1 L","productName": "Sprite 1 Liter","price": 1.99,"productId": 9}, {"qty": 11,"description": "Sprite 11 L","productName": "Sprite 11 Liter","price": 11.99,"productId": 19}],"storeName": "My awesome shop","status": 5,"timestamp": {"_seconds": 1590713204,"_nanoseconds": 916000000}}' json_data )
SELECT
  JSON_EXTRACT(json_data, '$.status') AS status,
  JSON_EXTRACT(json_data, '$.storeName') AS storeName,
  JSON_EXTRACT(cart_items, '$.productName') AS product,
  JSON_EXTRACT_SCALAR(cart_items, '$.qty') AS qty
FROM
  `myapp-1a602.firestore_orders.orders_raw_changelog`,
  UNNEST(JSON_EXTRACT_ARRAY(json_data, '$.cart')) AS cart_items
WHERE
  JSON_EXTRACT(json_data,'$.storeName') like "\"My awesome shop\"" AND 
  CAST(JSON_EXTRACT_SCALAR(json_data,'$.status') AS NUMERIC) = 5