如何将字符串解析为 HIVE 中的映射数组答案

【问题标题】：How to parse an string to an array of maps in HIVE如何将字符串解析为 HIVE 中的映射数组
【发布时间】：2021-09-09 18:26:19
【问题描述】：

我有一个从系统日志中提取的配置单元表。数据以一种奇怪的格式（映射数组）编码，其中数组的每个元素都包含field_name，它是value。列类型为 STRING。就像下面的例子一样：

select 1 as user_id, '[{"field":"name", "value":"Bob"}, {"field":"gender", "value":"M"}]' as user_info
union all
select 2 as user_id, '[{"field":"gender", "value":"F"}, {"field":"age", "value":22}, {"field":"name", "value":"Ana"}]' as user_info;

它创建了这样的东西：

user_id	user_info
1	[{"field":"name", "value":"Bob"}, {"field":"gender", "value":"M"}]
2	[{"field":"gender", "value":"F"}, {"field":"age", "value":22}, {"field":"name", "value":"Ana"}]

请注意，数组大小并不总是相同的。我正在尝试将地图数组转换为简单的地图。然后，这就是我所期望的结果：

user_id	user_info
1	{"name":"Bob", "gender":"M"}
2	{"name":"Ana", "gender":"F", "age":22}

我计划通过 3 个步骤实现：(1) 解析字符串列以创建地图数组，(2) 分解数组（使用横向视图），(3) 收集字段列表并将它们分组user_id

我正在努力完成第一步：解析字符串列以创建地图数组。任何帮助将不胜感激：D

【问题讨论】：

标签： sql json dictionary hive hiveql

【解决方案1】：

查看代码中的 cmets。要转换为映射的字符串数组由此split(user_info, '(?<=\\}) *, *(?=\\{)') 生成。然后将其分解，并将每个元素转换为地图。

with mydata as
(select 1 as user_id, '[{"field":"name", "value":"Bob"}, {"field":"gender", "value":"M"}]' as user_info
union all
select 2 as user_id, '[{"field":"gender", "value":"F"}, {"field":"age", "value":22}, {"field":"name", "value":"Ana"}]' as user_info
)

select user_id,
       --build new map
       str_to_map(concat('name:', name, nvl(concat(',','gender:', gender),''),  nvl(concat(',','age:', age),'') )) as user_info
from 
(
select user_id, 
      --get name, gender, age, aggregate by user_id
      max(case when user_info['field'] = 'name' then user_info['value'] end) name,
      max(case when user_info['field'] = 'gender' then user_info['value'] end) gender,
      max(case when user_info['field'] = 'age' then user_info['value'] end) age
      
from      
(
select s.user_id, 
       --remove {} and ", convert to map
       str_to_map(regexp_replace(e.element,'^\\{| *"|\\}$','')) as user_info 
from
(
select user_id, regexp_replace(user_info, '^\\[|\\]$','') as user_info -- remove []
 from mydata
)s lateral view outer explode(split(user_info, '(?<=\\}) *, *(?=\\{)'))e as element --split by comma between }{ with optional spaces in between
) s
group by user_id
)s

结果：

user_id   user_info 
1        {"name":"Bob","gender":"M"}
2        {"name":"Ana","gender":"F","age":"22"}

【讨论】：