折叠 Hive 中的行并保留非 Null 值

Question

我在 Hive 中有一个表，其中 athr_name 和 post_date 字段 90% 为空（在 Hive 中由“？”表示）。我想查询表和 GROUP BY athr_name、post_date、page_nm 和 visit_date 以获取访问次数和访问者数。但是，我还想将空值组合并替换为 athr_name 和 post_date 不为空的值（page_nm 包含唯一值，因此只能有正确的 athr_name 或空值）。

换句话说，我有这个：

   athr_name post_date         page_nm visit_date visit visitors
1      Steve  9/1/2019 /page1/content/   20191014    45       11
2      Steve  9/1/2019 /page1/content/   20191015    62       38
3      Steve  9/1/2019 /page1/content/   20191016    28       49
4      Steve  9/1/2019 /page1/content/   20191207    54       70
5      Steve  9/1/2019 /page1/content/   20191208    39       26
6          ?         ? /page1/content/   20191014    28       24
7          ?         ? /page1/content/   20191015    17       63
8          ?         ? /page1/content/   20191016    48       40
9          ?         ? /page1/content/   20191017    47       14
10         ?         ? /page1/content/   20191018    33        1

我想将该数据折叠成这个结果：

  athr_name post_date         page_nm visit_date visit visitors
1     Steve  9/1/2019 /page1/content/   20191014    73       35
2     Steve  9/1/2019 /page1/content/   20191015    79      101
3     Steve  9/1/2019 /page1/content/   20191016    76       89
4     Steve  9/1/2019 /page1/content/   20191017    47       14
5     Steve  9/1/2019 /page1/content/   20191018    33        1
6     Steve  9/1/2019 /page1/content/   20191207    54       70
7     Steve  9/1/2019 /page1/content/   20191208    39       26

如果它是列而不是行，它可以通过合并函数来解决。非常感谢任何帮助！

Answer 1

首先你需要用LAST_VALUE 填充空值。您的查询可能如下所示：

SELECT athr_name, 
       post_date, 
       page_nm, visit_date, 
       sum(visit), 
       sum(visitors)
from (
    select nvl(athr_name, LAST_VALUE(athr_name, TRUE)
                                              OVER (ORDER BY page_nm, athr_name NULLS LAST ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)) as athr_name,
           nvl(post_date, LAST_VALUE(post_date, TRUE)
                                              OVER (ORDER BY page_nm, post_date NULLS LAST ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)) as post_date,
           page_nm,
           visit_date,
           visit,
           visitors
    from your_table) as tmp_view
GROUP BY athr_name, post_date, page_nm, visit_date;

更新：

如果某些 page_nm 可能没有对应的 athr_name 或 post_date，则最好使用此查询来保留此信息：

SELECT athr_name, post_date, page_nm, visit_date, sum(visit), sum(visitors)
from (
         select name_view.athr_name as athr_name,
                date_view.post_date as post_date,
                main.page_nm,
                main.visit_date,
                main.visit,
                main.visitors
         from your_table main
                  LEFT JOIN (select athr_name, page_nm, row_number() over (PARTITION BY page_nm) as rn
                             from your_table
                             where athr_name is not null) name_view
                            ON main.page_nm = name_view.page_nm AND name_view.rn = 1
                  LEFT JOIN (select post_date, page_nm, row_number() over (PARTITION BY page_nm) as rn
                             from your_table
                             where post_date is not null) date_view
                            ON main.page_nm = date_view.page_nm AND date_view.rn = 1) as tmp_view
GROUP BY athr_name, post_date, page_nm, visit_date;

Answer 2

这是你想要的吗？

select max(athr_name), max(post_date), page_nm, 
       visit_date, sum(visit), sum(visitors)
from t
group by page_nm, visit_date;