使用 sql (Hive) 中的条件为每个 ID 选择随机行

Question

下面是一个有 85M 行的表，其中 ~7M 不同的 product_ids。日期范围为 2019 年 1 月 1 日至 2019 年 10 月 31 日（共 10 个月）。

product_id    status     date
60151028       LESS    2019-04-04
86069378       MORE    2019-08-12
99660145       LESS    2019-01-06
16775944       LESS    2019-05-12
52801157       LESS    2019-09-06
4493185        LESS    2019-09-13
39917883       LESS    2019-07-03
87716367       MORE    2019-05-24
90785856       MORE    2019-05-15
15412972       SAME    2019-07-11
12236638       SAME    2019-03-23
.
.
.
.
60151028       LESS    2019-10-12

我想为我的分析生成一个随机样本，并且我想为每个 ID 和状态选择 4 个随机行——例如，我想要：

1. 4 行，ID 为 60151028，status = "LESS"
2. 4 行，ID 为 60151028，status = "MORE"
3. 4 行 ID 为 60151028，状态 =“SAME”，所有产品 ID 都类似，这意味着有 12 行产品 ID 相同。

应在这 10 个月内的所有日期中随机选择数据。下面是结果视图：

product_id  status      date
60151028       LESS    2019-04-04
60151028       LESS    2019-07-18
60151028       LESS    2019-09-23
60151028       LESS    2019-01-25
60151028       SAME    2019-05-14
60151028       SAME    2019-10-11
60151028       SAME    2019-03-31
60151028       SAME    2019-08-30
60151028       MORE    2019-04-27
60151028       MORE    2019-09-19
60151028       MORE    2019-10-29
60151028       MORE    2019-06-04
.
.
.
39917883       SAME    2019-08-03
39917883       SAME    2019-02-26
39917883       SAME    2019-10-07
.
.
.

我一直在考虑以下查询（Hive），但我无法想到如何随机选择状态 =“LESS”或“MORE”等的行。任何帮助将不胜感激！！

另外，我知道有一个 rand() 函数是 hive sql 但我不知道如何在这里使用它。

select max(case when status = 'LESS' then 1 else 0 end) OVER (PARTITION by product_id) as flg
,detail1.*
from
(
select row_number() OVER
(partition by product_id order by date) as rn
from db.table 
) detail1

UNION ALL

select max(case when status = 'SAME' then 1 else 0 end) OVER (PARTITION by product_id) as flg
,detail2.*
from
(
select row_number() OVER
(partition by product_id order by date) as rn
from db.table  
) detail2

UNION ALL

select max(case when status = 'MORE' then 1 else 0 end) OVER (PARTITION by product_id) as flg
,detail3.*
from
(
select row_number() OVER
(partition by product_id order by date) as rn
from db.table  
) detail3

Answer 1

使用partition by:

select t.*
from (select t.*
             row_number() OVER (partition by product_id, status order by rand()) as seqnum
      from db.table  
     ) t
where seqnum <= 4;

编辑：

获取所有状态至少有 4 行的示例：

select t.*
from (select t.*
             row_number() OVER (partition by product_id, status order by rand()) as seqnum,
             max(ps_cnt) over (partition by product_id) as max_ps_cnt
      from (select t.*, count(*) over (partition by product_id, status) as ps_cnt
            from db.table t
           ) t
     ) t
where seqnum <= 4 and max_ps_cnt >= 4;