同一查询中的计算列 - Spark ， Hive

Question

我需要使用派生列中的前一个值来获得同一列的下一个结果。

每个名字的第 1 次出现默认为 1

派生列 = num(i) + 派生 (i-1)

Name, Num, derived
A       0       1
A       1       2
A       0       2      
B       0       1
B       0       1
B       1       2
C       0       1
C       0       1
D       1       1
D       1       2
D       1       3       

下面是我的一半代码——使用了窗口延迟功能但无法实现

spark.sql("""
select

z.name as name,

z.num as num,

case

  when z.row_num='1' then '1'

  else (lag(num) over(partition by z.name order by name))

  end as derived

from

(select name,num,row_number() over(partition by name order by name) as row_num

from result)z""").show()

Answer 1

你可以使用会话变量

select
  z.name as name,
  z.num as num,       
  @derived := case
  when z.row_num='1' then '1'
  else num + @derived
  end as derived
from
  (select name,num,row_number() over(partition by name order by name) as row_num
  from result)z,(SELECT @derived:= 0) d1

哪个会给你

姓名 |编号 |衍生的 :--- | --: | :------ 一个 | 0 | 1 一个 | 1 | 2 一个 | 0 | 2 乙| 0 | 1 乙| 0 | 1 乙| 1 | 2 C | 0 | 1 C | 0 | 1 D | 1 | 1 D | 1 | 2 D | 1 | 3

db小提琴here

Answer 2

为了解决这个问题，您需要一个可用于排序记录的列。我假设这样一个专栏并被称为id。

在 hive 中，你可以使用row_number() 并做一个窗口sum() 如下：

select
    name,
    num,
    sum(case when rn = 1 then 1 else num) over(partition by name order by id) derived
from (
    select
        t.*,
        row_number() over(partition by name order by id) rn
    from mytable t
) t