Pyspark 获取最新值作为新列

Question

我的输入数据框是；

Date        Client  Score
2020-10-26  1       NULL
2020-10-27  1       NULL
2020-10-28  1       3 
2020-10-29  1       10
2020-10-30  1       NULL
2020-10-31  1       NULL
2020-11-01  1       NULL
2020-11-02  1       NULL
2020-11-03  1       NULL
2020-11-04  1       NULL
2020-11-05  1       NULL
2020-11-06  1       NULL
2020-11-07  1       NULL
2020-11-08  1       NULL
2020-11-09  1       2
2020-10-26  2       NULL
2020-10-27  2       NULL
2020-10-28  2       NULL
2020-10-29  2       NULL
2020-10-30  2       NULL
2020-10-31  2       NULL
2020-11-01  2       NULL
2020-11-02  2       NULL
2020-11-03  2       NULL
2020-11-04  2       NULL
2020-11-05  2       NULL
2020-11-06  2       NULL
2020-11-07  2       NULL
2020-11-08  2       NULL
2020-11-09  2       NULL

输入数据框有每个客户和每一天的得分列。在某些日子里，此列值为空。

我想要的输出是；

Date        Client  Score   Last_5days  Last_10days Last_15days
2020-10-26  1       NULL    NULL        NULL        NULL
2020-10-27  1       NULL    NULL        NULL        NULL
2020-10-28  1          3      3         NULL        NULL
2020-10-29  1         10     10            3        NULL
2020-10-30  1       NULL     10            3        NULL
2020-10-31  1       NULL     10            3        NULL
2020-11-01  1       NULL     10            3        NULL
2020-11-02  1       NULL     10            3        NULL
2020-11-03  1       NULL     10            3        NULL
2020-11-04  1       NULL    NULL          10        3
2020-11-05  1       NULL    NULL          10        3
2020-11-06  1       NULL    NULL          10        3
2020-11-07  1       NULL    NULL          10        3
2020-11-08  1       NULL    NULL        NULL        10
2020-11-09  1          2       2        NULL        10
2020-10-26  2       NULL    NULL        NULL        NULL
2020-10-27  2         15      15        NULL        NULL
2020-10-28  2       NULL      15        NULL        NULL
2020-10-29  2       NULL      15        NULL        NULL
2020-10-30  2       NULL      15        NULL        NULL
2020-10-31  2       NULL      15        NULL        NULL
2020-11-01  2       NULL    NULL        15          NULL
2020-11-02  2       NULL    NULL        15          NULL
2020-11-03  2       NULL    NULL        15          NULL
2020-11-04  2       NULL    NULL        15          NULL
2020-11-05  2       NULL    NULL        15          NULL
2020-11-06  2       NULL    NULL        15          NULL
2020-11-07  2       NULL    NULL        NULL        15
2020-11-08  2       NULL    NULL        NULL        15
2020-11-09  2       NULL    NULL        NULL        15

我想添加一些新列。这些与客户指定 n 天（5、10 或 15）的最新分数有关。

我对 partitionBy 和 window 做了一些研究，但我无法处理。

你能帮我解决这个问题吗？

Answer 1

我不太明白其中的逻辑，但这是我试图获得与您想要的类似的东西：

from pyspark.sql import functions as F, Window

w = Window.partitionBy('Client').orderBy('Date')
df2 = df.withColumn(
    'last5days',
    F.collect_list('score').over(w.rowsBetween(-5,0))
).withColumn(
    'last10days',
    F.collect_list('score').over(w.rowsBetween(-10,-6))
).withColumn(
    'last15days',
    F.collect_list('score').over(w.rowsBetween(-15,-11))
).withColumn(
    'last15days', 
    F.when(F.size(F.col('last10days')) > 1, F.col('last10days')[0]).otherwise(F.element_at(F.col('last15days'), -1))
).withColumn(
    'last10days', 
    F.when(F.size(F.col('last5days')) > 1, F.col('last5days')[0]).otherwise(F.element_at(F.col('last10days'), -1))
).withColumn(
    'last5days',
    F.element_at(F.col('last5days'), -1)
)

df2.show(99,0)
+----------+------+-----+---------+----------+----------+
|Date      |Client|Score|last5days|last10days|last15days|
+----------+------+-----+---------+----------+----------+
|2020-10-26|1     |null |null     |null      |null      |
|2020-10-27|1     |null |null     |null      |null      |
|2020-10-28|1     |3    |3        |null      |null      |
|2020-10-29|1     |10   |10       |3         |null      |
|2020-10-30|1     |null |10       |3         |null      |
|2020-10-31|1     |null |10       |3         |null      |
|2020-11-01|1     |null |10       |3         |null      |
|2020-11-02|1     |null |10       |3         |null      |
|2020-11-03|1     |null |10       |3         |null      |
|2020-11-04|1     |null |null     |10        |3         |
|2020-11-05|1     |null |null     |10        |3         |
|2020-11-06|1     |null |null     |10        |3         |
|2020-11-07|1     |null |null     |10        |3         |
|2020-11-08|1     |null |null     |10        |3         |
|2020-11-09|1     |2    |2        |null      |10        |
|2020-10-26|2     |null |null     |null      |null      |
|2020-10-27|2     |15   |15       |null      |null      |
|2020-10-28|2     |null |15       |null      |null      |
|2020-10-29|2     |null |15       |null      |null      |
|2020-10-30|2     |null |15       |null      |null      |
|2020-10-31|2     |null |15       |null      |null      |
|2020-11-01|2     |null |15       |null      |null      |
|2020-11-02|2     |null |null     |15        |null      |
|2020-11-03|2     |null |null     |15        |null      |
|2020-11-04|2     |null |null     |15        |null      |
|2020-11-05|2     |null |null     |15        |null      |
|2020-11-06|2     |null |null     |15        |null      |
|2020-11-07|2     |null |null     |null      |15        |
|2020-11-08|2     |null |null     |null      |15        |
|2020-11-09|2     |null |null     |null      |15        |
+----------+------+-----+---------+----------+----------+