Spark数据帧groupby和订单组？

Question

我有以下数据，

+-------+----+----+
|user_id|time|item|
+-------+----+----+
|      1|   5| ggg|
|      1|   5| ddd|
|      1|  20| aaa|
|      1|  20| ppp|
|      2|   3| ccc|
|      2|   3| ttt|
|      2|  20| eee|
+-------+----+----+

这可以通过代码生成：

    val df = sc.parallelize(Array(
      (1, 20, "aaa"),
      (1, 5, "ggg"),
      (2, 3, "ccc"), 
      (1, 20, "ppp"), 
      (1, 5, "ddd"), 
      (2, 20, "eee"), 
      (2, 3, "ttt"))).toDF("user_id", "time", "item")

我怎样才能得到结果：

+---------+------+------+----------+
| user_id | time | item | order_id |
+---------+------+------+----------+
|       1 |    5 | ggg  |        1 |
|       1 |    5 | ddd  |        1 |
|       1 |   20 | aaa  |        2 |
|       1 |   20 | ppp  |        2 |
|       2 |    3 | ccc  |        1 |
|       2 |    3 | ttt  |        1 |
|       2 |   20 | eee  |        2 |
+---------+------+------+----------+

groupby user_id,time and order by time and rank the group,谢谢~

Answer 1

要对行进行排名，您可以使用dense_rank 窗口函数，并且可以通过最终的orderBy 转换来实现排序：

import org.apache.spark.sql.expressions.Window
import org.apache.spark.sql.functions.{dense_rank}

val w = Window.partitionBy("user_id").orderBy("user_id", "time")

val result = df
  .withColumn("order_id", dense_rank().over(w))
  .orderBy("user_id", "time")

result.show()
+-------+----+----+--------+
|user_id|time|item|order_id|
+-------+----+----+--------+
|      1|   5| ddd|       1|
|      1|   5| ggg|       1|
|      1|  20| aaa|       2|
|      1|  20| ppp|       2|
|      2|   3| ttt|       1|
|      2|   3| ccc|       1|
|      2|  20| eee|       2|
+-------+----+----+--------+

注意item列的顺序没有给出