消除列中的重复项答案

【问题标题】：elminate duplicates in a column消除列中的重复项
【发布时间】：2020-02-27 04:00:52
【问题描述】：

我可以消除多个值 i Column_3,Column_4

+--------+--------+--------+--------+
|Column_1|Column_2|Column_3|Column_4|
+--------+--------+--------+--------+
|       1|       x|     abc|     www|
|       1|       x|     abc|     sdf|
|       1|       x|     abc|     xyz|
|       1|       x|     def|     www|
|       1|       x|     def|     sdf|
|       1|       x|     def|     xyz|
+--------+--------+--------+--------+

预期输出

+--------+--------+--------+--------+
|Column_1|Column_2|Column_3|Column_4|
+--------+--------+--------+--------+
|       1|       x|     abc|     www|
|       1|       x|     def|     sdf|
|       1|       x|    null|     xyz|
+--------+--------+--------+--------+

【问题讨论】：

试试df.dropDuplicates(Array("Column_3"))。
嗨@Uservxn - 欢迎来到SO :) 几个问题： 1. 你的spark 版本是什么？ 2. 是否有任何规则可以保留Column_3 和Column_4 的组合，例如如何决定保留`abc | www` 或abc| sdf.

标签： scala apache-spark

【解决方案1】：

使用 df.dropDuplicates(Column_3,Column_4)

另外，从Removing duplicates from rows based on specific columns in an RDD/Spark DataFrame复制

【讨论】：

val df1 = Seq((1,"x","abc"),(1,"x","def")).toDF("Column_1","Column_2","Column_3 ") > val df2 = Seq((1,"x","xyz"),(1,"x","www"),(1,"x","sdf")).toDF("Column_1" ,"Column_2","Column_4") > val df3 = df1.join(df2, Seq("Column_1","Column_2"), "outer") > df3.show --------+--- -----+ |Column_1|Column_2|Column_3|Column_4| +--------+--------+--------+--------+ | 1| x| ABC|万维网| | 1| x| ABC| sdf| | 1| x| ABC| xyz| | 1| x|定义|万维网| | 1| x|定义| sdf| | 1| x|定义| xyz|
df3.dropDuplicates("column_3","column_4") res68: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] = [Column_1: int, Column_2:字符串 ... 2 个更多字段] scala> res68.show +--------+--------+--------+--------+ |Column_1|Column_2|Column_3|Column_4| +--------+--------+--------+--------+ | 1| x|定义| xyz| | 1| x|定义| sdf| | 1| x| ABC|万维网| | 1| x|定义|万维网| | 1| x| ABC| sdf| | 1| x| ABC| xyz| +--------+--------+--------+--------+