为什么我得到这个

Question

df = spark.createDataFrame([(1, "a", None), (2, None, 3.0), (3, "c", 4.0),(None, "c", None)], ["id", "x", "y"])enter image description here

df.na.drop(subset=["x"]).show() enter image description here
df.na.drop(subset=["y"]).show() enter image description here
当我删除 X 为什么它将其他行值显示为 null 并且在 Null 位置它生成了 C

Answer 1

这是您的数据框：

+----+----+----+
|  id|   x|   y|
+----+----+----+
|   1|   a|null|  <== Line with y null
|   2|null| 3.0|  <== Line with x null
|   3|   c| 4.0|
|null|   c|null|  <== Line with y null
+----+----+----+

和你的代码的输出：

df.na.drop(subset=["x"]).show()

+----+---+----+
|  id|  x|   y|
+----+---+----+
|   1|  a|null|
|   3|  c| 4.0|
|null|  c|null|
+----+---+----+
# Line with id=2 is dropped

和

df.na.drop(subset=["y"]).show()

+---+----+---+
| id|   x|  y|
+---+----+---+
|  2|null|3.0|
|  3|   c|4.0|
+---+----+---+
# Lines with id=1 and id=null are dropped

没有错，没有生成。它做你要求它做的事。