PySpark：与.dropna（）相反？

Question

我试图找出哪家商店有“空”的日子，即没有顾客来的日子。

我的表结构如下：

+----------+-------------+-------------+-------------+-------------+-------------+-------------+------------+
| shop     | 2020-10-15  | 2020-10-16  | 2020-10-17  | 2020-10-18  | 2020-10-19  | 2020-10-20  | 2020-10-21 |
+----------+-------------+-------------+-------------+-------------+-------------+-------------+------------+
| Paris    | 215         | 213         | 128         | 102         | 195         | 180         |        110 |
| London   | 145         | 106         | 102         | 83          | 127         | 111         |         56 |
| Beijing  | 179         | 245         | 134         | 136         | 207         | 183         |        136 |
| Sydney   | 0           | 0           | 0           | 0           | 0           | 6           |         36 |
+----------+-------------+-------------+-------------+-------------+-------------+-------------+------------+

使用 pandas，我可以执行 customers[customers== 0].dropna(how="all") 之类的操作，这将只保留存在 0 的行，我明白了：

+----------+-------------+-------------+-------------+-------------+-------------+-------------+------------+
| shop     | 2020-10-15  | 2020-10-16  | 2020-10-17  | 2020-10-18  | 2020-10-19  | 2020-10-20  | 2020-10-21 |
+----------+-------------+-------------+-------------+-------------+-------------+-------------+------------+
| Sydney   | 0           | 0           | 0           | 0           | 0           | NaN         |         NaN|
+----------+-------------+-------------+-------------+-------------+-------------+-------------+------------+

在 PySpark 中，我相信 .dropna() 会做类似的事情，但我想做相反的事情，保持 NA/0 值。我该怎么做？

Answer 1

创建样本数据集：

from pyspark.sql import SparkSession
from pyspark.sql import Row
from pyspark.sql import functions as f

df_list= [
  { "shop":"Paris", "2020-10-15" : 215,"2020-10-16": 213, "2020-10-17" : 128,"2020-10-18": 195,"2020-10-19":195},
{"shop":"London", "2020-10-15" : 145,"2020-10-16": 106, "2020-10-17" : 102,"2020-10-18": 127,"2020-10-19":127},
 { "shop":"Beijing ", "2020-10-15" : 179,"2020-10-16": 245, "2020-10-17" : 136,"2020-10-18": 207,"2020-10-19":207},

 {"shop":"Sydney", "2020-10-15" : 0,"2020-10-16": 0 ,"2020-10-17" : 0,"2020-10-18": 0, "2020-10-19":0}

]
spark = SparkSession.builder.getOrCreate()
df = spark.createDataFrame(Row(**x) for x in df_list)
df.show()

--

+--------+----------+----------+----------+----------+----------+
|    shop|2020-10-15|2020-10-16|2020-10-17|2020-10-18|2020-10-19|
+--------+----------+----------+----------+----------+----------+
|   Paris|       215|       213|       128|       195|       195|
|  London|       145|       106|       102|       127|       127|
|Beijing |       179|       245|       136|       207|       207|
|  Sydney|         0|         0|         0|         0|         0|
+--------+----------+----------+----------+----------+----------+

您可以应用过滤功能

df.filter(f.greatest(*[f.col(i).isin(0) for i in df.columns])).show()

结果：

+------+----------+----------+----------+----------+----------+
|  shop|2020-10-15|2020-10-16|2020-10-17|2020-10-18|2020-10-19|
+------+----------+----------+----------+----------+----------+
|Sydney|         0|         0|         0|         0|         0|
+------+----------+----------+----------+----------+----------+