向数据框中添加新列，计算某些组中的条目数

Question

假设我在 PySpark 中有一个数据框 df（我完全不熟悉这个界面），它有两列，一列带有标签“体育”，只需要 3 个值（“足球”、“篮球”、 'volleyball') 和另一个带有标签 'player_names' 的标签可以在其条目中使用任何字符串。

如何创建一个包含 3 列的新数据框，第一列只是“体育”列，第二列“计数”计算参加每项运动的唯一球员姓名的总数，第三列“约翰计数”是参加每项运动并且包含字符串“john”的唯一球员姓名的总数？

例如，如果我从包含 2 列 ['soccer', 'volleyball', 'basketball', 'basketball', 'soccer', 'soccer'], ['john doe', 'john wick', 'tom hanks', 'tom haverford', 'john doe', 'michael'] 的数据框开始，我想以包含 3 列 ['soccer', 'basketball', 'volleyball'], [2,2,1], [1,0,1] 的数据框结束。

在我的示例中，我的第一个专栏中实际上包含大量体育项目，因此我们将不胜感激避免使用小尺寸“体育”的解决方案。

---

编辑输入：

+----------+-------------+
|    sports| player_names|
+----------+-------------+
|    soccer|     john doe|
|volleyball|    john wick|
|basketball|    tom hanks|
|basketball|tom haverford|
|    soccer|     john doe|
|    soccer|      michael|
+----------+-------------+

预期输出：

+----------+------+-----------+
|    sports|counts|john_counts|
+----------+------+-----------+
|    soccer|     2|          1|
|basketball|     2|          0|
|volleyball|     1|          1|
+----------+------+-----------+

非常感谢！

Answer 1

groupby.agg 使用 SQL 语法：

import pyspark.sql.functions as f

df.dropDuplicates().groupby('sports').agg(
    f.expr('count(*) as counts'),
    f.expr('sum(case when player_names like "%john%" then 1 else 0 end) as john_counts')
).show()

+----------+------+-----------+
|    sports|counts|john_counts|
+----------+------+-----------+
|basketball|     2|          0|
|    soccer|     2|          1|
|volleyball|     1|          1|
+----------+------+-----------+

地点：

1. 首先删除重复项以保证唯一性；
2. groupby sports 并计算唯一玩家的数量；
3. 检查 player_name 是否包含 john 并求和；

---

或者，如果您更喜欢非 SQL 语法：

df.dropDuplicates().groupby('sports').agg(
    f.count('*').alias('counts'),
    f.sum(f.col('player_names').contains('john').cast('long')).alias('john_counts')
).show()

+----------+------+-----------+
|    sports|counts|john_counts|
+----------+------+-----------+
|basketball|     2|          0|
|    soccer|     2|          1|
|volleyball|     1|          1|
+----------+------+-----------+

Answer 2

您需要删除 dropduplicates 并按 count(*) 进行简单的分组，并与 case 相加以获得结果

代码

import pyspark.sql.functions as f
from pyspark.sql.functions import *
from pyspark.sql.types import *

data = [
("soccer" ,"john doe"),
("volleyball","john wick"),
("basketball","tom hanks"),
("basketball","tom haverford"),
("soccer", "john doe"),
("soccer",  "michael")
        ]

schema = StructType([
StructField('game', StringType(),True), \
StructField('name', StringType(),True)
])
df = spark.createDataFrame(data=data, schema=schema).dropDuplicates()
df.createOrReplaceTempView("Data")
spark.sql("select game,count(*),(sum ( case when name like '%john%' then 1 else 0 end ) ) as jo_cnt from Data group by game").show()

一行一行的数据-

>>> data = [
... ("soccer" ,"john doe"),
... ("volleyball","john wick"),
... ("basketball","tom hanks"),
... ("basketball","tom haverford"),
... ("soccer", "john doe"),
... ("soccer",  "michael")
...         ]
>>> schema = StructType([
... StructField('game', StringType(),True), \
... StructField('name', StringType(),True)
... ])

>>> df = spark.createDataFrame(data=data, schema=schema).dropDuplicates()
>>> df.show()
+----------+-------------+
|      game|         name|
+----------+-------------+
|volleyball|    john wick|
|basketball|    tom hanks|
|basketball|tom haverford|
|    soccer|     john doe|
|    soccer|      michael|
+----------+-------------+

>>> df.createOrReplaceTempView("Data")
>>> spark.sql("select game,count(*),(sum ( case when name like '%john%' then 1 else 0 end ) ) as jo_cnt from Data group by game").show()
+----------+--------+------+
|      game|count(1)|jo_cnt|
+----------+--------+------+
|basketball|       2|     0|
|    soccer|       2|     1|
|volleyball|       1|     1|
+----------+--------+------+