PySpark Dataframe 到 Json - 分组数据

Question

我们正在尝试从数据框创建一个 json。请在下面找到数据框，

+----------+--------------------+----------+--------------------+-----------------+--------------------+---------------+--------------------+---------------+--------------------+--------------------+
|    CustId|                 TIN|EntityType|    EntityAttributes|AddressPreference|      AddressDetails|EmailPreference|        EmailDetails|PhonePreference|        PhoneDetails|       MemberDetails|
+----------+--------------------+----------+--------------------+-----------------+--------------------+---------------+--------------------+---------------+--------------------+--------------------+
|1234567890|XXXXXXXXXXXXXXXXXX...|    Person|[{null, PRINCESS,...|        Alternate|[{Home, 460 M XXX...|        Primary|[{Home, HEREBY...|      Alternate|[{Home, {88888888...|[{7777777, 999999...|
|1234567890|XXXXXXXXXXXXXXXXXX...|    Person|[{null, PRINCESS,...|        Alternate|[{Home, 460 M XXX...|        Primary|[{Home, HEREBY...|        Primary|[{Home, {88888888...|[{7777777, 999999...|
|1234567890|XXXXXXXXXXXXXXXXXX...|    Person|[{null, PRINCESS,...|          Primary|[{Home, PO BOX 695020...|        Primary|[{Home, HEREBY...|      Alternate|[{Home, {88888888...|[{7777777, 999999...|
|1234567890|XXXXXXXXXXXXXXXXXX...|    Person|[{null, PRINCESS,...|          Primary|[{Home, PO BOX 695020...|        Primary|[{Home, HEREBY...|        Primary|[{Home, {88888888...|[{7777777, 999999...|
+----------+--------------------+----------+--------------------+-----------------+--------------------+---------------+--------------------+---------------+--------------------+--------------------+

因此，对于特定客户，初始列 custid、TIN、Entitytype、EntityAttributes 将是相同的，例如在我们的示例中为 1234567890。但他可能有多个地址/电话/电子邮件。能否请您帮助我们如何将它们分组到 1 个 json 下。

预期结构：

{ “客户 ID”：1234567890， "锡": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", “实体类型”：“人”， “实体属性”：[ { "名字": "公主", "姓氏": "XXXXXX", “生日”：“xxxx-xx-xx”， “已故标志”：“假” } ], “地址”： [ { "AddressPreference": "备用", “详细地址”： { “地址类型”：“家”， “地址1”：“460”， “城市”：“XXXX”， “状态”：“XXX”， “邮编”：“XXX” } }, { “地址首选项”：“主要”， “详细地址”： { “地址类型”：“家”， “地址 1”：“邮政信箱 695020”， “城市”：“XXX”， “状态”：“XXXX”， “邮编”：“695020”， } } ], “电话”： [ { “电话偏好”：“主要”， “电话详细信息”：{ “电话类型”：“家庭”， "电话号码": "xxxxx", "FormatPhoneNumber": "xxxxxx" } }, { "PhonePreference": "备用", “电话详细信息”：{ “电话类型”：“家庭”， "电话号码": "xxxx", "FormatPhoneNumber": "xxxxx" } }, {

],
"Email": [
    {
        "EmailPreference": "Primary",
        "EmailDetails": {
            "EmailType": "Home",
            "EmailAddress": "xxxxxxx@GMAIL.COM"
        }
    }
],
    }
]

}

更新

尝试使用以下推荐的分组方法，最终提供了 1 个客户详细信息，但该电子邮件在列表中重复了 4 次。理想情况下，它应该只有 1 封电子邮件。同样在地址首选项中，备用地址有 1 个地址，主地址有 1 个地址，但备用地址显示 2 个条目，主地址显示 2。请您提供理想的解决方案。

Answer 1

这应该可行。 id 就像您的示例中的 custid 一样，具有重复值。

>>> df.show()
+----+------------+----------+
|  id|     address|     email|
+----+------------+----------+
|1001|   address-a|   email-a|
|1001|   address-b|   email-b|
|1002|address-1002|email-1002|
|1003|address-1003|email-1002|
|1002|   address-c|   email-2|
+----+------------+----------+

聚合那些重复的列，然后转换为 JSON

>>> results = df.groupBy("id").agg(collect_list("address").alias("address"),collect_list("email").alias("email")).toJSON().collect()
>>> for i in results: print(i)
... 
{"id":"1003","address":["address-1003"],"email":["email-1002"]}
{"id":"1002","address":["address-1002","address-c"],"email":["email-1002","email-2"]}
{"id":"1001","address":["address-a","address-b"],"email":["email-a","email-b"]}