udf 解析 pyspark 数据帧中的字符串 json

Question

我有一个包含字符串 json 的 pyspark 数据框。如下所示：

+---------------------------------------------------------------------------+
|col                                                                        | 
+---------------------------------------------------------------------------+
|{"fields":{"list1":[{"list2":[{"list3":[{"type":false}]}]}]}}            | 
+----------------------------------------------------------------------------+--

我写了udfs尝试解析json然后计算匹配phone的值并返回df中的新列

def item_count(json,type):
    count=0
    for i in json.get("fields",{}).get("list1",[]):
        for j in i.get("list2",[]):
            for k in j.get("list3",[]):
                count+=k.get("type",None)==type
    return count

def item_phone_count(json):
    return item_count(json,False)

df2= df\
.withColumn('item_phone_count', (F.udf(lambda j: item_phone_count(json.loads(j)), t.StringType()))('col'))

但我得到了错误：

AttributeError: 'NoneType' object has no attribute 'get'

知道有什么问题吗？

Answer 1

检查无并跳过这些条目：

def item_count(json,type):
    count = 0
    if (json is None) or (json.get("fields",{}) is None):
        return count  
   
    for i in json.get("fields",{}).get("list1",[]):
        if i is None:
            continue
        for j in i.get("list2",[]):
            if j is None:
                continue 
            for k in j.get("list3",[]):
                if k is None:
                    continue 
                count += k.get("type",None) == type
    return count