展平数据框中的 JSON 列

Question

我在如下所示的数据框中有数据。我正在尝试展平数据，以便 JSON blob 中的值转换为列，如果 JSON blob 中有多个记录，它会创建一个新行。我想保留非 JSON 字段（即 ID）。

我特别关注：How to flatten a pandas dataframe with some columns as json? - 但在执行后，我使用未解析 JSON 的相同数据帧失败。下面是该帖子中的函数的 sn-p，它以前对我有用：

def flatten_nested_json_df(df):

    df = df.reset_index()

    print(f"original shape: {df.shape}")
    print(f"original columns: {df.columns}")


    # search for columns to explode/flatten
    s = (df.applymap(type) == list).all()
    list_columns = s[s].index.tolist()

    s = (df.applymap(type) == dict).all()
    dict_columns = s[s].index.tolist()

    print(f"lists: {list_columns}, dicts: {dict_columns}")
    while len(list_columns) > 0 or len(dict_columns) > 0:
        new_columns = []

        for col in dict_columns:
            print(f"flattening: {col}")
            # explode dictionaries horizontally, adding new columns
            horiz_exploded = pd.json_normalize(df[col]).add_prefix(f'{col}.')
            horiz_exploded.index = df.index
            df = pd.concat([df, horiz_exploded], axis=1).drop(columns=[col])
            new_columns.extend(horiz_exploded.columns) # inplace

        for col in list_columns:
            print(f"exploding: {col}")
            # explode lists vertically, adding new columns
            df = df.drop(columns=[col]).join(df[col].explode().to_frame())
            new_columns.append(col)

        # check if there are still dict o list fields to flatten
        s = (df[new_columns].applymap(type) == list).all()
        list_columns = s[s].index.tolist()

        s = (df[new_columns].applymap(type) == dict).all()
        dict_columns = s[s].index.tolist()

        print(f"lists: {list_columns}, dicts: {dict_columns}")

    print(f"final shape: {df.shape}")
    print(f"final columns: {df.columns}")
    return df

感谢任何帮助！

ID	PROPERTIES	FORMSUBMISSIONS
123	{"firstname":{"value":"FAKE"},"lastmodifieddate":{"value":"FAKE"},"lastname":{"value":"FAKE"}}	[{"contact-associated-by":["FAKE"],"conversion-id":"FAKE","form-id":"FAKE","form-type":"FAKE","meta-data":[],"portal-id": FAKE,"timestamp": FAKE,"title":"FAKE"}]

Answer 1

你可以使用pd.json_normalize，应该更简单。

>>> df
    ID                                         PROPERTIES                                    FORMSUBMISSIONS
0  123  {'firstname': {'value': 'FAKE'}, 'lastmodified...  [{'contact-associated-by': ['FAKE'], 'conversi...

>>> df = df.explode('FORMSUBMISSIONS')  # list to dict
>>> df
    ID                                         PROPERTIES                                    FORMSUBMISSIONS
0  123  {'firstname': {'value': 'FAKE'}, 'lastmodified...  {'contact-associated-by': ['FAKE'], 'conversio...

现在您可以在 FORMSUBMISSIONS 列上执行 json_normalize。为了保留其他列，我使用pd.concat

>>> df = pd.concat([df, pd.json_normalize(df['FORMSUBMISSIONS']), axis=1).drop('FORMSUBMISSIONS', axis=1)

>>> df
    ID                                         PROPERTIES contact-associated-by conversion-id form-id form-type meta-data portal-id timestamp title
0  123  {'firstname': {'value': 'FAKE'}, 'lastmodified...                [FAKE]          FAKE    FAKE      FAKE        []      FAKE      FAKE  FAKE

您可以在 PROPERTIES 列上执行相同的操作。

df = pd.concat([df, pd.json_normalize(df.PROPERTIES)], axis=1).drop('PROPERTIES', axis=1)