将 Json 转换为 CSV 文件，但分隔符因属性而异

Question

将 JSON 文件转换为 CSV，但所有属性值用逗号 (,) 分隔。

phone 是多值属性，因此 2 个或多个 phone 应由竖线 (|) 分隔 address 是复杂的多值属性，因此 address 中的每个值都应该用分号 (;) 分隔。

当我将 json 转换为 csv 时，我只有分隔符逗号，但无法分隔多值和复杂的多值属性。

代码尝试

df = pd.read_json("file")
df1 = df.to_csv("file", sep=",",index=False)

json中的输入文件

[
   {
      "parsed_address":[
         {
            "address_type":"primary",
            "address_line_1":"abc",
            "city":"jersey",
            "state":"nj",
            "postal_code":"073024588",
            "country":"usa"
         },
         {
            "address_type":"work",
            "address_line_1":"xyz",
            "city":"ny",
            "state":"ns",
            "postal_code":"073024533",
            "country":"london"
         }
      ],
      "phone":[
         {
            "phone":"+12177218280",
            "phone_type":"Mobile"
         },
         {
            "phone":"+1217721340",
            "phone_type":"Work"
         }
      ],
      "first_name":"saman",
      "last_name":"zonouz"
   },
]

以 CSV 格式输出文件

first_name,last_name,phone,parsed_address
samon,zonouz,+12177218280|+1217721340,abc;jersey;nj;073024588;usa|xyz;ny;ns;073024533;london

Answer 1

我认为最简单的方法是使用正确的键构建一个新的字典列表：

import pandas as pd

addresses = [
  {
      "parsed_address":[
         {
            "address_type":"primary",
            "address_line_1":"abc",
            "city":"jersey",
            "state":"nj",
            "postal_code":"073024588",
            "country":"usa"
         },
         {
            "address_type":"work",
            "address_line_1":"xyz",
            "city":"ny",
            "state":"ns",
            "postal_code":"073024533",
            "country":"london"
         }
      ],
      "phone":[
         {
            "phone":"+12177218280",
            "phone_type":"Mobile"
         },
         {
            "phone":"+1217721340",
            "phone_type":"Work"
         }
      ],
      "first_name":"saman",
      "last_name":"zonouz"
   }
]

formatted_addr = []
for addr in addresses:
    new_dic={}
    new_dic['first_name'] = addr['first_name']
    new_dic['last_name'] = addr['last_name']
    new_dic['phone'] = '|'.join([dic_phone['phone'] for dic_phone in addr['phone']])
    new_dic['parsed_address'] = '|'.join(
                                    [';'.join([dic_addr[key] 
                                    for key in dic_addr.keys() if key != 'address_type'])
                                    for dic_addr in addr['parsed_address']])
    formatted_addr.append(new_dic)

df = pd.DataFrame(formatted_addr)
df1 = df.to_csv('example.csv', sep=",",index=False)

输出：

first_name,last_name,phone,parsed_address
saman,zonouz,+12177218280|+1217721340,abc;jersey;nj;073024588;usa|xyz;ny;ns;073024533;london

Answer 2

您可以先加载 json，然后使用 pd.json_normalize 以正确格式获取数据。使用 record_path 和 meta_var 允许在 json 的嵌套级别中获取数据，同时仍然知道它们属于哪个条目：

>>> with open('f.json') as f:
...   data = json.load(f)
... 
>>> phones = pd.json_normalize(data, record_path='phone', meta=['first_name', 'last_name'])
>>> phones
          phone phone_type first_name last_name
0  +12177218280     Mobile      saman    zonouz
1   +1217721340       Work      saman    zonouz
>>> phones.groupby(['first_name', 'last_name']).agg('|'.join)
                                         phone   phone_type
first_name last_name                                       
saman      zonouz     +12177218280|+1217721340  Mobile|Work

地址只是稍微复杂一点，因为有一个额外的连接：

>>> addr = pd.json_normalize(data, record_path='parsed_address', meta=['first_name', 'last_name'])
>>> addr = addr[['first_name', 'last_name']].join(
...     addr.drop(columns=['first_name', 'last_name']).agg(';'.join, axis='columns').rename('address')
... )
>>> addr
  first_name last_name                              address
0      saman    zonouz  primary;abc;jersey;nj;073024588;usa
1      saman    zonouz      work;xyz;ny;ns;073024533;london
>>> addr.groupby(['first_name', 'last_name']).agg('|'.join)
                                                                address
first_name last_name                                                   
saman      zonouz     primary;abc;jersey;nj;073024588;usa|work;xyz;n...

所以最后你可以将这些数据框连接在一起：

>>> df = pd.merge(
...     addr.groupby(['first_name', 'last_name']).agg('|'.join), 
...     phones.groupby(['first_name', 'last_name'])['phone'].agg('|'.join), 
...     on=['first_name', 'last_name']
... )
>>> df
                                                                address                     phone
first_name last_name                                                                             
saman      zonouz     primary;abc;jersey;nj;073024588;usa|work;xyz;n...  +12177218280|+1217721340
>>> df.to_csv('test.csv')

这会产生以下文件：

first_name,last_name,address,phone
saman,zonouz,primary;abc;jersey;nj;073024588;usa|work;xyz;ny;ns;073024533;london,+12177218280|+1217721340

---

这段代码的一个完全通用的版本，但它保留了电话类型，如下：

# Load data naively
df = pd.json_normalize(data)

# Remove all columns that have nesting
nested_cols = df.columns[df.applymap(type).eq(list).any()]
df = df.drop(columns=nested_cols)

# Remaining columns identify csv rows
id_cols = df.columns.to_list()

# Columns we are not interested in, key is first-level nesting column
drop_cols = {
   'phone': ['phone_type'],
}

# Now for each nested column, join columns with ; then join rows with |
for col in nested_cols:
   nested = pd.json_normalize(data, record_path=col, meta=id_cols)
   nested = nested[id_cols].join(
      nested.drop(columns=id_cols + drop_cols.get(col, [])).agg(lambda s: s.str.cat(sep=';'), axis='columns').rename(col)
   ).groupby(id_cols).agg(lambda s: s.str.cat(sep='|'))
   df = df.merge(nested, on=id_cols)

这将为您提供以下数据框：

  first_name last_name                                     parsed_address                     phone
0      saman    zonouz  primary;abc;jersey;nj;073024588;usa|work;xyz;n...  +12177218280|+1217721340