在 Pandas Dataframe 中读取嵌套的 json 文件

Question

我有一个JSON文件，结构如下（不是完整的json文件，但结构是一样的）：

{"data":[{"referenced_tweets":[{"type":"retweeted","id":"xxxxxxx"}],"text":"abcdefghijkl","created_at":"2020-03-09T00:11:41.000Z","author_id":"xxxxx","id":"xxxxxxxxxxx"},{"referenced_tweets":[{"type":"retweeted","id":"xxxxxxxxxxxx"}],"text":"abcdefghijkl","created_at":"2020-03-09T00:11:41.000Z","author_id":"xxxxxxxx","id":"xxxxxxxxxxx"}]}
..... 

//The rest of json continues with the same structure, but referenced_tweets is not always present  

我的问题：如何将这些数据加载到具有以下列的数据框中：type、id(referenced_tweet id)、text、created_at、author_id 和 id (tweet id) ?

到目前为止我能做什么：我可以获得以下列：

referenced_tweets	text	cerated_at	author_id	id (tweet id)
[{'type': 'xx', 'id': 'xxx'}]	xxx	xxxx	xxxxx	xxxxxxxxxxxx

获取上表的代码如下：

with open('Test_SampleRetweets.json') as json_file:
    data_list = json.load(json_file)

df1 = json_normalize(data_list, 'data')
df1.head()

但是，我想在单独的列中获取 type 和 id（在 referenced_tweets 中），到目前为止我可以获得以下信息：

type	id (referenced_tweet id)
xxxx	xxxxxxxxxxxxxxxxxxxxxxx

这是获取上表的代码：

df2 = json_normalize(data_list, record_path=['data','referenced_tweets'], errors='ignore')
df2.head()

问题是什么？我想将所有内容都放在一个表中，即类似于此处的第一个表，但 type 和 id 在单独的列中（例如第二张表）。所以，最后的列应该是：type、id (referenced_tweet id)、text、created_at、author_id 和 id (tweet id)

感谢任何帮助

谢谢

Answer 1

import pandas as pd


with open('Test_SampleRetweets.json') as json_file:
    raw_data = json.load(json_file)


data = []
for item in raw_data["data"]:
    item["tweet_id"] = item["id"]
    item.update(item["referenced_tweets"][0])
    del item["referenced_tweets"]
    data.append(item)


df1 = pd.DataFrame(data)
print(df1.head())

Answer 2

使用json_normalize() 中的嵌套json 时，您需要使用meta 参数来获取元级别中的字段。所以，本质上你正在做的是将嵌套和规范化，然后从上面的级别加入其他几个字段。显然，您可以将其组合到多个嵌套字段中，请参阅this 以供参考。

import json
import pandas as pd

j = '{"data":[{"referenced_tweets":[{"type":"retweeted","id":"xxxxxxx"}],"text":"abcdefghijkl","created_at":"2020-03-09T00:11:41.000Z","author_id":"xxxxx","id":"xxxxxxxxxxx"},{"referenced_tweets":[{"type":"retweeted","id":"xxxxxxxxxxxx"}],"text":"abcdefghijkl","created_at":"2020-03-09T00:11:41.000Z","author_id":"xxxxxxxx","id":"xxxxxxxxxxx"}]}'
j = json.loads(j)

# since you have id twice, it's a bit more complicated and you need to 
# introduce a meta prefix
df = pd.json_normalize(
    j,
    record_path=["data", 'referenced_tweets'],
    meta_prefix="data.",
    meta=[["data", "text"], ["data", "created_at"], ["data", "author_id"], ["data", "id"]]
    )
print(df)

导致：

        type            id data.data.text      data.data.created_at  \
0  retweeted       xxxxxxx   abcdefghijkl  2020-03-09T00:11:41.000Z   
1  retweeted  xxxxxxxxxxxx   abcdefghijkl  2020-03-09T00:11:41.000Z   

  data.data.author_id data.data.id  
0               xxxxx  xxxxxxxxxxx  
1            xxxxxxxx  xxxxxxxxxx

我更喜欢这种方式，因为它看起来更容易处理

df = pd.json_normalize(
    j["data"],
    record_path=['referenced_tweets'],
    meta_prefix="data.",
    meta=["text", "created_at", "author_id", "id"]
    )
print(df)

导致：

        type            id     data.text           data.created_at  \
0  retweeted       xxxxxxx  abcdefghijkl  2020-03-09T00:11:41.000Z   
1  retweeted  xxxxxxxxxxxx  abcdefghijkl  2020-03-09T00:11:41.000Z   

  data.author_id      data.id  
0          xxxxx  xxxxxxxxxxx  
1       xxxxxxxx  xxxxxxxxxxx