如何从 json 数据创建 DataFrame - 数组中的字典、列表和数组

Question

我无法获取数据，只能获取来自 json 数据的标头

尝试使用 json_normalize 从 json 数据创建 DataFrame，但是当我尝试循环和附加数据时，结果是我只得到了标题。

import pandas as pd
import json
import requests
from pandas.io.json import json_normalize
import numpy as np

# importing json data

def get_json(file_path):
    r = requests.get('https://www.atg.se/services/racinginfo/v1/api/games/V75_2019-09-29_5_6')
    jsonResponse = r.json()
    with open(file_path, 'w', encoding='utf-8') as outfile:
        json.dump(jsonResponse, outfile, ensure_ascii=False, indent=None)

# Run the function and choose where to save the json file
get_json('../trav.json')

# Open the json file and print a list of the keys
with open('../trav.json', 'r') as json_data:
    d = json.load(json_data)

    print(list(d.keys()))

[Out]:
['@type', 'id', 'status', 'pools', 'races', 'currentVersion']

要获取一场比赛的所有开始数据，我可以使用 json_normalize 函数

race_1_starts = json_normalize(d['races'][0]['starts'])
race_1_starts_df = race_1_starts.drop('videos', axis=1)
print(race_1_starts_df)

[Out]:
    distance  driver.birth  ... result.prizeMoney  result.startNumber
0       1640          1984  ...             62500                   1
1       1640          1976  ...             11000                   2
2       1640          1968  ...               500                   3
3       1640          1953  ...            250000                   4
4       1640          1968  ...               500                   5
5       1640          1962  ...             18500                   6
6       1640          1961  ...              7000                   7
7       1640          1989  ...             31500                   8
8       1640          1960  ...               500                   9
9       1640          1954  ...               500                  10
10      1640          1977  ...            125000                  11
11      1640          1977  ...               500                  12

在上面我们得到一个 DataFrame，其中包含所有从一场比赛开始的数据。但是，当我尝试遍历范围内的所有比赛以获取所有比赛的所有开始数据时，我只获得每场比赛的标题，而不是每场比赛的开始数据：

all_starts = []

for t in range(len(d['races'])):

    all_starts.append([t+1, json_normalize(d['races'][t]['starts'])])

all_starts_df = pd.DataFrame(all_starts, columns = ['race', 'starts'])
print(all_starts_df)

[Out]:
   race                                             starts
0     1      distance  ...                             ...
1     2      distance  ...                             ...
2     3      distance  ...                             ...
3     4      distance  ...                             ...
4     5      distance  ...                             ...
5     6      distance  ...                             ...
6     7      distance  ...                             ...

在输出中，我想要一个 DataFrame，它是所有比赛的所有开始数据的合并。请注意，列数可能因种族而异，但我希望如果一场比赛有 21 列，而另一场比赛有 20 列 - 那么 all_starts_df 应该包含所有列，但如果一场比赛没有一列的数据应该说“NaN”。

预期结果：

[Out]:
race  distance  driver.birth  ... result.column_20     result.column_22
1       1640          1984  ...             12500                   1
1       1640          1976  ...             11000                   2
2       2140          1968  ...               NaN                   1
2       2140          1953  ...               NaN                   2
3       3360          1968  ...              1500                 NaN
3       3360          1953  ...            250000                 NaN

Answer 1

如果你想要所有列，你可以试试这个..（我发现超过 20 列，所以我可能有问题。）

all_starts = []
headers = []
for idx, race in enumerate(d['races']):
    df = json_normalize(race['starts'])
    df['race'] = idx
    all_starts.append(df.drop('videos', axis=1))
    headers.append(set(df.columns))

# Create set of all columns for all races
columns = set.union(*headers)

# If columns are missing from one dataframe add it (as np.nan)
for df in all_starts:
    for c in columns - set(df.columns):
        df[c] = np.nan

# Concatenate all dataframes for each race to make one dataframe
df_all_starts = pd.concat(all_starts, axis=0, sort=True)

或者，如果您知道要保留的列的名称，请尝试此操作

columns = ['race', 'distance', 'driver.birth', 'result.prizeMoney']
all_starts = []

for idx, race in enumerate(d['races']):
    df = json_normalize(race['starts'])
    df['race'] = idx
    all_starts.append(df[columns])

# Concatenate all dataframes for each race to make one dataframe
df_all_starts = pd.concat(all_starts, axis=0)