在 Python 中使用 json_normalize 解析具有多个记录路径的 json答案

【问题标题】：Using json_normalize in Python to parse json with multiple record paths在 Python 中使用 json_normalize 解析具有多个记录路径的 json
【发布时间】：2020-06-14 00:32:55
【问题描述】：

我已尽力让 json_normalize 处理以下 json 数据，但我做错了。我总是只得到 1 行嵌套 json 分组为对象 - 或错误。

问题：如何从以下json生成平面表：https://www.inegi.org.mx/app/api/indicadores/interna_v1_1//ValorIndicador/6200028395/0/null/en/null/null/3/null/0/null/null/null/null/json/563cbaa8-58bb-fef8-6763-1f1dae318f99

{"dimension":{"freq":{"category":{"index":["3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3","3"],"label":[{"Key":"3","Value":"Annual"}]},"label":"Freq"},"id":["indicator","state","periods"],"indicator":{"category":{"index":["I_6200028395"],"label":[{"Key":"I_6200028395","Value":"Criminal prevalence rate for every one hundred thousand inhabitants of 18 years and over, men"}]},"label":"Indicador"},"municipality":{"category":{"index":[""],"label":[{"Key":"","Value":"Estados Unidos Mexicanos"}]},"label":"Municipio"},"periods":{"category":{"index":["P1","P2","P3","P4","P5","P6","P7","P8","P9"],"label":[{"Key":"P1","Value":"2010"},{"Key":"P2","Value":"2011"},{"Key":"P3","Value":"2012"},{"Key":"P4","Value":"2013"},{"Key":"P5","Value":"2014"},{"Key":"P6","Value":"2015"},{"Key":"P7","Value":"2016"},{"Key":"P8","Value":"2017"},{"Key":"P9","Value":"2018"}]},"label":"periodos"},"role":null,"size":[1,33,9],"state":{"category":{"index":["MEX-0-0","MEX-1-0","MEX-2-0","MEX-3-0","MEX-4-0","MEX-5-0","MEX-6-0","MEX-7-0","MEX-8-0","MEX-9-0","MEX-10-0","MEX-11-0","MEX-12-0","MEX-13-0","MEX-14-0","MEX-15-0","MEX-16-0","MEX-17-0","MEX-18-0","MEX-19-0","MEX-20-0","MEX-21-0","MEX-22-0","MEX-23-0","MEX-24-0","MEX-25-0","MEX-26-0","MEX-27-0","MEX-28-0","MEX-29-0","MEX-30-0","MEX-31-0","MEX-32-0"],"label":[{"Key":"MEX-0-0","Value":"Estados Unidos Mexicanos-Estados Unidos Mexicanos"},{"Key":"MEX-1-0","Value":"Aguascalientes-Estatal"},{"Key":"MEX-2-0","Value":"Baja California-Estatal"},{"Key":"MEX-3-0","Value":"Baja California Sur-Estatal"},{"Key":"MEX-4-0","Value":"Campeche-Estatal"},{"Key":"MEX-5-0","Value":"Coahuila de Zaragoza-Estatal"},{"Key":"MEX-6-0","Value":"Colima-Estatal"},{"Key":"MEX-7-0","Value":"Chiapas-Estatal"},{"Key":"MEX-8-0","Value":"Chihuahua-Estatal"},{"Key":"MEX-9-0","Value":"Ciudad de México-Estatal"},{"Key":"MEX-10-0","Value":"Durango-Estatal"},{"Key":"MEX-11-0","Value":"Guanajuato-Estatal"},{"Key":"MEX-12-0","Value":"Guerrero-Estatal"},{"Key":"MEX-13-0","Value":"Hidalgo-Estatal"},{"Key":"MEX-14-0","Value":"Jalisco-Estatal"},{"Key":"MEX-15-0","Value":"México-Estatal"},{"Key":"MEX-16-0","Value":"Michoacán de Ocampo-Estatal"},{"Key":"MEX-17-0","Value":"Morelos-Estatal"},{"Key":"MEX-18-0","Value":"Nayarit-Estatal"},{"Key":"MEX-19-0","Value":"Nuevo León-Estatal"},{"Key":"MEX-20-0","Value":"Oaxaca-Estatal"},{"Key":"MEX-21-0","Value":"Puebla-Estatal"},{"Key":"MEX-22-0","Value":"Querétaro-Estatal"},{"Key":"MEX-23-0","Value":"Quintana Roo-Estatal"},{"Key":"MEX-24-0","Value":"San Luis Potosí-Estatal"},{"Key":"MEX-25-0","Value":"Sinaloa-Estatal"},{"Key":"MEX-26-0","Value":"Sonora-Estatal"},{"Key":"MEX-27-0","Value":"Tabasco-Estatal"},{"Key":"MEX-28-0","Value":"Tamaulipas-Estatal"},{"Key":"MEX-29-0","Value":"Tlaxcala-Estatal"},{"Key":"MEX-30-0","Value":"Veracruz de Ignacio de la Llave-Estatal"},{"Key":"MEX-31-0","Value":"Yucatán-Estatal"},{"Key":"MEX-32-0","Value":"Zacatecas-Estatal"}]},"label":"Entidad federativa"},"unit":null},"exceptions":null,"label":"Criminal prevalence rate for every one hundred thousand inhabitants of 18 years and over, men","noteValues":null,"precition":null,"source":"INEGI","sourceValues":null,"status":{},"typechart":"3","updated":"\/Date(-62135575200000-0600)\/","value":["26,682","25,868","29,560","30,285","29,430","30,181","30,124","31,419","29,650","38,699","26,305","28,929","28,958","34,789","33,472","34,865","37,404","30,061","32,097","30,340","36,046","41,483","36,532","29,854","40,536","36,138","32,897","27,864","30,124","29,827","25,150","31,445","24,414","27,411","26,028","25,029","20,411","19,147","23,261","25,215","21,992","21,848","23,452","25,284","24,845","26,603","23,745","19,783","24,916","18,616","22,458","20,427","23,770","21,412","19,236","20,541","25,575","22,715","22,872","25,014","27,609","23,170","24,537","12,369","12,932","14,456","18,819","15,742","16,477","17,499","16,180","17,497","37,270","32,613","34,859","27,867","21,956","25,158","26,514","25,455","24,943","36,308","35,662","33,860","35,861","35,949","40,515","36,428","46,343","45,130","21,918","22,085","24,239","22,733","23,120","23,039","21,126","19,418","21,148","24,412","27,395","32,985","28,631","32,489","29,417","30,941","28,269","29,159","20,527","21,050","28,344","27,322","28,638","33,505","32,589","33,817","26,393","22,308","24,214","24,633","20,627","19,306","17,924","22,677","22,381","22,827","27,621","26,482","34,051","33,622","32,602","36,749","31,313","33,410","31,674","31,454","33,889","45,506","53,217","49,874","51,555","49,907","49,177","44,778","17,480","21,510","23,668","21,049","21,052","21,331","22,172","20,758","20,093","26,209","25,185","30,540","27,077","32,304","30,151","28,588","34,061","33,886","22,807","20,138","21,888","20,051","22,003","19,736","20,925","24,595","23,252","33,346","27,488","34,306","27,762","24,236","26,790","26,698","31,112","25,776","17,008","17,948","14,308","16,904","20,158","18,983","19,323","20,558","17,704","26,642","23,643","24,860","24,790","25,286","24,222","29,592","34,349","31,239","21,251","22,180","26,031","25,595","27,909","29,607","26,122","29,837","32,986","33,793","28,321","31,904","29,531","30,246","30,808","29,001","26,148","27,007","25,411","24,752","30,095","26,128","26,841","22,086","22,467","26,833","27,062","28,303","24,758","27,517","24,196","22,998","19,807","19,331","24,259","21,930","28,911","30,055","29,605","27,977","26,394","32,865","36,526","30,555","30,225","27,306","18,154","20,843","24,178","26,415","27,216","29,039","36,440","31,797","24,197","19,634","24,438","18,241","24,244","17,741","22,091","18,677","19,364","23,257","19,295","20,008","24,327","30,483","30,460","27,318","31,630","33,913","17,560","17,358","17,393","20,965","16,558","19,060","17,262","17,186","16,414","19,934","16,571","19,953","20,101","22,794","22,355","21,071","20,822","18,600","29,219","24,194","21,839","21,568","24,145","21,570","21,300","26,265","24,241"],"valueDecimalFormat":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]}

我感兴趣的只有三个部分打印在下面 - 句点、状态、数据。我可以隔离它们并从中构建 DataFrame，但 json_normalize 似乎是一种更有效的方法。尽我所能，我似乎无法将这些参数正确地传递给函数。

This is how the table should look，虽然标题不是必须的：

import requests
from pandas.io.json import json_normalize

# the url
url = r'https://www.inegi.org.mx/app/api/indicadores/interna_v1_1//ValorIndicador/6200028395/0/null/en/null/null/3/null/0/null/null/null/null/json/563cbaa8-58bb-fef8-6763-1f1dae318f99'

response = requests.get(url)
json_response = response.json()

# the years
print(json_response['dimension']['periods']['category']['label'])

# the states
print(json_response['dimension']['state']['category']['label'])

# the data
print(json_response['value'])

【问题讨论】：

标签： python json pandas

【解决方案1】：

json_normalize 在这里没有帮助，因为数据被分成不同的部分，值在不同的部分；您当前的步骤可能是最好的方法：

#extract states and years
states = [entry['Value']
          for entry in
          data['dimension']['state']['category']['label']]

years =  [entry['Value']
          for entry in
          data['dimension']['periods']['category']['label']]

from itertools import product

#get product of states and years
year_state = product(states, years)

#merge with values data
outcome = [(state, year, value)
           for (state, year), value in 
           zip(year_state,data['value'])]

#reshape data into dataframe
result = (pd.DataFrame(outcome, columns=['state','year','value'])
          .astype({"state":pd.api.types.CategoricalDtype(categories=states,ordered=True)})
          .pivot("state","year","value")
         )

【讨论】：

感谢@sammywemmy 证实了我的怀疑，并给出了很好的答案。我猜 json_normalize() 确实适用于仍然自包含的嵌套对象，就像文档pandas.pydata.org/pandas-docs/stable/reference/api/… 中的所有示例一样。此 json 文件提供两个维度 - 状态和年份 - 与数据分开，其嵌套方式与两个维度不同。因此，解析需要您在答案中描述的那种特定逻辑。感谢您的宝贵时间，谢谢！

【解决方案2】：

您也可以查看flatten_json。您可以使用它来展平任何类型的 JSON。 json_normalize 有多个记录路径的限制。

【讨论】：