我有混合级别的 JSON 在 python 中解析，键有问题

Question

我有一组嵌套的 JSON，到目前为止我正在执行以下操作：

r = session.get(search_url, auth=HTTPKerberosAuth(mutual_authentication=OPTIONAL), verify=False)
json_data = json.loads(r.content)
flattened_data = json_normalize(json_data['documents'])
print(list(flattened_data))

这会输出以下结果：

['affected_users', 'aggregatedLabels', 'aliases', 'assignedFolder', 'assigneeIdentity', 'attachments', 'authorizations', 'autoUpgrade.workingHours', 'conversation', 'createDate', 'dedupes', 'deleted', 'description', 'descriptionContentType', 'editCount', 'engagementList', 'extensions.backlog.priority', 'extensions.effort.effortEstimatedLocal.effort', 'extensions.effort.effortEstimatedLocal.unit', 'extensions.effort.effortEstimatedRecursiveSum.effort', 'extensions.effort.effortEstimatedRecursiveSum.unit', 'extensions.effort.effortRemainingLocalSum.effort', 'extensions.effort.effortRemainingLocalSum.unit', 'extensions.effort.effortRemainingRecursiveSum.effort', 'extensions.effort.effortRemainingRecursiveSum.unit', 'extensions.effort.effortSpentLocalSum.effort', 'extensions.effort.effortSpentLocalSum.unit', 'extensions.effort.effortSpentRecursiveSum.effort', 'extensions.effort.effortSpentRecursiveSum.unit', 'extensions.tt.assignedGroup', 'extensions.tt.building', 'extensions.tt.caseType', 'extensions.tt.category', 'extensions.tt.city', 'extensions.tt.endCode', 'extensions.tt.ecd', 'extensions.tt.impact', 'extensions.tt.item', 'extensions.tt.justification', 'extensions.tt.migrationStatus', 'extensions.tt.minImpact', 'extensions.tt.resolution', 'extensions.tt.rootCause', 'extensions.tt.rootCauseDetails', 'extensions.tt.status', 'extensions.tt.type', 'frames', 'id', 'identityTimestamped', 'inheritedLabels', 'isTicket', 'labels', 'lastAssignedDate', 'lastResolvedByIdentity', 'lastResolvedDate', 'lastUpdatedActualDate', 'lastUpdatedConversationDate', 'lastUpdatedDate', 'lastUpdatedIdentity', 'next_step.action', 'next_step.exceptions', 'next_step.owner', 'parentTasks', 'requesterIdentity', 'rootCauses', 'rulesReceipt', 'schedule.estimatedCompletionDate', 'schedule.estimatedStartDate', 'schedule.needByDate', 'schema', 'slaReceipts', 'status', 'stickyThreadId', 'submitterIdentity', 'subtasks', 'tags', 'threads', 'title', 'watchers']

从此列表中，我试图仅将某些键及其值放入数据框中：

    print(flattened_data['assigneeIdentity',
#                         'createDate',
#                         'description',
#                         'extensions.tt.assignedGroup',
#                         'extensions.tt.category',
#                         'extensions.tt.endCode',
#                         'extensions.tt.ecd',
#                         'extensions.tt.impact',
#                         'extensions.tt.item',
#                         'extensions.tt.justification',
#                         'extensions.tt.resolution',
#                         'extensions.tt.rootCause',
#                         'extensions.tt.rootCauseDetails',
#                         'extensions.tt.status',
#                         'extensions.tt.type',
#                         'id',
#                         'labels',
#                         'lastAssignedDate',
#                         'lastResolvedByIdentity',
#                         'lastResolvedDate',
#                         'lastUpdatedActualDate',
#                         'lastUpdatedConversationDate',
#                         'lastUpdatedDate',
#                         'lastUpdatedIdentity',
#                         'requesterIdentity',
#                         'submitterIdentity',
#                         'title',
#                         'watchers'])

当我这样做时，我得到一个关键错误。因此，对于我上面列出的字段以及每个字段的嵌套级别的想法，进来的基本 JSON 如下所示；每个“项目”都是文档元素下的整数，然后我需要更多嵌套元素：

documents:
          0:
             extensions:
                         tt:
                             category:
                             type:
                             item:
                             assignedGroup:
                             impact:
                             justification:
                             endCode:
                             rootCause:
                             rootCauseDetails:
                             status:
              id:
              title:
              lastAssignedDate:
              createDate:
              lastUpdatedActualDate:
              lastResolvedDate:
              lastResolvedByIdentity:
              lastUpdatedIdentity:
              assigneeIdentity:
              submitterIdentity:
              requesterIdentity:
              identityTimestamped:
              lastUpdatedConversationDate:
              lastUpdatedDate:
          1:
             extensions:
                         tt:
                             category:
                             type:
                             item:
                             assignedGroup:
                             impact:
                             justification:
                             endCode:
                             rootCause:
                             rootCauseDetails:
                             status:
              id:
              title:
              lastAssignedDate:
              createDate:
              lastUpdatedActualDate:
              lastResolvedDate:
              lastResolvedByIdentity:
              lastUpdatedIdentity:
              assigneeIdentity:
              submitterIdentity:
              requesterIdentity:
              identityTimestamped:
              lastUpdatedConversationDate:
              lastUpdatedDate:

如何将这个和值放入数据框中。

Answer 1

flattened_data 应该已经是一个有效的 DataFrame。错误似乎是您正在尝试打印flattened_data["key1", "key2", ...]，这将在flattened_data 中查找列命名 ["key1", "key2", ...]。本质上，您是在告诉 DataFrame “获取名称为 是此列表的列”。

要从 DataFrame 中获取列列表，您应该尝试flattened_data[["key1", "key2", ...]]，而不是说“获取名称在此列表中的所有列”.

这里还可能发生的情况是，您有一个包含 ["0.id", "0.title", ..., "1.id", "1.title", ...] 列的 DataFrame，其中只有一行：分配给 JSON 对象中每个路径的值。

但是，pandas.io.json.normalize_json() 可以将字典列表作为参数，因此不要使用 flattened_data = json_normalize(json_data['documents'])，而是使用 json_data['documents'] 中的子字典列表（例如，json_data['documents'].values()）应该返回正确的数据帧。

records = list(json_data['documents'].values())
flattened_data = json_normalize(records)

然后，您可以检索所需的列：

 print(flattened_data[['assigneeIdentity', 'createDate', 'description', 'extensions.tt.assignedGroup', ...]])

Answer 2

引用我今天刚刚评论的fantastic response 中的一些内容。也许这会有所帮助：

import pandas as pd

r = session.get(search_url, auth=HTTPKerberosAuth(mutual_authentication=OPTIONAL), verify=False)
data = r.json()
df = pd.DataFrame(data)
mask = df['assigneeIdentity'].apply(lambda x: '<your value to filter here>' in x)
df1 = df[mask] # The mask will return values that are True (i.e. - what you want)