从列表列表创建字典

Question

我有一个我读入的文本文件。这是一个日志文件，因此它遵循特定的模式。我最终需要创建一个 JSON，但是通过研究这个问题，一旦它在一个 dict 中，就可以使用json.loads() 或json.dumps()。

文本文件的示例如下。

INFO:20180606_141527:submit:is_test=False
INFO:20180606_141527:submit:username=Mary
INFO:20180606_141527:env:sys.platform=linux2
INFO:20180606_141527:env:os.name=ubuntu

我最终要寻找的 dict 结构是

{
  "INFO": {
    "submit": {
      "is_test": false,
      "username": "Mary"
    },
    "env": {
      "sys.platform": "linux2",
      "os.name": "ubuntu"
    }
  }
}

我现在忽略每个列表中的时间戳信息。

这是我正在使用的代码的 sn-p，

import csv
tree_dict = {}
with open('file.log') as file:
    for row in file:
        for key in reversed(row.split(":")):
            tree_dict = {key: tree_dict}

这会导致不希望的输出，

{'INFO': {'20180606_141527': {'submit': {'os.name=posix\n': {'INFO': {'20180606_141527': {'submit': {'sys.platform=linux2\n': {'INFO': {'20180606_141527': {'submit': {'username=a227874\n': {'INFO': {'20180606_141527': {'submit': {'is_test=False\n': {}}}}}}}}}}}}}}}}}

我需要动态填充字典，因为我不知道实际的字段/键名。

Answer 1

你可以使用itertools.groupby:

import itertools, re
content = [re.split('\=|:', i.strip('\n')) for i in open('filename.txt')]
new_content = [[a, *c] for a, _, *c in content]
def group_vals(d):
  new_d = [[a, [c for _, *c in b]] for a, b in itertools.groupby(sorted(d, key=lambda x:x[0]), key=lambda x:x[0])]
  return {a:b[0][0] if len(b) ==1 else group_vals(b) for a, b in new_d}

---

import json
print(json.dumps(group_vals(new_content), indent=4))

输出：

{
 "INFO": {
     "env": {
        "os.name": "ubuntu",
        "sys.platform": "linux2"
     },
     "submit": {
         "is_test": "False",
         "username": "Mary"
     }
  }
}

Answer 2

with open('demo.txt') as f:
    lines = f.readlines()

dct = {}

for line in lines:
    # param1 == INFO
    # param2 == submit or env
    # params3 == is_test=False etc.
    param1, _, param2, params3 = line.strip().split(':')

    # create dct[param1] = {} if it is not created
    dct.setdefault(param1, {})

    # create dct[param1][param2] = {} if it is no created
    dct[param1].setdefault(param2, {})

    # for example params3 == is_test=False
    # split it by '=' and now we unpack it
    # k == is_test
    # v == False
    k, v = params3.split('=')

    # and update our `dict` with the new values
    dct[param1][param2].update({k: v})

print(dct)

输出

{
'INFO': {
    'submit': {
        'is_test': 'False', 'username': 'Mary'
        }, 
    'env': {
        'sys.platform': 'linux2', 'os.name': 'ubuntu'
        }
    }
}  

Answer 3

您可以在这里使用嵌套的collections.defaultdict()：

from collections import defaultdict
from pprint import pprint

d = defaultdict(lambda: defaultdict(dict))
with open('sample.txt') as in_file:
    for line in in_file:
        info, _, category, pair = line.strip().split(':')
        props, value = pair.split('=')
        d[info][category][props] = value

pprint(d)

这给出了以下内容：

defaultdict(<function <lambda> at 0x7ff8a341aea0>,
            {'INFO': defaultdict(<class 'dict'>,
                                 {'env': {'os.name': 'ubuntu',
                                          'sys.platform': 'linux2'},
                                  'submit': {'is_test': 'False',
                                             'username': 'Mary'}})})

注意： defaultdict() 是内置 dict 的子类，因此最终结果不是将其转换为 dict 的理由。此外，defaultdict() 也可以使用json.dumps() 序列化为 JSON。

Answer 4

来源：

import os

with open('file.log') as file:
    tree_dict = {}
    is_test = False
    username = ""              
    sysplatform = ""
    osname = ""
    for row in file: 
        row = row.rstrip('\n')
        for key in reversed(row.split(":")):            
            if not key.find('is_test'):
                is_test = key.split('=')[1]
            elif not key.find('username'):
                username =key.split('=')[1]
            elif not key.find('sys.platform'):
                sysplatform = key.split('=')[1]
            elif not key.find('os.name'):
                osname = key.split('=')[1]    

     tree_dict = {
         "INFO": {
              "submit": {
                       "is_test": is_test,
                        "username": username
              },
              "env": {
                      "sys.platform":  sysplatform,
                      "os.name": osname
             }
        }
    }
    print(tree_dict)

结果：

 {'INFO': {'submit': {'is_test': 'False', 'username': 'Mary'}, 'env': {'sys.platform': 'linux2', 'os.name': 'ubuntu'}}}

Answer 5

import re
from functools import reduce

with open('file.txt') as f:
    lines = f.readlines()

def rec_merge(d1, d2):
    for k, v in d1.items():
        if k in d2:
            d2[k] = rec_merge(v, d2[k])
    d3 = d1.copy()
    d3.update(d2)
    return d3

lst_of_tup = re.findall(r'^([^:]*):[\d_]+:([^:]*):([^=]*)=(.*)$', lines, re.MULTILINE)
lst_of_dct = [reduce(lambda x,y: {y:x}, reversed(t)) for t in lst_of_tup]

dct = reduce(rec_merge, lst_of_dct)

pprint(dct)
# {'INFO': {'env': {'os.name': 'ubuntu', 'sys.platform': 'linux2'},
#           'submit': {'is_test': 'False', 'username': 'Mary'}}}

Answer 6

这是 Python 中递归似乎合适且有用的罕见情况之一。以下函数将value 添加到keys 列表指定的分层字典d：

def add_to_dict(d, keys, value): 
    if len(keys) == 1: # The last key
        d[keys[0]] = value
        return
    if keys[0] not in d:
        d[keys[0]] = {} # Create a new subdict
    add_to_dict(d[keys[0]], keys[1:], value)

该函数适用于任意深度的字典。剩下的就是调用函数的事情了：

d = {}
for line in file:
    keys, value = line.split("=")
    keys = keys.split(":")
    add_to_dict(d, keys, value.strip())

结果：

{'INFO': {'20180606_141527': {
                       'submit': {'is_test': 'False', 
                                  'username': 'Mary'}, 
                       'env': {'sys.platform': 'linux2', 
                               'os.name': 'ubuntu'}}}}

您可以修改代码以排除某些级别（如时间戳）。

Answer 7

检查是否存在密钥：

import csv
import json

tree_dict = {}
with open('file.log') as file:
    tree_dict = {}
    for row in file:
        keys = row.split(":")

        if keys[0] not in tree_dict:
            tree_dict[keys[0]] = {}

        if keys[-2] not in tree_dict[keys[0]]:
            tree_dict[keys[0]][keys[-2]] = {}

        key, value = keys[-1].split("=")

        if value == "False":
            value = False
        if value == "True":
            value = True

        tree_dict[keys[0]][keys[-2]][key] = value

dumped = json.dumps(tree_dict)