【问题标题】:Reducing the complexity of getting unique values from dict in Python降低 Python 中从 dict 获取唯一值的复杂性
【发布时间】:2019-12-06 11:25:31
【问题描述】:

我对 Python 相当陌生(并且编写了良好且高效的算法),并且不太熟悉可用于有效迭代大量数据的不同数据结构。我需要从嵌套字典中找到唯一的一组值,并编写了以下代码:

data = {'c14da622-7fb8-4da3-a2fb-d8c632957fbe': {'25': {'label': 'no plane'}, '50': {'label': 'no plane'}, '125': {'label': 'no plane'}, '150': {'label': 'no plane'}, '175': {'label': 'plane'}, '200': {'label': 'plane'}, '275': {'label': 'plane'}, '300': {'label': 'plane'}, '325': {'label': 'plane'}, '350': {'label': 'plane'}, '375': {'label': 'plane'}, '400': {'label': 'plane'}, '425': {'label': 'plane'}, '450': {'label': 'plane'}, '475': {'label': 'plane'}, '500': {'label': 'plane'}, '525': {'label': 'plane'}, '550': {'label': 'plane'}, '575': {'label': 'plane'}, '600': {'label': 'plane'}, '625': {'label': 'plane'}, '650': {'label': 'plane'}, '875': {'label': 'plane'}, '900': {'label': 'plane'}, '925': {'label': 'plane'}, '950': {'label': 'plane'}, '975': {'label': 'plane'}, '1000': {'label': 'plane'}, '1025': {'label': 'plane'}, '1050': {'label': 'plane'}, '1075': {'label': 'plane'}, '1100': {'label': 'plane'}, '1125': {'label': 'plane'}, '1150': {'label': 'plane'}, '1175': {'label': 'plane'}}, '60cb59c7-6b0a-4225-b00f-2d888a9d5250': {'30': {'label': 'no plane'}, '60': {'label': 'no plane'}, '90': {'label': 'no plane'}, '120': {'label': 'no plane'}, '150': {'label': 'no plane'}, '180': {'label': 'plane'}, '210': {'label': 'plane'}, '240': {'label': 'plane'}, '270': {'label': 'plane'}, '300': {'label': 'plane'}, '330': {'label': 'plane'}, '360': {'label': 'plane'}, '390': {'label': 'plane'}, '420': {'label': 'plane'}, '450': {'label': 'plane'}, '480': {'label': 'plane'}, '510': {'label': 'plane'}, '570': {'label': 'plane'}, '600': {'label': 'plane'}, '660': {'label': 'plane'}, '690': {'label': 'plane'}, '720': {'label': 'plane crash'}, '750': {'label': 'plane crash'}, '780': {'label': 'plane crash'}, '810': {'label': 'plane crash'}, '840': {'label': 'plane crash'}, '870': {'label': 'plane crash'}, '900': {'label': 'plane crash'}, '930': {'label': 'plane crash'}, '960': {'label': 'plane crash'}, '990': {'label': 'no plane'}, '1020': {'label': 'plane crash'}, '1050': {'label': 'plane crash'}, '1080': {'label': 'plane crash'}, '1110': {'label': 'plane crash'}, '1140': {'label': 'plane crash'}, '1170': {'label': 'plane crash'}, '1200': {'label': 'plane crash'}, '1230': {'label': 'plane crash'}, '1260': {'label': 'plane crash'}, '1290': {'label': 'plane crash'}, '1320': {'label': 'plane crash'}, '1350': {'label': 'plane crash'}, '1380': {'label': 'plane crash'}, '1410': {'label': 'plane crash'}, '1560': {'label': 'plane crash'}, '1590': {'label': 'plane crash'}, '1620': {'label': 'plane crash'}, '1650': {'label': 'plane crash'}, '1680': {'label': 'plane crash'}, '1710': {'label': 'plane crash'}}}

def parse_label_categories(data):
    tuples = list(data.values())
    unique_labels = []
    for labels in tuples:
        labels_dump = list(labels.values())
        for dump in labels_dump:
            label = list(dump.values())
            new = label.pop()
            unique_labels.append(new)
    return list(set(unique_labels))

parse_label_categories(data)

返回三个唯一值:

['plane crash', 'plane', 'no plane']

我有一个嵌套的 for 循环,总的来说我的代码非常糟糕,但我一直很难在 Python 中找到一个更优雅、更有效的解决方案。

任何帮助/建议将不胜感激:-)

【问题讨论】:

  • 如果您不将.values 转换为列表,您将降低空间复杂度 - 转换会创建新对象。您还可以对集合 unique_labels 进行操作,而不是列表 - 现在列表包含循环中的每个标签,而不是唯一标签。
  • 另一个问题 - 你不必弹出,你知道“labels_dump”有多少项目 - 正是一个,标记为“label”。 |两个for循环没问题。通过我提到的修复,您甚至可以将其转换为单行集合理解:unique_labels = {labelled["label"] for subdict in data.values() for labelled in subdict.values()}(或者让您的名字远离循环:unique_labels = {labels_dump["label"] for labels in data.values() for labels_dump in labels.values()} - 但我认为我的名字更合乎逻辑:P)
  • 感谢这些建议 :-) 我正在摆弄重写。我喜欢单行集合理解。

标签: python list data-structures time-complexity


【解决方案1】:

专业提示:jsonlint 会将数据格式化为可读格式,即使该 JSON 已被解析为 python 列表/字典。

data = {'c14da622-7fb8-4da3-a2fb-d8c632957fbe': {'25': {'label': 'no plane'}, '50': {'label': 'no plane'}, '125': {'label': 'no plane'}, '150': {'label': 'no plane'}, '175': {'label': 'plane'}, '200': {'label': 'plane'}, '275': {'label': 'plane'}, '300': {'label': 'plane'}, '325': {'label': 'plane'}, '350': {'label': 'plane'}, '375': {'label': 'plane'}, '400': {'label': 'plane'}, '425': {'label': 'plane'}, '450': {'label': 'plane'}, '475': {'label': 'plane'}, '500': {'label': 'plane'}, '525': {'label': 'plane'}, '550': {'label': 'plane'}, '575': {'label': 'plane'}, '600': {'label': 'plane'}, '625': {'label': 'plane'}, '650': {'label': 'plane'}, '875': {'label': 'plane'}, '900': {'label': 'plane'}, '925': {'label': 'plane'}, '950': {'label': 'plane'}, '975': {'label': 'plane'}, '1000': {'label': 'plane'}, '1025': {'label': 'plane'}, '1050': {'label': 'plane'}, '1075': {'label': 'plane'}, '1100': {'label': 'plane'}, '1125': {'label': 'plane'}, '1150': {'label': 'plane'}, '1175': {'label': 'plane'}}, '60cb59c7-6b0a-4225-b00f-2d888a9d5250': {'30': {'label': 'no plane'}, '60': {'label': 'no plane'}, '90': {'label': 'no plane'}, '120': {'label': 'no plane'}, '150': {'label': 'no plane'}, '180': {'label': 'plane'}, '210': {'label': 'plane'}, '240': {'label': 'plane'}, '270': {'label': 'plane'}, '300': {'label': 'plane'}, '330': {'label': 'plane'}, '360': {'label': 'plane'}, '390': {'label': 'plane'}, '420': {'label': 'plane'}, '450': {'label': 'plane'}, '480': {'label': 'plane'}, '510': {'label': 'plane'}, '570': {'label': 'plane'}, '600': {'label': 'plane'}, '660': {'label': 'plane'}, '690': {'label': 'plane'}, '720': {'label': 'plane crash'}, '750': {'label': 'plane crash'}, '780': {'label': 'plane crash'}, '810': {'label': 'plane crash'}, '840': {'label': 'plane crash'}, '870': {'label': 'plane crash'}, '900': {'label': 'plane crash'}, '930': {'label': 'plane crash'}, '960': {'label': 'plane crash'}, '990': {'label': 'no plane'}, '1020': {'label': 'plane crash'}, '1050': {'label': 'plane crash'}, '1080': {'label': 'plane crash'}, '1110': {'label': 'plane crash'}, '1140': {'label': 'plane crash'}, '1170': {'label': 'plane crash'}, '1200': {'label': 'plane crash'}, '1230': {'label': 'plane crash'}, '1260': {'label': 'plane crash'}, '1290': {'label': 'plane crash'}, '1320': {'label': 'plane crash'}, '1350': {'label': 'plane crash'}, '1380': {'label': 'plane crash'}, '1410': {'label': 'plane crash'}, '1560': {'label': 'plane crash'}, '1590': {'label': 'plane crash'}, '1620': {'label': 'plane crash'}, '1650': {'label': 'plane crash'}, '1680': {'label': 'plane crash'}, '1710': {'label': 'plane crash'}}}

def parse_label_categories(data):
    seen = set()
    for some_lable, data_dict in data.items():
        for some_number, outcome in data_dict.items():
            seen.add(outcome['label'])
    return seen

a = parse_label_categories(data)

我认为在 Python 中没有更有效的方法。您也许可以使用 pandas 并可能将循环推送到 C 中,因为它将 JSON 扩展为数据框,但我不相信。


自从熊猫方法确实出现后,我做了时间安排:

import pandas as pd


data = {'c14da622-7fb8-4da3-a2fb-d8c632957fbe': {'25': {'label': 'no plane'}, '50': {'label': 'no plane'}, '125': {'label': 'no plane'}, '150': {'label': 'no plane'}, '175': {'label': 'plane'}, '200': {'label': 'plane'}, '275': {'label': 'plane'}, '300': {'label': 'plane'}, '325': {'label': 'plane'}, '350': {'label': 'plane'}, '375': {'label': 'plane'}, '400': {'label': 'plane'}, '425': {'label': 'plane'}, '450': {'label': 'plane'}, '475': {'label': 'plane'}, '500': {'label': 'plane'}, '525': {'label': 'plane'}, '550': {'label': 'plane'}, '575': {'label': 'plane'}, '600': {'label': 'plane'}, '625': {'label': 'plane'}, '650': {'label': 'plane'}, '875': {'label': 'plane'}, '900': {'label': 'plane'}, '925': {'label': 'plane'}, '950': {'label': 'plane'}, '975': {'label': 'plane'}, '1000': {'label': 'plane'}, '1025': {'label': 'plane'}, '1050': {'label': 'plane'}, '1075': {'label': 'plane'}, '1100': {'label': 'plane'}, '1125': {'label': 'plane'}, '1150': {'label': 'plane'}, '1175': {'label': 'plane'}}, '60cb59c7-6b0a-4225-b00f-2d888a9d5250': {'30': {'label': 'no plane'}, '60': {'label': 'no plane'}, '90': {'label': 'no plane'}, '120': {'label': 'no plane'}, '150': {'label': 'no plane'}, '180': {'label': 'plane'}, '210': {'label': 'plane'}, '240': {'label': 'plane'}, '270': {'label': 'plane'}, '300': {'label': 'plane'}, '330': {'label': 'plane'}, '360': {'label': 'plane'}, '390': {'label': 'plane'}, '420': {'label': 'plane'}, '450': {'label': 'plane'}, '480': {'label': 'plane'}, '510': {'label': 'plane'}, '570': {'label': 'plane'}, '600': {'label': 'plane'}, '660': {'label': 'plane'}, '690': {'label': 'plane'}, '720': {'label': 'plane crash'}, '750': {'label': 'plane crash'}, '780': {'label': 'plane crash'}, '810': {'label': 'plane crash'}, '840': {'label': 'plane crash'}, '870': {'label': 'plane crash'}, '900': {'label': 'plane crash'}, '930': {'label': 'plane crash'}, '960': {'label': 'plane crash'}, '990': {'label': 'no plane'}, '1020': {'label': 'plane crash'}, '1050': {'label': 'plane crash'}, '1080': {'label': 'plane crash'}, '1110': {'label': 'plane crash'}, '1140': {'label': 'plane crash'}, '1170': {'label': 'plane crash'}, '1200': {'label': 'plane crash'}, '1230': {'label': 'plane crash'}, '1260': {'label': 'plane crash'}, '1290': {'label': 'plane crash'}, '1320': {'label': 'plane crash'}, '1350': {'label': 'plane crash'}, '1380': {'label': 'plane crash'}, '1410': {'label': 'plane crash'}, '1560': {'label': 'plane crash'}, '1590': {'label': 'plane crash'}, '1620': {'label': 'plane crash'}, '1650': {'label': 'plane crash'}, '1680': {'label': 'plane crash'}, '1710': {'label': 'plane crash'}}}

def parse_label_categories(data):
    seen = set()
    for some_lable, data_dict in data.items():
        for some_number, outcome in data_dict.items():
            seen.add(outcome['label'])
    return seen


def pandas_approach(d):
    all_df=None 
    for id, d in data.items():
        df = pd.DataFrame.from_dict(d, orient="index")
        if all_df is None:
            all_df = df
        else:
            all_df = pd.concat([all_df, df])

这给出了:

%timeit parse_label_categories(data)
18 µs ± 2.31 µs per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit pandas_approach(data)
2.7 ms ± 156 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

【讨论】:

  • 谢谢你 - 我一般不喜欢使用 pandas,因为我以前尝试过使用它们,而且它们似乎总是非常慢。没想到jsonlist的方法,很优雅。而且看起来超级快:-) 非常感谢!如果我可以问,你是如何计时的?
  • @ulsha 很多关于 pandas API 的事情都是不直观的。它实际上确实在许多不同的任务上执行得更快,但在这种情况下,你不需要很多额外的开销
  • @ulsha 我使用 Spyder 作为我的 IDE。它带有 IPython 和它拥有的 "magic" 方法。 timeit 也是一个模块,但 IPython 让一些事情变得更容易
  • 很好 - 我会检查 timeit。谢谢!
【解决方案2】:

您可以将其转换为数据框并显着减少您需要执行的循环量。您仍然需要在第一级循环然后连接结果,但由于其余部分将在后台使用矢量化,因此速度可能会明显更快:

import pandas as pd
all_df=None 
for id, d in data.items():
    df = pd.DataFrame.from_dict(d, orient="index")
    if all_df is None:
        all_df = df
    else:
        all_df = pd.concat([all_df, df])

print(all_df["label"].unique())

【讨论】:

  • 大声笑,你在我回答的最后测试了这个理论。现在别无选择,只能计时:)
猜你喜欢
  • 1970-01-01
  • 1970-01-01
  • 2021-09-22
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
  • 1970-01-01
相关资源
最近更新 更多