如何检查整个 json 行中的重复项？

Question

我有一个包含 1000 行的 json，在这里我不会让你厌烦，但这是数据集的一个简单示例。

{
    "places": [
        {
            "place_name": "123 YOU N ME PRESCHOOL",
            "address": "809 W DETWEILLER DR STE A",
            "city": "PEORIA",
            "state": "IL",
            "zip": "61614",
            "geo_location": "40.89564657,-89.60566821",
            "data": {}
        },
        {
            "place_name": "123 YOU N ME PRESCHOOL",
            "address": "809 W DETWEILLER DR STE A",
            "city": "PEORIA",
            "state": "IL",
            "zip": "61615",
            "geo_location": "40.78878653,-89.605669034",
            "data": {}
        },
        {
            "place_name": "18144 GLEN TERRACE ST.",
            "address": "18144 GLEN TERRACE ST.",
            "city": "LANSING",
            "state": "IL",
            "zip": "60438",
            "geo_location": "41.565952019,-87.556316006",
            "data": {}
        }
    ]
}

如您所见，前两个地方几乎相同，但它们的拉链和地理位置不同，因此它们不是重复的。我目前有一个脚本，它只查看被重复的 place_name，但这会返回我实际上并不重复的地方。本质上，我希望我的脚本查看每一行，如果一行与另一行相同，则删除该行。 到目前为止，这是我的代码。

import collections
import pandas as pd
places = pd.read_json("test.json")
place_names = [item['place_name'] for item in places['places']]
print([item for item, count in collections.Counter(place_names).items() if count > 1])

Answer 1

正如@user696969 所指出的，that post 中已经有有用的答案。 @jcollado 接受的答案效果很好，但是你有一个字典作为字典中的值，它是键 data 的值。因此，在不对数据集进行任何更改的情况下运行那段代码将为您提供TypeError: unhashable type: 'dict'。要解决此问题，您可以实施以下检查

new_d = d.copy()
for k, v in d.items(): 
    if type(v) is dict: # if a value is a dictionary, convert that value to a tuple
        new_d[k] = tuple(v.items())

所以，整个代码就变成了

seen = set()
new_l = []
for d in l:
    new_d = d.copy()
    for k, v in d.items():
        if type(v) is dict:
            new_d[k] = tuple(v.items())
            
    t = tuple(new_d.items())
    if t not in seen:
        seen.add(t)
        new_l.append(d)

例如，设l为

l = [{
        "place_name": "123 YOU N ME PRESCHOOL",
        "address": "809 W DETWEILLER DR STE A",
        "city": "PEORIA",
        "state": "IL",
        "zip": "61614",
        "geo_location": "40.89564657,-89.60566821",
        "data": {}
    },
    {
        "place_name": "123 YOU N ME PRESCHOOL",
        "address": "809 W DETWEILLER DR STE A",
        "city": "PEORIA",
        "state": "IL",
        "zip": "61615",
        "geo_location": "40.78878653,-89.605669034",
        "data": {"hey": 1}
    },
    {
        "place_name": "18144 GLEN TERRACE ST.",
        "address": "18144 GLEN TERRACE ST.",
        "city": "LANSING",
        "state": "IL",
        "zip": "60438",
        "geo_location": "41.565952019,-87.556316006",
        "data": {"hey": 2}
    },
    {
        "place_name": "123 YOU N ME PRESCHOOL",
        "address": "809 W DETWEILLER DR STE A",
        "city": "PEORIA",
        "state": "IL",
        "zip": "61615",
        "geo_location": "40.78878653,-89.605669034",
        "data": {"hey": 1}
    },
]

我运行代码后，new_l 变成了

>>> new_l
    [{'place_name': '123 YOU N ME PRESCHOOL',
      'address': '809 W DETWEILLER DR STE A',
      'city': 'PEORIA',
      'state': 'IL',
      'zip': '61614',
      'geo_location': '40.89564657,-89.60566821',
      'data': {}},
     {'place_name': '123 YOU N ME PRESCHOOL',
      'address': '809 W DETWEILLER DR STE A',
      'city': 'PEORIA',
      'state': 'IL',
      'zip': '61615',
      'geo_location': '40.78878653,-89.605669034',
      'data': {'hey': 1}},
     {'place_name': '18144 GLEN TERRACE ST.',
      'address': '18144 GLEN TERRACE ST.',
      'city': 'LANSING',
      'state': 'IL',
      'zip': '60438',
      'geo_location': '41.565952019,-87.556316006',
      'data': {'hey': 2}}]

"data": {"hey": 1} 的字典现在只出现一次。如果您知道"data" 的值是唯一可以是字典的值，则删除for loop 并执行

seen = set()
new_l = []
for d in l:
    d["data"] = tuple(d["data"].items())
    t = tuple(d.items())
    if t not in seen:
        seen.add(t)
        d["data"] = {k: v for k, v in d["data"]} # back to dictionary
        new_l.append(d)

这将提高整体性能。

Answer 2

我认为没有更有效的方法。通常，要从列表中删除重复项，您可以使用 list(set(list_variable)) 但仅当 list_variable 仅包含可散列值时才有效，因为字典对象不可散列，因此您不能使用该方法。 this post 中提到了很多方法。我能想到的另一种方法是将每个place 的字典对象转换为字符串（str(place)），然后将它们用作字典的键，并且由于字典仅存储唯一的键字符串，因此您只会获得唯一的位置新字典

unique_places = list({str(place): place for place in places["places"]}.values())