【问题标题】:How to normalize my nested json properly?如何正确规范我的嵌套 json?
【发布时间】:2020-09-11 02:55:06
【问题描述】:

我有 json 文件(名为 list_of_beers),它是 2 种啤酒的列表:

[
    {
        "id": 1,
        "name": "Buzz",
        "tagline": "A Real Bitter Experience.",
        "first_brewed": "09/2007",
        "description": "A light, crisp and bitter IPA brewed with English and American hops. A small batch brewed only once.",
        "image_url": "https://images.punkapi.com/v2/keg.png",
        "abv": 4.5,
        "ibu": 60,
        "target_fg": 1010,
        "target_og": 1044,
        "ebc": 20,
        "srm": 10,
        "ph": 4.4,
        "attenuation_level": 75,
        "volume": {
            "value": 20,
            "unit": "litres"
        },
        "boil_volume": {
            "value": 25,
            "unit": "litres"
        },
        "method": {
            "mash_temp": [
                {
                    "temp": {
                        "value": 64,
                        "unit": "celsius"
                    },
                    "duration": 75
                }
            ],
            "fermentation": {
                "temp": {
                    "value": 19,
                    "unit": "celsius"
                }
            },
            "twist": null
        },
        "ingredients": {
            "malt": [
                {
                    "name": "Maris Otter Extra Pale",
                    "amount": {
                        "value": 3.3,
                        "unit": "kilograms"
                    }
                },
                {
                    "name": "Caramalt",
                    "amount": {
                        "value": 0.2,
                        "unit": "kilograms"
                    }
                },
                {
                    "name": "Munich",
                    "amount": {
                        "value": 0.4,
                        "unit": "kilograms"
                    }
                }
            ],
            "hops": [
                {
                    "name": "Fuggles",
                    "amount": {
                        "value": 25,
                        "unit": "grams"
                    },
                    "add": "start",
                    "attribute": "bitter"
                },
                {
                    "name": "First Gold",
                    "amount": {
                        "value": 25,
                        "unit": "grams"
                    },
                    "add": "start",
                    "attribute": "bitter"
                },
                {
                    "name": "Fuggles",
                    "amount": {
                        "value": 37.5,
                        "unit": "grams"
                    },
                    "add": "middle",
                    "attribute": "flavour"
                },
                {
                    "name": "First Gold",
                    "amount": {
                        "value": 37.5,
                        "unit": "grams"
                    },
                    "add": "middle",
                    "attribute": "flavour"
                },
                {
                    "name": "Cascade",
                    "amount": {
                        "value": 37.5,
                        "unit": "grams"
                    },
                    "add": "end",
                    "attribute": "flavour"
                }
            ],
            "yeast": "Wyeast 1056 - American Ale\u2122"
        },
        "food_pairing": [
            "Spicy chicken tikka masala",
            "Grilled chicken quesadilla",
            "Caramel toffee cake"
        ],
        "brewers_tips": "The earthy and floral aromas from the hops can be overpowering. Drop a little Cascade in at the end of the boil to lift the profile with a bit of citrus.",
        "contributed_by": "Sam Mason <samjbmason>"
    },
    {
        "id": 2,
        "name": "Trashy Blonde",
        "tagline": "You Know You Shouldn't",
        "first_brewed": "04/2008",
        "description": "A titillating, neurotic, peroxide punk of a Pale Ale. Combining attitude, style, substance, and a little bit of low self esteem for good measure; what would your mother say? The seductive lure of the sassy passion fruit hop proves too much to resist. All that is even before we get onto the fact that there are no additives, preservatives, pasteurization or strings attached. All wrapped up with the customary BrewDog bite and imaginative twist.",
        "image_url": "https://images.punkapi.com/v2/2.png",
        "abv": 4.1,
        "ibu": 41.5,
        "target_fg": 1010,
        "target_og": 1041.7,
        "ebc": 15,
        "srm": 15,
        "ph": 4.4,
        "attenuation_level": 76,
        "volume": {
            "value": 20,
            "unit": "litres"
        },
        "boil_volume": {
            "value": 25,
            "unit": "litres"
        },
        "method": {
            "mash_temp": [
                {
                    "temp": {
                        "value": 69,
                        "unit": "celsius"
                    },
                    "duration": null
                }
            ],
            "fermentation": {
                "temp": {
                    "value": 18,
                    "unit": "celsius"
                }
            },
            "twist": null
        },
        "ingredients": {
            "malt": [
                {
                    "name": "Maris Otter Extra Pale",
                    "amount": {
                        "value": 3.25,
                        "unit": "kilograms"
                    }
                },
                {
                    "name": "Caramalt",
                    "amount": {
                        "value": 0.2,
                        "unit": "kilograms"
                    }
                },
                {
                    "name": "Munich",
                    "amount": {
                        "value": 0.4,
                        "unit": "kilograms"
                    }
                }
            ],
            "hops": [
                {
                    "name": "Amarillo",
                    "amount": {
                        "value": 13.8,
                        "unit": "grams"
                    },
                    "add": "start",
                    "attribute": "bitter"
                },
                {
                    "name": "Simcoe",
                    "amount": {
                        "value": 13.8,
                        "unit": "grams"
                    },
                    "add": "start",
                    "attribute": "bitter"
                },
                {
                    "name": "Amarillo",
                    "amount": {
                        "value": 26.3,
                        "unit": "grams"
                    },
                    "add": "end",
                    "attribute": "flavour"
                },
                {
                    "name": "Motueka",
                    "amount": {
                        "value": 18.8,
                        "unit": "grams"
                    },
                    "add": "end",
                    "attribute": "flavour"
                }
            ],
            "yeast": "Wyeast 1056 - American Ale\u2122"
        },
        "food_pairing": [
            "Fresh crab with lemon",
            "Garlic butter dipping sauce",
            "Goats cheese salad",
            "Creamy lemon bar doused in powdered sugar"
        ],
        "brewers_tips": "Be careful not to collect too much wort from the mash. Once the sugars are all washed out there are some very unpleasant grainy tasting compounds that can be extracted into the wort.",
        "contributed_by": "Sam Mason <samjbmason>"
    }
]

我正在尝试使用 Pandas 库中的 json_normalize() 对其进行规范化:

beers_after_normalization = json_normalize(data=list_of_beers)

输出是:

| id  |     name       |          tagline           | first_brewed  |                                                                                                                                                                                                                          description                                                                                                                                                                                                                           |               image_url                | abv  | ibu   | target_fg  | target_og  | ebc   | srm   | ph   | attenuation_level  |                                                    food_pairing                                                      |                                                                                      brewers_tips                                                                                       |     contributed_by      | volume.value  | volume.unit  | boil_volume.value  | boil_volume.unit  |                        method.mash_temp                         | method.fermentation.temp.value  | method.fermentation.temp.unit  | method.twist  |                                                                                                       ingredients.malt                                                                                                        |                                                                                                                                                                                                                                                               ingredients.hops                                                                                                                                                                                                                                                                 |      ingredients.yeast      |
+=====+================+============================+===============+================================================================================================================================================================================================================================================================================================================================================================================================================================================================+========================================+======+=======+============+============+=======+=======+======+====================+======================================================================================================================+=========================================================================================================================================================================================+=========================+===============+==============+====================+===================+=================================================================+=================================+================================+===============+===============================================================================================================================================================================================================================+================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================+=============================+
|  1  | Buzz           | A Real Bitter Experience.  | 09/2007       | A light, crisp and bitter IPA brewed with English and American hops. A small batch brewed only once.                                                                                                                                                                                                                                                                                                                                                           | https://images.punkapi.com/v2/keg.png  | 4.5  | 60.0  |    1010.0  |    1044.0  | 20.0  | 10.0  | 4.4  |              75.0  | [Spicy chicken tikka masala, Grilled chicken quesadilla, Caramel toffee cake]                                        | The earthy and floral aromas from the hops can be overpowering. Drop a little Cascade in at the end of the boil to lift the profile with a bit of citrus.                               | Sam Mason <samjbmason>  |           20  | litres       |                25  | litres            | [{'temp': {'value': 64, 'unit': 'celsius'}, 'duration': 75}]    |                           19.0  | celsius                        | None          | [{'name': 'Maris Otter Extra Pale', 'amount': {'value': 3.3, 'unit': 'kilograms'}}, {'name': 'Caramalt', 'amount': {'value': 0.2, 'unit': 'kilograms'}}, {'name': 'Munich', 'amount': {'value': 0.4, 'unit': 'kilograms'}}]   | [{'name': 'Fuggles', 'amount': {'value': 25, 'unit': 'grams'}, 'add': 'start', 'attribute': 'bitter'}, {'name': 'First Gold', 'amount': {'value': 25, 'unit': 'grams'}, 'add': 'start', 'attribute': 'bitter'}, {'name': 'Fuggles', 'amount': {'value': 37.5, 'unit': 'grams'}, 'add': 'middle', 'attribute': 'flavour'}, {'name': 'First Gold', 'amount': {'value': 37.5, 'unit': 'grams'}, 'add': 'middle', 'attribute': 'flavour'}, {'name': 'Cascade', 'amount': {'value': 37.5, 'unit': 'grams'}, 'add': 'end', 'attribute': 'flavour'}]  | Wyeast 1056 - American Ale™ |
+-----+----------------+----------------------------+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------+------+-------+------------+------------+-------+-------+------+--------------------+----------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+---------------+--------------+--------------------+-------------------+-----------------------------------------------------------------+---------------------------------+--------------------------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------+
|  2  | Trashy Blonde  | You Know You Shouldn't     | 04/2008       | A titillating, neurotic, peroxide punk of a Pale Ale. Combining attitude, style, substance, and a little bit of low self esteem for good measure; what would your mother say? The seductive lure of the sassy passion fruit hop proves too much to resist. All that is even before we get onto the fact that there are no additives, preservatives, pasteurization or strings attached. All wrapped up with the customary BrewDog bite and imaginative twist.  | https://images.punkapi.com/v2/2.png    | 4.1  | 41.5  |    1010.0  |    1041.7  | 15.0  | 15.0  | 4.4  |              76.0  | [Fresh crab with lemon, Garlic butter dipping sauce, Goats cheese salad, Creamy lemon bar doused in powdered sugar]  | Be careful not to collect too much wort from the mash. Once the sugars are all washed out there are some very unpleasant grainy tasting compounds that can be extracted into the wort.  | Sam Mason <samjbmason>  |           20  | litres       |                25  | litres            | [{'temp': {'value': 69, 'unit': 'celsius'}, 'duration': None}]  |                           18.0  | celsius                        | None          | [{'name': 'Maris Otter Extra Pale', 'amount': {'value': 3.25, 'unit': 'kilograms'}}, {'name': 'Caramalt', 'amount': {'value': 0.2, 'unit': 'kilograms'}}, {'name': 'Munich', 'amount': {'value': 0.4, 'unit': 'kilograms'}}]  | [{'name': 'Amarillo', 'amount': {'value': 13.8, 'unit': 'grams'}, 'add': 'start', 'attribute': 'bitter'}, {'name': 'Simcoe', 'amount': {'value': 13.8, 'unit': 'grams'}, 'add': 'start', 'attribute': 'bitter'}, {'name': 'Amarillo', 'amount': {'value': 26.3, 'unit': 'grams'}, 'add': 'end', 'attribute': 'flavour'}, {'name': 'Motueka', 'amount': {'value': 18.8, 'unit': 'grams'}, 'add': 'end', 'attribute': 'flavour'}]                                                                                                                | Wyeast 1056 - American Ale™ |
+-----+----------------+----------------------------+---------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------+------+-------+------------+------------+-------+-------+------+--------------------+----------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------------------------+---------------+--------------+--------------------+-------------------+-----------------------------------------------------------------+---------------------------------+--------------------------------+---------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------+

但正如您所见,仍有几列需要规范化:method.mash_tempingredients.maltingredients.hops。我的问题是如何做到这一点?我尝试使用:

json_normalize(data=list_of_beers, record_path = ['method.mash_temp', 'ingredients.malt',
'ingredients.hops'], meta=['id', 'name', 'tagline', 'first_brewed', 'description', 'image_url',
       'abv', 'ibu', 'target_fg', 'target_og', 'ebc', 'srm', 'ph',
       'attenuation_level', 'food_pairing', 'brewers_tips', 'contributed_by',
       'volume.value', 'volume.unit', 'boil_volume.value', 'boil_volume.unit',
       'method.fermentation.temp.value',
       'method.fermentation.temp.unit', 'method.twist',
       'ingredients.yeast'])

但不幸的是,有一些 KeyErrors ;/ 我还尝试首先规范化我的 3 个嵌套列(即 method.mash_tempingredients.maltingredients.hops),然后将每一列加入我的表中,但老实说我将我的表与这 3 列合并-.- 如果有人向我展示解决此问题的不同方法,那就太好了。提前致谢!

【问题讨论】:

    标签: python json pandas nested normalization


    【解决方案1】:

    您必须为同一级别的每个记录路径创建单独的数据框,然后根据键合并它们,或者将它们保留为单独的数据框:

    df_hops = pd.json_normalize(data['listofbeers'],
                           record_path=['ingredients', 'hops'], 
                           meta=[['id'], ['name']],
                           record_prefix='hops.')
    
    df_malt = pd.json_normalize(data['listofbeers'],
                           record_path=['ingredients', 'malt'], 
                           meta=[['id'], ['name']],
                           record_prefix='malt.')
    
    df_method = pd.json_normalize(data['listofbeers'],
                           record_path=['method', 'mash_temp'], 
                           meta=[['id'], ['name']],
                           record_prefix='method.')
    
    df = pd.concat([df_hops, df_malt], join='inner', axis=1)
    print(df)
    

    输出:

       hops.name hops.add hops.attribute  hops.amount.value hops.amount.unit  ...               malt.name malt.amount.value malt.amount.unit  id           name
    0     Fuggles    start         bitter               25.0            grams  ...  Maris Otter Extra Pale              3.30        kilograms   1           Buzz
    1  First Gold    start         bitter               25.0            grams  ...                Caramalt              0.20        kilograms   1           Buzz
    2     Fuggles   middle        flavour               37.5            grams  ...                  Munich              0.40        kilograms   1           Buzz
    3  First Gold   middle        flavour               37.5            grams  ...  Maris Otter Extra Pale              3.25        kilograms   2  Trashy Blonde
    4     Cascade      end        flavour               37.5            grams  ...                Caramalt              0.20        kilograms   2  Trashy Blonde
    5    Amarillo    start         bitter               13.8            grams  ...                  Munich              0.40        kilograms   2  Trashy Blonde
    

    另一种方法是使用flatten_json,文档在此处-> Flatten JSON docs,您可以基于此模块创建自己的递归函数。

    【讨论】:

      猜你喜欢
      • 2021-05-19
      • 2018-04-09
      • 1970-01-01
      • 2021-07-23
      • 2017-10-27
      • 2018-04-03
      • 2020-11-09
      • 1970-01-01
      • 1970-01-01
      相关资源
      最近更新 更多