【问题标题】:Pandas semi structured JSON data frame to simple Pandas dataframePandas 半结构化 JSON 数据帧到简单 Pandas 数据帧
【发布时间】:2015-11-12 18:43:25
【问题描述】:

我有一个从红移集群中获取的数据。 前 4 列由“|”分隔那么 2 列是 JSON。

XXX|ABANDONED|1197|11|"{""currency"":""EUR""    item_id"":""143""   type"":""FLIGHT""   name"":""PAR-FEZ""  price"":1111    origin"":""PAR""    destination"":""FEZ""   merchant"":""GOV""  flight_type"":""OW""    flight_segment"":[{ origin"":""ORY""    destination"":""FEZ""   departure_date_time"":""2015-08-02T07:20""  arrival_date_time"":""2015-08-02T09:05""    carrier"":""AT""    f_class"":""ECONOMY""}]}"|"{""type"":""FLIGHT"" name"":""FI_ORY-OUD""   item_id"":""FLIGHT""    currency"":""EUR""  price"":111 origin"":""ORY""    destination"":""OUD""   flight_type"":""OW""    flight_segment"":[{""origin"":""ORY""   destination"":""OUD""   departure_date_time"":""2015-08-02T13:55""  arrival_date_time"":""2015-08-02T15:30""    flight_number"":""AT625""   carrier"":""AT""    f_class"":""ECONOMIC_DISCOUNTED""}]}"   

在 Python 2.7 中工作想要分离出 JSON 值并将其转换为 Pandas 数据帧,但我在 pyparsing 方面缺乏经验。

我的方法是使用“|”将文件作为 Pandas 数据框读取作为分隔符,而不是采用包含 JSON 的列并使用 'JSON_normalise' 将其展平,但 JSON_normalise 不会索引熊猫的列

我发现了herehere 的解决方案,但是一个不适合我的“混合数据”,另一个是对于一个相当大的 JSON 文件来说过于简单

关于如何在这些数据上部署 Pyparsing 的任何提示都会非常有帮助。 谢谢

Pyparsing: Parsing semi-JSON nested plaintext data to a list

Parsing semi-structured json data(Python/R)

【问题讨论】:

    标签: python json python-2.7 pyparsing


    【解决方案1】:

    将上面的输入字符串作为一个名为“data”的变量,这个 Python+pyparsing 代码会理解它。不幸的是,第四个'|'右边的那个东西不是真正的 JSON。幸运的是,它的格式足够好,可以在没有过度不适的情况下对其进行解析。请参阅下面程序中的嵌入式 cmets:

    from pyparsing import *
    from datetime import datetime
    
    # for the most part, we suppress punctuation - it's important at parse time
    # but just gets in the way afterwards
    LBRACE,RBRACE,COLON,DBLQ,LBRACK,RBRACK = map(Suppress, '{}:"[]')
    DBLQ2 = DBLQ + DBLQ
    
    # define some scalar value expressions, including parse-time conversion parse actions
    realnum = Regex(r'[+-]?\d+\.\d*').setParseAction(lambda t:float(t[0]))
    integer = Regex(r'[+-]?\d+').setParseAction(lambda t:int(t[0]))
    timestamp = Regex(r'""\d{4}-\d{2}-\d{2}T\d{2}:\d{2}""')
    timestamp.setParseAction(lambda t: datetime.strptime(t[0][2:-2],'%Y-%m-%dT%H:%M'))
    string_value = QuotedString('""')
    
    # define our base key ':' value expression; use a Forward() placeholder
    # for now for value, since these things can be recursive
    key = Optional(DBLQ2) + Word(alphas, alphanums+'_') + DBLQ2
    value = Forward()
    key_value = Group(key + COLON + value)
    
    # objects can be values too - use the Dict class to capture keys as field names
    obj = Group(Dict(LBRACE + OneOrMore(key_value) + RBRACE))
    objlist = (LBRACK + ZeroOrMore(obj) + RBRACK)
    
    # define expression for previously-declared value, using <<= operator
    value <<= timestamp | string_value | realnum | integer | obj | Group(objlist)
    
    # the outermost objects are enclosed in "s, and list of them can be given with '|' delims
    quotedObj = DBLQ + obj + DBLQ
    obsList = delimitedList(quotedObj, delim='|')
    

    现在将该解析器应用于您的“数据”:

    fields = data.split('|',4)
    result = obsList.parseString(fields[-1])
    
    # we get back a list of objects, dump them out
    for r in result:
        print r.dump()
        print
    

    给予:

    [['currency', 'EUR'], ['item_id', '143'], ['type', 'FLIGHT'], ['name', 'PAR-FEZ'], ['price', 1111], ['origin', 'PAR'], ['destination', 'FEZ'], ['merchant', 'GOV'], ['flight_type', 'OW'], ['flight_segment', [[['origin', 'ORY'], ['destination', 'FEZ'], ['departure_date_time', datetime.datetime(2015, 8, 2, 7, 20)], ['arrival_date_time', datetime.datetime(2015, 8, 2, 9, 5)], ['carrier', 'AT'], ['f_class', 'ECONOMY']]]]]
    - currency: EUR
    - destination: FEZ
    - flight_segment: 
      [0]:
        [['origin', 'ORY'], ['destination', 'FEZ'], ['departure_date_time', datetime.datetime(2015, 8, 2, 7, 20)], ['arrival_date_time', datetime.datetime(2015, 8, 2, 9, 5)], ['carrier', 'AT'], ['f_class', 'ECONOMY']]
        - arrival_date_time: 2015-08-02 09:05:00
        - carrier: AT
        - departure_date_time: 2015-08-02 07:20:00
        - destination: FEZ
        - f_class: ECONOMY
        - origin: ORY
    - flight_type: OW
    - item_id: 143
    - merchant: GOV
    - name: PAR-FEZ
    - origin: PAR
    - price: 1111
    - type: FLIGHT
    
    [['type', 'FLIGHT'], ['name', 'FI_ORY-OUD'], ['item_id', 'FLIGHT'], ['currency', 'EUR'], ['price', 111], ['origin', 'ORY'], ['destination', 'OUD'], ['flight_type', 'OW'], ['flight_segment', [[['origin', 'ORY'], ['destination', 'OUD'], ['departure_date_time', datetime.datetime(2015, 8, 2, 13, 55)], ['arrival_date_time', datetime.datetime(2015, 8, 2, 15, 30)], ['flight_number', 'AT625'], ['carrier', 'AT'], ['f_class', 'ECONOMIC_DISCOUNTED']]]]]
    - currency: EUR
    - destination: OUD
    - flight_segment: 
      [0]:
        [['origin', 'ORY'], ['destination', 'OUD'], ['departure_date_time', datetime.datetime(2015, 8, 2, 13, 55)], ['arrival_date_time', datetime.datetime(2015, 8, 2, 15, 30)], ['flight_number', 'AT625'], ['carrier', 'AT'], ['f_class', 'ECONOMIC_DISCOUNTED']]
        - arrival_date_time: 2015-08-02 15:30:00
        - carrier: AT
        - departure_date_time: 2015-08-02 13:55:00
        - destination: OUD
        - f_class: ECONOMIC_DISCOUNTED
        - flight_number: AT625
        - origin: ORY
    - flight_type: OW
    - item_id: FLIGHT
    - name: FI_ORY-OUD
    - origin: ORY
    - price: 111
    - type: FLIGHT
    

    请注意,不是字符串的值(整数、时间戳等)已经转换为 Python 类型。由于字段名称保存为 dict 键,因此您可以按名称访问字段,如下所示:

    res[0].currency
    res[0].price
    res[0].destination
    res[0].flight_segment[0].origin
    len(res[0].flight_segment) # gives how many segments
    

    【讨论】:

    • 感谢一百万!
    猜你喜欢
    • 2019-02-01
    • 1970-01-01
    • 2018-12-31
    • 1970-01-01
    • 2018-10-26
    • 1970-01-01
    • 1970-01-01
    • 2023-04-10
    • 1970-01-01
    相关资源
    最近更新 更多