Python：每行一个 JSON obj 的文件答案

【问题标题】：Python : File with one JSON obj on each linePython：每行一个 JSON obj 的文件
【发布时间】：2017-01-30 15:59:19
【问题描述】：

来自东京的问候。

让我解释一下我想用 python 2.7 实现什么：

我有一个文件，每行都有一个 JSON Dict，这是一个捕获：

1 {"res":0, "res_message":"OK", "debug_info":{"id-info":"9089"}, "visits":[{"id":"237000080507750613","siteId":1551642,"startTime":1483217576324,"endTime":1483217696000,"clientIPs":["69.61.12.70"],"country"    :["United States"],"countryCode":["US"],"clientType":"Vulnerability Scanner","clientApplication":"Grabber","clientApplicationId":780,"httpVersion":"1.1","clientApplicationVersion":"null","userAgent":"Mozi    lla/5.0 CommonCrawler Node 3AEHGF7VNEKJUWOPKJJIJ7ODKPM4XXVZQUTHNWS5B2O5AEAGHIG4HVC42LLEUSO.CQYXO3ZFD.GB5RZ5EG2SRWW335PUSOSIVLZUXPCTJUGV2MDJGQJDJPE5UH.cdn0.common.crawl.zone","os":"","osVersion":"","suppor    tsCookies":false,"supportsJavaScript":false,"hits":1,"pageViews":0,"entryReferer":"","servedVia":["Ashburn,VA"],"securitySummary": {"api.threats.bot_access_control":1},"actions":[{"postData":"","requestResult":"api.request_result.req_blocked_security","isSecured":false,"responseTime":0,"thinkTime":0,"incidentId":"237000080507750613-304992946328    764549","threats":[{"securityRule":"api.threats.bot_access_control","alertLocation":"api.alert_location.alert_location_path","attackCodes":["200.0"],"securityRuleAction    ":"api.rule_action_type.rule_action_block"}]}]}, ...

2 {"res":0, "res_message":"OK", "debug_info":{"id-info":"9089"}, "visits":[{"id":"520000110618442601","siteId":1551642,"startTime":1482666233524,"endTime":1482666353000,"clientIPs":["93.175.201.18"],"countr    y":["Ukraine"],"countryCode":["UA"],"clientType":"Spam Bot","clientApplication":"DTS Agent","clientApplicationId":99,"httpVersion":"1.1","clientApplicationVersion":"null","userAgent":"Mozilla/4.0 (compati    ble; MSIE 5.0; Windows NT; DigExt; DTS Agent","os":"","osVersion":"","supportsCookies":false,"supportsJavaScript":false,"hits":1,"pageViews":0,"entryReferer":"","served    Via":["Warsaw, Poland"],"securitySummary":{"api.threats.bot_access_control":1},"actions":[{"postData":"","requestResult":"api.request_result.req_blocked_security","isSecured":false,"responseTime":2,"thinkTime":1,"incidentId":"520000110618442601-1233371267206742195","threats":[{"securityRule":"api.threats.bot_access_control","alertLocation":"api.alert_location.alert_location_path","attackCodes":["200.0"],"securityRuleAction":"api.rule_action_type.rule_action_block"}]}]}, ...

3 {"res":0, "res_message":"OK", "debug_info":{"id-info":"9089"}, "visits":[{"id":"520000110602830007","siteId":1551642,"startTime":1482429957001,"endTime":1482430077000,"clientIPs":["93.175.201.18"],"countr    y":["Ukraine"],"countryCode":["UA"],"clientType":"Spam Bot","clientApplication":"DTS Agent","clientApplicationId":99,"httpVersion":"1.1","clientApplicationVersion":"null","userAgent":"Mozilla/4.0 (compati    ble; MSIE 5.0; Windows NT; DigExt; DTS Agent","os":"","osVersion":"","supportsCookies":false,"supportsJavaScript":false,"hits":1,"pageViews":0,"entryReferer":"","served    Via":["Warsaw, Poland"],"securitySummary":{"api.threats.bot_access_control":1},"actions":[{"postData":"","requestResult":"api.request_result.req_blocked_security","isSecured":false","responseTime":4,"thinkTime":4,"incidentId":"520000110602830007-3073954101470953658","threats":[{"securityRule":"api.threats.bot_access_control","alertLocation":"api.alert_location.alert_location_path","attackCodes":["200.0"],"securityRuleAction":"api.rule_action_type.rule_action_block"}]}]}, ...

我尝试使用 json.loads() 继续整个文件，但没有任何成功。

这是我的代码

g = open('monthlyLogShort.txt', 'w')
with open("page.txt") as f:
         data = f.read()
         parse = json.loads(data)        # <-load the JSON dict
         field_list = parse["visits"]
         for fields in field_list:       # <-extract the the following field
                 print >> g , "visit_id=",(fields["id"]),",","src_country=",(fields["country"]),",", "event_timestamp=",(fields["startTime"]),",","src_ip=",(fields["clientIPs"]),",","dest_name=", rwdname,"    ,","dest_id=",(fields["siteId"]),",","signature=",(fields["securitySummary"])
g.close()

如你所想，我只能用这段代码解析一行。处理整个文件的最佳（pythonic）方法是什么？

感谢阅读

【问题讨论】：

标签： python json parsing dictionary

【解决方案1】：

文件整体不是有效的JSON，但你可以逐行解析：

with open("page.txt") as f:
    for line in f:
        obj = json.loads(line.split(" ", 1)[1])
        print(obj["visits"])

【讨论】：

哇，感谢 alecxe 的快速回复。明天在工作中尝试这个东西，如果顺利的话，接受答案！ :)
仍然出现解析错误：ValueError: Extra data: line 1 column 14 - line 2 column 1 (char 13 - 121954)
@Emka 你确定每行第一个空格后都有有效的 json 字符串吗？此外，在 JSON 字符串之后是否还有其他以空格分隔的值？谢谢。
我很确定 json 字符串是有效的。在 Visits 键之前，有 3 个空格。访问 Key 后，值内部有大量空格，但外部没有。

【解决方案2】：

因为行数总是一样的，所以我想出了这个解决方案：

g = open('monthlyLogShort.txt', 'w')
with open('page.txt','r') as f:
        data = f.readlines()
        countp = 0
        page = 0
        while countp < 10:
                parse = json.loads(data[page])  # load the JSON dict
                field_list = parse["visits"]
                for fields in field_list:       # extract the the following field
                        print >> g , "visit_id=",(fields["id"]),",","src_country=",(fields["country"]),",", "event_timestamp=",(fields["startTime"]),",","src_ip=",(fields["clientIPs"]),",","dest_name=", dname,",","dest_id=",(fields["siteId"]),",","signature=",(fields['securitySummary'])
                countp = countp + 1
                page = page + 1
        else:
                g.close()

它就像一个魅力。

【讨论】：