如何修复可能损坏的 json 文件？花括号字符“{”的问题（Python3）

Question

这是一个奇怪的问题，我什至不知道如何问，但我会尝试。我有一些 json 文件，其中包含网络抓取数据，每个文件有多个条目，它们看起来像这样：

{
"doc_id": "some_number",
"url": "www.seedurl1.com",
"scrape_date": "2019-10-22 16:17:22",
"publish_date": "unknown",
"author": "unknown",
"urls_out": [
"https://www.something.com",
"https://www.sometingelse.com/smth"
],
"text": "lots of text here"
}
{
"doc_id": "some_other_number",
"url": "www.seedurl2.com/smth",
"scrape_date": "2019-10-22 17:44:40",
"publish_date": "unknown",
"author": "unknown",
"urls_out": [
"www.anotherurl.com/smth",
"http://urlx.com/smth.htm"
],
"text": "lots more text over here."
}

我试图格式化它们，以便每个条目都在自己的行上，如下所示：

{"doc_id": blah blah....} 
{"doc_id": blah blah blah...}

所以我这样做了：

    # Read the file
    f = codecs.open(file, 'r', encoding='utf-8-sig', errors='replace')
    text = f.read()
    f.close()

    # Check if }{ was found; 
    # this prints nothing for original files but finds everything in a hand written file
    pattern = '}{'
    print('Before editing: ', (re.findall(pattern, text)))

    # Getting rid of excess newlines and whitespaces
    newtext = " ".join(text.split())

    # Check if } { was found;
    # this prints nothing for original files but finds everything in a hand written file
    pattern = '} {'
    print('After editing: ', (re.findall(pattern, newtext)))

    # Put newlines in the right places
    finaltext = re.sub('} {', '}\n{', newtext)

    # Write the new JSON
    newfile = file[:-5]+'_ED.json'
    nf = codecs.open(newfile, 'w', encoding='utf-8', errors='replace')
    nf.write(finaltext)
    nf.close()

问题是，代码在具有相同结构的手写测试文件上完美运行，但不适用于原始文件或源自原始文件的较小测试文件。

我尝试在文本编辑器中分别简单地搜索“}”和“{”，结果没问题。但是，如果我尝试搜索“}{”或“} {”，则什么也找不到。虽然我可以看到他们清楚地在那里。

最后一个发现：我试图在 Linux 的 Nano 中打开我的小测试文件的编辑版本，然后移到了问题区域。出于某种原因，需要按两次右箭头键才能移过“{”大括号。所以那里显然有一些奇怪的东西。我怎样才能知道什么？或任何其他可能有帮助的建议？

Answer 1

最简单的解决方案就是创建一个 JSON 数组以...

否则，我建议不要替换任何内容，只需计算匹配的括号即可。

count = 0
objects = 0
with open('file.txt') as f:
    for i, c in enumerate(f.read()):
      if c == '\n':
        continue
      elif c == '{':
        if i > 0 and count == 0:
          print()  # start new line before printing bracket
        count += 1
      elif c == '}':
        count -= 1
        if count == 0:  # found a complete JSON object
          objects += 1

      print(c, end='')
    print(f'\n\nfound {objects} objects')  # for debugging 

对于给定的文本，我最终得到了这个

{"doc_id": "some_number","url": "www.seedurl1.com","scrape_date": "2019-10-22 16:17:22","publish_date": "unknown","author": "unknown","urls_out": ["https://www.something.com","https://www.sometingelse.com/smth"],"text": "lots of text here"}
{"doc_id": "some_other_number","url": "www.seedurl2.com/smth","scrape_date": "2019-10-22 17:44:40","publish_date": "unknown","author": "unknown","urls_out": ["www.anotherurl.com/smth","http://urlx.com/smth.htm"],"text": "lots more text over here."}

found 2 objects

Answer 2

这是一种方法。

例如：

import json

with open(filename) as infile:
    data = json.loads("[" + infile.read().replace("}\n{", "},\n{") + "]")
    for i in data:
        print(i)

输出：

{'doc_id': 'some_number', 'url': 'www.seedurl1.com',.....
{'doc_id': 'some_other_number', 'url': 'www.seedurl2.com/smth',.....

Answer 3

这是另一种解决方案，与您尝试的方法有点接近

import json

with open('test.txt') as f:
    file = f.readlines()
file = ['{'+i+'}'for i in "".join("".join(file).split("\n"))[1:-1].split("}{")]

for i in file:
    print(json.loads(i))

json 在这里仅用于验证单个 JSON。这给了

{'doc_id': 'some_number', 'url': 'www.seedurl1.com', 'scrape_date': '2019-10-22 16:17:22', 'publish_date': 'unknown', 'author': 'unknown', 'urls_out': ['https://www.something.com', 'https://www.sometingelse.com/smth'], 'text': 'lots of text here'}
{'doc_id': 'some_number', 'url': 'www.seedurl1.com', 'scrape_date': '2019-10-22 16:17:22', 'publish_date': 'unknown', 'author': 'unknown', 'urls_out': ['https://www.something.com', 'https://www.sometingelse.com/smth'], 'text': 'lots of text here'}