如何解决从错误的 json 格式解码的问题

Question

每个人。需要帮助打开和阅读文件。

得到这个 txt 文件 - https://yadi.sk/i/1TH7_SYfLss0JQ

这是一本字典

{"id0":"url0", "id1":"url1", ..., "idn":"urln"}

但它是使用json写入txt文件的。

#This is how I dump the data into a txt    
json.dump(after,open(os.path.join(os.getcwd(), 'before_log.txt'), 'a')) 

所以，文件结构是 {"id0":"url0", "id1":"url1", ..., "idn":"urln"}{"id2":"url2", "id3":"url3", ..., "id4":"url4"}{"id5":"url5", "id6":"url6", ..., "id7":"url7"}

而且都是字符串....

我需要打开它并检查重复的ID，删除并再次保存。

但是得到 - json.loads 显示 ValueError: Extra data

试过这些： How to read line-delimited JSON from large file (line by line) Python json.loads shows ValueError: Extra data json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 190)

但仍然出现该错误，只是在不同的地方。

现在我做到了：

with open('111111111.txt', 'r') as log:
    before_log = log.read()
before_log = before_log.replace('}{',', ').split(', ')

mu_dic = []
for i in before_log:
    mu_dic.append(i)

这消除了连续多个 {}{}{} 字典/json 的问题。

也许有更好的方法来做到这一点？

附：文件是这样制作的：

json.dump(after,open(os.path.join(os.getcwd(), 'before_log.txt'), 'a'))    

Answer 1

您的文件大小为 9,5M，因此您需要一段时间才能打开它并手动调试它。 因此，使用 head 和 tail 工具（通常在任何 Gnu/Linux 发行版中都可以找到）你会看到：

# You can use Python as well to read chunks from your file
# and see the nature of it and what it's causing a decode problem
# but i prefer head & tail because they're ready to be used :-D
$> head -c 217 111111111.txt
{"1933252590737725178": "https://instagram.fiev2-1.fna.fbcdn.net/vp/094927bbfd432db6101521c180221485/5CC0EBDD/t51.2885-15/e35/46950935_320097112159700_7380137222718265154_n.jpg?_nc_ht=instagram.fiev2-1.fna.fbcdn.net",
$> tail -c 219 111111111.txt
, "1752899319051523723": "https://instagram.fiev2-1.fna.fbcdn.net/vp/a3f28e0a82a8772c6c64d4b0f264496a/5CCB7236/t51.2885-15/e35/30084016_2051123655168027_7324093741436764160_n.jpg?_nc_ht=instagram.fiev2-1.fna.fbcdn.net"}
$> head -c 294879 111111111.txt | tail -c 12
net"}{"19332

所以第一个猜测是您的文件是一组格式错误的JSON 数据，最好的猜测是用\n 分隔}{ 以便进一步操作。

因此，这里有一个示例，说明如何使用 Python 解决您的问题：

import json

input_file = '111111111.txt'
output_file = 'new_file.txt'

data = ''
with open(input_file, mode='r', encoding='utf8') as f_file:
    # this with statement part can be replaced by 
    # using sed under your OS like this example:
    # sed -i 's/}{/}\n{/g' 111111111.txt
    data = f_file.read()
    data = data.replace('}{', '}\n{')


seen, total_keys, to_write = set(), 0, {}
# split the lines of the in memory data
for elm in data.split('\n'):
    # convert the line to a valid Python dict
    converted = json.loads(elm)
    # loop over the keys
    for key, value in converted.items():
        total_keys += 1
        # if the key is not seen then add it for further manipulations
        # else ignore it
        if key not in seen:
            seen.add(key)
            to_write.update({key: value})

# write the dict's keys & values into a new file as a JSON format
with open(output_file, mode='a+', encoding='utf8') as out_file:
    out_file.write(json.dumps(to_write) + '\n')

print(
    'found duplicated key(s): {seen} from {total}'.format(
        seen=total_keys - len(seen),
        total=total_keys
    )
)

输出：

found duplicated key(s): 43836 from 45367

最后，输出文件将是一个有效的JSON 文件，并且重复的键将连同它们的值一起被删除。

Answer 2

文件结构和实际 json 格式之间的基本区别是缺少逗号，并且这些行没有包含在 [ 中。所以同样可以用下面的代码sn-p来实现

with open('json_file.txt') as f:
    # Read complete file
    a = (f.read())

    # Convert into single line string
    b = ''.join(a.splitlines())

    # Add , after each object
    b = b.replace("}", "},")

    # Add opening and closing parentheses and ignore last comma added in prev step
    b = '[' + b[:-1] + ']'

x = json.loads(b)