【发布时间】：2019-08-10 18:08:56
【问题描述】：
我正在寻找一种方法来使用 json 保存 numpy 数据，同时保留 numpy 的人类可读的漂亮打印格式。
受this answer 的启发，我选择使用pprint 而不是base64 以我想要的格式写入数据，因此：
import numpy as np
data = np.random.random((1,3,2))
磁盘上的结果文件应如下所示：
{
"__dtype__": "float64",
"__ndarray__": [[[0.7672818918130646 , 0.6846412220229668 ],
[0.7082023466738064 , 0.0896531267221291 ],
[0.43898454934160147, 0.9245898883694668 ]]]
}
出现了一些小问题。
虽然
json可以读取格式为[[...]]的列表列表，但numpy的浮点格式存在问题。例如[[0., 0., 0.]]在读回时会产生错误，而[[0.0, 0.0, 0.0]]会很好。pformat将输出array([[0., 0., 0.]])，其中array()必须被解析，否则json在读回数据时会引发错误。
为了解决这些问题，我必须进行一些字符串解析，这导致我当前的代码如下：
import json, sys
import numpy as np
import pprint as pp
# Set numpy's printoptions to display all the data with max precision
np.set_printoptions(threshold=np.inf,
linewidth=sys.maxsize,
suppress=True,
nanstr='0.0',
infstr='0.0',
precision=np.finfo(np.longdouble).precision)
# Modified version of Adam Hughes's https://stackoverflow.com/a/27948073/1429402
def save_formatted(fname,data):
class NumpyEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, np.ndarray):
return {'__ndarray__': self.numpy_to_string(obj),
'__dtype__': str(obj.dtype)}
return json.JSONEncoder.default(self, obj)
def numpy_to_string(self,data):
''' Use pprint to generate a nicely formatted string
'''
# Get rid of array(...) and keep only [[...]]
f = pp.pformat(data, width=sys.maxsize)
f = f[6:-1].splitlines() # get rid of array(...) and keep only [[...]]
# Remove identation caused by printing "array("
for i in xrange(1,len(f)):
f[i] = f[i][6:]
return '\n'.join(f)
# Parse json stream and fix formatting.
# JSON doesn't support float arrays written as [0., 0., 0.]
# so we look for the problematic numpy print syntax and correct
# it to be readable natively by JSON, in this case: [0.0, 0.0, 0.0]
with open(fname,'w') as io:
for line in json.dumps(data, sort_keys=False, indent=4, cls=NumpyEncoder).splitlines():
if '"__ndarray__": "' in line:
index = line.index('"__ndarray__": "')
lines = line.split('"__ndarray__": "')[-1][:-1]
lines = lines.replace('. ','.0') # convert occurences of ". " to ".0" ex: 3. , 2. ]
lines = lines.replace('.,','.0,') # convert occurences of ".," to ".0," ex: 3., 2.,
lines = lines.replace('.]','.0]') # convert occurences of ".]" to ".0]," ex: 3., 2.]
lines = lines.split('\\n')
# write each lines with appropriate indentation
for i in xrange(len(lines)):
if i == 0:
indent = ' '*index
io.write(('%s"__ndarray__": %s\n"'%(indent,lines[i]))[:-1])
else:
indent = ' '*(index+len('"__ndarray__": "')-1)
io.write('%s%s\n'%(indent,lines[i]))
else:
io.write('%s\n'%line)
def load_formatted(fname):
def json_numpy_obj_hook(dct):
if isinstance(dct, dict) and '__ndarray__' in dct:
return np.array(dct['__ndarray__']).astype(dct['__dtype__'])
return dct
with open(fname,'r') as io:
return json.load(io, object_hook=json_numpy_obj_hook)
测试：
data = np.random.random((200,3,1000))
save_formatted('test.data', data)
data_ = load_formatted('test.data')
print np.allclose(data,data_) # Returns True
问题
我的解决方案适合我，但它的字符串解析方面使得处理大型数据数组时速度变慢。有没有更好的方法来达到预期的效果？ regular expression 可以替换我的序列 str.replace() 调用吗？或者也许pprint 可以用来正确格式化我的字符串？有没有更好的方法让json 写出类似numpy 的打印格式的列表？
【问题讨论】：