【发布时间】：2019-12-18 15:24:38
【问题描述】：
我正在编写一个用于控制示波器的 python 库/包。其中一项功能是从示波器下载轨迹。数据以二进制形式发送并由 numpy 转换为浮点数。在 test_script 中，它将在（几乎）无限循环中用于保存记录的数据（当设备上没有空间时结束）。但在它耗尽空间之前，python 脚本会耗尽内存。
这是我的库的故障版本，即使被告知要释放它，它也会遇到同样的问题，即囤积内存而不释放它：
import time
import gc
import numpy
from pympler import muppy, summary
class Measurement:
data = None
def __del__(self):
del self.data
print("Deleting Measurement")
class Oscilloscope:
def GetData(self):
# Simulating the acquiration of the data by reading it in by a file
p = open(r"rawdata.bin",mode="rb")
rawdata = p.read()
mes = Measurement()
mes.data = numpy.array(numpy.frombuffer(bytes(rawdata), dtype='B'), dtype=float)
p.close()
return mes
def __del__(self):
print("Deleting Oscilloscope")
if __name__ == "__main__":
myosci = Oscilloscope()
for i in range(0,10):
var = myosci.GetData()
data = var.data
# In the real script something will be done with the data
print("Saving and evaluating Traces")
time.sleep(0.5)
del var
gc.collect()
print("Collected everything")
# Show MemoryUsage
all_objects = muppy.get_objects()
sum1 = summary.summarize(all_objects)
# Prints out a summary of the large objects
summary.print_(sum1)
print("Done")
执行时输出如下：
Saving and evaluating Traces
Deleting Measurement
Collected everything
types | # objects | total size
============================ | =========== | ============
numpy.ndarray | 40 | 9.16 MB
str | 18144 | 3.39 MB
dict | 4685 | 2.21 MB
code | 6556 | 925.30 KB
type | 1060 | 868.78 KB
tuple | 4209 | 272.90 KB
wrapper_descriptor | 2209 | 172.58 KB
set | 139 | 131.41 KB
weakref | 1339 | 104.61 KB
method_descriptor | 1402 | 98.58 KB
list | 396 | 88.62 KB
builtin_function_or_method | 1165 | 81.91 KB
abc.ABCMeta | 84 | 81.50 KB
getset_descriptor | 1050 | 73.83 KB
int | 1994 | 60.12 KB
Saving and evaluating Traces
Deleting Measurement
Collected everything
types | # objects | total size
============================ | =========== | ============
numpy.ndarray | 41 | 18.31 MB
str | 22639 | 3.70 MB
dict | 4685 | 2.21 MB
list | 4895 | 959.05 KB
code | 6556 | 925.30 KB
type | 1060 | 868.78 KB
tuple | 4212 | 273.09 KB
wrapper_descriptor | 2209 | 172.58 KB
set | 139 | 131.41 KB
weakref | 1339 | 104.61 KB
method_descriptor | 1402 | 98.58 KB
int | 2940 | 85.98 KB
builtin_function_or_method | 1165 | 81.91 KB
abc.ABCMeta | 84 | 81.50 KB
getset_descriptor | 1050 | 73.83 KB
Saving and evaluating Traces
Deleting Measurement
Collected everything
types | # objects | total size
============================ | =========== | ============
numpy.ndarray | 42 | 27.47 MB
str | 27134 | 4.00 MB
dict | 4685 | 2.21 MB
list | 9392 | 1.90 MB
code | 6556 | 925.30 KB
type | 1060 | 868.78 KB
tuple | 4212 | 273.09 KB
wrapper_descriptor | 2209 | 172.58 KB
set | 139 | 131.41 KB
int | 3879 | 111.66 KB
weakref | 1339 | 104.61 KB
method_descriptor | 1402 | 98.58 KB
builtin_function_or_method | 1165 | 81.91 KB
abc.ABCMeta | 84 | 81.50 KB
getset_descriptor | 1050 | 73.83 KB
Saving and evaluating Traces
Deleting Measurement
Collected everything
types | # objects | total size
============================ | =========== | ============
numpy.ndarray | 43 | 36.63 MB
str | 31629 | 4.31 MB
list | 13889 | 2.94 MB
dict | 4685 | 2.21 MB
code | 6556 | 925.30 KB
type | 1060 | 868.78 KB
tuple | 4212 | 273.09 KB
wrapper_descriptor | 2209 | 172.58 KB
int | 4818 | 137.34 KB
set | 139 | 131.41 KB
weakref | 1339 | 104.61 KB
method_descriptor | 1402 | 98.58 KB
builtin_function_or_method | 1165 | 81.91 KB
abc.ABCMeta | 84 | 81.50 KB
getset_descriptor | 1050 | 73.83 KB
Saving and evaluating Traces
Deleting Measurement
Collected everything
types | # objects | total size
============================ | =========== | ============
numpy.ndarray | 44 | 45.78 MB
str | 36124 | 4.62 MB
list | 18386 | 4.05 MB
dict | 4685 | 2.21 MB
code | 6556 | 925.30 KB
type | 1060 | 868.78 KB
tuple | 4212 | 273.09 KB
wrapper_descriptor | 2209 | 172.58 KB
int | 5757 | 163.01 KB
set | 139 | 131.41 KB
weakref | 1339 | 104.61 KB
method_descriptor | 1402 | 98.58 KB
builtin_function_or_method | 1165 | 81.91 KB
abc.ABCMeta | 84 | 81.50 KB
getset_descriptor | 1050 | 73.83 KB
Saving and evaluating Traces
Deleting Measurement
Collected everything
types | # objects | total size
============================ | =========== | ============
numpy.ndarray | 45 | 54.94 MB
list | 22883 | 5.25 MB
str | 40619 | 4.92 MB
dict | 4685 | 2.21 MB
code | 6556 | 925.30 KB
type | 1060 | 868.78 KB
tuple | 4212 | 273.09 KB
int | 6696 | 188.69 KB
wrapper_descriptor | 2209 | 172.58 KB
set | 139 | 131.41 KB
weakref | 1339 | 104.61 KB
method_descriptor | 1402 | 98.58 KB
builtin_function_or_method | 1165 | 81.91 KB
abc.ABCMeta | 84 | 81.50 KB
getset_descriptor | 1050 | 73.83 KB
Saving and evaluating Traces
Deleting Measurement
Collected everything
types | # objects | total size
============================ | =========== | ============
numpy.ndarray | 46 | 64.09 MB
list | 27380 | 6.55 MB
str | 45114 | 5.23 MB
dict | 4685 | 2.21 MB
code | 6556 | 925.30 KB
type | 1060 | 868.78 KB
tuple | 4212 | 273.09 KB
int | 7635 | 214.36 KB
wrapper_descriptor | 2209 | 172.58 KB
set | 139 | 131.41 KB
weakref | 1339 | 104.61 KB
method_descriptor | 1402 | 98.58 KB
builtin_function_or_method | 1165 | 81.91 KB
abc.ABCMeta | 84 | 81.50 KB
getset_descriptor | 1050 | 73.83 KB
Saving and evaluating Traces
Deleting Measurement
Collected everything
types | # objects | total size
============================ | =========== | ============
numpy.ndarray | 47 | 73.25 MB
list | 31877 | 7.85 MB
str | 49609 | 5.54 MB
dict | 4685 | 2.21 MB
code | 6556 | 925.30 KB
type | 1060 | 868.78 KB
tuple | 4212 | 273.09 KB
int | 8574 | 240.04 KB
wrapper_descriptor | 2209 | 172.58 KB
set | 139 | 131.41 KB
weakref | 1339 | 104.61 KB
method_descriptor | 1402 | 98.58 KB
builtin_function_or_method | 1165 | 81.91 KB
abc.ABCMeta | 84 | 81.50 KB
getset_descriptor | 1050 | 73.83 KB
Saving and evaluating Traces
Deleting Measurement
Collected everything
types | # objects | total size
============================ | =========== | ============
numpy.ndarray | 48 | 82.40 MB
list | 36374 | 9.26 MB
str | 54104 | 5.84 MB
dict | 4685 | 2.21 MB
code | 6556 | 925.30 KB
type | 1060 | 868.78 KB
tuple | 4212 | 273.09 KB
int | 9513 | 265.71 KB
wrapper_descriptor | 2209 | 172.58 KB
set | 139 | 131.41 KB
weakref | 1339 | 104.61 KB
method_descriptor | 1402 | 98.58 KB
builtin_function_or_method | 1165 | 81.91 KB
abc.ABCMeta | 84 | 81.50 KB
getset_descriptor | 1050 | 73.83 KB
Saving and evaluating Traces
Deleting Measurement
Collected everything
types | # objects | total size
============================ | =========== | ============
numpy.ndarray | 49 | 91.56 MB
list | 40871 | 10.79 MB
str | 58599 | 6.15 MB
dict | 4685 | 2.21 MB
code | 6556 | 925.30 KB
type | 1060 | 868.78 KB
int | 10452 | 291.39 KB
tuple | 4212 | 273.09 KB
wrapper_descriptor | 2209 | 172.58 KB
set | 139 | 131.41 KB
weakref | 1339 | 104.61 KB
method_descriptor | 1402 | 98.58 KB
builtin_function_or_method | 1165 | 81.91 KB
abc.ABCMeta | 84 | 81.50 KB
getset_descriptor | 1050 | 73.83 KB
Done
Deleting Oscilloscope
Process finished with exit code 0
即使 numpyarrays 收到删除命令并且垃圾收集器被触发，内存也会被囤积。在实际脚本中，内存使用量每一步都会增加大约 150 mB。运行我的库的 Raspberry Pi 很快就会耗尽内存。
如果你想自己测试，这里是使用的 rawdata.bin 文件： https://mega.nz/#!ps4ACAJD!0cqTJJmMU5RjSx5BqggM1afz47PqcR67hzKKnI8LgTs
【问题讨论】：
-
如果这是 NumPy 的真正错误，您是否考虑过联系开发人员/维护人员？
标签： python numpy memory-leaks garbage-collection out-of-memory