在一个非常大的项目中发现内存泄漏

Question

我有一个相当大的多线程 Python 项目，显然某处存在内存泄漏。 DoctorThread 向我展示了这些（缩短的）结果：

Partition of a set of 418 objects. Total size = 96792 bytes.
 Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)
     0     43  10    22792  24     22792  24 guppy.etc.Glue.Interface
     1     66  16    18480  19     41272  43 dict of guppy.etc.Glue.Owner
     2     25   6    18344  19     59616  62 dict of guppy.etc.Glue.Share
     3      8   2     8384   9     68000  70 guppy.etc.Glue.Share
     4     86  21     6696   7     74696  77 dict (no owner)
     5     22   5     6160   6     80856  84 guppy.etc.Glue.Owner
     6     37   9     2608   3     83464  86 dict (no owner), dict of guppy.etc.Glue.Interface
     7     28   7     2464   3     85928  89 guppy.heapy.heapyc.HeapView
     8     11   3     1840   2     87768  91 <Nothing>
     9      2   0     1112   1     88880  92 __builtin__.cell
<24 more rows. Type e.g. '_.more' to view.>
Partition of a set of 23178 objects. Total size = 1604224 bytes.
 Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)
     0  11135  48   801440  50    801440  50 list
     1  11153  48   602408  38   1403848  88 tuple
     [...]
<95 more rows. Type e.g. '_.more' to view.>
Partition of a set of 45140 objects. Total size = 2987568 bytes.
 Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)
     0  22114  49  1591936  53   1591936  53 list
     1  22133  49  1195328  40   2787264  93 tuple
     [...]
<95 more rows. Type e.g. '_.more' to view.>
Partition of a set of 66115 objects. Total size = 4337720 bytes.
 Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)
     0  32524  49  2341216  54   2341216  54 list
     1  32513  49  1755848  40   4097064  94 tuple
     [...]
<104 more rows. Type e.g. '_.more' to view.>
Partition of a set of 88355 objects. Total size = 5739128 bytes.
 Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)
     0  43644  49  3141856  55   3141856  55 list
     1  43633  49  2356328  41   5498184  96 tuple
     [...]
<104 more rows. Type e.g. '_.more' to view.>
Partition of a set of 110380 objects. Total size = 7097992 bytes.
 Index  Count   %     Size   % Cumulative  % Referrers by Kind (class / dict of class)
     0  54734  50  3940576  56   3940576  56 list
     1  54753  50  2956808  42   6897384  97 tuple
     [...]
<97 more rows. Type e.g. '_.more' to view.>

如您所见，list 和tuple 推荐人的数量逐渐增加。而且它永远不会停止增加。这两个条目是唯一不断增加的条目。

DoctorThread 类如下所示：

class DoctorThread(threading.Thread):
    def __init__(self):
        super(DoctorThread, self).__init__()
        self.daemon = True
        self.hp = guppy.hpy()

    def run(self):
        time.sleep(5)
        logging.info("Doctor Thread started - taking heap snapshots")
        before_heap = self.hp.heap()

        while not PippinNetwork.is_shutdown():
            gc.collect()
            leftover = self.hp.heap() - before_heap
            print(leftover.byrcs)
            time.sleep(2.0)

内存消耗同样增加。我怎样才能找到这次泄漏的罪魁祸首？

更新：已解决。

事后看来，这是一个猖獗的list.append((object, object))。孔雀鱼的结果可以这样解释。

Answer 1

如果大型源代码中出现内存泄漏，您可以在 python 中使用 'pympler' 来跟踪以供将来工作参考。

from pympler import muppy

 from pympler import summary

all_objects = muppy.get_objects()

sum1 = summary.summarize(all_objects)
 summary.print_(sum1) 

您也可以过滤对象。只包括那些你有疑问的，然后你可以看到哪个对象或变量导致内存泄漏，然后查看其背后的逻辑。

我建议这种方法，因为这样你就不必削减逻辑（因为你的项目会有大量代码）

如有澄清，您可以在评论中询问。