C ++进程正在缓慢泄漏内存答案

【问题标题】：C++ process is leaking memory slowlyC ++进程正在缓慢泄漏内存
【发布时间】：2020-07-14 15:35:21
【问题描述】：

我们用 C++ 编写的一个进程正在非常缓慢地泄漏内存。 Valgrind 在所有 3 个部分中显示 0 字节（肯定丢失、间接丢失和可能丢失）。此问题大约需要 6-7 个月才能消耗 70% 的系统内存和 8GB 的 RAM。我试图在测试机器上重现该问题，并且在多次请求后能够看到内存消耗缓慢。

以下是我得到的 valgrind 结果：

==20868==
--20868-- REDIR: 0x4019f70 (ld-linux-x86-64.so.2:strlen) redirected to 0x58059dd1 (???)
--20868-- REDIR: 0x4019d40 (ld-linux-x86-64.so.2:index) redirected to 0x58059deb (???)
--20868-- Reading syms from /usr/lib64/valgrind/vgpreload_core-amd64-linux.so
--20868-- Reading syms from /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so
==20868== WARNING: new redirection conflicts with existing -- ignoring it
--20868--     old: 0x04019f70 (strlen              ) R-> (0000.0) 0x58059dd1 ???
--20868--     new: 0x04019f70 (strlen              ) R-> (2007.0) 0x04c2ba90 strlen
--20868-- REDIR: 0x4019ef0 (ld-linux-x86-64.so.2:strcmp) redirected to 0x4c2cbe0 (strcmp)
--20868-- REDIR: 0x401aae0 (ld-linux-x86-64.so.2:mempcpy) redirected to 0x4c2fc60 (mempcpy)
--20868-- Reading syms from /lib64/libpthread-2.17.so
--20868--   Considering /usr/lib/debug/.build-id/d9/73c39d1900dc61d8519c653c3bc405692de563.debug ..
--20868--   .. build-id is valid
--20868-- Reading syms from /usr/lib64/libcurl.so.4.4.0
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /usr/lib64/libcassandra.so.2.9.0
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /usr/lib64/libstdc++.so.6.0.24
--20868--   Considering /usr/lib/debug/.build-id/87/91ddd49348603cd50b74652c5b25354d8fd06e.debug ..
--20868--   .. build-id is valid
--20868-- Reading syms from /lib64/libm-2.17.so
--20868--   Considering /usr/lib/debug/.build-id/07/fb92afef1756f093371ce60c3ae85dd3a06325.debug ..
--20868--   .. build-id is valid
--20868-- Reading syms from /lib64/libgcc_s-7-20170915.so.1
--20868--   Considering /usr/lib/debug/.build-id/a0/3c9a80e995ed5f43077ab754a258fa0e34c3cd.debug ..
--20868--   .. build-id is valid
--20868-- Reading syms from /lib64/libc-2.17.so
--20868--   Considering /usr/lib/debug/.build-id/af/310f56618fc1ef9158973484f60942f11cc0fb.debug ..
--20868--   .. build-id is valid
--20868-- Reading syms from /usr/lib64/libnghttp2.so.14.13.1
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /usr/lib64/libidn2.so.0.1.4
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /usr/lib64/libssh2.so.1.0.1
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /usr/lib64/libpsl.so.0.2.3
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /usr/lib64/libssl3.so
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /usr/lib64/libsmime3.so
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /usr/lib64/libnss3.so
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /usr/lib64/libnssutil3.so
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /lib64/libplds4.so
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /lib64/libplc4.so
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /lib64/libnspr4.so
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /lib64/libdl-2.17.so
--20868--   Considering /usr/lib/debug/.build-id/0e/5cd5baa5ee8bf3648a5031b088f9a78c89364f.debug ..
--20868--   .. build-id is valid
--20868-- Reading syms from /usr/lib64/libgssapi_krb5.so.2.2
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /usr/lib64/libkrb5.so.3.3
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /usr/lib64/libk5crypto.so.3.1
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /usr/lib64/libcom_err.so.2.1
--20868--   Considering /usr/lib/debug/.build-id/ab/007f5df96c66e515542598f5be1429ed63d86f.debug ..
--20868--   .. build-id is valid
--20868-- Reading syms from /lib64/liblber-2.4.so.2.10.3
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /lib64/libldap-2.4.so.2.10.3
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /lib64/libz.so.1.2.8
--20868--   Considering /usr/lib/debug/.build-id/89/c6af118b6b4fb6a73ae1813e2c8bdd722956d1.debug ..
--20868--   .. build-id is valid
--20868-- Reading syms from /usr/lib64/libuv.so.1.0.0
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /usr/lib64/libssl.so.1.0.2k
--20868--   Considering /usr/lib/debug/.build-id/9c/4eb34a346260f2a77746f4e5ed837619137db7.debug ..
--20868--   .. build-id is valid
--20868-- Reading syms from /usr/lib64/libcrypto.so.1.0.2k
--20868--   Considering /usr/lib/debug/.build-id/0b/7f7487280fe68af9302a282fae37776a99bc80.debug ..
--20868--   .. build-id is valid
--20868-- Reading syms from /usr/lib64/libunistring.so.0.1.2
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /usr/lib64/libicuuc.so.50.1.2
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /lib64/librt-2.17.so
--20868--   Considering /usr/lib/debug/.build-id/f2/701e2a24459d5b55df5549d585f091e7bcf07a.debug ..
--20868--   .. build-id is valid
--20868-- Reading syms from /usr/lib64/libkrb5support.so.0.1
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /lib64/libkeyutils.so.1.5
--20868--   Considering /usr/lib/debug/.build-id/37/a58210fa50c91e09387765408a92909468d25b.debug ..
--20868--   .. build-id is valid
--20868-- Reading syms from /lib64/libresolv-2.17.so
--20868--   Considering /usr/lib/debug/.build-id/9e/5e0bf5f22de7555bc4b9853240817147489258.debug ..
--20868--   .. build-id is valid
--20868-- Reading syms from /usr/lib64/libsasl2.so.2.0.23
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /lib64/libnsl-2.17.so
--20868--   Considering /usr/lib/debug/.build-id/af/2bb3cd9fcda0b820c88aab5d80639ca5287921.debug ..
--20868--   .. build-id is valid
--20868-- Reading syms from /usr/lib64/libicudata.so.50.1.2
--20868--    object doesn't have a symbol table
--20868-- Reading syms from /usr/lib64/libselinux.so.1
--20868--   Considering /usr/lib/debug/.build-id/f5/054dc94443326819fbf3065cfdf5e4726f57ee.debug ..
--20868--   .. build-id is valid
--20868-- Reading syms from /lib64/libcrypt-2.17.so
--20868--   Considering /usr/lib/debug/.build-id/9a/9be1d2234d3641b9cfe17bb7a40b82973a2802.debug ..
--20868--   .. build-id is valid
--20868-- Reading syms from /lib64/libfreebl3.so
--20868--    object doesn't have a symbol table
--20868-- REDIR: 0x5f79e40 (libc.so.6:strcasecmp) redirected to 0x4a24780 (_vgnU_ifunc_wrapper)
--20868-- REDIR: 0x5f76bc0 (libc.so.6:strnlen) redirected to 0x4a24780 (_vgnU_ifunc_wrapper)
--20868-- REDIR: 0x5f7c110 (libc.so.6:strncasecmp) redirected to 0x4a24780 (_vgnU_ifunc_wrapper)
--20868-- REDIR: 0x5f79620 (libc.so.6:memset) redirected to 0x4a24780 (_vgnU_ifunc_wrapper)
--20868-- REDIR: 0x5f795d0 (libc.so.6:memcpy@GLIBC_2.2.5) redirected to 0x4a24780 (_vgnU_ifunc_wrapper)
--20868-- REDIR: 0x5f75040 (libc.so.6:strcmp) redirected to 0x4a24780 (_vgnU_ifunc_wrapper)
--20868-- REDIR: 0x5f76ae0 (libc.so.6:__GI_strlen) redirected to 0x4c2b9f0 (__GI_strlen)
--20868-- REDIR: 0x5f785b0 (libc.so.6:__GI_strrchr) redirected to 0x4c2b450 (__GI_strrchr)
--20868-- REDIR: 0x5f6f800 (libc.so.6:malloc) redirected to 0x4c28b5c (malloc)
--20868-- REDIR: 0x5f8f630 (libc.so.6:__GI_strstr) redirected to 0x4c2fef0 (__strstr_sse2)
--20868-- REDIR: 0x5f78cb0 (libc.so.6:memchr) redirected to 0x4c2cc80 (memchr)
--20868-- REDIR: 0x5f7e850 (libc.so.6:__GI_memcpy) redirected to 0x4c2d580 (__GI_memcpy)
--20868-- REDIR: 0x5f8fbf0 (libc.so.6:strstr) redirected to 0x4a24780 (_vgnU_ifunc_wrapper)
--20868-- REDIR: 0x602a620 (libc.so.6:__strstr_sse42) redirected to 0x4c2ff80 (__strstr_sse42)
--20868-- REDIR: 0x5f6fc20 (libc.so.6:free) redirected to 0x4c29c56 (free)
--20868-- REDIR: 0x5f76a90 (libc.so.6:strlen) redirected to 0x4a24780 (_vgnU_ifunc_wrapper)
--20868-- REDIR: 0x6030670 (libc.so.6:__strlen_sse42) redirected to 0x4c2ba50 (__strlen_sse42)
--20868-- REDIR: 0x56dd640 (libstdc++.so.6:operator new(unsigned long)) redirected to 0x4c2917c (operator new(unsigned long))
--20868-- REDIR: 0x5f7e7e0 (libc.so.6:memcpy@@GLIBC_2.14) redirected to 0x4a24780 (_vgnU_ifunc_wrapper)
--20868-- REDIR: 0x6036480 (libc.so.6:__memcpy_ssse3) redirected to 0x4c2cfa0 (memcpy@@GLIBC_2.14)
--20868-- REDIR: 0x5f74f80 (libc.so.6:index) redirected to 0x4a24780 (_vgnU_ifunc_wrapper)
--20868-- REDIR: 0x6028640 (libc.so.6:__strchr_sse42) redirected to 0x4c2b640 (index)
--20868-- REDIR: 0x56db6f0 (libstdc++.so.6:operator delete(void*)) redirected to 0x4c2a106 (operator delete(void*))
--20868-- REDIR: 0x5f79040 (libc.so.6:__GI_memcmp) redirected to 0x4c2e4d0 (__GI_memcmp)
--20868-- REDIR: 0x60286f0 (libc.so.6:__strcmp_sse42) redirected to 0x4c2cb90 (__strcmp_sse42)
--20868-- REDIR: 0x5f70210 (libc.so.6:calloc) redirected to 0x4c2a8df (calloc)
--20868-- REDIR: 0x5f800d0 (libc.so.6:strchrnul) redirected to 0x4c2f780 (strchrnul)
--20868-- REDIR: 0x5f797f0 (libc.so.6:__GI_mempcpy) redirected to 0x4c2f990 (__GI_mempcpy)
--20868-- REDIR: 0x5f74d80 (libc.so.6:strcat) redirected to 0x4a24780 (_vgnU_ifunc_wrapper)
--20868-- REDIR: 0x5f88a90 (libc.so.6:__strcat_ssse3) redirected to 0x4c2b680 (strcat)
--20868-- REDIR: 0xffffffffff600400 (???:???) redirected to 0x58059dbd (???)
--20868-- REDIR: 0x5f79680 (libc.so.6:__GI_memset) redirected to 0x4c2ee80 (memset)
--20868-- REDIR: 0xffffffffff600000 (???:???) redirected to 0x58059db3 (???)
--20868-- REDIR: 0x5f79000 (libc.so.6:bcmp) redirected to 0x4a24780 (_vgnU_ifunc_wrapper)
--20868-- REDIR: 0x6053910 (libc.so.6:__memcmp_sse4_1) redirected to 0x4c2e610 (__memcmp_sse4_1)
--20868-- REDIR: 0x5f79440 (libc.so.6:__GI_memmove) redirected to 0x4c2f1d0 (__GI_memmove)
--20868-- REDIR: 0x5f7fec0 (libc.so.6:__GI___rawmemchr) redirected to 0x4c2f7e0 (__GI___rawmemchr)
--20868-- REDIR: 0x56dd700 (libstdc++.so.6:operator new[](unsigned long)) redirected to 0x4c29821 (operator new[](unsigned long))
--20868-- REDIR: 0x603b820 (libc.so.6:__memmove_ssse3) redirected to 0x4c2cd40 (memcpy@GLIBC_2.2.5)
--20868-- REDIR: 0x5f76ce0 (libc.so.6:__GI_strncmp) redirected to 0x4c2c220 (__GI_strncmp)
--20868-- REDIR: 0x5f79ce0 (libc.so.6:__GI_stpcpy) redirected to 0x4c2e8d0 (__GI_stpcpy)
--20868-- REDIR: 0x56db720 (libstdc++.so.6:operator delete[](void*)) redirected to 0x4c2a5b6 (operator delete[](void*))
==20868== Warning: invalid file descriptor -1 in syscall close()
==20868==    at 0x4E4281D: ??? (syscall-template.S:81)
==20868==    by 0x49969C: RTSPServer::~RTSPServer() (RTSPServer.cpp:394)
==20868==    by 0x4A9DB3: RTSPServerWithREGISTERProxying::~RTSPServerWithREGISTERProxying() (RTSPServer.cpp:2816)
==20868==    by 0x45D2F5: iv::Server::~Server() (Server.cpp:292)
==20868==    by 0x45D329: iv::Server::~Server() (Server.cpp:311)
==20868==    by 0x4804D9: MediaLookupTable::remove(char const*) (Media.cpp:151)
==20868==    by 0x480100: Medium::close(UsageEnvironment&, char const*) (Media.cpp:53)
==20868==    by 0x48013F: Medium::close(Medium*) (Media.cpp:59)
==20868==    by 0x45CE03: iv::Server::releaseServer(iv::Server*&) (Server.cpp:269)
==20868==    by 0x45A5B8: main (Main.cpp:543)
==20868==
==20868== FILE DESCRIPTORS: 5 open at exit.
==20868== Open file descriptor 2: /var/log/ivrtspproxyserver/log/ivrtspproxyserver.log
==20868==    at 0x5FDA3A7: dup2 (syscall-template.S:81)
==20868==    by 0x476FBD: Logger::startNewSession(char) (Logger.cpp:712)
==20868==    by 0x4751CC: Logger::createLogFile(std::string, std::string, char) (Logger.cpp:441)
==20868==    by 0x4570FD: LogFile() (Main.cpp:84)
==20868==    by 0x458164: main (Main.cpp:283)
==20868==
==20868== Open file descriptor 1: /var/log/ivrtspproxyserver/log/ivrtspproxyserver.log
==20868==    at 0x5FDA3A7: dup2 (syscall-template.S:81)
==20868==    by 0x476FA7: Logger::startNewSession(char) (Logger.cpp:711)
==20868==    by 0x4751CC: Logger::createLogFile(std::string, std::string, char) (Logger.cpp:441)
==20868==    by 0x4570FD: LogFile() (Main.cpp:84)
==20868==    by 0x458164: main (Main.cpp:283)
==20868==
==20868== Open file descriptor 4: /var/log/ivrtspproxyserver/log/ivrtspproxyserver.log
==20868==    at 0x4E42F40: __open_nocancel (syscall-template.S:81)
==20868==    by 0x476E4B: Logger::startNewSession(char) (Logger.cpp:696)
==20868==    by 0x4751CC: Logger::createLogFile(std::string, std::string, char) (Logger.cpp:441)
==20868==    by 0x4570FD: LogFile() (Main.cpp:84)
==20868==    by 0x458164: main (Main.cpp:283)
==20868==
==20868== Open file descriptor 3: /home/ec2-user/akshay/mem_leak/IV-proxyserver/ProxyServer/leak_details.txt
==20868==    <inherited from parent>
==20868==
==20868== Open file descriptor 0: /dev/pts/2
==20868==    <inherited from parent>
==20868==
==20868==
==20868== HEAP SUMMARY:
==20868==     in use at exit: 261 bytes in 4 blocks
==20868==   total heap usage: 838 allocs, 834 frees, 289,041 bytes allocated
==20868==
==20868== Searching for pointers to 4 not-freed blocks
==20868== Checked 1,354,824 bytes
==20868==
==20868== 46 bytes in 1 blocks are still reachable in loss record 1 of 4
==20868==    at 0x4C29203: operator new(unsigned long) (vg_replace_malloc.c:334)
==20868==    by 0x571E858: allocate (new_allocator.h:111)
==20868==    by 0x571E858: std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) (basic_string.tcc:1057)
==20868==    by 0x57202E4: char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) (basic_string.tcc:578)
==20868==    by 0x572070B: _S_construct_aux<char const*> (basic_string.h:5031)
==20868==    by 0x572070B: _S_construct<char const*> (basic_string.h:5052)
==20868==    by 0x572070B: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) (basic_string.tcc:665)
==20868==    by 0x4570BB: LogFile() (Main.cpp:84)
==20868==    by 0x458164: main (Main.cpp:283)
==20868==
==20868== 55 bytes in 1 blocks are still reachable in loss record 2 of 4
==20868==    at 0x4C29203: operator new(unsigned long) (vg_replace_malloc.c:334)
==20868==    by 0x571E858: allocate (new_allocator.h:111)
==20868==    by 0x571E858: std::string::_Rep::_S_create(unsigned long, unsigned long, std::allocator<char> const&) (basic_string.tcc:1057)
==20868==    by 0x57202E4: char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) (basic_string.tcc:578)
==20868==    by 0x572070B: _S_construct_aux<char const*> (basic_string.h:5031)
==20868==    by 0x572070B: _S_construct<char const*> (basic_string.h:5052)
==20868==    by 0x572070B: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) (basic_string.tcc:665)
==20868==    by 0x4570E1: LogFile() (Main.cpp:84)
==20868==    by 0x458164: main (Main.cpp:283)
==20868==
==20868== 72 bytes in 1 blocks are still reachable in loss record 3 of 4
==20868==    at 0x4C29203: operator new(unsigned long) (vg_replace_malloc.c:334)
==20868==    by 0x4782AC: IVUtils::Object<IVUtils::Mutex>::CreateNew() (Object.h:21)
==20868==    by 0x4781DF: IVUtils::CreateMutex() (Mutex.cpp:73)
==20868==    by 0x474CCE: Logger::Logger() (Logger.cpp:404)
==20868==    by 0x456FEF: LogFile() (Main.cpp:74)
==20868==    by 0x458164: main (Main.cpp:283)
==20868==
==20868== 88 bytes in 1 blocks are still reachable in loss record 4 of 4
==20868==    at 0x4C29203: operator new(unsigned long) (vg_replace_malloc.c:334)
==20868==    by 0x456FE4: LogFile() (Main.cpp:74)
==20868==    by 0x458164: main (Main.cpp:283)
==20868==
==20868== LEAK SUMMARY:
==20868==    definitely lost: 0 bytes in 0 blocks
==20868==    indirectly lost: 0 bytes in 0 blocks
==20868==      possibly lost: 0 bytes in 0 blocks
==20868==    still reachable: 261 bytes in 4 blocks
==20868==                       of which reachable via heuristic:
==20868==                         stdstring          : 101 bytes in 2 blocks
==20868==         suppressed: 0 bytes in 0 blocks
==20868==
==20868== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)
==20868== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

如此缓慢的内存消耗还有什么原因？

【问题讨论】：

也许与其说是内存泄漏，不如说是一种逐渐增长的数据结构。 Valgrind 不一定会将其显示为泄漏。
根据您的 valgrind 输出，您的程序没有泄漏任何内存。因此，如果程序的 RAM 使用量随着时间的推移而增长，那一定是它的数据结构越来越大。以前当我看到这种行为时，结果是例如std::vector 代码一直无限期地附加项目，但从未清除或删除任何项目。（当然任何可变大小的数据结构都可能有同样的问题；我只是用std::vector作为一个简单的例子）
所以你认为问题出在LogFile？将有助于查看该代码。
gdb 能否帮助我确定哪个数据结构正在逐渐增长，或者哪个数据结构正在填充并且在使用该条目后未被清除？
gdb 没有你想的那么有用。您必须定期停止程序并检查所有可能增长的对象，并证明或进一步检查自上次检查以来增长的对象。如果您可以加速程序以强制更快地消耗资源，这可能是合理的。如果不是，您可能会发现自己在发现并确认坏人之前运行了几个月的调试实例。定期记录感兴趣对象的大小可能更实用。

标签： c++ linux memory-leaks valgrind

【解决方案1】：

从你的跟踪来看，你有 4 个东西仍在使用 - 记录器、记录器的互斥锁和两个 std::strings。

我建议您使用客户端请求检测您的代码（请参阅here）。这样你就可以看到内存在哪里被增量使用。我特别想尝试使用VALGRIND_DO_ADDED_LEAK_CHECK 宏。

平均而言，您每秒添加大约 300-350 个字节。在 Valgrind 下运行时，使用您对记录事件时间的了解 - 您希望它运行足够长的时间以至少捕获一些事件。

【讨论】：

【解决方案2】：

查看此类错误的一种方法是使用https://github.com/vmware/chap

您根本不需要检测您的流程。只需使用 gcore 收集您的流程的实时核心。可能不止一次这样做，间隔一个小时。

对于每个核心，在 chap 中打开它然后尝试：

redirect on
summarize used

或

redirect on
summarize used /sortby bytes

比较结果将使您了解正在累积的分配类型，您可以进一步查看这些分配。例如，假设您发现 Foo 类型的分配有所增加。然后你可以做

enumerate used Foo

要获取此类分配的所有地址的列表，然后选择一个此类地址并反复使用类似以下内容来找出它们为何保存在内存中：

describe incoming <address>

在这种情况下，您可能使用的初始地址是枚举中的一个。

【讨论】：