如何在不泄漏内存的情况下正确关闭 FFmpeg 流和 AVFormatContext？

Question

我已经构建了一个应用程序，它使用 FFmpeg 连接到远程 IP 摄像机，以便通过 RTSP 2.0 接收视频和音频帧。

该应用是使用 Xcode 10-11 和 Objective-C 构建的，并带有自定义 FFmpeg 构建配置。

架构如下：

MyApp


Document_0

    RTSPContainerObject_0
        RTSPObject_0

    RTSPContainerObject_1
        RTSPObject_1

    ...
Document_1
...

目标：

1. 关闭Document_0 后不会泄露FFmpeg 对象。
2. 关闭进程应该停止帧读取并销毁所有使用FFmpeg的对象。

问题：

1. 不知何故，Xcode 的内存调试器显示了 MyApp 的两个实例。

事实：

• macOS 的活动监视器不显示 MyApp 的两个实例。

• macOS 的活动监视器没有 FFmpeg 或其他子进程的任何实例。

• 由于内存快照延迟，问题与一些剩余内存无关，因为它可以轻松复制。

• Xcode 的内存调试器显示第二个实例只有 RTSPObject's AVFormatContext 而没有其他对象。

  1. 第二个实例有一个AVFormatContext，而RTPSObject 仍然有一个指向AVFormatContext 的指针。

事实：

• 打开和关闭第二个文档Document_1 会导致同样的问题并且有两个对象泄漏。这意味着存在一个会产生可扩展问题的错误。越来越多的内存被使用和不可用。

这是我的终止代码：

   - (void)terminate
{
    // * Video and audio frame provisioning termination *
    [self stopVideoStream];
    [self stopAudioStream];
    // *

    // * Video codec termination *
    avcodec_free_context(&_videoCodecContext); // NULL pointer safe.
    self.videoCodecContext = NULL;
    // *

// * Audio codec termination *
avcodec_free_context(&_audioCodecContext); // NULL pointer safe.
self.audioCodecContext = NULL;
// *

if (self.packet)
{
    // Free the packet that was allocated by av_read_frame.
    av_packet_unref(&packet); // The documentation doesn't mention NULL safety.
    self.packet = NULL;
}

if (self.currentAudioPacket)
{
    av_packet_unref(_currentAudioPacket);
    self.currentAudioPacket = NULL;
}

// Free raw frame data.
av_freep(&_rawFrameData); // NULL pointer safe.

// Free the swscaler context swsContext.
self.isFrameConversionContextAllocated = NO;
sws_freeContext(scallingContext); // NULL pointer safe.

[self.audioPacketQueue removeAllObjects];

self.audioPacketQueue = nil;

self.audioPacketQueueLock = nil;
self.packetQueueLock = nil;
self.audioStream = nil;
BXLogInDomain(kLogDomainSources, kLogLevelVerbose, @"%s:%d: All streams have been terminated!", __FUNCTION__, __LINE__);

// * Session context termination *
AVFormatContext *pFormatCtx = self.sessionContext;
BOOL shouldProceedWithInputSessionTermination = self.isInputStreamOpen && self.shouldTerminateStreams && pFormatCtx;
NSLog(@"\nTerminating session context...");
if (shouldProceedWithInputSessionTermination)
{
    NSLog(@"\nTerminating...");
    //av_write_trailer(pFormatCtx);
    // Discard all internally buffered data.
    avformat_flush(pFormatCtx); // The documentation doesn't mention NULL safety.
    // Close an opened input AVFormatContext and free it and all its contents.
    // WARNING: Closing an non-opened stream will cause avformat_close_input to crash.
    avformat_close_input(&pFormatCtx); // The documentation doesn't mention NULL safety.
    NSLog(@"Logging leftovers - %p, %p  %p", self.sessionContext, _sessionContext, pFormatCtx);
    avformat_free_context(pFormatCtx);

    NSLog(@"Logging content = %c", *self.sessionContext);
    //avformat_free_context(pFormatCtx); - Not needed because avformat_close_input is closing it.
    self.sessionContext = NULL;
}
// *

}

重要提示：终止顺序为：

    New frame will be read.
-[(RTSPObject)StreamInput currentVideoFrameDurationSec]
-[(RTSPObject)StreamInput frameDuration:]
-[(RTSPObject)StreamInput currentCGImageRef]
-[(RTSPObject)StreamInput convertRawFrameToRGB]
-[(RTSPObject)StreamInput pixelBufferFromImage:]
-[(RTSPObject)StreamInput cleanup]
-[(RTSPObject)StreamInput dealloc]
-[(RTSPObject)StreamInput stopVideoStream]
-[(RTSPObject)StreamInput stopAudioStream]

Terminating session context...
Terminating...
Logging leftovers - 0x109ec6400, 0x109ec6400  0x109ec6400
Logging content = \330
-[Document dealloc]

无效的解决方案：

• 更改对象释放的顺序（AVFormatContext 已被首先释放，但并未导致任何更改）。
• 更快地调用RTSPObject's cleanup 方法，让FFmpeg 有更多时间来处理对象释放。
• 阅读大量 SO 答案和 FFmpeg 文档以找到一个干净的清理过程或更新的代码，这可能会突出显示对象释放未正确发生的原因。

我目前正在阅读AVFormatContext 上的文档，因为我相信我忘记发布某些内容了。这是基于AVFormatContext 仍然存在的内存调试器输出。

这是我的创建代码：

#pragma mark # Helpers - Start

- (NSError *)openInputStreamWithVideoStreamId:(int)videoStreamId
                                audioStreamId:(int)audioStreamId
                                     useFirst:(BOOL)useFirstStreamAvailable
                                       inInit:(BOOL)isInitProcess
{
    // NSLog(@"%s", __PRETTY_FUNCTION__); // RTSP
    self.status = StreamProvisioningStatusStarting;
    AVCodec *decoderCodec;
    NSString *rtspURL = self.streamURL;
    NSString *errorMessage = nil;
    NSError *error = nil;

    self.sessionContext = NULL;
    self.sessionContext = avformat_alloc_context();

    AVFormatContext *pFormatCtx = self.sessionContext;
    if (!pFormatCtx)
    {
        // Create approp error.
        return error;
    }


    // MUST be called before avformat_open_input().
    av_dict_free(&_sessionOptions);

        self.sessionOptions = 0;
        if (self.usesTcp)
        {
            // "rtsp_transport" - Set RTSP transport protocols.
            // Allowed are: udp_multicast, tcp, udp, http.
            av_dict_set(&_sessionOptions, "rtsp_transport", "tcp", 0);
        }
        av_dict_set(&_sessionOptions, "rtsp_transport", "tcp", 0);

    // Open an input stream and read the header with the demuxer options.
    // WARNING: The stream must be closed with avformat_close_input()
    if (avformat_open_input(&pFormatCtx, rtspURL.UTF8String, NULL, &_sessionOptions) != 0)
    {
        // WARNING: Note that a user-supplied AVFormatContext (pFormatCtx) will be freed on failure.
        self.isInputStreamOpen = NO;
        // Create approp error.
        return error;
    }

    self.isInputStreamOpen = YES;

    // user-supplied AVFormatContext pFormatCtx might have been modified.
    self.sessionContext = pFormatCtx;

    // Retrieve stream information.
    if (avformat_find_stream_info(pFormatCtx,NULL) < 0)
    {
        // Create approp error.
        return error;
    }

    // Find the first video stream
    int streamCount = pFormatCtx->nb_streams;

    if (streamCount == 0)
    {
        // Create approp error.
        return error;
    }

    int noStreamsAvailable = pFormatCtx->streams == NULL;

    if (noStreamsAvailable)
    {
        // Create approp error.
        return error;
    }

    // Result. An Index can change, an identifier shouldn't.
    self.selectedVideoStreamId = STREAM_NOT_FOUND;
    self.selectedAudioStreamId = STREAM_NOT_FOUND;

    // Fallback.
    int firstVideoStreamIndex = STREAM_NOT_FOUND;
    int firstAudioStreamIndex = STREAM_NOT_FOUND;

    self.selectedVideoStreamIndex = STREAM_NOT_FOUND;
    self.selectedAudioStreamIndex = STREAM_NOT_FOUND;

    for (int i = 0; i < streamCount; i++)
    {
        // Looking for video streams.
        AVStream *stream = pFormatCtx->streams[i];
        if (!stream) { continue; }
        AVCodecParameters *codecPar = stream->codecpar;
        if (!codecPar) { continue; }

        if (codecPar->codec_type==AVMEDIA_TYPE_VIDEO)
        {
            if (stream->id == videoStreamId)
            {
                self.selectedVideoStreamId = videoStreamId;
                self.selectedVideoStreamIndex = i;
            }

            if (firstVideoStreamIndex == STREAM_NOT_FOUND)
            {
                firstVideoStreamIndex = i;
            }
        }
        // Looking for audio streams.
        if (codecPar->codec_type==AVMEDIA_TYPE_AUDIO)
        {
            if (stream->id == audioStreamId)
            {
                self.selectedAudioStreamId = audioStreamId;
                self.selectedAudioStreamIndex = i;
            }

            if (firstAudioStreamIndex == STREAM_NOT_FOUND)
            {
                firstAudioStreamIndex = i;
            }
        }
    }

    // Use first video and audio stream available (if possible).

    if (self.selectedVideoStreamIndex == STREAM_NOT_FOUND && useFirstStreamAvailable && firstVideoStreamIndex != STREAM_NOT_FOUND)
    {
        self.selectedVideoStreamIndex = firstVideoStreamIndex;
        self.selectedVideoStreamId = pFormatCtx->streams[firstVideoStreamIndex]->id;
    }

    if (self.selectedAudioStreamIndex == STREAM_NOT_FOUND && useFirstStreamAvailable && firstAudioStreamIndex != STREAM_NOT_FOUND)
    {
        self.selectedAudioStreamIndex = firstAudioStreamIndex;
        self.selectedAudioStreamId = pFormatCtx->streams[firstAudioStreamIndex]->id;
    }

    if (self.selectedVideoStreamIndex == STREAM_NOT_FOUND)
    {
        // Create approp error.
        return error;
    }

    // See AVCodecID for codec listing.

    // * Video codec setup:
    // 1. Find the decoder for the video stream with the gived codec id.
    AVStream *stream = pFormatCtx->streams[self.selectedVideoStreamIndex];
    if (!stream)
    {
        // Create approp error.
        return error;
    }
    AVCodecParameters *codecPar = stream->codecpar;
    if (!codecPar)
    {
        // Create approp error.
        return error;
    }

    decoderCodec = avcodec_find_decoder(codecPar->codec_id);
    if (decoderCodec == NULL)
    {
        // Create approp error.
        return error;
    }

    // Get a pointer to the codec context for the video stream.
    // WARNING: The resulting AVCodecContext should be freed with avcodec_free_context().
    // Replaced:
    // self.videoCodecContext = pFormatCtx->streams[self.selectedVideoStreamIndex]->codec;
    // With:
    self.videoCodecContext = avcodec_alloc_context3(decoderCodec);
    avcodec_parameters_to_context(self.videoCodecContext,
                                  codecPar);

    self.videoCodecContext->thread_count = 4;
    NSString *description = [NSString stringWithUTF8String:decoderCodec->long_name];

    // 2. Open codec.
    if (avcodec_open2(self.videoCodecContext, decoderCodec, NULL) < 0)
    {
        // Create approp error.
        return error;
    }

    // * Audio codec setup:
    if (self.selectedAudioStreamIndex > -1)
    {
        [self setupAudioDecoder];
    }

    // Allocate a raw video frame data structure. Contains audio and video data.
    self.rawFrameData = av_frame_alloc();

    self.outputWidth = self.videoCodecContext->width;
    self.outputHeight = self.videoCodecContext->height;

    if (!isInitProcess)
    {
        // Triggering notifications in init process won't change UI since the object is created locally. All
        // objects which need data access to this object will not be able to get it. Thats why we don't notifiy anyone about the changes.
        [NSNotificationCenter.defaultCenter postNotificationName:NSNotification.rtspVideoStreamSelectionChanged
                                                          object:nil userInfo: self.selectedVideoStream];

        [NSNotificationCenter.defaultCenter postNotificationName:NSNotification.rtspAudioStreamSelectionChanged
                                                          object:nil userInfo: self.selectedAudioStream];
    }

    return nil;
}

更新 1

允许使用任何给定线程的初始架构。下面的大部分代码将主要在主线程上运行。此解决方案不合适，因为打开流输入可能需要几秒钟，在此期间主线程在等待FFmpeg 内的网络响应时被阻塞。为了解决这个问题，我实施了以下解决方案：

• 仅允许在 background_thread 上创建和初始设置（请参见下面的代码 sn-p "1"）。
• current_thread(Any) 允许更改。
• current_thread(Any) 允许终止。

将main thread 检查和dispatch_asyncs 删除到后台线程后，泄漏已停止，我无法再重现该问题：

// Code that produces the issue.   
dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{
    // 1 - Create and do initial setup. 
    // This block creates the issue. 
[self.rtspObject = [[RTSPObject alloc] initWithURL: ... ];
[self.rtspObject openInputStreamWithVideoStreamId: ...
                                audioStreamId: ...
                                     useFirst: ...
                                       inInit: ...];
});

我还是不明白为什么Xcode的内存调试器说这个块被保留了？

欢迎任何建议或想法。

Answer 1

如果你使用 av_format_open_input 打开一个文件，你必须使用 avformat_close_input 来释放它。使用 free_context 将泄漏所有与 io 相关的分配。