如何在 AVCaptureSession 中为视频的每一帧应用过滤器？答案

【问题标题】：How can I apply filter for each frame of a video in AVCaptureSession?如何在 AVCaptureSession 中为视频的每一帧应用过滤器？
【发布时间】：2023-03-09 01:34:01
【问题描述】：

我正在编写一个应用程序，它需要对使用 AVCaptureSession 捕获的视频应用过滤器。过滤后的输出被写入输出文件。我目前使用 CIFilter 和 CIImage 过滤每个视频帧。代码如下：

func captureOutput(_ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, from connection: AVCaptureConnection!) {
    ...
    let pixelBuffer = CMSampleBufferGetImageBuffer(samples)!
    let options = [kCVPixelBufferPixelFormatTypeKey as String : kCVPixelFormatType_420YpCbCr8BiPlanarFullRange]
    let cameraImage = CIImage(cvImageBuffer: pixelBuffer, options: options)
    let filter = CIFilter(name: "CIGaussianBlur")!
    filter.setValue((70.0), forKey: kCIInputRadiusKey)
    filter.setValue(cameraImage, forKey: kCIInputImageKey)
    let result = filter.outputImage!
    var pixBuffer:CVPixelBuffer? = nil;
    let fmt = CVPixelBufferGetPixelFormatType(pixelBuffer)
    CVPixelBufferCreate(kCFAllocatorSystemDefault,
                        CVPixelBufferGetWidth(pixelBuffer),
                        CVPixelBufferGetHeight(pixelBuffer),
                        fmt,
                        CVBufferGetAttachments(pixelBuffer, .shouldPropagate),
                        &pixBuffer);

    CVBufferPropagateAttachments(pixelBuffer, pixBuffer!)
    let eaglContext = EAGLContext(api: EAGLRenderingAPI.openGLES3)!
    eaglContext.isMultiThreaded = true
    let contextOptions = [kCIContextWorkingColorSpace : NSNull(), kCIContextOutputColorSpace: NSNull()]
    let context = CIContext(eaglContext: eaglContext, options: contextOptions)
    CVPixelBufferLockBaseAddress( pixBuffer!, CVPixelBufferLockFlags(rawValue: 0))
    context.render(result, to: pixBuffer!)
    CVPixelBufferUnlockBaseAddress( pixBuffer!, CVPixelBufferLockFlags(rawValue: 0))
    var timeInfo = CMSampleTimingInfo(duration: sampleBuffer.duration,
                                      presentationTimeStamp: sampleBuffer.presentationTimeStamp,
                                      decodeTimeStamp: sampleBuffer.decodeTimeStamp)
    var sampleBuf:CMSampleBuffer? = nil;
    CMSampleBufferCreateReadyWithImageBuffer(kCFAllocatorDefault,
                                             pixBuffer!,
                                             samples.formatDescription!,
                                             &timeInfo,
                                             &sampleBuf)

    // write to video file
    let ret = assetWriterInput.append(sampleBuf!)
    ...
}

来自 AVAssetWriterInput.append 的 ret 始终为 false。我在这里做错了什么？另外，我使用的方法效率很低。沿途会创建一些临时副本。可以就地实现吗？

【问题讨论】：

你在修改sampleBuffer吗？
我目前没有修改 sampleBuffer。不过，如果可以的话，那就太好了。这将使我免于为过滤后的输出创建新缓冲区。
检查this。它说我们不能编辑它。使用那里告诉的方法看看错误是什么。

标签： ios avcapturesession cifilter ciimage

【解决方案1】：

我使用了几乎相同的代码来解决同样的问题。我发现为渲染创建的像素缓冲区有问题。 append(sampleBuffer:) 总是返回 false 而 assetWriter.error 是

Error Domain=AVFoundationErrorDomain Code=-11800 "该操作可能未完成” UserInfo={NSUnderlyingError=0x17024ba30 {错误域=NSOSStatusErrorDomain 代码=-12780 "(null)"}, NSLocalizedFailureReason=发生未知错误（-12780）， NSLocalizedDescription=操作无法完成}

他们说这是一个错误（如here 所述），已经发布：https://bugreport.apple.com/web/?problemID=34574848。

但出乎意料的是，当使用原始像素缓冲区进行渲染时，我发现问题消失了。见以下代码：

let sourcePixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
let sourceImage = CIImage(cvImageBuffer: sourcePixelBuffer)
let filter = CIFilter(name: "CIGaussianBlur", withInputParameters: [kCIInputImageKey: sourceImage])!
let filteredImage = filter.outputImage!

var pixelBuffer: CVPixelBuffer? = nil
let width = CVPixelBufferGetWidth(sourcePixelBuffer)
let height = CVPixelBufferGetHeight(sourcePixelBuffer)
let pixelFormat = CVPixelBufferGetPixelFormatType(sourcePixelBuffer)
let attributes = CVBufferGetAttachments(sourcePixelBuffer, .shouldPropagate)!
CVPixelBufferCreate(nil, width, height, pixelFormat, attributes, &pixelBuffer)
CVBufferPropagateAttachments(sourcePixelBuffer, pixelBuffer!)

var filteredPixelBuffer = pixelBuffer!      // this never works
filteredPixelBuffer = sourcePixelBuffer     // 0_0

let context = CIContext(options: [kCIContextOutputColorSpace: CGColorSpace(name: CGColorSpace.sRGB)!])
context.render(filteredImage, to: filteredPixelBuffer)  // modifying original image buffer here!

let presentationTimestamp = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
var timing = CMSampleTimingInfo(duration: kCMTimeInvalid, presentationTimeStamp: presentationTimestamp, decodeTimeStamp: kCMTimeInvalid)

var processedSampleBuffer: CMSampleBuffer? = nil
var formatDescription: CMFormatDescription? = nil
CMVideoFormatDescriptionCreateForImageBuffer(nil, filteredPixelBuffer, &formatDescription)
CMSampleBufferCreateReadyWithImageBuffer(nil, filteredPixelBuffer, formatDescription!, &timing, &processedSampleBuffer)

print(assetInput!.append(processedSampleBuffer!))

当然，我们都知道您不允许修改样本缓冲区，但不知何故，这种方法可以提供正常处理的视频。 Trick 很脏，我不能说在你有预览层或一些并发处理例程的情况下它是否会很好。

【讨论】：