将 ProresRAW 格式解码为原生拜耳表示

Question

我正在尝试解码 Prores 视频文件，但它不起作用。我总是得到

Optional(Error Domain=AVFoundationErrorDomain Code=-11821 "Cannot Decode" UserInfo={NSLocalizedFailureReason=媒体数据无法解码。可能已损坏。, NSLocalizedDescription=无法解码, NSUnderlyingError=0x600002a982a0 {Error Domain=NSOSStatusErrorDomain Code =-12137 "(null)"}})

这里有一个完整的解码器：

class Decoder {

private let assetReader: AVAssetReader?
private let output: AVAssetReaderTrackOutput

init() throws {
    
    VTRegisterProfessionalVideoWorkflowVideoDecoders()
    VTRegisterProfessionalVideoWorkflowVideoEncoders()
    
        let assetReader = try AVAssetReader(asset: movieAsset)
        let tracks = movieAsset.tracks(withMediaType: .video)
        
        guard let firstTrack = tracks.first else {
            print("No video tracks found")
            throw NSError()
        }
    
        let out = AVAssetReaderTrackOutput(track: firstTrack, outputSettings: outputSettings)
        out.alwaysCopiesSampleData = true
        
        assetReader.add(out)
        
        self.assetReader = assetReader
        self.output = out

   
}

func run(){
    
    guard let assetReader = assetReader, assetReader.startReading() else {
        print("Failed to stard asset reader")
        return
    }
    

    while(assetReader.status == .reading) {
        guard let sampleBuffer = output.copyNextSampleBuffer() else {
            print(assetReader.status.rawValue)
            print(assetReader.error)
            continue
        }
          
        print("Decoding success!")
    }
}

}

Answer 1

不清楚你为什么想要拜耳，我不确定你所说的“本地”是什么意思，但我猜你可能希望你的数据是

1. 尽可能高的定义，或
2. 以其最自然/高效/最少处理的格式
3. 拜耳别再问我问题了

所以我认为有两种可能。

如果您喜欢高清数据，请尝试将您的AVAssetReaderTrackOutput 像素格式设置为kCVPixelFormatType_444YpCbCr16VideoRange_16A_TriPlanar、kCVPixelFormatType_4444AYpCbCr16 或kCVPixelFormatType_64RGBALE 或AVAssetReaderTrackOutput 中提到的其他格式之一。我认为AVAssetReader 不会无缘无故截断数据的可能性很大。

在使用 ProRes RAW 时，我不知道自然或高效的表示，但如果你真的想要 Bayer 输出，你可以将你的 outputSettings 设置为 nil 并使用 VTDecompressionSession 将原始样本缓冲区转换为kCVPixelFormatType_16VersatileBayer（或kCVPixelFormatType_64RGBAHalf，kCVPixelFormatType_128RGBAFloat，如果您仍然使用AVAssetReader 出于某种原因不喜欢的高范围格式），但不是 kCVPixelFormatType_64RGBA_DownscaledProResRAW 因为那似乎不是工作。

无论如何，您可以稍微修改您的代码以解码为kCVPixelFormatType_16VersatileBayer，如下所示：

import AVFoundation
import VideoToolbox

class Decoder {
    private let assetReader: AVAssetReader?
    private let output: AVAssetReaderTrackOutput
    private var decompressionSession: VTDecompressionSession!
        
    init() throws {
        let movieUrl = URL(fileURLWithPath: "/Users/xxxx/ProresRAW_Video.MOV")
        let movieAsset = AVAsset(url: movieUrl)
        
        do {
            let assetReader = try AVAssetReader(asset: movieAsset)
            let tracks = movieAsset.tracks(withMediaType: .video)
            
            guard let firstTrack = tracks.first else {
                print("No video tracks found")
                throw NSError()
            }
            
            let out = AVAssetReaderTrackOutput(track: firstTrack, outputSettings: nil)
            out.alwaysCopiesSampleData = true
            
            assetReader.add(out)
            
            self.assetReader = assetReader
            self.output = out
            
        } catch {
            print(error)
            throw error
        }
        
    }
    
    func run(){
        guard let assetReader = assetReader, assetReader.startReading() else {
            print("Failed to stard asset reader")
            return
        }
        
        while(assetReader.status == .reading) {
            guard let sampleBuffer = output.copyNextSampleBuffer() else {
                print(assetReader.status.rawValue)
                print(assetReader.error)
                continue
            }
            
            print("Decoding success! \(sampleBuffer)")
            
            if let formatDescription = CMSampleBufferGetFormatDescription(sampleBuffer) {
                if decompressionSession == nil {
                    let imageBufferAttributes: [CFString: Any] = [
                        kCVPixelBufferPixelFormatTypeKey: kCVPixelFormatType_16VersatileBayer
                    ]
                    var outputCallback = VTDecompressionOutputCallbackRecord(decompressionOutputCallback: { _, _, status, infoFlags, imageBuffer, presentationTimeStamp, presentationDuration in
                        assert(noErr == status)
                        print("decode callback status: \(status), bayer imageBuffer \(String(describing: imageBuffer)), flags: \(infoFlags), pts: \(presentationDuration), ptsd: \(presentationDuration)")
                    }, decompressionOutputRefCon: nil)
                    let status = VTDecompressionSessionCreate(allocator: nil, formatDescription: formatDescription, decoderSpecification: nil, imageBufferAttributes: imageBufferAttributes as CFDictionary, outputCallback: &outputCallback, decompressionSessionOut: &decompressionSession)
                    assert(noErr == status)
                }

                let status = VTDecompressionSessionDecodeFrame(decompressionSession, sampleBuffer: sampleBuffer, flags: [], frameRefcon: nil, infoFlagsOut: nil)
                assert(noErr == status)
            }
        }
    }
}

我不明白为什么AVAssetReader（它可能在后台使用VTDecompressionSession）并不能简单地让您首先请求kCVPixelFormatType_16VersatileBayer。也许这是血腥的思想，或者也许没有意义？ p.s.你想做什么？