CGImageRef 访问像素数据的更快方法？

Question

我目前的方法是：

CGDataProviderRef provider = CGImageGetDataProvider(imageRef);
imageData.rawData = CGDataProviderCopyData(provider);
imageData.imageData = (UInt8 *) CFDataGetBytePtr(imageData.rawData);

我每秒只能获得大约 30 帧。我知道部分性能损失是复制数据，如果我可以访问字节流而不让它自动为我创建副本，那就太好了。

我正在尝试让它尽快处理 CGImageRefs，有更快的方法吗？

这是我的工作解决方案 sn-p：

- (void)applicationDidFinishLaunching:(NSNotification *)aNotification {
    // Insert code here to initialize your application
    //timer = [NSTimer scheduledTimerWithTimeInterval:1.0/60.0 //2000.0
    //                                         target:self
    //                                       selector:@selector(timerLogic)
    //                                       userInfo:nil
    //                                        repeats:YES];
    leagueGameState = [LeagueGameState new];

    [self updateWindowList];
    lastTime = CACurrentMediaTime();






    // Create a capture session
    mSession = [[AVCaptureSession alloc] init];

    // Set the session preset as you wish
    mSession.sessionPreset = AVCaptureSessionPresetMedium;

    // If you're on a multi-display system and you want to capture a secondary display,
    // you can call CGGetActiveDisplayList() to get the list of all active displays.
    // For this example, we just specify the main display.
    // To capture both a main and secondary display at the same time, use two active
    // capture sessions, one for each display. On Mac OS X, AVCaptureMovieFileOutput
    // only supports writing to a single video track.
    CGDirectDisplayID displayId = kCGDirectMainDisplay;

    // Create a ScreenInput with the display and add it to the session
    AVCaptureScreenInput *input = [[AVCaptureScreenInput alloc] initWithDisplayID:displayId];
    input.minFrameDuration = CMTimeMake(1, 60);

    //if (!input) {
    //    [mSession release];
    //    mSession = nil;
    //    return;
    //}
    if ([mSession canAddInput:input]) {
        NSLog(@"Added screen capture input");
        [mSession addInput:input];
    } else {
        NSLog(@"Couldn't add screen capture input");
    }

    //**********************Add output here
    //dispatch_queue_t _videoDataOutputQueue;
    //_videoDataOutputQueue = dispatch_queue_create( "com.apple.sample.capturepipeline.video", DISPATCH_QUEUE_SERIAL );
    //dispatch_set_target_queue( _videoDataOutputQueue, dispatch_get_global_queue( DISPATCH_QUEUE_PRIORITY_HIGH, 0 ) );

    AVCaptureVideoDataOutput *videoOut = [[AVCaptureVideoDataOutput alloc] init];
    videoOut.videoSettings = @{ (id)kCVPixelBufferPixelFormatTypeKey : @(kCVPixelFormatType_32BGRA) };
    [videoOut setSampleBufferDelegate:self queue:dispatch_get_main_queue()];

    // RosyWriter records videos and we prefer not to have any dropped frames in the video recording.
    // By setting alwaysDiscardsLateVideoFrames to NO we ensure that minor fluctuations in system load or in our processing time for a given frame won't cause framedrops.
    // We do however need to ensure that on average we can process frames in realtime.
    // If we were doing preview only we would probably want to set alwaysDiscardsLateVideoFrames to YES.
    videoOut.alwaysDiscardsLateVideoFrames = YES;

    if ( [mSession canAddOutput:videoOut] ) {
        NSLog(@"Added output video");
        [mSession addOutput:videoOut];
    } else {NSLog(@"Couldn't add output video");}


    // Start running the session
    [mSession startRunning];

    NSLog(@"Set up session");
}




- (void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection
{
    //NSLog(@"Captures output from sample buffer");
    //CMFormatDescriptionRef formatDescription = CMSampleBufferGetFormatDescription( sampleBuffer );
/*
        if ( self.outputVideoFormatDescription == nil ) {
            // Don't render the first sample buffer.
            // This gives us one frame interval (33ms at 30fps) for setupVideoPipelineWithInputFormatDescription: to complete.
            // Ideally this would be done asynchronously to ensure frames don't back up on slower devices.
            [self setupVideoPipelineWithInputFormatDescription:formatDescription];
        }
        else {*/
            [self renderVideoSampleBuffer:sampleBuffer];
        //}
}

- (void)renderVideoSampleBuffer:(CMSampleBufferRef)sampleBuffer
{
    //CVPixelBufferRef renderedPixelBuffer = NULL;
    //CMTime timestamp = CMSampleBufferGetPresentationTimeStamp( sampleBuffer );

    //[self calculateFramerateAtTimestamp:timestamp];

    // We must not use the GPU while running in the background.
    // setRenderingEnabled: takes the same lock so the caller can guarantee no GPU usage once the setter returns.
    //@synchronized( _renderer )
    //{
    //    if ( _renderingEnabled ) {
    CVPixelBufferRef sourcePixelBuffer = CMSampleBufferGetImageBuffer( sampleBuffer );

    const int kBytesPerPixel = 4;

    CVPixelBufferLockBaseAddress( sourcePixelBuffer, 0 );

    int bufferWidth = (int)CVPixelBufferGetWidth( sourcePixelBuffer );
    int bufferHeight = (int)CVPixelBufferGetHeight( sourcePixelBuffer );
    size_t bytesPerRow = CVPixelBufferGetBytesPerRow( sourcePixelBuffer );
    uint8_t *baseAddress = CVPixelBufferGetBaseAddress( sourcePixelBuffer );

    int count = 0;
    for ( int row = 0; row < bufferHeight; row++ )
    {
        uint8_t *pixel = baseAddress + row * bytesPerRow;
        for ( int column = 0; column < bufferWidth; column++ )
        {
            count ++;
            pixel[1] = 0; // De-green (second pixel in BGRA is green)
            pixel += kBytesPerPixel;
        }
    }

    CVPixelBufferUnlockBaseAddress( sourcePixelBuffer, 0 );


    //NSLog(@"Test Looped %d times", count);

    CIImage *ciImage = [CIImage imageWithCVImageBuffer:sourcePixelBuffer];


    /*
    CIContext *temporaryContext = [CIContext contextWithCGContext:
                                             [[NSGraphicsContext currentContext] graphicsPort]
                                                          options: nil];

    CGImageRef videoImage = [temporaryContext
                             createCGImage:ciImage
                             fromRect:CGRectMake(0, 0,
                                                 CVPixelBufferGetWidth(sourcePixelBuffer),
                                                 CVPixelBufferGetHeight(sourcePixelBuffer))];

    */

    //UIImage *uiImage = [UIImage imageWithCGImage:videoImage];

    // Create a bitmap rep from the image...
    NSBitmapImageRep *bitmapRep = [[NSBitmapImageRep alloc] initWithCIImage:ciImage];
    // Create an NSImage and add the bitmap rep to it...
    NSImage *image = [[NSImage alloc] init];
    [image addRepresentation:bitmapRep];
    // Set the output view to the new NSImage.
    [imageView setImage:image];

    //CGImageRelease(videoImage);



    //renderedPixelBuffer = [_renderer copyRenderedPixelBuffer:sourcePixelBuffer];
    //    }
    //    else {
    //        return;
    //    }
    //}

    //Profile code? See how fast it's running?
    if (CACurrentMediaTime() - lastTime > 3) //10 seconds
    {
        float time = CACurrentMediaTime() - lastTime;
        [fpsText setStringValue:[NSString stringWithFormat:@"Elapsed Time: %f ms, %f fps", time * 1000 / loopsTaken, (1000.0)/(time * 1000.0 / loopsTaken)]];
        lastTime = CACurrentMediaTime();
        loopsTaken = 0;
        [self updateWindowList];
        if (leagueGameState.leaguePID == -1) {
            [statusText setStringValue:@"No League Instance Found"];
        }
    }
    else
    {
        loopsTaken++;
    }

}

即使在循环数据之后，我也能获得非常好的每秒 60 帧。

它捕获屏幕，获取数据，修改数据并重新显示数据。

Answer 1

您指的是哪个“字节流”？ CGImage 代表最终的位图数据，但在后台它可能仍被压缩。位图当前可能存储在 GPU 上，因此获取它可能需要 GPU->CPU 获取（这很昂贵，当您不需要它时应该避免使用它）。

如果您尝试以超过 30 fps 的速度执行此操作，您可能需要重新考虑解决问题的方法，并使用为此设计的工具，例如 Core Image、Core Video 或 Metal。 Core Graphics 针对显示进行了优化，而不是处理（绝对不是实时处理）。 Core Image 等工具的一个关键区别在于，您可以在 GPU 上执行更多工作，而无需将数据重新传输回 CPU。这对于维护快速管道绝对至关重要。尽可能避免获取实际字节数。

如果你已经有一个CGImage，你可以用imageWithCGImage:将它转换成一个CIImage，然后使用CIImage进一步处理它。如果您确实需要访问字节，您的选项就是您正在使用的选项，或者使用CGContextDrawImage 将其渲染为位图上下文（这也需要复制）。无法保证 CGImage 在任何给定时间都有一堆位图字节挂在你可以查看的地方，并且它不提供像 Core Video 等实时框架中那样的“锁定缓冲区”方法.

WWDC 视频中对高速图像处理的一些非常好的介绍：

• WWDC 2013 Session 509 核心图像效果和技术
• WWDC 2014 Session 514 核心图像进展
• WWDC 2014 会议 603-605 使用金属