从 SurfaceTexture (Android) 获取帧

Question

我正在使用SurfaceTexture 在后台运行设备的摄像头并从中获取帧。我看到onFrameAvailable 回调没有带来框架，而是SurfaceTexture。我想得到它的框架或图像表示，但我不知道该怎么做。当我搜索时，我发现我需要使用GraphicBuffer，但它似乎太复杂了，我不清楚如何使用它。

我也在这里查看了解决方案：

Texture Image processing on the GPU?

Android SDK: Get raw preview camera image without displaying it

但不清楚如何在代码中做到这一点。这是我的代码：

public class BackgroundService extends Service  {

    private Camera camera = null;
    private int  NOTIFICATION_ID= 1;
    private static final String TAG = "OCVSample::Activity";
    // Binder given to clients
    private final IBinder mBinder = new LocalBinder();
    private WindowManager windowManager;
    private SurfaceTexture mSurfaceTexture= new SurfaceTexture (10);
    Intent intializerIntent;





    public class LocalBinder extends Binder {
        BackgroundService getService() {
            // Return this instance of this service so clients can call public methods
            return BackgroundService.this;
        }
    }//end inner class that returns an instance of the service.

    @Override
    public IBinder onBind(Intent intent) {
        intializerIntent = intent;
        return mBinder;
    }//end onBind.


    @Override
    public void onCreate() {

        Log.i(TAG, "onCreate is called");
        // Start foreground service to avoid unexpected kill

        startForeground(NOTIFICATION_ID, buildNotification());

        Thread thread = new Thread() {
            public void run() {

                camera = Camera.open(1);

                mSurfaceTexture.setOnFrameAvailableListener(new SurfaceTexture.OnFrameAvailableListener() {
                    @Override
                    public void onFrameAvailable(SurfaceTexture surfaceTexture) {
                        Log.i(TAG, "frame captured from texture");


                        if (camera!=null) {
                            //camera.setPreviewCallbackWithBuffer(null);
                            //camera.setPreviewCallback(null);
                            //camera.setOneShotPreviewCallback(null);

                            //if the following two lines are not called, many frames will be droped. 
                            camera.stopPreview();
                            camera.startPreview();
                        }


                    }
                });
                //now try to set the preview texture of the camera which is actually the  surfaceTexture that has just been created.

                try {
                    camera.setPreviewTexture(mSurfaceTexture);
                } catch (IOException e) {
                    Log.e(TAG, "Error in setting the camera surface texture");
                }



                camera.startPreview();
            }


        };
        thread.start();

    }

    private Notification buildNotification () {

        NotificationCompat.Builder notificationBuilder=new NotificationCompat.Builder(this);

        notificationBuilder.setOngoing(true); //this notification should be ongoing

        notificationBuilder.setContentTitle(getString(R.string.notification_title))
                .setContentText(getString (R.string.notification_text_and_ticker))
                .setSmallIcon(R.drawable.vecsat_logo)
                .setTicker(getString(R.string.notification_text_and_ticker));
        return(notificationBuilder.build());

    }


    @Override
    public void onDestroy() {

        Log.i(TAG, "surfaceDestroyed method");

        camera.stopPreview();
        //camera.lock();

        camera.release();
        mSurfaceTexture.detachFromGLContext();
        mSurfaceTexture.release();
        stopService(intializerIntent) ;
        //windowManager.removeView(surfaceView);

    }
}

如何获取帧并在 GPU 或 CPU 上处理它们？如果有办法在 GPU 上做到这一点，然后从那里得到结果，那就太好了，因为它看起来更有效率。

谢谢。

Answer 1

SurfaceTexture 获取发送到 Surface 的任何内容并将其转换为 OpenGL ES“外部”纹理。如果您想从软件访问这些像素，您需要将纹理渲染到帧缓冲区，然后使用glReadPixels() 读取像素。这方面的一个例子是 bigflake ExtractMpegFramesTest，它将解码的视频帧转换为 PNG。

通过在 GPU 上进行所有处理可以获得更好的性能（参见例如this demo），但这并不总是可行的。