如何使用 ARMatteGenerator 将 CIFilter 添加到 MTLTexture？

Question

我正在处理 Apple 的 sample project 与使用 ARMatteGenerator 生成可用作人员遮挡技术中的遮挡遮罩的 MTLTexture 相关的工作。

我想确定如何通过 CIFilter 运行生成的遮罩。在我的代码中，我正在“过滤”这样的遮罩；

func updateMatteTextures(commandBuffer: MTLCommandBuffer) {
    guard let currentFrame = session.currentFrame else {
        return
    }
    var targetImage: CIImage?
    alphaTexture = matteGenerator.generateMatte(from: currentFrame, commandBuffer: commandBuffer)
    dilatedDepthTexture = matteGenerator.generateDilatedDepth(from: currentFrame, commandBuffer: commandBuffer)
    targetImage = CIImage(mtlTexture: alphaTexture!, options: nil)
    monoAlphaCIFilter?.setValue(targetImage!, forKey: kCIInputImageKey)
    monoAlphaCIFilter?.setValue(CIColor.red, forKey: kCIInputColorKey)
    targetImage = (monoAlphaCIFilter?.outputImage)!
    let drawingBounds = CGRect(origin: .zero, size: CGSize(width: alphaTexture!.width, height: alphaTexture!.height))
    context.render(targetImage!, to: alphaTexture!, commandBuffer: commandBuffer, bounds: drawingBounds, colorSpace: CGColorSpaceCreateDeviceRGB())

}

当我合成哑光纹理和背景时，没有对哑光应用过滤效果。这就是合成纹理的方式；

func compositeImagesWithEncoder(renderEncoder: MTLRenderCommandEncoder) {
    guard let textureY = capturedImageTextureY, let textureCbCr = capturedImageTextureCbCr else {
        return
    }

    // Push a debug group allowing us to identify render commands in the GPU Frame Capture tool
    renderEncoder.pushDebugGroup("CompositePass")

    // Set render command encoder state
    renderEncoder.setCullMode(.none)
    renderEncoder.setRenderPipelineState(compositePipelineState)
    renderEncoder.setDepthStencilState(compositeDepthState)

    // Setup plane vertex buffers
    renderEncoder.setVertexBuffer(imagePlaneVertexBuffer, offset: 0, index: 0)
    renderEncoder.setVertexBuffer(scenePlaneVertexBuffer, offset: 0, index: 1)

    // Setup textures for the composite fragment shader
    renderEncoder.setFragmentBuffer(sharedUniformBuffer, offset: sharedUniformBufferOffset, index: Int(kBufferIndexSharedUniforms.rawValue))
    renderEncoder.setFragmentTexture(CVMetalTextureGetTexture(textureY), index: 0)
    renderEncoder.setFragmentTexture(CVMetalTextureGetTexture(textureCbCr), index: 1)
    renderEncoder.setFragmentTexture(sceneColorTexture, index: 2)
    renderEncoder.setFragmentTexture(sceneDepthTexture, index: 3)
    renderEncoder.setFragmentTexture(alphaTexture, index: 4)
    renderEncoder.setFragmentTexture(dilatedDepthTexture, index: 5)

    // Draw final quad to display
    renderEncoder.drawPrimitives(type: .triangleStrip, vertexStart: 0, vertexCount: 4)
    renderEncoder.popDebugGroup()
}

如何将 CIFilter 仅应用于 ARMatteGenerator 生成的 alphaTexture？

Answer 1

我认为您不想将CIFilter 应用于alphaTexture。我假设您使用的是 Apple 的 Effecting People Occlusion in Custom Renderers 示例代码。如果您观看今年的 Bringing People into AR WWDC 会议，他们会谈论使用 ARMatteGenerator 生成分段遮罩，这就是使用 alphaTexture = matteGenerator.generateMatte(from: currentFrame, commandBuffer: commandBuffer) 所做的事情。 alphaTexture 是一个 MTLTexture，它本质上是一个 alpha 蒙版，用于在相机帧中检测到人类的位置（即在人类所在的地方完全不透明，在没有人类的地方完全透明）。

向 Alpha 纹理添加过滤器不会过滤最终渲染的图像，而只会影响合成中使用的蒙版。如果您尝试实现your previous question 中链接的视频，我建议您调整发生合成的金属着色器。在会议中，他们指出他们比较了dilatedDepth 和renderedDepth，看看他们是否应该从相机中绘制虚拟内容或像素：

fragment half4 customComposition(...) {
    half4 camera = cameraTexture.sample(s, in.uv);
    half4 rendered = renderedTexture.sample(s, in.uv);
    float renderedDepth = renderedDepthTexture.sample(s, in.uv);
    half4 scene = mix(rendered, camera, rendered.a);
    half matte = matteTexture.sample(s, in.uv);
    float dilatedDepth = dilatedDepthTexture.sample(s, in.uv);

    if (dilatedDepth < renderedDepth) { // People in front of rendered
        // mix together the virtual content and camera feed based on the alpha provided by the matte
        return mix(scene, camera, matte);
    } else {
        // People are not in front so just return the scene
        return scene
    }
}

不幸的是，这在示例代码中的实现方式略有不同，但修改起来仍然相当容易。打开Shaders.metal。找到compositeImageFragmentShader 函数。在函数的末尾，您将看到 half4 occluderResult = mix(sceneColor, cameraColor, alpha); 这与我们在上面看到的 mix(scene, camera, matte); 基本相同。我们正在根据分割遮罩决定是否应该使用场景中的像素或相机输入中的像素。我们可以通过将cameraColor 替换为代表颜色的half4，轻松地将相机图像像素替换为任意rgba 值。例如，我们可以使用half4(float4(0.0, 0.0, 1.0, 1.0)) 将分割内的所有像素绘制成哑光蓝色：

…
// Replacing camera color with blue
half4 occluderResult = mix(sceneColor, half4(float4(0.0, 0.0, 1.0, 1.0)), alpha);
half4 mattingResult = mix(sceneColor, occluderResult, showOccluder);
return mattingResult;

当然，您也可以应用其他效果。动态灰度静态很容易实现。

以上compositeImageFragmentShader加：

float random(float offset, float2 tex_coord, float time) {
    // pick two numbers that are unlikely to repeat
    float2 non_repeating = float2(12.9898 * time, 78.233 * time);

    // multiply our texture coordinates by the non-repeating numbers, then add them together
    float sum = dot(tex_coord, non_repeating);

    // calculate the sine of our sum to get a range between -1 and 1
    float sine = sin(sum);

    // multiply the sine by a big, non-repeating number so that even a small change will result in a big color jump
    float huge_number = sine * 43758.5453 * offset;

    // get just the numbers after the decimal point
    float fraction = fract(huge_number);

    // send the result back to the caller
    return fraction;
}

（取自@twostraws ShaderKit）

然后将compositeImageFragmentShader修改为：

…
float randFloat = random(1.0, cameraTexCoord, rgb[0]);

half4 occluderResult = mix(sceneColor, half4(float4(randFloat, randFloat, randFloat, 1.0)), alpha);
half4 mattingResult = mix(sceneColor, occluderResult, showOccluder);
return mattingResult;

你应该得到：

最后，调试器似乎很难跟上应用的步伐。对我来说，在运行附加的 Xcode 时，应用程序会在启动后不久冻结，但在单独运行时通常很流畅。