将 CGImage 转换为 MTLTexture 而不进行预乘

Question

我有一个之前从 png 文件创建的 UIImage：

let strokeUIImage = UIImage(data: pngData)

我想将 strokeImage（具有不透明度）转换为 MTLTexture 以在 MTKView 中显示，但进行转换似乎会执行不需要的预乘，这会使所有半透明边缘变暗。

我的混合设置如下：

pipelineDescriptor.colorAttachments[0].isBlendingEnabled = true
pipelineDescriptor.colorAttachments[0].rgbBlendOperation = .add
pipelineDescriptor.colorAttachments[0].alphaBlendOperation = .add
pipelineDescriptor.colorAttachments[0].sourceRGBBlendFactor = .one
pipelineDescriptor.colorAttachments[0].sourceAlphaBlendFactor = .one
pipelineDescriptor.colorAttachments[0].destinationRGBBlendFactor = .oneMinusSourceAlpha
pipelineDescriptor.colorAttachments[0].destinationAlphaBlendFactor = .oneMinusSourceAlpha

我尝试了两种转换方法：

let stampTexture = try! MTKTextureLoader(device: self.device!).newTexture(cgImage: strokeUIImage.cgImage!, options: nil)

还有更精细的dataProvider驱动的方法：

let image = strokeUIImage.cgImage!
    let imageWidth = image.width
    let imageHeight = image.height
    let bytesPerPixel:Int! = 4
    let rowBytes = imageWidth * bytesPerPixel        

    let texDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .rgba8Unorm_srgb,
                                                              width: imageWidth,
                                                              height: imageHeight,
                                                              mipmapped: false)

    guard let stampTexture = device!.makeTexture(descriptor: texDescriptor) else { return }

    let srcData: CFData! = image.dataProvider?.data
    let pixelData = CFDataGetBytePtr(srcData)

    let region = MTLRegionMake2D(0, 0, imageWidth, imageHeight)
    stampTexture.replace(region: region, mipmapLevel: 0, withBytes: pixelData!, bytesPerRow: Int(rowBytes))

两者都会产生相同的不想要的预乘结果。

我尝试了后者，因为有一些帖子表明旧的 swift3 方法 CGDataProviderCopyData() 从未预乘的图像中提取原始像素数据。可悲的是，相当于：

let srcData: CFData! = image.dataProvider?.data

似乎没有奏效。我错过了什么吗？

任何指针将不胜感激。

Answer 1

经过大量试验，我找到了解决 CoreGraphics 图像中固有的预乘问题的解决方案。感谢 Warren 关于使用 Accelerate 函数（特别是 vImageUnpremultiplyData_ARGB8888）的提示，我想，为什么不使用 vImage_CGImageFormat 构建一个 CGImage，这将允许我使用指定如何解释 alpha 的 bitmapInfo 设置......结果并不完美，如下图所示：

不知何故，在翻译中，alpha 值被略微提高了（可能也是 rgb，但不是很明显）。顺便提一下，png像素格式是sRGB，我使用的MTKView设置为MTLPixelFormat.rgba16Float（app要求）

下面是我实现的完整 metalDrawStrokeUIImage 例程。特别值得注意的是：

bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.last.rawValue)

它基本上取消了 alpha 的关联（我认为）而不调用 vImageUnpremultiplyData_ARGB8888。查看生成的图像肯定看起来像未预乘的图像...

最后，为了在 MTKView 端取回预乘纹理，我让片段着色器处理预乘：

fragment float4 premult_fragment(VertexOut interpolated [[stage_in]],
                                 texture2d<float> texture [[texture(0)]],
                                 sampler sampler2D [[sampler(0)]]) {
  float4 sampled = texture.sample(sampler2D, interpolated.texCoord);

  // this fragment shader premultiplies incoming rgb with texture's alpha

  return float4(sampled.r * sampled.a,
                sampled.g * sampled.a,
                sampled.b * sampled.a,
                sampled.a );


} // end of premult_fragment

结果非常接近输入源，但图像的不透明度可能比传入的 png 高 5%。同样，png 像素格式是 sRGB，我用来显示的 MTKView 设置为 MTLPixelFormat.rgba16Float 。所以，我敢肯定，有些东西在某处变得糊涂了。如果有人有任何指点，我一定会很感激。

下面是其余的相关代码：

func metalDrawStrokeUIImage (strokeUIImage: UIImage, strokeBbox: CGRect) {

self.metalSetupRenderPipeline(compStyle: compMode.strokeCopy) // needed so stampTexture is not modified by fragmentFunction

let bytesPerPixel = 4
let bitsPerComponent = 8

let width = Int(strokeUIImage.size.width)
let height = Int(strokeUIImage.size.height)

let rowBytes = width * bytesPerPixel
//
let texDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .rgba8Unorm_srgb,
                                                             width: width,
                                                             height: height,
                                                             mipmapped: false)
guard let stampTexture = device!.makeTexture(descriptor: texDescriptor) else { return }

//let cgImage: CGImage = strokeUIImage.cgImage!
//let sourceColorSpace = cgImage.colorSpace else {
guard
  let cgImage = strokeUIImage.cgImage,
  let sourceColorSpace = cgImage.colorSpace else {
    print("Unable to initialize cgImage or colorSpace.")
    return
}

var format = vImage_CGImageFormat(
  bitsPerComponent: UInt32(cgImage.bitsPerComponent),
  bitsPerPixel: UInt32(cgImage.bitsPerPixel),
  colorSpace: Unmanaged.passRetained(sourceColorSpace),
  bitmapInfo: CGBitmapInfo(rawValue: CGImageAlphaInfo.last.rawValue),
  version: 0, decode: nil,
  renderingIntent: CGColorRenderingIntent.defaultIntent)
var sourceBuffer = vImage_Buffer()

defer {
  free(sourceBuffer.data)
}

var error = vImageBuffer_InitWithCGImage(&sourceBuffer, &format, nil, cgImage, numericCast(kvImageNoFlags))

guard error == kvImageNoError else {
  print ("[MetalBrushStrokeView]: can't vImageBuffer_InitWithCGImage")
  return

}

//vImagePremultiplyData_RGBA8888(&sourceBuffer, &sourceBuffer, numericCast(kvImageNoFlags))


// create a CGImage from vImage_Buffer
var destCGImage = vImageCreateCGImageFromBuffer(&sourceBuffer, &format, nil, nil, numericCast(kvImageNoFlags), &error)?.takeRetainedValue()


guard error == kvImageNoError else {
  print ("[MetalBrushStrokeView]: can't vImageCreateCGImageFromBuffer")
  return
}

let dstData: CFData = (destCGImage!.dataProvider!.data)!
let pixelData = CFDataGetBytePtr(dstData)

destCGImage = nil

let region = MTLRegionMake2D(0, 0, Int(width), Int(height))
stampTexture.replace(region: region, mipmapLevel: 0, withBytes: pixelData!, bytesPerRow: Int(rowBytes))
let stampColor = UIColor.white
let stampCorners = self.stampSetVerticesFromBbox(bbox: strokeBbox)
self.stampAppendToVertexBuffer(stampLayer: stampLayerMode.stampLayerFG, stampCorners: stampCorners, stampColor: stampColor)
self.metalRenderStampSingle(stampTexture: stampTexture)
self.initializeStampArray() // clears out the stamp array so we always draw 1 stamp at a time


} // end of func metalDrawStrokeUIImage (strokeUIImage: UIImage, strokeBbox: CGRect)