ios - Live camera is getting stretched while rendering using CIFilter. - Swift 4

Question

I want to apply camera filter while rendering, my code is...

func session(_ session: ARSession, didUpdate frame: ARFrame) {
let image = CIImage(cvPixelBuffer: frame.capturedImage)
        var r: CGFloat = 0, g: CGFloat = 0, b: CGFloat = 0, a: CGFloat = 0

        color.getRed(&r, green: &g, blue: &b, alpha: &a)
        filter.setDefaults()
        filter.setValue(image, forKey: kCIInputImageKey)
        filter.setValue(CIVector(x: r, y: 0, z: 0, w: 0), forKey: "inputRVector")
        filter.setValue(CIVector(x: 0, y: g, z: 0, w: 0), forKey: "inputGVector")
        filter.setValue(CIVector(x: 0, y: 0, z: b, w: 0), forKey: "inputBVector")
        filter.setValue(CIVector(x: 0, y: 0, z: 0, w: a), forKey: "inputAVector")

        if let result = filter.outputImage,
            let cgImage = context.createCGImage(result, from: result.extent) {
            sceneView.scene.background.contents = cgImage
            sceneView.scene.background.contentsTransform = SCNMatrix4MakeRotation(.pi / 2, 0, 0, 1)
        }
}

at runtime, output is getting stretched. I have attached two images,

1. Normal Camera rendering
2. With filter camera rendering.

Please help me to resolve it, it would be great help if you provide any demo code or project. Thank you.

Answer 1

First of all, i recommend you make a copy of the ARFrame, otherwise currentFrame will get all sort of weirdness if not processed soon enough, because the memory will be discarded by the hardware.

Second, the stretching is because you are using the buffer image that is different from the viewport image, grab a session.snapshot and save it to you mac, and then grab your cgImage and save it too, open them in Preview, you will see the sizes differ, so, when you want to put that wider image back to the viewport size, it stretches.

You do not need a demo project for this one, just print your scene frame size, and your buffer size, check the ratios between both and you will understand your issue.