avfoundation - Determine the corners of a sheet of paper with iOS 5 AV Foundation and core-image in realtime

Question

I am currently building a camera app prototype which should recognize sheets of paper lying on a table. The clue about this is that it should do the recognition in real time, so I capture the video stream of the camera, which in iOS 5 can easily be done with the AV foundation. I looked at here and here

They are doing some basic object recognition there.

I have found out that using OpenCV library in this realtime environment does not work in a performant way.

So what I need is an algorithm to determine the edges of an image without OpenCV.

Does anyone have some sample code snippets which lay out how to do this or point me in the right direction.

Any help would be appreciated.

Answer 1

You're not going to be able to do this with the current Core Image implementation in iOS, because corner detection requires some operations that Core Image doesn't yet support there. However, I've been developing an open source framework called GPUImage that does have the required capabilities.

For finding the corners of an object, you can use a GPU-accelerated implementation of the Harris corner detection algorithm that I just got working. You might need to tweak the thresholds, sensitivities, and input image size to work for your particular application, but it's able to return corners for pieces of paper that it finds in a scene:

It also finds other corners in that scene, so you may need to use a binary threshold operation or some later processing to identify which corners belong to a rectangular piece of paper and which to other objects.

I describe the process by which this works over at Signal Processing, if you're interested, but to use this in your application you just need to grab the latest version of GPUImage from GitHub and make the GPUImageHarrisCornerDetectionFilter the target for a GPUImageVideoCamera instance. You then just have to add a callback to handle the corner array that's returned to you from this filter.

On an iPhone 4, the corner detection process itself runs at ~15-20 FPS on 640x480 video, but my current CPU-bound corner tabulation routine slows it down to ~10 FPS. I'm working on replacing that with a GPU-based routine which should be much faster. An iPhone 4S currently handles everything at 20-25 FPS, but again I should be able to significantly improve the speed there. Hopefully, that would qualify as being close enough to realtime for your application.