opencv - Otsu thresholding for depth image

Question

I am trying to substract background from depth images acquired with kinect. When I learned what otsu thresholding is I thought that it could with it. Converting the depth image to grayscale i can hopefully apply otsu threshold to binarize the image.

However I implemented (tried to implemented) this with OpenCV 2.3, it came in vain. The output image is binarized however, very unexpectedly. I did the thresholding continuously (i.e print the result to screen to analyze for each frame) and saw that for some frames threshold is found to be 160ish and sometimes it is found to be 0. I couldn't quite understand why this is happening. May it be due to the high number of 0's in the depth image returned by kinect, which corresponds to pixels that can not be measured. Is there a way that I could tell the algorithm to ignore pixels having the value 0? Or otsu thresholding is not good for what I am trying to do?

Here are some outputs and segment of the related code. You may notice that the second screenshot looks like it could do some good binarization, however i want to achieve one that distincly differentiates between pixels corresponding to the chair in the scene and the backgroung.

Thanks.

            cv::Mat1s depthcv(depth->getHeight(), depth->getWidth());
            cv::Mat1b depthcv8(depth->getHeight(), depth->getWidth());
            cv::Mat1b depthcv8_th(depth->getHeight(), depth->getWidth());
            depthcv.data =(uchar*) depth->getDepthMetaData().Data();
            depthcv.convertTo(depthcv8,CV_8U,255/5000.f);

            //apply otsu thresholding
            cv::threshold(depthcv8, depthcv8_th, 128, 255, CV_THRESH_BINARY|CV_THRESH_OTSU);
            std::ofstream output;
            output.open("output.txt");
            //output << "M = "<< endl << " "  << depthcv8 << endl << endl;
            cv::imshow("lab",depthcv8_th);
            cv::waitKey(1);

Answer 1

Otsu is probably good enough for what you are trying to do, but you do need to mask out the zero values before computing the optimal threshold with the Otsu algorithm, otherwise the distribution of intensity values will be skewed lower than what you want.

OpenCV does not provide a mask argument for the cv::threshold function, so you will have to remove those values yourself. I would recommend putting all the non-zero values in a 1 by N matrix, and calling the cv::threshold function with CV_THRESH_OTSU and saving the return value (which is the estimated optimal threshold), and then running the cv::threshold function again on the original image with just the CV_THRESH_BINARY flag and the computed threshold.

Here is one possible implementation:

// move zeros to the back of a temp array
cv::Mat copyImg = origImg;
uint8* ptr = copyImg.datastart;
uint8* ptr_end = copyImg.dataend;
while (ptr < ptr_end) {
  if (*ptr == 0) { // swap if zero
    uint8 tmp = *ptr_end;
    *ptr_end = *ptr;
    *ptr = tmp;
    ptr_end--; // make array smaller
  } else {
    ptr++;
  }
}

// make a new matrix with only valid data
cv::Mat nz = cv::Mat(std::vector<uint8>(copyImg.datastart,ptr_end),true);

// compute optimal Otsu threshold
double thresh = cv::threshold(nz,nz,0,255,CV_THRESH_BINARY | CV_THRESH_OTSU);

// apply threshold
cv::threshold(origImg,origImg,thresh,255,CV_THRESH_BINARY_INV);