android - how to check ray intersection with object in ARCore

Question

Is there a way to check if I touched the object on the screen ? As I understand the HitResult class allows me to check if I touched the recognized and maped surface. But I want to check this I touched the object that is set on that surface.

Answer 1

ARCore doesn't really have a concept of an object, so we can't directly provide that. I suggest looking at ray-sphere tests for a starting point.

However, I can help with getting the ray itself (to be added to HelloArActivity):

/** 
 * Returns a world coordinate frame ray for a screen point.  The ray is
 * defined using a 6-element float array containing the head location
 * followed by a normalized direction vector.
 */
float[] screenPointToWorldRay(float xPx, float yPx, Frame frame) {
    float[] points = new float[12];  // {clip query, camera query, camera origin}
    // Set up the clip-space coordinates of our query point
    // +x is right:
    points[0] = 2.0f * xPx / mSurfaceView.getMeasuredWidth() - 1.0f;
    // +y is up (android UI Y is down):
    points[1] = 1.0f - 2.0f * yPx / mSurfaceView.getMeasuredHeight(); 
    points[2] = 1.0f; // +z is forwards (remember clip, not camera)
    points[3] = 1.0f; // w (homogenous coordinates)

    float[] matrices = new float[32];  // {proj, inverse proj}
    // If you'll be calling this several times per frame factor out
    // the next two lines to run when Frame.isDisplayRotationChanged().
    mSession.getProjectionMatrix(matrices, 0, 1.0f, 100.0f);
    Matrix.invertM(matrices, 16, matrices, 0);
    // Transform clip-space point to camera-space.
    Matrix.multiplyMV(points, 4, matrices, 16, points, 0);
    // points[4,5,6] is now a camera-space vector.  Transform to world space to get a point
    // along the ray.
    float[] out = new float[6];
    frame.getPose().transformPoint(points, 4, out, 3);
    // use points[8,9,10] as a zero vector to get the ray head position in world space.
    frame.getPose().transformPoint(points, 8, out, 0);
    // normalize the direction vector:
    float dx = out[3] - out[0];
    float dy = out[4] - out[1];
    float dz = out[5] - out[2];
    float scale = 1.0f / (float) Math.sqrt(dx*dx + dy*dy + dz*dz);
    out[3] = dx * scale;
    out[4] = dy * scale;
    out[5] = dz * scale;
    return out;
}

If you're calling this several times per frame see the comment about the getProjectionMatrix and invertM calls.