com - How to pass POINT structure to ElementFromPoint method in Python?

Question

I'm trying to use method IUIAutomation::ElementFromPoint in Python using comtypes package. There are many examples how to use it in C++, but not in Python. This simple code reproduces the problem on 64-bit Windows 10 (Python 2.7 32-bit):

import comtypes.client

UIA_dll = comtypes.client.GetModule('UIAutomationCore.dll')
UIA_dll.IUIAutomation().ElementFromPoint(10, 10)

I get the following error:

TypeError: Expected a COM this pointer as first argument

Creating the POINT structure this way doesn't help as well:

from ctypes import Structure, c_long

class POINT(Structure):
    _pack_ = 4
    _fields_ = [
        ('x', c_long),
        ('y', c_long),
    ]

point = POINT(10, 10)
UIA_dll.IUIAutomation().ElementFromPoint(point) # raises the same exception

Answer 1

You can reuse existing POINT structure definition directly, like this:

import comtypes
from comtypes import *
from comtypes.client import *

comtypes.client.GetModule('UIAutomationCore.dll')
from comtypes.gen.UIAutomationClient import *

# get IUIAutomation interface
uia = CreateObject(CUIAutomation._reg_clsid_, interface=IUIAutomation)

# import tagPOINT from wintypes
from ctypes.wintypes import tagPOINT
point = tagPOINT(10, 10)
element = uia.ElementFromPoint(point)

rc = element.currentBoundingRectangle # of type ctypes.wintypes.RECT
print("Element bounds left:", rc.left, "right:", rc.right, "top:", rc.top, "bottom:", rc.bottom)

To determine what's the expected type for ElementFromPoint, you can just go to your python setup directory (for me it was C:Users<user>AppDataLocalProgramsPythonPython36Libsite-packagescomtypesgen) and check the files in there. It should contains files automatically generated by comtypes, including the one for UIAutomationCore.dll. The interesting file name starts with _944DE083_8FB8_45CF_BCB7_C477ACB2F897 (the COM type lib's GUID).

The file contains this:

COMMETHOD([], HRESULT, 'ElementFromPoint',
          ( ['in'], tagPOINT, 'pt' ),

This tells you that it expects a tagPOINT type. And this type is defined a the beginning of the file like this:

from ctypes.wintypes import tagPOINT

It's named tagPOINT because that's how it's defined in original Windows header.