python - Passing a ctypes callback function around in C causes memory related problems

Question

I've come across a slightly complicated bug in some C code that I'm porting to Python using ctypes. I've been able to reproduce the problem in a minimal working example and was hoping to get some help with it. The essence of the problem is that using a ctypes callback causes memory related issues if the call made is passed on to another C function via a pointer.

So here's C code:

// the function pointer that we want to use as a callback from python
typedef void(*fn)(double *, double *, double *);

// a place to store the func (among other things in the real code)
typedef struct {
    fn func;
} Parameters;


// wrapper around calls to allocate instance of Parameters from Python
Parameters **makeParameters() {
    Parameters **pp;
    pp = (Parameters **) malloc(sizeof(Parameters *));
    *pp = (Parameters *) malloc(sizeof(Parameters));
    return pp;
}

// clean up memory
void freeParameters(Parameters **parameters) {
    free(*parameters);
    free(parameters);
}

// collects input parameters. Store the callback inside the Parameters struct
// for later use in other functions. 
void setup_fn(Parameters **parameters, fn callback_fn) {
    (*parameters)->func = callback_fn;
}

// Worker function that actually calls the callback
void do_work_with_func(Parameters **parameters){
    double *x = (double *) malloc(sizeof(double) * 2);
    double *y = (double *) malloc(sizeof(double));
    double *z = (double *) malloc(sizeof(double));

    x[0] = 1.0;
    x[1] = 2.0;
    *y = 3.0;
    *z = 0;

    (*parameters)->func(x, y, z);

    printf("From C: Output from do_work_with_func fn: %f", *z);
    free(x);
    free(y);
    free(z);
}

Compiling this in C with

void callback_fn(double* x, double* y, double* z){
    printf("From C: I'm a callback function
");
    *z = x[0] + (x[1] * *y); // output 7 with test numbers.
}

int main(){
    Parameters ** params = makeParameters();
    setup_fn(params, callback_fn);
    do_work_with_func(params);
    freeParameters(params);
}

produces

From C: I'm a callback function
From C: Output from do_work_with_func fn: 7.000000
Process finished with exit code 0

All well and good. So lets now try and use this from Python using the ctypes library.

# load the library
lib = ct.CDLL("library_name")

# load the functions
makeParameters = lib.makeParameters
makeParameters.argtypes = []
makeParameters.restype = ct.c_int64 # raw pointer

freeParameters = lib.freeParameters
freeParameters.argtypes = [ct.c_int64]
freeParameters.restype = None

FN_PTR = ct.CFUNCTYPE(None, ct.POINTER(ct.c_double*2), ct.POINTER(ct.c_double), ct.POINTER(ct.c_double))

setup_fn = lib.setup_fn
setup_fn.argtypes = [ct.c_int64, FN_PTR]
setup_fn.restype = None

do_work_with_func = lib.do_work_with_func
do_work_with_func.argtypes = [ct.c_int64]
do_work_with_func.restype = None

# main program 
def callback_fn(x, y, z):
    print("From Python: I'm a callback function")
    z.contents.value = x.contents[0] + (x.contents[1] * y.value)

params = makeParameters()
setup_fn(params, FN_PTR(callback_fn))
do_work_with_func(params)

sometimes results in

OSError: [WinError -1073741795] Windows Error 0xc000001d

or sometimes results in

OSError: exception: access violation reading 0xFFFFFFFFFFFFFFFF

Can anybody explain what is happening here and how to fix the program? The expected output should be

From Python: I'm a callback function
From C: Output from do_work_with_func fn: 7.000000

question from:https://stackoverflow.com/questions/65941540/passing-a-ctypes-callback-function-around-in-c-causes-memory-related-problems

Answer 1

setup_fn(params, FN_PTR(callback_fn)) wraps callback_fn passes the wrapped object to the function, then frees the FN_PTR object. You have to maintain the object for the lifetime it could be called. The easiest way to do this is permanently decorate the function:

FN_PTR = ct.CFUNCTYPE(None, ct.POINTER(ct.c_double*2), ct.POINTER(ct.c_double), ct.POINTER(ct.c_double))

@FN_PTR  # decorate here
def callback_fn(x, y, z):
    print("From Python: I'm a callback function")
    z.contents.value = x.contents[0] + (x.contents[1] * y.value)

setup_fn(params, callback_fn) # call with decorated function.