Python, VTK and GIL release

Question

I'm trying to parallelise processing of large datasets in VTK using its Python interface. For that, I want to use joblib since I have a (large) number of independent snapshots that I want to process and gather in a large numpy matrix, i.e. something like:

import vtk
from vtk.numpy_interface import dataset_adapter as dsa
for i,snap in enumerate(snapshot_list):
   myVtkFilter.SetInputData(snap)
   result = myVtkFilter.GetOutput()
   output[i, :] = dsa.WrapDataObject(result).CellData['myArray']

However, I'm facing some issues:

1. If I use the default loky backend, Python complains about the output matrix not being writable (it's a very large dataset, a matrix like (100, 1000000, 3)). The fact that it needs to be serialised by loky would be a major overhead anyway;
2. If I want to use Python threads, the code runs alright, but it seems to me that VTK doesn't release the GIL for most of the time (only one core is used for the largest part of the time). I would expect that, like numpy, VTK calls should release the GIL (as per this release note http://vtk.1045678.n5.nabble.com/Announce-vtk-7-0-0-release-candidate-1-is-ready-td5735575.html), but it doesn't seem to be the case.

I'm using Python 3.7.3 with VTK 8.1.2. Any suggestions?

question from:https://stackoverflow.com/questions/65928669/python-vtk-and-gil-release

Answer 1

There was some issue with the GIL in VTK 8.2.0, they have been fixed here: https://gitlab.kitware.com/paraview/paraview/-/issues/14169 and the fix is present in VTK 9.0.1.

Update to VTK 9.0.1 and use the VTK_PYTHON_FULL_THREADSAFE=ON CMake option to fix your problem.