python - What is the recommended way to persist (pickle) custom sklearn pipelines?

Question

I have built an sklearn pipeline that combines a standard support vector regression component with some custom transformers that create features. This pipeline is then put into an object which is trained and then pickled (this seems to be the recommended way). The unpickled object is used to make predictions.

For distribution, this is turned into an executable file with pyinstaller.

When I call the unpickled regression object from a unit test, it works fine.

However, when I attempt to use the PyInstaller binary to make predictions, I get a long stack trace that ends with:

module = loader.load_module(fullname)   File "messagestream.pxd", line 5, in init scipy.optimize._trlib._trlib ImportError: No module named 'scipy._lib.messagestream'

This feels like some kind of pickling error, probably due to the interaction of pickling with pyinstaller. How can I refactor my code so that my custom pipeline runs just as easily and robustly as a standard sklearn regressor after unpickling?

Answer 1

OK, after some googling around it seems to be the case that the root cause is not pickling, it is simply a pyinstaller "hidden imports" issue, but for some reason it only shows up when pickling (don't ask me why).

The following solved the immediate issue for me: edit the .spec file to add the following hidden import with Scipy:

 hiddenimports=['scipy._lib.messagestream']

I also needed some other hidden imports related to other libraries

 hiddenimports=['sklearn.neighbors.typedefs',
                'scipy._lib.messagestream',
                'pandas._libs.tslibs.timedeltas'   ]