python - Keras model conflicting with multiprocessing

Question

I encountered a weird issue when I was trying to run 2 class methods concurrently in a third method. After eliminating large chunks of code, one at a time, I ended up with the example below.

Notes:

• I must have a model as a class attribute, I cannot change that.
• I need both tasks to run concurrently and I cannot get these 2 tasks out of the class because they interact with other class members
• I get the same error using multiprocessing.Process(), so that's not going to fix the problem.

---

from concurrent.futures import ProcessPoolExecutor, as_completed

from tensorflow.keras.models import Model


class Example:
    def __init__(self):
        self.model = Model()
        # comment out the line above and uncomment the line below, the error is gone
        # self.model = None

    def task1(self):
        pass

    def task2(self):
        pass

    def process(
        self,
    ):
        with ProcessPoolExecutor(2) as executor:
            future_items = [
                executor.submit(self.task1),
                executor.submit(self.task2),
            ]
            results = [
                future_item.result() for future_item in as_completed(future_items)
            ]
            print(results)


if __name__ == '__main__':
    ex = Example()
    ex.process()

Result:

2021-01-10 08:10:04.315386: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2021-01-10 08:10:04.315897: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.8.7/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/queues.py", line 239, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/usr/local/Cellar/[email protected]/3.8.7/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot pickle 'weakref' object
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/emadboctor/Desktop/code/drl-algos/scratch.py", line 34, in <module>
    ex.process()
  File "/Users/emadboctor/Desktop/code/drl-algos/scratch.py", line 26, in process
    results = [
  File "/Users/emadboctor/Desktop/code/drl-algos/scratch.py", line 27, in <listcomp>
    future_item.result() for future_item in as_completed(future_items)
  File "/usr/local/Cellar/[email protected]/3.8.7/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/local/Cellar/[email protected]/3.8.7/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "/usr/local/Cellar/[email protected]/3.8.7/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/queues.py", line 239, in _feed
    obj = _ForkingPickler.dumps(obj)
  File "/usr/local/Cellar/[email protected]/3.8.7/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: cannot pickle 'weakref' object

question from:https://stackoverflow.com/questions/65650650/keras-model-conflicting-with-multiprocessing

Answer 1

Waitting for answers