python - Multiprocessing on Windows breaks

Question

I develop with Python on Linux and have never really seen this sort of problem with Windows. I'm using the multiprocessing library to speed up computations, which works very well for me on Linux.

On Windows, however, things don't run as smoothly:

 * [INFO] Parsing 1 file using 2 threads

Traceback (most recent call last):
  File "main.py", line 170, in <module>
    master = ParsingMaster(parser, list(input_file), output_list, threads=num_threads)
Traceback (most recent call last):
  File "main.py", line 39, in __init__
  File "<string>", line 1, in <module>
    self.input_process.start()
  File "C:Python26libmultiprocessingforking.py", line 342, in main
  File "C:Python26libmultiprocessingprocess.py", line 104, in start
        self._popen = Popen(self)
self = load(from_parent)
  File "C:Python26libmultiprocessingforking.py", line 239, in __init__
  File "C:Python26libpickle.py", line 1370, in load
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:Python26libmultiprocessingforking.py", line 162, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:Python26libpickle.py", line 224, in dump
    return Unpickler(file).load()
    self.save(obj)
 File "C:Python26libpickle.py", line 858, in load
  File "C:Python26libpickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:Python26libpickle.py", line 419, in save_reduce
    dispatch[key](self)
  File "C:Python26libpickle.py", line 880, in load_eof
    save(state)
  File "C:Python26libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
    r aise EOFError
 File "C:Python26libpickle.py", line 649, in save_dict
EOFError
    self._batch_setitems(obj.iteritems())
  File "C:Python26libpickle.py", line 681, in _batch_setitems
    save(v)
  File "C:Python26libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:Python26libmultiprocessingforking.py", line 40, in dispatcher
    self.save_reduce(obj=obj, *rv)
  File "C:Python26libpickle.py", line 401, in save_reduce
    save(args)
  File "C:Python26libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:Python26libpickle.py", line 548, in save_tuple
    save(element)
  File "C:Python26libpickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:Python26libpickle.py", line 419, in save_reduce
    save(state)
  File "C:Python26libpickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:Python26libpickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:Python26libpickle.py", line 681, in _batch_setitems
    save(v)
  File "C:Python26libpickle.py", line 306, in save
    rv = reduce(self.proto)
  File "C:Python26libmultiprocessingmanagers.py", line 458, in __reduce__
    return type(self).from_address, 
AttributeError: type object 'SyncManager' has no attribute 'from_address'

I'm testing on both Python 2.6 and 2.7 on Windows 7 and get this same error over and over. Does anybody know what it means?

Answer 1

There are restrictions on Windows, here is the relevant parts to the errors you are seeing:

Since Windows lacks os.fork() it has a few extra restrictions:

More picklability

Ensure that all arguments to Process.__init__() are picklable. This means, in particular, that bound or unbound methods cannot be used directly as the target argument on Windows — just define a function and use that instead.

Also, if you subclass Process then make sure that instances will be picklable when the Process.start() method is called.

This means that something that is being passed as an argument to Process.__init__() isn't able to be pickled or unpickled ( a serialization in Python ). What is SyncManager it is complaining about not being able to find attributes on that object AttributeError: type object 'SyncManager' has no attribute 'from_address', it is probably your root cause. Can that SyncManager object actually be pickled, does it meet the pickle rules?

If you are running this from the command line on Windows, you can't do that either apparently.

Don't do that. Save the code in a file and run it from the file instead, with the command:

python myfile.py

That will solve your issue.