python multiprocessing - How to pickle/ serialize a swigpyobject?

Question

I'm working on a application which connects with kernel module. for this communication i'm using netlink libnl3.5.0 in python which wraps "c" into py using swig. Problem is when I try do multiprocessing and trying to share the msg queue to different processes. Python is not able to pickle/serialize the messages which are in SwigPyObject format.

I tried to find the high level class implementation of this object but I'm confused. Saw other posts which says to implement some getstate and reduce functions in the object but I don't understand where to use this.

Tried dill package but I guess dill is using pickle for underlying implementations. so this is returning same error.

Below is main part of my implementation

def queue_packet(msg,args):
    global a
    nlh=nl.nlmsg_hdr(msg)
    ghdr=genl.genlmsg_hdr(nlh)
    if(ghdr.cmd==2):
        _,attr=genl.py_genlmsg_parse(nlh,0,8,None)
        a.put(attr)
        #print(a.qsize())
    return 0

def worker(a):
    print("worker started")
    while True:
        if a.qsize()==0:
            continue
        attr=a.get()
        process_messages_cb(attr)
        a.task_done()
    return 0
a=multiprocessing.Manager.Queue()
while 1:
   nl.nl_recvmsgs(s,rx_cb)

This is main part of receiving and putting the messages to queue. and different workers will get the message from the queue and process them further. So how can I put the messages to queue to share between different workers/sub-processes. I'm using this https://github.com/thom311/libnl/tree/master/python API.

Thanks

question from:https://stackoverflow.com/questions/65942415/how-to-pickle-serialize-a-swigpyobject

Answer 1

For an object to pickable, it needs to implement a __getstate__ and a __setstate__ function.

What I usually do is the following (at the end of your interface file). Say that the object MyObject is the object that you have wrapped with SWIG for which the wrapper isn't pickable.

#pragma once
class MyOBject {
 public:
  MyObject() {}
  SetA(float a) {m_a = a;}
  SetB(float b) {m_b = b;}
 private:
  float m_a;
  float m_b;
};

Then in your interface file, do the following

%extend MyObject {
%pythoncode %{
def __getstate__(self):
  args = (a, b)
  return args
def __setstate__(self, state)
  self.__init__() # construct object
  (a,b) = state
  self.SetA(a)
  self.SetB(b)
%}
}

For an object to be pickable, you must be able to recover it from the serialized data (in some way). In the above example, the recovering is made by calling a pair of functions. In your case, you need to find a way to serialize the state of your object and re-create it from the serialized data.

This shows how you can make a wrapper such that the objects become pickable, but this requires you to change the wrapper for libnl

Another option is to wrap the python objects that are passed around in a python class for which you implement __getstate__ and __setstate__. The idea is the same, you need to be able to serialize and recover the state of your object. I am not sure which of the two approaches is the simplest one.