python - input() is blocking the use of processes

Question

I have a problem with multiprocessing. If I am waiting for input in a thread the process is not starting.

The class to put the input into a queue in the background:

class InputCatcher(Thread):
    def __init__(self, input_queue):
        Thread.__init__(self)
        self.input_queue = input_queue

    def run(self):
        while True:
            self.input_queue.put(input()) # <<-- Without this it works!

The class that will not start:

class Usb(Process):
    def __init__(self, port, ctrl=Controller()):
        Process.__init__(self)
        self.usb_port = port
        self.ctrl = ctrl


    def run(self):
        self.ctrl.usb_ports.append(self.usb_port)
        ser = Serial(self.usb_port, 115200)
        while True:
            dsl = ser.readline()
            self.ctrl.get_dataset_queue().put(['USBDS', dsl])
            print(dsl)

Starting with:

ic = InputCatcher(self.input_queue)
ic.setDaemon(True)
ic.start()

usbs = []
for port in usb_ports():
    if not port in ctrl.xbee_ports:
        usbs.append(Usb(port, ctrl))

for usb in usbs:
    usb.daemon = True
    usb.start()

Answer 1

When you call input, it is blocking the entire Python process, not just the thread it runs in. This happens because reading from STDIN, like reading from any other file-like object, involves making a blocking syscall - that is, input blocking to wait for user input happens at the OS level, rather than inside Python's own thread management code. Python threads are essentially invisible to the OS process scheduler, so Python itself gets blocked.

The usual way around blocking problems like this is to use processes instead of threads. If you make InputCatcher into a process rather than a thread, then it becomes a separate OS-level process that the OS can schedule independently, and so the syscall will only block that process and not the main one.

---

Except, that Python automatically closes STDIN when you spawn a process.

So, you would need to have the producer for the queue in the main process, and only the consumer in another one. This is also a trivial adaption - don't start the producer (InputCatcher) running until after all of the consumer processes have spawned. That involves moving the line:

ic.start()

to below the two loops. But in this case, there's no need for that to be backgrounded at all - it doesn't run simultaneously with other things. So, you can forget about the InputCatcher class entirely, and just write your code like this:

for usb in usbs:
    usb.daemon = True
    usb.start()

while True:
    input_queue.put(input())

You might also want to consider a particular input - say, the empty string - to end the program. Having the input in the main run makes this really easy by just ending the loop:

while True:
    data = input('Type data; or enter to exit: ')
    if not data:
        break
    input_queue.put(data)