multithreading - a new thread for running a cell in ipython/jupyter notebook

Question

Sometimes it takes a long time to run a single cell, while it is running, I would like to write and run other cells in the same notebook, accessing the variables in the same context.

Is there any ipython magic that can be used such that when it is added to a cell, running the cell will automatically create a new thread and run with shared global data in the notebook?

Answer 1

It may not be an answer, but rather the direction to it. I did not saw anything like that, still I'm interested in this too.

My current findings suggesting that one need to define it's own custom cell magic. Good references would be the custom cell magic section in the documentation and two examples that I would consider:

• memit: magic memory usage benching for IPython https://gist.github.com/vene/3022718
• Illustrating Python multithreading vs multiprocessing: http://nathangrigg.net/2015/04/python-threading-vs-processes/

Both those links wrapping the code in a thread. That could be a starting point.

UPDATE: ngcm-tutorial at github has description of background jobs class

##github.com/jupyter/ngcm-tutorial/blob/master/Day-1/IPython%20Kernel/Background%20Jobs.ipynb
from IPython.lib import backgroundjobs as bg
jobs = bg.BackgroundJobManager()

def printfunc(interval=1, reps=5):
    for n in range(reps):
        time.sleep(interval)
        print('In the background... %i' % n)
        sys.stdout.flush()
    print('All done!')
    sys.stdout.flush()

jobs.new('printfunc(1,3)')
jobs.status()

UPDATE 2: Another option:

from IPython.display import display
from ipywidgets import IntProgress

import threading

class App(object):
    def __init__(self, nloops=2000):
        self.nloops = nloops
        self.pb = IntProgress(description='Thread loops', min=0, max=self.nloops)

    def start(self):
        display(self.pb)
        while self.pb.value < self.nloops:
            self.pb.value += 1 
        self.pb.color = 'red'

app = App(nloops=20000)

t = threading.Thread(target=app.start)

t.start()
#t.join()