multithreading - How do I run a long-running job in the background in Python

Question

I have a web-service that runs long-running jobs (in the order of several hours). I am developing this using Flask, Gunicorn, and nginx.

What I am thinking of doing is to have the route which takes a long time to complete, call a function that creates a thread. The function will then return a guid back to the route, and the route will return a url (using the guid) that the user can use to check progress. I am making the thread a daemon (thread.daemon = True) so that the thread exits if my calling code exits (unexpectedly).

Is this the correct approach to use? It works, but that doesn't mean that it is correct.

my_thread = threading.Thread(target=self._run_audit, args=())
my_thread.daemon = True
my_thread.start()

Answer 1

Celery and RQ is overengineering for simple task. Take a look at this docs - https://docs.python.org/3/library/concurrent.futures.html

Also check example, how to run long-running jobs in background for Flask app - https://stackoverflow.com/a/39008301/5569578