I'm trying to do a simple task queue with Flask and without any DB.
In the most simple version, I have two endpoints. Submit job and Check status.
Submit job will add request to queue and check status will return the status of a job id (queued, running, failed, finished).
The workflow is as follows:
- user submits a job
- job is added to queue
- user will check status of the job every 5 seconds
- every check of status will trigger a function that checks if the number of running jobs is smaller than the maximum number of jobs (from config). If the number is smaller, it will span another thread with the job on top of queue.
This is the simplified code:
app = Flask(__name__)
def finish_job(job_id):
finished.append(job_id)
last = running.pop(job_id)
last.close()
def remove_finished():
for j in list(running.keys()):
if not running[j].is_alive():
finish_job(j)
def start_jobs():
while len(running) < config.threads and len(queue_list) > 0:
print('running now', len(running))
next_job = queue.pop()
queue_list.remove(next_job[0])
start_job(*next_job)
@app.route("/Simulation", methods=['POST'])
@authenticate
def submit_job():
# create id
job_id = str(uuid.uuid4())
job_data = request.data.decode('utf-8')
queue.append((job_id, job_data))
queue_list.add(job_id)
return 'QUEUED', 200
@app.route("/Simulation/<uuid:job_id>", methods=['GET'])
@authenticate
def check_status(job_id: uuid):
job_id = str(job_id)
remove_finished()
start_jobs()
if job_id in running:
r = 'RUNNING'
elif job_id in queue_list:
r = 'QUEUED'
elif job_id in finished:
r = 'COMPLETED'
else:
r = 'FAILED'
return status_response(r), 200
running = {}
finished = []
queue = []
queue_list = set()
app.run()
Now, the problem is, that if multiple users submit a check status request at the same time, and there is only one slot free for running a task, both requests will spawn the job.
Is there some way to force Flask to only run one instance of a function at a time?
Thank you
question from:
https://stackoverflow.com/questions/65846578/python-flask-simple-task-queue-without-external-libraries-not-working 与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…