Add a job to SLURM queue with higher priority as previously submitted jobs

Question

I want to submit and run a job X to a SLURM queue while already having other jobs YZ waiting in that queue. Basically, I want to avoid doing scontrol hold YZ manually or find an automated way to scontrol hold YZ with the submission of X and scontrol release YZ as soon as the job X finishes.

Cheers

question from:https://stackoverflow.com/questions/65887458/add-a-job-to-slurm-queue-with-higher-priority-as-previously-submitted-jobs

Answer 1

There is the scontrol top <jobID> command, which puts a job on top of other jobs of the same user ID. But it has to be enabled by the system administrator.

To quote the scontrol man-page:

top job_list

Move the specified job IDs to the top of the queue of jobs belonging to the identical user ID, partition name, account, and QOS. The job_list argument is a comma separated ordered list of job IDs. Any job not matching all of those fields will not be effected. Only jobs submitted to a single partition will be effected. This operation changes the order of jobs by adjusting job nice values. The net effect on that user's throughput will be negligible to slightly negative. This operation is disabled by default for non-privileged (non-operator, admin, SlurmUser, or root) users. This operation may be enabled for non-privileged users by the system administrator by including the option "enable_user_top" in the SchedulerParameters configuration parameter.