asp.net - How to prevent a Hangfire recurring job from restarting after 30 minutes of continuous execution

Question

I am working on an asp.net mvc-5 web application, and I am facing a problem in using Hangfire tool to run long running background jobs. the problem is that if the job execution exceed 30 minutes, then hangfire will automatically initiate another job, so I will end up having two similar jobs running at the same time.

Now I have the following:-

1. Asp.net mvc-5
2. IIS-8
3. Hangfire 1.4.6
4. Windows server 2012

Now I have defined a hangfire recurring job to run at 17:00 each day. The background job mainly scan our network for servers and vms and update the DB, and the recurring job will send an email after completing the execution. The recurring job used to work well when its execution was less than 30 minutes. But today as our system grows, the recurring job completed after 40 minutes instead of 22-25 minutes as it used to be. and I received 2 emails instead of one email (and the time between the emails was around 30 minutes). Now I re-run the job manually and I have noted that that the problem is as follow:-

"when the recurring job reaches 30 minutes of continuous execution, a new instance of the recurring job will start, so I will have two instances instead of one running at the same time, so that why I received 2 emails."

Now if the recurring job takes less than 30 minutes (for example 29 minute) I will not face any problem, but if the recurring job execution exceeds 30 minutes then for a reason or another hangfire will initiate a new job. although when I access the hangfire dashboard during the execution of the job, I can find that there is only one active job, when I monitor our DB I can see from the sql profiler that there are two jobs accessing the DB. this happens after 30 minutes from the beginning of the recurring job (at 17:30 in our case), and that why I received 2 emails which mean 2 recurring jobs were running in the background instead of one.

So can anyone advice on this please, how I can avoid hangfire from automatically initiating a new recurring job if the current recurring job execution exceeds 30 minutes? Thanks

Answer 1

Did you look at InvisibilityTimeout setting from the Hangfire docs?

Default SQL Server job storage implementation uses a regular table as a job queue. To be sure that a job will not be lost in case of unexpected process termination, it is deleted only from a queue only upon a successful completion.

To make it invisible from other workers, the UPDATE statement with OUTPUT clause is used to fetch a queued job and update the FetchedAt value (that signals for other workers that it was fetched) in an atomic way. Other workers see the fetched timestamp and ignore a job. But to handle the process termination, they will ignore a job only during a specified amount of time (defaults to 30 minutes).

Although this mechanism ensures that every job will be processed, sometimes it may cause either long retry latency or lead to multiple job execution. Consider the following scenario:

1. Worker A fetched a job (runs for a hour) and started it at 12:00.
2. Worker B fetched the same job at 12:30, because the default invisibility timeout was expired.
3. Worker C (did not fetch) the same job at 13:00, because (it will be deleted after successful performance.)

If you are using cancellation tokens, it will be set for Worker A at 12:30, and at 13:00 for Worker B. This may lead to the fact that your long-running job will never be executed. If you aren’t using cancellation tokens, it will be concurrently executed by WorkerA and Worker B (since 12:30), but Worker C will not fetch it, because it will be deleted after successful performance.

So, if you have long-running jobs, it is better to configure the invisibility timeout interval:

var options = new SqlServerStorageOptions
{
    InvisibilityTimeout = TimeSpan.FromMinutes(30) // default value
};

GlobalConfiguration.Configuration.UseSqlServerStorage("<name or connection string>", options);

As of Hangfire 1.5 this option is now Obsolete. Jobs that are being worked on are invisible to other workers.

Say goodbye to confusing invisibility timeout with unexpected background job retries after 30 minutes (by default) when using SQL Server. New Hangfire.SqlServer implementation uses plain old transactions to fetch background jobs and hide them from other workers.

Even after ungraceful shutdown, the job will be available for other workers instantly, without any delays.