Hangfire: How to skip concurrent executions without failing the job
I've got a minutely recurring job that uses DisableConcurrentExecution(timeout:0) (0 retries), as only one worker should be processing it at a given time, specially if the job takes more than a minute to complete (it varies).
I don't want the "skipped" jobs to be flagged as failed on the dashboard, so I wanted to add a SkipConcurrentExecutionAttribute that would be almost identical to DisableConcurrentExecutionAttribute, except that it would try/catch around the lock acquisition, and swallow DistributedLockTimeoutException and set the job to DeletedState with the reason why.
But I can't do this properly, because GetResource() is private, and its implementation refers to other methods that are internal to the library.
$"{job.Type.ToGenericTypeString()}.{job.Method.Name}" seems to be the safe way of producing non-conflicting distributed lock names, but I can't refer to it.
Can we have in the lib a public method that generates the job's distributed lock name?
cf., https://discuss.hangfire.io/t/disableconcurentexecution-for-job-groups/1389/4
dgaspar
All 4 comments
I just noticed that filterContext.BackgroundJob.Job.ToString() would give the same string as the internal TypeExtensions.cs and DisableConcurrentExecutionAttribute.cs (re)implementations would... problem solved.
dgaspar
on 11 Jul 2017
@dgaspar, you can tell recurring job scheduler to skip creation of the next recurring job, when previous one is still running, please see this gist: https://gist.github.com/odinserj/a6ad7ba6686076c9b9b2e03fcf6bf74e.
odinserj
on 11 Jul 2017
Thanks for the suggestion @odinserj. I noticed that if the worker crashes during execution, that "Running" row will remain in the Hash table. Will it prevent the job from being re-scheduled, until I manually delete that row? Or is there an expiration/cleanup that I'm missing? (btw, I'm using new SqlServerStorageOptions { SlidingInvisibilityTimeout = TimeSpan.FromMinutes(5) }, if that matters)
dgaspar
on 11 Jul 2017
When a worker is terminated during the execution of a background job, that background job will be rescheduled automatically, because all message queues are transactional (at least the official ones, other storage implementations should act in the same way).
odinserj
on 11 Jul 2017
Related issues
vikramjb
·
3Comments
shorbachuk
·
4Comments
nigel-dewar
·
3Comments
nathvi
·
4Comments
dealproc
·
3Comments
Most helpful comment
@dgaspar, you can tell recurring job scheduler to skip creation of the next recurring job, when previous one is still running, please see this gist: https://gist.github.com/odinserj/a6ad7ba6686076c9b9b2e03fcf6bf74e.