Using Batches¶
Pro Only
This feature is a part of Hangfire Pro package set
Batches allow you to create a bunch of background jobs atomically. This means that if there was an exception during the creation of background jobs, none of them will be processed. Consider you want to send 1000 emails to your clients, and they really want to receive these emails. Here is the old way:
for (var i = 0; i < 1000; i++)
{
BackgroundJob.Enqueue(() => SendEmail(i));
// What to do on exception?
}
But what if storage become unavailable on i == 500? 500 emails may be already sent, because worker threads will pick up and process jobs once they created. If you re-execute this code, some of your clients may receive annoying duplicates. So if you want to handle this correctly, you should write more code to track what emails were sent.
But here is a much simpler method:
BatchJob.StartNew(x =>
{
for (var i = 0; i < 1000; i++)
{
x.Enqueue(() => SendEmail(i));
}
});
In case of exception, you may show an error to a user, and simply ask to retry her action after some minutes. No other code required!
Installation¶
Batches are available in the Hangfire.Pro package, and you can install it using NuGet Package Manager Console window as usually:
PM> Install-Package Hangfire.Pro
Batches require to add some additional job filters, some new pages to the Dashboard, and some new navigation menu items. But thanks to the new GlobalConfiguration class, it is now as simple as a one method call:
GlobalConfiguration.Configuration.UseBatches();
Limited storage support
Only official Hangfire.InMemory, Hangfire.SqlServer and Hangfire.Pro.Redis job storage implementations are currently supported. There is nothing special for batches, but some new storage methods should be implemented.
Configuration¶
The default batch job expiration/retention time if the batch succeeds is 7 days, but you can configure it when calling the UseBatches method:
GlobalConfiguration.Configuration.UseBatches(TimeSpan.FromDays(2));
Chaining Batches¶
Continuations allow you to chain multiple batches together. They will be executed once all background jobs of a parent batch finished. Consider the previous example where you have 1000 emails to send. If you want to make final action after sending, just add a continuation:
var id1 = BatchJob.StartNew(/* for (var i = 0; i < 1000... */);
var id2 = BatchJob.ContinueBatchWith(id1, x =>
{
x.Enqueue(() => MarkCampaignFinished());
x.Enqueue(() => NotifyAdministrator());
});
So batches and batch continuations allow you to define workflows and configure what actions will be executed in parallel. This is very useful for heavy computational methods as they can be distributed to different machines.
Complex Workflows¶
Create action does not restrict you to create jobs only in Enqueued state. You can schedule jobs to execute later, add continuations, add continuations to continuations, etc..
var batchId = BatchJob.StartNew(x =>
{
x.Enqueue(() => Console.Write("1a... "));
var id1 = x.Schedule(() => Console.Write("1b... "), TimeSpan.FromSeconds(1));
var id2 = x.ContinueJobWith(id1, () => Console.Write("2... "));
x.ContinueJobWith(id2, () => Console.Write("3... "));
});
BatchJob.ContinueBatchWith(batchId, x =>
{
x.Enqueue(() => Console.WriteLine("4..."));
});
Nested Batches¶
Since version 2.0, batches can consist of other batches, not only of background jobs. Outer batch is called as parent, inner batch is a child one (for continuations, it’s an antecedent/continuation relationship). You can mix both batches and background jobs together in a single batch.
BatchJob.StartNew(parent =>
{
parent.Enqueue(() => Console.WriteLine("First"));
parent.StartNew(child => child.Enqueue(() => Console.WriteLine("Second")));
});
Multiple nesting levels are supported, so each child batch can, in turn, become a parent for another batch, allowing you to create very complex batch hierarchies.
BatchJob.StartNew(batch1 =>
{
batch1.StartNew(batch2 =>
{
batch2.StartNew(batch3 => batch3.Enqueue(() => Console.WriteLine("Nested")));
});
});
The whole hierarchy, including parent batch, all of its child batches and background jobs are created in a single transaction. So this feature not only allows you to see a group of related batches on a single dashboard page, but also create multiple batches atomically.
var antecedentId = BatchJob.StartNew(batch =>
{
batch.StartNew(inner => inner.Enqueue(() => Console.WriteLine("First")));
batch.StartNew(inner => inner.Enqueue(() => Console.WriteLine("Second")));
});
Parent batch is succeeded, if all of its background jobs and batches are succeeded. Parent batch is finished, if all of its batches and background jobs are in a final state. So you can create continuation for multiple batches, not just for a single one. Batch continuations also support the nesting feature.
BatchJob.ContinueBatchWith(antecedentId, continuation =>
{
continuation.StartNew(inner => inner.Enqueue(() => Console.WriteLine("First")));
continuation.StartNew(inner => inner.Enqueue(() => Console.WriteLine("Second")));
});
Starting from Hangfire.Pro 2.1.0 it’s also possible to use continuations in batches, both standalone and nested ones, for both batches and background jobs.
BatchJob.StartNew(parent =>
{
var nested1 = parent.StartNew(nested =>
{
nested.Enqueue(() => Console.WriteLine("Nested 1"));
});
var nested2 = parent.ContinueBatchWith(nested1, () => Console.WriteLine("Nested 2"));
var nested3 = parent.ContinueJobWith(nested2, nested =>
{
nested.Enqueue(() => Console.WriteLine("Nested 3"));
});
string nested5 = null;
var nested4 = parent.ContinueBatchWith(nested3, nested =>
{
nested5 = nested.Enqueue(() => Console.WriteLine("Nested 4"));
});
parent.ContinueJobWith(nested5, () => Console.WriteLine("Nested 5"));
});
Batch Modification¶
This is another interesting feature available from version 2.0 that allows you to modify existing batches by attaching new background jobs and child batches to them. You can add background jobs in any states, as well as nested batches. If a modified batch has already been finished, it will be moved back to the started state.
var batchId = BatchJob.StartNew(batch => batch.Enqueue(() => Console.WriteLine("First")));
BatchJob.Attach(batchId, batch => batch.Enqueue(() => Console.WriteLine("Second")));
This feature helps, if you want a list of records you want to process in parallel, and then execute a continuation. Previously you had to generate a very long chain of continuations, and it was very hard to debug them. So you can create the structure, and modify a batch later.
var batchId = BatchJob.StartNew(batch => batch.Enqueue(() => ProcessHugeList(batch.Id, ListId)));
BatchJob.ContinueBatchWith(batchId, batch => batch.Enqueue(() => SendNotification(ListId)));
// ProcessHugeList
BatchJob.Attach(batchId, batch =>
{
foreach (var record in records)
{
batch.Enqueue(() => ProcessRecord(ListId, record.Id)));
}
});
Batches can be created without any background jobs. Initially such an empty batches are considered as completed, and once some background jobs or child batches are added, they move a batch to the started state (or to another, depending on their state).
var batchId = BatchJob.StartNew(batch => {});
BatchJob.Attach(batchId, batch => batch.Enqueue(() => Console.WriteLine("Hello, world!")));
More Continuations¶
Since version 2.0 it’s possible to continue batch with a regular background job without creating a batch that consists only of a single background job. Unfortunately we can’t add extension methods for static classes, so let’s create a client first.
var backgroundJob = new BackgroundJobClient();
var batchId = BatchJob.StartNew(/* ... */);
backgroundJob.ContinueBatchWith(batchId, () => Console.WriteLine("Continuation"));
You can use the new feature in other way, and create batch continuations for regular background jobs. So you are free to define workflows, where synchronous actions are continued by a group of parallel work, and then continue back to a synchronous method.
var jobId = BackgroundJob.Enqueue(() => Console.WriteLine("Antecedent"));
BatchJob.ContinueJobWith(jobId, batch => batch.Enqueue(() => Console.WriteLine("Continuation")));
Cancellation of a Batch¶
If you want to stop a batch with millions of background jobs from being executed, not a problem, you can call the Cancel method, or click the corresponding button in dashboard.
var batchId = BatchJob.StartNew(/* a lot of jobs */);
BatchJob.Cancel(batchId);
This method does not iterate through all the jobs, it simply sets a property of a batch. When a background job is about to execute, job filter checks for a batch status, and move a job to the Deleted state, if a batch has cancelled.