Background queues in .NET

Plenty of work should not happen inside a request: sending an email, rendering a PDF, pushing a webhook, recalculating a report. Doing it inline makes the request slow and lets a downstream failure take down a page that had nothing to do with it.

For a single process, System.Threading.Channels plus a BackgroundService is the smallest thing that works well: no extra infrastructure, proper async back-pressure, and clean shutdown.

The queue

Bounded is the important word. An unbounded channel accepts work faster than it is drained and eventually eats memory until the process dies. With a bound, producers wait — which is the correct behaviour under load.

public sealed record OutboxMessage(Guid Id, string Kind, string Payload);

public sealed class BackgroundQueue
{
    private readonly Channel<OutboxMessage> _channel =
        Channel.CreateBounded<OutboxMessage>(new BoundedChannelOptions(1_000)
        {
            FullMode = BoundedChannelFullMode.Wait,
            SingleReader = true,
            SingleWriter = false
        });

    public ValueTask EnqueueAsync(OutboxMessage message, CancellationToken ct) =>
        _channel.Writer.WriteAsync(message, ct);

    public IAsyncEnumerable<OutboxMessage> ReadAllAsync(CancellationToken ct) =>
        _channel.Reader.ReadAllAsync(ct);

    public void Complete() => _channel.Writer.TryComplete();
}

Register it as a singleton — it is shared state, and a scoped queue would give every request its own empty one:

builder.Services.AddSingleton<BackgroundQueue>();
builder.Services.AddHostedService<QueueWorker>();

The consumer

BackgroundService runs once at startup and lives until the host stops. Scoped services cannot be injected into it, because the host has no request scope — resolve them per message through IServiceScopeFactory.

public sealed class QueueWorker(
    BackgroundQueue queue,
    IServiceScopeFactory scopes,
    ILogger<QueueWorker> logger) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await foreach (var message in queue.ReadAllAsync(stoppingToken))
        {
            await using var scope = scopes.CreateAsyncScope();
            var handler = scope.ServiceProvider.GetRequiredKeyedService<IMessageHandler>(message.Kind);

            try
            {
                await handler.HandleAsync(message, stoppingToken);
            }
            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
            {
                break;
            }
            catch (Exception ex)
            {
                logger.LogError(ex, "Message {Id} ({Kind}) failed", message.Id, message.Kind);
                // retry policy or dead-letter here
            }
        }
    }

    public override async Task StopAsync(CancellationToken cancellationToken)
    {
        queue.Complete();
        await base.StopAsync(cancellationToken);
    }
}

Three details matter here. The try/catch is inside the loop, so one bad message does not kill the worker. The cancellation check distinguishes shutdown from failure. And Complete() on stop lets the reader finish the buffer instead of dropping in-flight work.

Back-pressure without long requests

If the queue is full, WriteAsync waits. In an HTTP handler that waiting is bounded by the request timeout, which gives you a natural circuit breaker: under sustained overload the API returns 503 instead of silently queueing gigabytes of work.

app.MapPost("/reports", async (ReportRequest request, BackgroundQueue queue, CancellationToken ct) =>
{
    var message = new OutboxMessage(Guid.NewGuid(), "report", JsonSerializer.Serialize(request));

    using var timeout = CancellationTokenSource.CreateLinkedTokenSource(ct);
    timeout.CancelAfter(TimeSpan.FromSeconds(2));

    try
    {
        await queue.EnqueueAsync(message, timeout.Token);
        return Results.Accepted($"/reports/{message.Id}");
    }
    catch (OperationCanceledException)
    {
        return Results.StatusCode(StatusCodes.Status503ServiceUnavailable);
    }
});

The honest limitation: this queue is not durable

An in-process channel disappears with the process. If the worker crashes between 202 Accepted and the actual send, that message is gone. That is acceptable for cache warming or analytics pings; it is not acceptable for invoices or order confirmations.

When the work must survive a restart, combine two things:

  • Write the intent first. Save the message to the same database, in the same transaction as the business change — the transactional outbox pattern.
  • Drain it with a durable consumer. That can be the same BackgroundService polling unprocessed rows, or a broker such as RabbitMQ, Azure Service Bus or Kafka with an at-least-once contract.

Hangfire and Quartz.NET are also reasonable when what you actually need is scheduling, retries and a dashboard rather than a stream of work.

Choosing

  • Channel + BackgroundService — one process, work can be lost on crash, no extra infrastructure.
  • Outbox table + poller — one process, work must not be lost, no broker to operate.
  • Real broker — several services, fan-out, independent scaling, or long-running work.

A queue does not remove work; it moves the failure. Decide up front whether losing a message is tolerable, and write that decision down next to the code.

« SOLID principles All posts