What Breaks a Job Queue at Scale (and How to Make It Boring)

Every backend eventually grows a job queue. Some work is too slow to do inside a request - send the email, run the export, call the model - so you push it onto a queue and let a worker handle it later. The first version takes an afternoon and works beautifully in the demo.

Then it meets production, where jobs fail halfway, workers get killed mid-task, and one noisy tenant tries to enqueue ten thousand jobs at once. This post is about the gap between those two, and the four changes that close it.

The demo version

Here’s roughly what everyone writes first:

queue.process(async (job) => {
  await doTheWork(job.data);
});

It’s correct, right up until it isn’t. There’s no limit on how many run at once, no plan for failure, and no answer to the question “what happens to a job whose worker died?” Let’s fix those one at a time.

1. Bounded concurrency

Unbounded concurrency is how a queue takes down the very services it depends on. If ten thousand jobs each open a database connection, your database - not your queue - is the thing that falls over.

The fix is a hard ceiling on how many jobs run in parallel, sized to the slowest downstream dependency:

const worker = new Worker('tasks', handler, {
  concurrency: 8, // never more than 8 in flight per worker
});

Eight is not a magic number. It’s “how many concurrent calls can the downstream survive,” measured, not guessed. The queue’s job is to absorb bursts, not forward them.

2. Retries with exponential backoff

Transient failures - a timeout, a rate limit, a brief blip - should not be fatal. But retrying immediately just hammers a service that’s already struggling. Back off, and add jitter so a thousand jobs don’t all retry on the same tick:

await queue.add('task', data, {
  attempts: 3,
  backoff: { type: 'exponential', delay: 1000 }, // 1s, 2s, 4s
});

Three attempts with exponential backoff handles the overwhelming majority of transient failures. What it must not do is retry things that will never succeed - which brings us to the next point.

3. Stalled-job recovery

This is the one people forget. A worker picks up a job, then the process is killed - a deploy, an OOM, a crash. The job is now in limbo: marked “active,” but nobody is working on it. Without recovery, it sits there forever.

The pattern is a heartbeat plus a reaper. Active jobs renew a lock on an interval; if the lock expires, the job is considered stalled and returned to the queue:

const worker = new Worker('tasks', handler, {
  lockDuration: 30_000,     // a job must renew its lock every 30s
  stalledInterval: 30_000,  // the reaper checks for expired locks this often
  maxStalledCount: 1,       // recover once, then fail loudly
});

maxStalledCount matters: a job that stalls repeatedly is probably crashing the worker, and blindly re-queuing it forever is how one poison job takes down your whole fleet. Recover once, then move it to a dead-letter queue and page a human.

4. Deterministic state

The quiet requirement underneath all of this: at any instant, a job is in exactly one well-defined state, and transitions only happen in one direction per outcome.

waiting -> active -> completed
                  -> failed -> waiting   (retry, if attempts remain)
                            -> dead-letter (attempts exhausted)

If two workers can grab the same job, or a “completed” job can silently slide back to “active,” you get double-sends and lost work - the bugs that are almost impossible to reproduce because they depend on timing. Enforce the transitions in one place, make them atomic, and never let application code set state directly.

The point

None of these four is clever. That’s the point. A good job queue is boring: bounded so it can’t amplify load, patient so it rides out blips, self-healing so a dead worker doesn’t strand work, and strict about state so nothing runs twice.

Boring is the goal. Boring is what lets you deploy on a Friday.