Deep Dive: Worker Heartbeats and Job Recovery

Three hours into production, a worker crashed mid-analysis. JetStream redelivered the message. The new worker looked at the database, saw “processing,” and skipped it. The queue did its job correctly and the work was lost anyway.

That’s where the real boundary is. JetStream gives me at-least-once delivery, but delivery is not ownership. A worker can pull a message, crash, and leave the database believing the job is still in flight. On redelivery the next worker sees an in-progress job that belongs to a dead process and politely backs off.

So I moved ownership into the database and left the queue as transport. The queue delivers messages; the database decides who owns the work. After that, recovery stopped being a web of edge cases and became a handful of conditionals.

The Invariant

A job is owned by exactly one worker at a time. If that stops being true, the system leaks work. Everything below exists to keep it true across crashes, restarts, and network partitions.

The Ownership Record

Jobs get two extra fields: a worker identity and a last-activity timestamp. That’s the whole heartbeat model.

ALTER TABLE jobs ADD COLUMN worker_id TEXT;
ALTER TABLE jobs ADD COLUMN last_activity_at TIMESTAMP WITH TIME ZONE;

CREATE INDEX idx_jobs_processing_activity
ON jobs(status, last_activity_at)
WHERE status = 'processing';

The index matters as much as the columns. Heartbeat queries run constantly, so a partial index on status = 'processing' keeps the recovery scan tight as the jobs table grows.

Worker Identity

A worker’s identity is hostname-pid-random:

• Hostname identifies the pod.
• PID distinguishes restarts inside the pod.
• Random bytes avoid collisions when processes restart fast.

const WORKER_ID = `${os.hostname()}-${process.pid}-${crypto.randomBytes(4).toString('hex')}`;

Not globally unique, but unique enough to tell one process instance from the next, which is all recovery needs. A UUID would work too; this format means I can see which pod a job belongs to without a lookup.

Claiming Is a Conditional Update

Claiming is a single UPDATE ... WHERE that encodes the ownership rules. If the update touches no rows, you don’t own the job.

async function claimJob(jobId) {
  const result = await db.query(`
    UPDATE jobs
    SET worker_id = $1,
        last_activity_at = NOW(),
        status = 'processing'
    WHERE id = $2
      AND status IN ('queued', 'processing')
      AND (
        worker_id IS NULL
        OR worker_id = $1
        OR last_activity_at < NOW() - INTERVAL '5 minutes'
      )
    RETURNING id
  `, [WORKER_ID, jobId]);

  return result.rows.length > 0;
}

The WHERE clause is the contract. A worker can claim a job if:

1. It is queued and unowned (worker_id IS NULL)
2. The worker already owns it (re-entrancy, useful for retries)
3. The previous owner stopped updating for 5 minutes (stale detection)

That INTERVAL '5 minutes' bounds both recovery time and false positives. Shorter, and a transient network blip triggers an unnecessary re-queue. Longer, and real failures take too long to recover. Five minutes is long enough for a git clone to stall on a slow network and short enough that nobody is staring at a stuck job.

Heartbeats Piggyback on Progress

There’s no separate heartbeat thread. Every progress update is a heartbeat, so progress reporting is the liveness signal.

async function updateProgress(jobId, stage, progress, message) {
  await db.query(`
    UPDATE jobs
    SET stage = $1,
        progress = $2,
        message = $3,
        last_activity_at = NOW()
    WHERE id = $4
  `, [stage, progress, message, jobId]);
}

If a worker is stuck, updates stop. That’s the only failure mode I need to detect, and it costs nothing extra to detect it.

The constraint this creates: jobs must make visible progress at least once every 5 minutes. Cloning and analyzing are both chatty, so that’s easy here. A system with legitimately long silent stretches would need periodic no-op heartbeats instead.

Startup Reconciliation

When a worker starts, it looks for jobs owned by previous incarnations of the same pod and clears them. The match is by hostname prefix, since the PID and random suffix change on restart.

async function reconcileOrphanedJobs() {
  const hostPattern = `${os.hostname()}-%`;

  const orphanedJobs = await db.query(`
    SELECT id, repo_url, status, stage, worker_id
    FROM jobs
    WHERE worker_id LIKE $1
      AND worker_id != $2
      AND status IN ('queued', 'processing')
  `, [hostPattern, WORKER_ID]);

  for (const job of orphanedJobs.rows) {
    await clearAndReenqueueJob(job.id, job.repo_url, 'Worker restarted');
  }
}

A worker that comes back cleans up its own mess before taking new work. Waiting for the 5-minute stale threshold would leave jobs stranded longer than necessary after an ordinary pod restart.

Stale Job Detection

Every 30 seconds, any healthy worker scans for stale jobs and re-enqueues them. No leader election, no coordinator; the database is the arbiter.

const HEARTBEAT_INTERVAL_MS = 30 * 1000;
const STALE_THRESHOLD_MS = 5 * 60 * 1000;

setInterval(async () => {
  const staleJobs = await db.query(`
    SELECT id, repo_url, worker_id, last_activity_at
    FROM jobs
    WHERE status = 'processing'
      AND (
        last_activity_at IS NULL
        OR last_activity_at < NOW() - INTERVAL '5 minutes'
      )
    ORDER BY last_activity_at ASC NULLS FIRST
    LIMIT 100
  `);

  for (const job of staleJobs.rows) {
    await clearAndReenqueueJob(job.id, job.repo_url,
      `Worker ${job.worker_id} stopped responding`
    );
  }
}, HEARTBEAT_INTERVAL_MS);

The LIMIT 100 keeps the scan bounded, so a system that’s on fire still makes forward progress instead of trying to recover everything at once. The ORDER BY prioritises whatever has been stuck longest.

Every worker runs this scan, so two workers can re-enqueue the same job. The claim step makes that harmless.

Re-enqueue Is a Reset, Not a Resume

Clearing a job sets it back to queued and republishes it to NATS. There’s no partial resume.

async function clearAndReenqueueJob(jobId, repoUrl, reason) {
  const job = await db.query('SELECT status FROM jobs WHERE id = $1', [jobId]);
  if (job.rows[0]?.status === 'completed' || job.rows[0]?.status === 'failed') return;

  await db.query(`
    UPDATE jobs SET
      status = 'queued',
      worker_id = NULL,
      last_activity_at = NULL,
      started_at = NULL,
      message = $1
    WHERE id = $2
  `, [`Cleared: ${reason}`, jobId]);

  await js.publish('jobs.analyze', JSON.stringify({
    job_id: jobId,
    repo_url: repoUrl,
    cleared: true,
    clear_reason: reason,
  }));
}

The cleared: true flag tells downstream code this is a retry, which helps with metrics and debugging. I’d rather have resumable jobs — checkpoint after each commit analysis — but without deterministic checkpoints, starting over is the safest recovery. The repo cache makes re-cloning cheap, so a reset costs the analysis time already spent, not the whole job.

Why Duplicates Are Harmless

Stale detection races. Two workers can flag the same stale job and both re-enqueue it, and JetStream may deliver the result to two workers. Only one UPDATE ... WHERE succeeds.

Worker A: detects job 123 is stale, re-enqueues
Worker B: detects job 123 is stale, re-enqueues (race)
NATS: delivers job 123 to Worker C
NATS: delivers job 123 to Worker D (duplicate)
Worker C: claims job 123, UPDATE returns 1 row, proceeds
Worker D: claims job 123, UPDATE returns 0 rows, skips

The queue can deliver duplicates; the database refuses them.

Tuning the Timers

Three numbers matter:

• STALE_THRESHOLD_MS is 5 minutes.
• HEARTBEAT_INTERVAL_MS is 30 seconds (the stale scan interval).
• NATS ack_wait is 10 minutes, set in nanoseconds.

Their ordering is what matters. ack_wait must exceed the stale threshold, or queue redelivery races database-driven recovery. The claim logic handles two workers arriving at once, but the metrics get confusing: you can’t tell whether recovery came from the queue or the database.

NATS as the backstop with the longer timeout, the database as the primary recovery mechanism with the shorter one.

What This Still Doesn’t Solve

• Partial network failure: a worker that can reach NATS but not the database looks dead. I’ll take that; partial work should be retried, and treating queue acknowledgement as ownership creates worse failure modes.
• Clock skew: not a factor, because all timestamps come from NOW() in PostgreSQL. Workers never compare local clocks.
• Hung processes: a deadlock that keeps the process alive looks exactly like a crash. No progress means no worker, as far as this system is concerned. Detecting genuinely hung processes would need out-of-band health checks, which don’t exist here.

The Model That Holds

The queue delivers, the database owns. That’s the sentence I hold onto during an incident, and it’s short enough to stay correct under stress.

The surprise was how much recovery logic disappeared once the queue stopped being the source of truth. JetStream is excellent at durable delivery, and durable delivery is not exclusive ownership. Giving ownership to the database turned the implementation into a list of conditionals.

Worker crashes now recover in under 6 minutes: the 5-minute threshold plus one scan interval. The path is the same whether a worker died, a pod restarted, or a partition healed.

---

See also: Building a Code Evolution Analyzer in a Weekend — the full project story