Designing a system that reliably runs scheduled and delayed jobs (cron-like and one-off) exactly once across a fleet of workers — leader election for the scheduler itself, a time-bucketed job store, and exactly-once execution despite worker crashes.
Published September 23, 2026
Design a system that reliably executes scheduled jobs — recurring ("run every day at 2am") and one-off delayed jobs ("send this reminder in 3 hours") — across a fleet of worker machines, guaranteeing each job runs (close to) exactly once even as workers crash and restart.
Functional: schedule a recurring job (cron expression) or a one-off delayed job; execute jobs at their scheduled time; support job cancellation/rescheduling; retry a failed job with backoff. Non-functional: no job is EVER skipped entirely; no job runs MORE than once concurrently (even with many worker instances); the scheduler itself has no single point of failure; handles millions of scheduled jobs.
Single machine cron: trivial — one process, one clock, no coordination needed
Distributed: MULTIPLE worker instances exist for redundancy — but a job must run
EXACTLY ONCE across all of them, not once per worker instance
The entire design challenge is this single tension: redundancy (many workers, so a crashed worker doesn't mean missed jobs) directly conflicts with exactly-once execution (many workers must NOT all pick up and run the same job). This is structurally the same problem Distributed Lock Service solves generally — a distributed task scheduler is, in large part, an application of distributed locking to the specific problem of job execution.
[Job API] → [Job Store] (persisted, durable — the source of truth for what's scheduled)
[Scheduler Leader] (elected via Distributed Lock Service / consensus)
— polls the Job Store for jobs due to run
— for each due job, PUBLISHES it to a [Task Queue]
[Worker Pool] (many instances) — consumes from the Task Queue, executes jobs
— a message queue's own delivery/ack semantics (Message Queue System) provide
the "only one worker processes a given queued job" guarantee at THIS layer
Splitting the design into two distinct roles is the key insight: a single elected Scheduler Leader decides WHICH jobs are due and enqueues them (avoiding multiple schedulers double-enqueueing the same job), while a separate, horizontally-scaled Worker Pool actually executes them, relying on the task queue's own single-delivery guarantees for exactly-once execution at that layer.
Only one Scheduler Leader should be actively polling and enqueueing jobs at a time — this is a direct application of Distributed Lock Service: the scheduler role itself is protected by a lock (or a consensus-based leader election, e.g. via ZooKeeper/etcd), and if the current leader crashes, its lock/session expires and another scheduler instance takes over automatically. Multiple standby scheduler instances exist purely for failover — only the current leader is actually active.
job_id, cron_expression | run_at, next_run_time (INDEXED), status, payload
Scheduler Leader query (runs periodically, e.g. every second):
SELECT * FROM jobs WHERE next_run_time <= NOW() AND status = 'SCHEDULED'
FOR UPDATE SKIP LOCKED -- avoids re-selecting a job another process is already handling
An index on next_run_time is what makes the leader's "what's due now" query fast even against millions of scheduled jobs — without it, every poll would require scanning the entire job table. FOR UPDATE SKIP LOCKED (a real database feature, not pseudocode) lets the query atomically claim rows without blocking on rows another process is already processing — relevant even with a single leader, since the leader's own polling and a manual admin action could otherwise race.
Q: What happens if the Scheduler Leader is up but the Task Queue is temporarily unavailable?
A: The leader should retry enqueueing with backoff rather than dropping the due job — since the Job Store (not the queue) is the durable source of truth for what's scheduled, a job's next_run_time isn't advanced/cleared until it's confirmed enqueued, so a temporarily failed enqueue attempt is simply retried on the next poll cycle rather than lost.
Q: How do you handle a job that takes longer to execute than the interval between its own scheduled runs (e.g. a job scheduled every minute that takes 90 seconds)? A: This needs an explicit policy decision, not an accidental default: either skip the overlapping run entirely (simplest, avoids pileup), queue it to run immediately after the current execution finishes (risks a growing backlog if the job is consistently slow), or run it concurrently if the job is genuinely safe to run in parallel with itself — silently allowing unbounded concurrent executions of the same job is the one option that's never correct.
Q: Why not just have every worker independently check 'is it time for this job yet' rather than a dedicated leader? A: Without a single leader (or equivalent coordination), every worker checking independently would all decide a due job needs to run and could all enqueue/execute it simultaneously — exactly the double-execution problem this design exists to prevent; the leader role exists specifically to centralize the "is this job due" decision to one place at a time.
Q: How does this design change for VERY high-frequency jobs (thousands scheduled per second) vs a small number of infrequent jobs? A: At high frequency, the time-bucketed job table itself may need sharding (by job ID hash, with each shard having its own leader-elected sub-scheduler) to keep the 'what's due now' query fast — for a smaller job volume, a single leader polling a single, well-indexed table is entirely sufficient and simpler to operate.