design-distributed-job-scheduler.md

Design a distributed job scheduler (like Cron)

Run jobs at scheduled times or intervals across a fleet, reliably and exactly as intended.

Requirements

• Schedule jobs to run at a specific time or on a recurring interval.
• Run them reliably across a fleet of workers.
• Avoid running the same job twice (or handle it safely).
• Scale to many jobs and survive worker failures.

Key ideas

• Storage: persist jobs with their next run time, partitioned by time so the scheduler scans only the near-future window (related to the reminder system).
• Dispatch: due jobs are placed on a queue; workers pull and execute them.
• Exactly-once vs at-least-once: distributed scheduling usually guarantees at-least-once, so jobs should be idempotent, or use a lock (see Chubby) so only one worker runs a given job.
• Reliability: retries with backoff, and a dead letter path for jobs that keep failing.

High-level design

flowchart LR
    Store[(Jobs by fire time)] --> Sched[Scheduler]
    Sched --> Q[Queue]
    Q --> W1[Worker]
    Q --> W2[Worker]
    Lock[Lock Service] -.-> Sched

Loading

Go deeper

• Quick, focused prep: System Design Interview Crash Course
• Full course: Grokking the System Design Interview