Designing the messaging infrastructure itself (a Kafka/RabbitMQ-like system) — partitioning for throughput, consumer groups and offset tracking, at-least-once vs exactly-once delivery, and how a queue survives broker failure.
Published September 23, 2026
Many designs in this course (Distributed Task Scheduler, the video transcoding pipeline in Video Streaming Platform, async order processing) simply USE a message queue as a building block. This lesson designs the queue itself.
Design a distributed message queue (Kafka/RabbitMQ-like) that lets producers publish messages and consumers process them asynchronously and reliably, decoupling producers from consumers in both time and load.
Functional: publish a message to a topic; consume messages from a topic, optionally as part of a consumer group; support multiple independent consumers of the same messages. Non-functional: high write throughput (millions of messages/sec at scale); durability (a published message survives a broker crash); ordering guarantees within a defined scope (per-partition); horizontal scalability for both producers and consumers.
Topic "orders" split into 8 partitions:
partition = hash(orderId) % 8
Partition 0: [msg1, msg5, msg9, ...] (each partition is an ORDERED, append-only log)
Partition 1: [msg2, msg6, msg10, ...]
...
A topic is split into multiple partitions, each an independently ordered, append-only log — this is what makes both write and read throughput scale horizontally: different partitions can be written to and read from in parallel, on different broker machines. The tradeoff this creates: ordering is only guaranteed WITHIN a partition, not across the whole topic — messages that must be strictly ordered relative to each other (e.g. all events for one order) need to be routed to the same partition, typically by hashing a consistent key (orderId) exactly like consistent hashing in load balancing.
Consumer Group "order-processors" (3 consumer instances, 8 partitions):
Consumer A: partitions 0, 1, 2
Consumer B: partitions 3, 4, 5
Consumer C: partitions 6, 7
Each consumer tracks its OFFSET (position) per partition:
partition 0, offset 4521 -- "I've processed everything up to message 4521"
A consumer group lets multiple consumer instances split the work of consuming a topic — each partition is consumed by exactly ONE consumer within a group at a time (this is what prevents duplicate processing within the group), while DIFFERENT consumer groups can each independently consume the SAME full topic (e.g. one group indexing orders for search, a completely separate group sending order-confirmation emails — both read every message, independently, without interfering with each other). Offsets being tracked per-partition per-group (not globally) is what makes independent re-reading/re-processing by different consumers possible at all.
At-most-once: message might be LOST, never processed twice (rarely acceptable)
At-least-once: message is NEVER lost, but might be processed MORE than once (the common default)
Exactly-once: message processed exactly once (hardest to guarantee, real systems achieve
this via idempotent consumers on top of at-least-once delivery, not a magic
different delivery mechanism)
At-least-once is the practical default for most systems: a consumer only advances its offset AFTER successfully processing a message, so a crash mid-processing means the message is redelivered (safe — nothing lost) but potentially processed twice (the consumer's own logic needs to tolerate this). "Exactly-once" in practice is almost always at-least-once delivery COMBINED WITH an idempotent consumer (the same idempotency-key pattern from Payment — Idempotency Implementation, applied to message processing) — it's rarely a property the queue alone can provide unconditionally.
Partition 0 replicated across 3 brokers: 1 leader (handles all reads/writes),
2 followers (replicate from the leader)
"acks=all": a write is only confirmed to the producer once ALL replicas have it
— durable even if the leader crashes immediately after
Each partition is replicated across multiple broker machines (a leader-follower model, similar in shape to database read replicas but here every replica exists purely for durability/failover, not read scaling) — a message isn't considered durably published until it's replicated to a configurable number of replicas, directly trading write latency (waiting for replication) for durability guarantees.
Q: What happens when a consumer in a group crashes? A: The group triggers a REBALANCE — the crashed consumer's partitions are reassigned to the remaining live consumers in the group, using their last-committed offsets, so no partition goes unconsumed; this rebalancing is itself a coordination problem (often solved via a dedicated group-coordinator broker or a consensus mechanism).
Q: How many partitions should a topic have? A: More partitions increase maximum parallelism (more consumers can work in parallel) but each partition adds some overhead (open file handles, replication traffic) and, since ordering only holds within a partition, over-partitioning can also fragment ordering guarantees more than needed — a common starting heuristic is sizing partition count to the expected number of concurrent consumers, not maximizing it arbitrarily.
Q: Why can't the queue itself just guarantee true exactly-once delivery, avoiding the need for idempotent consumers? A: True exactly-once delivery across an unreliable network is provably very difficult (related to the same fundamental issue behind Payment — Requirements' 'no true undo' — a response confirming delivery can itself be lost) — most production systems accept at-least-once at the transport layer and push the idempotency requirement to the consumer, where it's a well-understood, solvable problem rather than an unsolvable one.
Q: How does a message queue relate to a distributed task scheduler's design? A: Directly — Distributed Task Scheduler uses a message queue as its Task Queue layer specifically to get single-delivery-per-message semantics for job execution; that design is best understood as this queue design PLUS a leader-elected component deciding what to enqueue and when.