Kafka vs RabbitMQ vs SQS compared on ordering and delivery semantics, then a genuine deep-dive into Kafka's internals: partitions, consumer groups, rebalancing, ISR, idempotent producers, DLQs, consumer lag, and ZooKeeper vs KRaft.
Published September 23, 2026
Message Queue System (in the System Design track) designed a Kafka-like queue from first principles. This lesson is the practical comparison — Kafka vs RabbitMQ vs SQS — plus a genuine deep-dive into Kafka's internals, since Kafka is the default choice for most teams and its internals come up constantly in both interviews and real operational work.
Kafka: high throughput, an ORDERED LOG retained for replay — ideal for event
sourcing (OMS — State Machine Design) and streaming pipelines
RabbitMQ: flexible ROUTING (exchanges/queues, topic/fanout/direct routing) —
strong fit for traditional task queues and complex routing logic
SQS: fully managed, simple — the good default when you don't want to
operate broker infrastructure at all
Ordering guarantees, compared precisely: Kafka guarantees order only PER-PARTITION (Message Queue System's partitioning discussion applies directly); RabbitMQ guarantees order per-queue, but only with a SINGLE consumer on that queue (multiple consumers on the same queue lose ordering, since messages are distributed across them); SQS offers ordering ONLY on FIFO queues specifically (standard SQS queues make no ordering guarantee at all).
Delivery semantics, compared: at-least-once is the common baseline across all three — none of them give you true exactly-once delivery for free; each requires the same idempotent-consumer discipline covered in Message Queue System and Payment — Idempotency Implementation to actually achieve effectively-exactly-once processing.
Cluster: multiple BROKERS
Topic "orders": a named LOG, split into PARTITIONS for parallelism
Partition 0: [msg@offset0, msg@offset1, msg@offset2, ...] — each message's
position within ITS partition is its OFFSET
Each partition has ONE LEADER broker (handles all reads/writes for that
partition) and N FOLLOWER replicas (for fault tolerance)
Topic, partition, offset: a topic is the named log; partitions split that log across brokers for parallel throughput; the offset is simply each message's sequential position within its own partition — critically, offsets are only meaningful WITHIN a partition, not globally across the topic.
acks=0: fire and forget — producer doesn't wait for any confirmation at all
(fastest, weakest durability — a broker failure can silently lose the message)
acks=1: producer waits for the PARTITION LEADER to confirm the write
(moderate — lost only if the leader fails before followers replicate it)
acks=all (-1): producer waits for the leader AND all IN-SYNC REPLICAS (ISR) to confirm
(strongest durability, highest latency)
This is a direct, tunable trade-off between latency and durability, and the correct setting depends entirely on the data's importance — a high-volume metrics stream might reasonably use acks=1 or even acks=0 (losing an occasional metric point is a tolerable cost for lower latency); an order or payment event (OMS — State Machine Design, Payment — Core Flow) should use acks=all, since silently losing that message is a real correctness failure, not a minor inconvenience.
The ISR is the SET of replicas that are genuinely caught up with the partition leader at any given moment — acks=all specifically waits for every replica currently in the ISR (not necessarily every replica that theoretically exists, since a lagging replica temporarily falls out of the ISR) to confirm before acknowledging the write. This distinction matters: a replica that's fallen behind (network issue, temporary slowness) is excluded from the ISR, meaning acks=all doesn't wait indefinitely for a genuinely struggling replica — it waits for the currently-healthy replica set.
As introduced in Message Queue System, each partition is owned by exactly ONE consumer within a group at a time — this is what horizontally scales consumption up to the partition count, and it's also why two consumers in the SAME group never both receive the same message (use SEPARATE consumer groups if you genuinely need every message delivered to multiple independent consumers). Rebalancing is triggered whenever a consumer joins or leaves the group, or partition count changes — partition ownership is redistributed among the (now-different) set of consumers, which causes a brief PAUSE in consumption while the rebalance completes; frequent rebalancing (e.g. from consumers crashing and restarting repeatedly) is a real, measurable throughput cost worth monitoring.
At-most-once: no retry on failure — simplest, but a processing failure means the
message is just gone
At-least-once: retry on failure — the common default, but retries can produce
DUPLICATE delivery, requiring an idempotent consumer
Exactly-once: idempotent producer (below) + transactional consumer-producer —
genuinely achievable within Kafka specifically, unlike the broader
cross-system exactly-once problem discussed in Message Queue System
Idempotent producer (enable.idempotence=true): Kafka assigns each producer a unique ID and tracks a sequence number per partition, letting the BROKER itself detect and silently drop a duplicate retry from the same producer — this is a genuinely different mechanism from application-level idempotency keys (Payment — Idempotency Implementation); it solves duplicate PRODUCTION specifically (a producer's own retry after an ambiguous timeout), not duplicate consumption.
Main topic → processing fails → retry topic (with backoff) → still fails after N tries
→ Dead Letter Queue (DLQ) — held for manual inspection/replay, doesn't block the
main partition's later messages
A message that fails processing shouldn't simply block the entire partition (every message BEHIND it in that partition would also be stuck, since Kafka's per-partition ordering means the consumer can't skip ahead) — routing a failed message to a dedicated RETRY topic (with backoff between attempts) and, after exhausting retries, to a DLQ keeps the main partition flowing while still preserving the failed message for later inspection, rather than either silently dropping it or blocking everything behind it.
Consumer lag — the gap between the latest PRODUCED offset and the latest COMMITTED consumer offset — is the single most important Kafka-specific metric for "are consumers keeping up with producers." Growing lag over time means consumers can't process fast enough relative to incoming volume (needing more consumer instances, up to the partition count, or faster per-message processing) — this is a direct, Kafka-specific instance of Metrics & Monitoring's general monitoring principles, and a natural CloudWatch-alarm-style (Monitoring in Production) candidate for any Kafka-backed pipeline.
Kafka originally depended on ZooKeeper (a separate distributed coordination service, conceptually similar to Distributed Lock Service's consensus-based coordination) for cluster metadata and leader election. KRaft (Kafka Raft) removes this external dependency, managing that same metadata via a Raft consensus protocol built directly INTO Kafka itself — simplifying operations (one less separate system to run and keep available) and removing ZooKeeper-specific failure modes (like split-brain risk during ZooKeeper/controller failover) entirely. Modern Kafka deployments increasingly default to KRaft.
Max message size: capped by message.max.bytes on the broker (commonly ~1MB by default) and max.request.size on the producer — a genuinely large payload (an image, a full document) is almost always better handled by REFERENCING it (an S3 URL, connecting to the S3 lesson) than forcing it through Kafka inline.
Performance tuning: batching producer sends (linger.ms, batch.size — deliberately waiting a small window to batch multiple messages into one network request), compression (snappy/lz4), increasing partition count for more parallelism, and tuning consumer fetch sizes — always validated against ACTUAL observed throughput/latency (Metrics & Monitoring), never applied as blind, unverified "best practice" tuning.
Common redundancy issues in practice: under-replicated partitions (a follower falling behind its ISR window), split-brain risk during ZooKeeper/controller failover (largely eliminated by KRaft), and duplicate message delivery from at-least-once semantics — this last one is worth naming explicitly: a duplicate showing up downstream is usually a symptom of a MISSING idempotent consumer, not a Kafka bug to chase.
Q: When would RabbitMQ's flexible routing genuinely beat Kafka's simpler topic/partition model? A: When the ROUTING LOGIC itself is complex and needs to live in the broker — RabbitMQ's exchanges support content-based and pattern-based routing to multiple queues declaratively; achieving equivalent routing in Kafka typically means the CONSUMER applies filtering logic itself after receiving from a topic, pushing that complexity into application code rather than broker configuration.
Q: Is SQS's simplicity ever a real limitation, not just a convenience? A: Yes — SQS lacks Kafka's replay capability (once consumed and deleted, a standard SQS message is gone, with no equivalent to re-reading from an arbitrary earlier offset) and lacks Kafka's ordered-log-retention model entirely for standard queues — appropriate when you genuinely just need reliable task queuing, not when you need an auditable, replayable event history like OMS — State Machine Design's event sourcing.
Q: How does consumer lag interact with the DLQ mechanism? A: They address different failure modes — lag means consumers are falling behind on VOLUME (too slow, or under-provisioned); DLQ means specific INDIVIDUAL messages are failing to process correctly regardless of speed; a spike in lag right after deploying new consumer code, though, is a common signal worth checking against DLQ volume too, since a bug causing widespread processing failures often shows up as BOTH growing lag (failed messages retry, consuming time) and a growing DLQ simultaneously.
Q: Does moving from ZooKeeper to KRaft change any application-level code? A: No — it's purely an internal cluster-coordination change; producers and consumers interact with Kafka through the same client APIs regardless of which metadata-management mode the cluster runs, making a ZooKeeper-to-KRaft migration an operations-team concern, not something application teams need to account for in their own code.