Kafka for Backend Engineers: Partitions, Ordering and Delivery Guarantees

Kafka shows up in almost every modern backend, and in almost every senior interview. You don't need to know every broker setting; you do need a precise model of partitions, ordering, consumer groups and delivery guarantees. That model prevents the most common production bugs.

The core model

• A topic is a named stream of records, such as orders.
• Each topic is split into partitions: ordered, append-only logs. Each record in a partition gets a sequential offset.
• Partitions are replicated across brokers (a leader plus followers) for durability.
• Records are retained for a configured time or size, whether or not anyone read them. Consumers can re-read ("replay") from any offset.

Kafka is not a queue that deletes messages when consumed. It's a durable log, and each consumer group tracks its own position in it.

Ordering: only within a partition

This is the most important rule: Kafka guarantees order only within a single partition.

A record's partition is chosen by its key: the same key always goes to the same partition (as long as the partition count doesn't change). With no key, records are spread across partitions.

// ❌ No key: "PAID" might be consumed before "CREATED"
kafkaTemplate.send("orders", event);

// ✅ Keyed by orderId: all events for one order stay in one partition, in order
kafkaTemplate.send("orders", order.getId(), event);

Rule of thumb: key by the entity whose events must stay ordered (orderId, accountId, userId). Choose keys with enough distinct values to spread load. A single "hot" key pins all its traffic to one partition.

Consumer groups: how work is shared

Consumers with the same group.id form a consumer group. Each partition is assigned to exactly one consumer in the group.

• 6 partitions, 3 consumers → 2 partitions each.
• 6 partitions, 8 consumers → 2 consumers sit idle. Partitions cap your parallelism.
• A different group (say analytics) reads the same topic independently, with its own offsets. That's how one event feeds many services.

When consumers join or leave, the group rebalances (reassigns partitions). Use the cooperative sticky assignor (the default in recent clients) and static membership to reduce disruption.

Delivery guarantees

On the producer side

• acks=all waits for all in-sync replicas before acknowledging a write. Combine it with the topic setting min.insync.replicas=2 (and replication factor 3) so an acknowledged write survives a broker failure.
• Idempotent producer (enable.idempotence=true, the default since Kafka 3.0) prevents duplicates caused by producer retries.

On the consumer side: when you commit the offset

• At-most-once: commit, then process. A crash after the commit loses the message.
• At-least-once: process, then commit. A crash before the commit reprocesses the message. This is the standard choice, so your processing must be idempotent.
• Exactly-once: Kafka transactions give exactly-once for read-process-write within Kafka (consume from topic A, produce to topic B, and commit offsets atomically). Once you write to an external database, you're back to at-least-once plus idempotency.

Making consumers idempotent

@KafkaListener(topics = "payments", groupId = "ledger")
@Transactional
public void on(PaymentEvent e) {
    if (processedRepo.existsById(e.eventId())) return;   // duplicate: already handled
    ledger.apply(e);
    processedRepo.save(new ProcessedEvent(e.eventId()));  // same DB transaction as the effect
}

Store the processed event IDs in the same transaction as the side effect, or make the operation naturally idempotent (an upsert with a version check).

Publishing reliably: the outbox pattern

Writing to your database and sending to Kafka are two separate systems, so a crash between them loses an event or sends one for a rolled-back change. The transactional outbox fixes this: write the event to an outbox table in the same database transaction as the business change, then a relay (a poller, or Debezium change data capture) publishes it to Kafka.

Errors and retries in Spring Kafka

• Use DefaultErrorHandler with backoff for transient errors, and a dead-letter topic for messages that keep failing.
• For long retry delays without blocking the partition, use @RetryableTopic (retry topics).
• Classify exceptions: deserialisation and validation errors go straight to the DLT, because retrying them never helps.

Sizing partitions

• Target throughput ÷ per-partition throughput, then add headroom.
• More partitions means more parallelism, but also more files, longer recovery and rebalancing costs.
• Increasing partitions later changes key → partition mapping for new records, which breaks ordering assumptions across the change. Plan ahead.

Follow-up questions this topic invites — and their answers

Q: How do you guarantee global ordering across a topic? A: Use a single partition, which limits throughput to one consumer. Usually you only need ordering per entity, which keys provide.

Q: What happens if a consumer is slow? A: Its lag grows (the offset gap to the latest record). If processing takes longer than max.poll.interval.ms, the consumer is considered dead and a rebalance happens. Monitor lag, and scale consumers up to the partition count.

Q: Kafka or RabbitMQ? A: Kafka for high-throughput event streams, replay, and many independent consumers of the same data. RabbitMQ for task queues with flexible routing, per-message acknowledgements and priorities.

Q: What's log compaction? A: A retention mode that keeps at least the latest record per key. It's ideal for "current state" topics, such as the latest price per product.

Learn more in our microservices chapters and the Kafka interview questions.