Kafka Internals & Delivery Semantics — Interview Questions

How to use this lesson

Kafka interviews at senior level focus on guarantees and their costs:

• ordering (per partition, by key);
• durability (acks, the ISR, min.insync.replicas);
• delivery semantics (at-least-once plus idempotency is the practical default);
• operations (lag, rebalances, retention).

Kafka 4.0 removed ZooKeeper (KRaft only), so mention that when you describe the architecture.

Q1. What's the difference between a consumer group and a consumer instance?

Short answer:

• A consumer instance: one KafkaConsumer (one thread, or one listener container thread), which reads from the partitions assigned to it.
• A consumer group: a set of instances sharing a group.id. Each partition is assigned to exactly one instance in the group, which gives load balancing and ordering per partition. The group has one committed offset per partition.
• Different groups each get a full copy of the stream: that's pub/sub fan-out.
• More instances than partitions means idle instances. Partitions are the unit of parallelism. Membership changes trigger a rebalance: use cooperative sticky assignment (CooperativeStickyAssignor), static membership (group.instance.id), and Kafka 4's new consumer rebalance protocol (KIP-848) to reduce the disruption.

Learn it in depth → Messaging Technology Choices

Q2. How does offset management work? What's a committed offset vs the current offset?

Short answer:

• The current (position) offset: the offset of the next record the consumer will fetch, held in memory by the consumer, and advanced by poll().
• The committed offset: the position saved to Kafka (the internal __consumer_offsets topic) for the group and partition. It's where the group resumes after a restart or rebalance. It's committed automatically (enable.auto.commit=true, every 5 seconds by default, which risks loss or duplicates) or manually (commitSync/commitAsync; in Spring Kafka, AckMode.RECORD/BATCH/MANUAL_IMMEDIATE).
• Lag is the log-end offset minus the committed (or current) offset.
• Semantics depend on when you commit:
  • commit after processing gives at-least-once (a crash between processing and committing means reprocessing);
  • commit before processing gives at-most-once (a crash means loss).
• auto.offset.reset (earliest/latest) applies only when no committed offset exists, or it's out of range (expired by retention).

Q3. How does Kafka support reprocessing and replay?

Short answer: Kafka is a durable, replayable log: records stay for the retention period, regardless of consumption. To replay:

• reset the group's offsets: kafka-consumer-groups.sh --reset-offsets --to-datetime/--to-earliest/--shift-by --execute (with the group stopped), or programmatically with seek/seekToBeginning/offsetsForTimes. Spring Kafka offers ConsumerSeekAware;
• a new consumer group reads from the earliest offset (for example, to rebuild a read model or backfill a new service);
• compacted topics keep the latest value per key forever, which is ideal for rebuilding state (changelogs, reference data).

Design for replay: idempotent consumers (replays produce duplicates), and deterministic processing (avoid side effects like emails on replay, or guard them). Version the event schemas.

Q4. Compare at-least-once, at-most-once and exactly-once delivery in Kafka. How do you ensure exactly-once?

Short answer:

• At-most-once: commit offsets before processing, or fire-and-forget producers (acks=0, no retries). Messages can be lost, never duplicated. It's acceptable for lossy telemetry.
• At-least-once: producer acks=all plus retries, and consumers commit after processing. There's no loss, but duplicates are possible (producer retries, consumer crashes before committing, rebalances). This is the common default, made safe by idempotent consumers.
• Exactly-once semantics (EOS):
  • within Kafka, for read → process → write to Kafka pipelines: idempotent producers (no duplicates from retries per partition), plus transactions (transactional.id; atomic writes to several partitions, plus committing the consumer offsets in the same transaction through sendOffsetsToTransaction), plus consumers reading with isolation.level=read_committed. Kafka Streams enables it with processing.guarantee=exactly_once_v2;
  • for external side effects (databases, emails, payment APIs), Kafka's EOS doesn't apply. You need idempotent handling (deduplication by event ID, upserts, unique constraints), or the outbox/inbox patterns, to get effectively-once outcomes.

Learn it in depth → Event-Driven Architecture Patterns

Q5. What is the ISR (in-sync replicas)? What happens when it shrinks, or is empty?

Short answer: Each partition has a leader, and followers that replicate its log. The ISR is the set of replicas fully caught up with the leader (within replica.lag.time.max.ms).

• With acks=all, a write is acknowledged only when all the ISR members have it, and min.insync.replicas (for example 2, with replication factor 3) sets the minimum ISR size required to accept writes.
• When the ISR shrinks below min.insync.replicas, producers with acks=all get NotEnoughReplicasException: writes are rejected, in favour of durability. Reads continue from the leader.
• When the ISR is empty (the leader dies, with no in-sync follower): the partition is offline, unless unclean leader election (unclean.leader.election.enable=true) lets an out-of-sync replica become the leader. That restores availability, but loses data. It's off by default. That's the classic C-versus-A choice.
• The durable configuration: RF=3, min.insync.replicas=2, acks=all, idempotent producers, and unclean election disabled.

Q6. How do you configure Kafka for high throughput? What are the best practices for tuning producers and consumers?

Short answer:

• Producers:
  • batching: batch.size (for example 64–256 KB) and linger.ms (5–20 ms), trading a little latency for much bigger batches;
  • compression: compression.type=zstd or lz4;
  • buffer.memory;
  • max.in.flight.requests.per.connection ≤ 5, with idempotence (which preserves ordering);
  • asynchronous sends with callbacks, never get() per message.
• Consumers:
  • fetch.min.bytes/fetch.max.wait.ms for bigger fetches;
  • max.poll.records;
  • batch listeners, processed efficiently (bulk database writes);
  • enough partitions, and consumer concurrency up to the partition count;
  • keep processing within max.poll.interval.ms.
• Topics and brokers:
  • the partition count sized for the target throughput (per-partition throughput × partitions), without going wild (partitions cost memory, file handles and recovery time);
  • replication factor 3;
  • broker network and I/O threads;
  • fast disks, and relying on the page cache.
• Measure with kafka-producer-perf-test/kafka-consumer-perf-test, and watch the broker request latency, under-replicated partitions, and consumer lag.

Q7. What is an idempotent producer?

Short answer: With enable.idempotence=true (the default since Kafka 3.0), the broker assigns the producer a producer ID (PID), and the producer attaches a sequence number per partition to each batch. The broker de-duplicates retried batches (the same PID and sequence), and rejects out-of-order sequences. So retries can't create duplicates or reorder records within a partition, for a single producer session. It requires acks=all, retries > 0, and max.in.flight.requests.per.connection ≤ 5. Its limits: it covers only a single producer session and one partition. Across restarts, or several partitions atomically, you need transactions (transactional.id). It doesn't de-duplicate application-level re-sends of the same business event, so include an event ID for consumers.

Q8. What happens if a consumer crashes?

Short answer:

1. Detection: its heartbeats stop, and after session.timeout.ms (45 seconds by default) the group coordinator removes it. If processing hangs without polling, max.poll.interval.ms expiry kicks it out.
2. Rebalance: its partitions are reassigned to the remaining group members (with cooperative rebalancing, only the affected partitions move).
3. Resumption: the new owner starts from the last committed offset. Records processed but not committed by the crashed consumer are processed again, so duplicates. That's why at-least-once needs idempotent consumers.
4. With static membership (group.instance.id), a quick restart within the session timeout avoids a rebalance entirely.

Handle it: commit after processing, keep processing short, store external side effects idempotently, and monitor rebalance rates and lag.

Q9. How do you handle duplicate messages? How do you design an idempotent consumer?

Short answer: Assume at-least-once delivery, and make processing idempotent:

• Deduplicate by event ID: a processed_events(event_id PRIMARY KEY, processed_at) table, inserted in the same database transaction as the business change. On a duplicate key, skip. This is the inbox pattern. Keep a TTL, or clean up old IDs.
• Natural idempotency: upserts and set-state operations (UPDATE order SET status='SHIPPED' WHERE id=?) instead of increments, or conditional updates using versions or sequence numbers (apply only if event.version > stored.version).
• Unique constraints on business keys (payment reference).
• External calls: pass idempotency keys to downstream APIs (payment providers).
• Kafka-only pipelines: EOS transactions.
• Caches (Redis SET NX with a TTL) for cheap deduplication, if a small risk of loss on eviction is acceptable.

@KafkaListener(topics = "payments.completed", groupId = "orders")
@Transactional
public void on(PaymentCompleted e) {
    if (inbox.alreadyProcessed(e.eventId())) return;             // INSERT … ON CONFLICT DO NOTHING returns 0 rows
    orders.markPaid(e.orderId(), e.paymentRef());
}

Q10. How do you scale Kafka?

Short answer:

• Consumers: add instances, up to the partition count. Increase the partitions when you need more parallelism (but adding partitions changes key-to-partition mapping, which breaks per-key ordering for existing keys; plan partition counts up front). Optimise the processing itself (batching, asynchronous I/O, and parallel processing with ordering by key: Confluent Parallel Consumer).
• Producers: batching, compression, and more producer instances.
• Brokers: add brokers, and reassign partitions (kafka-reassign-partitions.sh, Cruise Control) to spread the load. Scale disks (tiered storage, KIP-405, offloads old segments to object storage). Scale the network.
• Topic design: enough partitions per topic for peak throughput, a good key distribution (no hot keys), and separate topics for workloads with different SLAs.
• Clusters: separate clusters per domain or criticality, and multi-region replication (MirrorMaker 2, Cluster Linking). Managed services (MSK, Confluent Cloud) with auto-balancing.
• KRaft (Kafka 4) removes ZooKeeper's scaling limits on partitions and metadata.

Q11. What is Kafka's retention policy?

Short answer: Retention decides how long records stay in a topic, independent of consumption:

• time-based: retention.ms (7 days by default);
• size-based: retention.bytes per partition;
• deletion happens per log segment (segment.ms/segment.bytes), so data can outlive the limit until its segment rolls.
• cleanup.policy=delete (the default) drops old segments. compact keeps the latest record per key (plus tombstones for deletes, for delete.retention.ms), for changelogs and state. compact,delete combines both.
• Tiered storage keeps long retention cheap (hot local disk, older data in object storage).
• Consider: consumer downtime tolerance (retention must exceed the worst-case outage, or the consumers lose data), replay needs, compliance (GDPR deletion needs tombstones and compaction, or crypto-shredding), and disk costs.

Q12. What is Kafka Streams, and how does it differ from the consumer API? How do you join streams?

Short answer:

• Kafka Streams: a client library (not a cluster) for stateful stream processing, with a DSL (KStream, KTable, GlobalKTable): filter, map, groupBy, aggregations, windowing (tumbling, hopping, session), joins, and exactly-once processing. It keeps local state stores (RocksDB) backed by changelog topics (fault-tolerant, restorable), and scales by running more instances (tasks per partition). Interactive queries expose the state.

• The plain consumer API: you poll records and handle everything yourself: state, windowing, fault tolerance, rebalancing of state, and EOS. It's fine for simple per-record processing (and Spring's @KafkaListener).

• Joins:

  • KStream–KStream: a windowed join (records within X minutes: orders plus payments);
  • KStream–KTable: enriches events with the latest table value (orders enriched with the customer profile), non-windowed;
  • KTable–KTable: a changelog join (a materialised view);
  • KStream–GlobalKTable: a join against a fully replicated table, with no co-partitioning needed (small reference data).

  Stream–stream and stream–table joins require co-partitioning: the same number of partitions, and the same key, so repartition (selectKey + repartition) if needed.

KStream<String, Order> orders = builder.stream("orders");
KTable<String, Customer> customers = builder.table("customers");
orders.selectKey((k, o) -> o.customerId())
      .join(customers, (order, customer) -> new EnrichedOrder(order, customer.tier()))
      .to("orders.enriched");

Q13. How do you handle back-pressure in a Kafka consumer?

Short answer: Kafka consumers are pull-based, so they naturally consume only as fast as they poll. The real risks are processing slower than the poll deadline, and unbounded in-memory buffering:

• Tune max.poll.records and processing time to stay within max.poll.interval.ms, or you get evicted, then rebalance, then reprocess.
• Pause and resume partitions (consumer.pause(...), or Spring Kafka's container pause()/resume()) when downstream systems are slow or overloaded (a full database pool, an open circuit breaker), while still polling to keep the group membership.
• Bounded internal queues if you hand off to worker threads, and commit only completed offsets.
• Scale out: more consumers or partitions, or faster processing (batching).
• Reactive Kafka (reactor-kafka) propagates Reactive Streams demand into fetches.
• Monitor the lag, and alert on sustained growth, rather than trying to consume at any cost.

Q14. How would you design a booking event flow with Kafka?

Short answer:

1. Command side: the Booking service validates the request, reserves inventory in its database (atomically), and writes BookingRequested/BookingHeld to an outbox in the same transaction. Debezium or a relay publishes it to bookings.events (keyed by booking ID, for per-booking ordering).
2. The payment saga: the Payment service consumes BookingHeld, charges with an idempotency key, and emits PaymentCompleted or PaymentFailed. The Booking service consumes those, and confirms (BookingConfirmed) or releases the hold (BookingCancelled, a compensation). Hold expiry through a scheduler or delayed topic cancels unpaid holds.
3. Fan-out consumers (their own groups): Notification (email or SMS), Loyalty, Analytics (to the warehouse), Search or availability read models, and partner channel sync (OTA updates for hotels).
4. Reliability: idempotent consumers (an inbox table), retry topics plus a DLT per consumer, schema-registry-versioned events (BookingConfirmed v2), and an event ID plus correlation ID in the headers.
5. Observability: trace context propagation, lag alerts per group, and dashboards of the booking state transitions.

Follow-up questions this topic invites — and their answers

Q: How is ordering guaranteed in Kafka, and when is it lost? A: Only within a partition. The producer key determines the partition, so events for one entity stay ordered. It's lost if you change the partition count, use no key or random keys, retry without idempotence and with max.in.flight > 1 (old versions), or process one partition's records concurrently in the consumer.

Q: What did KRaft change? A: Kafka stores its metadata in an internal Raft-based quorum of controllers, instead of ZooKeeper. It gives simpler operations, faster controller failover, and support for far more partitions. Kafka 4.0 removed ZooKeeper mode entirely.

Q: What is a tombstone? A: A record with a key and a null value. In compacted topics, it marks the key for deletion (it's removed after delete.retention.ms), and sink connectors use it to delete the downstream rows.

Q: How many partitions should a topic have? A: Enough to meet the peak throughput (target MB/s divided by per-partition consumer throughput), with headroom for consumer parallelism. Avoid huge counts without need: they cost broker memory and file handles, and lengthen failover. It's easier to over-provision moderately at creation than to repartition later.