Producers and consumers, how producers send data (batching, keys, idempotence), consumer read strategies and offset commits, consumer groups and scaling, fault tolerance and replication, preventing data loss end to end, acks settings, Kafka Streams, how it differs from other engines, state stores and its challenges.
Published September 25, 2026
"Kafka is reliable" isn't an answer. Interviewers want to know which settings make it reliable (acks, idempotence, min.insync.replicas, commit strategy), and where data can still be lost or duplicated. Walk through the full path: producer → broker → consumer.
Short answer: Producers publish records (key, value, headers, timestamp) to topics. Consumers pull records from partitions, and track their position with offsets. Consumers usually belong to a consumer group, so the partitions are shared out among them. Producers and consumers are fully decoupled: neither knows about the other, and they can run at different speeds.
Learn it in depth → Message Queue System
Short answer:
batch.size, linger.ms), and optionally compresses them (lz4, zstd).acks.It retries automatically on transient failures.
Properties p = new Properties();
p.put(ProducerConfig.ACKS_CONFIG, "all");
p.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true); // no duplicates from retries (default true since 3.0)
p.put(ProducerConfig.LINGER_MS_CONFIG, 10);
p.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "zstd");
producer.send(new ProducerRecord<>("orders.placed", order.id(), event), (meta, ex) -> {
if (ex != null) log.error("publish failed for {}", order.id(), ex); // handle async failures!
});
Short answer:
auto.offset.reset = earliest or latest.AckMode.BATCH does this.max.poll.records to control the batch size.Key points to cover:
Short answer: Within a group, each partition is consumed by exactly one member. Adding consumers spreads the partitions across more instances (horizontal scaling), up to one consumer per partition. Extra consumers sit idle. Different groups each get every record independently, which gives pub/sub fan-out. When members join or leave, the group rebalances the partitions.
Key points to cover:
group.instance.id), so rolling deploys don't trigger full stop-the-world rebalances.Short answer:
Short answer: Replication keeps redundant copies of each partition, so a broker failure causes no data loss and only a brief interruption for leader election. Writes go to the leader. Followers replicate them, and an acknowledged write (acks=all) has reached all in-sync replicas. Combined with min.insync.replicas, replication defines your durability guarantee.
Common trap: "replication helps balance reads because consumers read from different copies". By default, consumers read from the leader. Follower fetching exists, but it's an opt-in, rack-aware optimisation.
Short answer: Only when the whole chain is configured for it:
| Stage | Setting |
|---|---|
| Producer | acks=all, enable.idempotence=true, sensible retries / delivery.timeout.ms, handle send callbacks |
| Topic | replication.factor=3, min.insync.replicas=2, unclean.leader.election.enable=false |
| Brokers | Replicas spread across racks/AZs; monitor under-replicated partitions |
| Consumer | Commit offsets after successful processing; no auto-commit for critical data |
| Application | The outbox pattern, so events are published if and only if the database commit succeeds |
Key points to cover:
acks setting?Short answer: It defines when a write counts as successful:
acks=0: fire-and-forget. Fastest, but data can be lost silently.acks=1: the leader has written it. Data is lost if the leader fails before the followers replicate it.acks=all (-1): all in-sync replicas have it. That's the strongest durability, and it's the default since Kafka 3.0 (together with idempotence).Key points to cover:
acks=all is only meaningful with min.insync.replicas ≥ 2. Otherwise "all ISR" can shrink to just the leader.acks=all producers get a NotEnoughReplicas error. That's safety over availability, by design.Short answer: Kafka Streams is a Java library for building stream-processing applications that read from Kafka topics, transform, join, aggregate and window the data, and write the results back to Kafka. It runs inside your own application, with no separate cluster. Use cases:
StreamsBuilder b = new StreamsBuilder();
b.stream("payments", Consumed.with(Serdes.String(), paymentSerde))
.filter((k, p) -> p.amount().compareTo(new BigDecimal("100000")) > 0)
.groupBy((k, p) -> p.cardId(), Grouped.with(Serdes.String(), paymentSerde))
.windowedBy(TimeWindows.ofSizeWithNoGrace(Duration.ofMinutes(5)))
.count()
.toStream()
.filter((windowedCard, count) -> count >= 3)
.to("fraud-alerts");
Short answer:
processing.guarantee=exactly_once_v2).The trade-off: it's Kafka-in, Kafka-out. For multiple sources and sinks, very large state, or complex event-time handling at scale, Apache Flink is often the better fit.
Short answer: Stateful operations (aggregations, joins, windows) keep their state in local state stores: RocksDB by default, or in memory. Each store is backed by a compacted changelog topic in Kafka. On a restart or rebalance, a task restores its store by replaying the changelog. Standby replicas (num.standby.replicas) keep warm copies on other instances, for faster failover. State is partitioned in the same way as the input topics.
Short answer:
groupBy/selectKey add latency and storage.TopologyTestDriver).Q: What does producer idempotence guarantee? A: Retries can't create duplicates in the log, and order is preserved per partition. The broker deduplicates using a producer ID plus per-partition sequence numbers. It covers a single producer session. For atomic writes across partitions, use transactions.
Q: What's consumer lag, and why does it matter? A: Lag is the difference between the latest offset in a partition and the consumer group's committed offset. Growing lag means consumers can't keep up, and data is getting stale. It's the key metric for autoscaling and alerting.
Q: When should you commit offsets? A: After the record's side effects are durable: after the database write succeeds. Committing before processing risks losing records on a crash. Committing after means occasional duplicates, so make the handlers idempotent.
Q: KStream vs KTable?
A: A KStream is an unbounded stream of events, where every record is a fact. A KTable is a changelog view, keeping the latest value per key, like a continuously updated table. Joining a stream to a table enriches each event with the current state.