Kafka Production Scenarios — Interview Questions

How to use this lesson

These are the "you're on call" questions. Strong answers name the specific configuration or metric (min.insync.replicas, UnderReplicatedPartitions, max.poll.interval.ms), explain why it helps, and state the trade-off it introduces.

Q1. How do you configure Kafka for high availability and fault tolerance?

Short answer:

Key points to cover:

• With replication factor 3 and min.insync.replicas=2, you can lose one broker and keep both reads and acks=all writes. Losing two stops writes (safety over availability).
• For disaster recovery across regions, use MirrorMaker 2 or cluster linking.

Q2. How do you handle duplicates caused by consumer rebalances or producer retries?

Short answer:

• Producer retries: keep idempotence enabled (the default since 3.0), so broker-side sequence numbers drop the duplicates.
• Consumer rebalances and crashes: with at-least-once delivery, records processed but not yet committed are redelivered. You can't avoid that, so:
  • Commit offsets after processing (and on partition revocation, through a ConsumerRebalanceListener or Spring Kafka's handling).
  • Make handlers idempotent: an event ID stored in a processed-events table in the same transaction as the side effect, or natural upserts.
• Kafka-to-Kafka pipelines: use transactions or Streams exactly-once v2.

Common trap: "set exactly-once and duplicates disappear". That's only within Kafka. External side effects still need idempotency.

Q3. Your messages are bigger than the default ~1 MB limit, and they're causing problems. What do you do?

Short answer: Prefer not to put large payloads into Kafka:

• Claim-check pattern: store the payload in object storage (S3), and send a small message with a reference (the key or URL, plus a checksum).
• Compress (zstd), and trim the payload.
• Split into chunks, with a correlation ID and sequence, only if you must.

If you really must raise the limit, raise it consistently everywhere:

• the topic's max.message.bytes (or the broker's message.max.bytes);
• the producer's max.request.size (and buffer.memory);
• the consumer's max.partition.fetch.bytes / fetch.max.bytes;
• the broker's replica.fetch.max.bytes.

Otherwise, replication or consumption fails.

Key points to cover:

• Large messages hurt batching, memory, replication and latency for everyone on the broker. That's why the default limit is small.

Q4. A consumer group is falling far behind. How do you handle it?

Short answer:

1. Diagnose with the lag per partition (kafka-consumer-groups.sh --describe, Burrow, or Prometheus exporters).
2. If processing is slow: optimise the handlers, batch the downstream writes, and parallelise per key.
3. If the consumer count < the partition count: add consumers.
4. If you're already at one consumer per partition: increase partitions (considering the ordering impact), or use a parallel consumer.
5. Check for rebalance loops. Processing that takes longer than max.poll.interval.ms gets consumers kicked out repeatedly.
6. Temporary relief: scale out, pause non-critical consumers, or, if the business allows, skip ahead (reset offsets to a timestamp).

Learn it in depth → Metrics & Monitoring

Q5. A broker fails unexpectedly. How do you keep the cluster available?

Short answer: It's mostly automatic, if the cluster was configured correctly beforehand:

• The controller elects new leaders from the ISR for the partitions the failed broker led. Clients refresh their metadata, and reconnect to the new leaders.
• With replication factor 3 / min.insync.replicas 2, writes continue with two replicas.
• Operator actions:
  1. Restore or replace the broker. It re-joins and catches up.
  2. Watch under-replicated partitions go back to zero.
  3. Rebalance leadership (preferred leader election, or Cruise Control).
  4. Check that no partition has gone offline, which would mean all its replicas were lost.

Q6. How do you keep data consistent when several consumers read the same topic?

Short answer: It depends on what "several consumers" means:

• The same consumer group: each partition goes to exactly one member, so a record is processed by one consumer. Per-key ordering holds if keys map to partitions. Commit after processing, and make handlers idempotent, for crash safety.
• Different groups (separate services): each group processes every record independently, and their views of the data become eventually consistent. Use the same event ordering (keys), versioned events, and idempotent upserts, so each service converges on the same state.

Common trap: saying consumer groups give "exactly once". They give one consumer per record per group, but redelivery can still happen.

Q7. Which metrics matter for Kafka performance, and how do you troubleshoot a throughput drop?

Short answer:

• Brokers:
  • UnderReplicatedPartitions, OfflinePartitionsCount, ActiveControllerCount;
  • request latency (RequestMetrics produce/fetch totalTimeMs);
  • RequestHandlerAvgIdlePercent and NetworkProcessorAvgIdlePercent (saturation);
  • bytes in and out, disk usage, ISR shrink/expand rate.
• Producers: record send rate, error and retry rates, batch size, request-latency.
• Consumers: lag, records consumed per second, rebalance rate, commit latency.
• Hosts: CPU, disk I/O and latency, network, GC.

Troubleshooting a throughput drop: check what changed (deploys, traffic mix, a new large-message producer), then look for:

• hot partitions (skewed keys);
• an overloaded or slow broker (disk, GC);
• network limits;
• ISR shrinkage, which slows acks=all;
• rebalance storms;
• compression or batching misconfiguration;
• client throttling from quotas.

Q8. What happens when a partition leader fails, and how does leader election work?

Short answer:

1. The controller detects the failure (broker heartbeat or session loss).
2. It chooses a new leader from the partition's ISR, preferring the first live in-sync replica in the assignment order.
3. It updates the metadata. In KRaft, that's a record in the replicated metadata log.
4. Brokers and clients learn the new leader, and producers and consumers resume against it.

Key points to cover:

• The unavailability window is typically short: seconds at most, and faster with KRaft.
• If no in-sync replica is alive, the partition goes offline, unless unclean election is enabled. Unclean election trades data loss for availability.

Q9. Why do consumer-group rebalances sometimes take long, and how do you shorten them?

Short answer: Classic ("eager") rebalances are stop-the-world. Every member revokes all its partitions, commits, waits for the slowest member to rejoin, then gets a new assignment. Slow members (long processing, beyond max.poll.interval.ms), large groups, and state restoration (in Streams) all make it worse.

To shorten them:

• CooperativeStickyAssignor (incremental rebalancing): only the moved partitions pause.
• Static membership (group.instance.id): restarts during deployments don't trigger rebalances at all, within session.timeout.ms.
• Keep poll() loops fast, or reduce max.poll.records.
• Kafka 4.0's new consumer protocol (KIP-848), with server-side, incremental assignment.
• In Streams, use standby replicas, to avoid long state restoration.

Q10. Can you lose data in Kafka even with replication? How?

Short answer: Yes:

• acks=0 or acks=1: the leader acknowledges, then dies before followers replicate.
• min.insync.replicas=1: acks=all degenerates to "leader only" when followers fall behind.
• Unclean leader election: an out-of-sync replica becomes leader, and truncates the acknowledged data.
• Correlated failures: all replicas in one AZ or rack, or losing a majority of brokers at once before data is flushed.
• The producer ignores send failures: asynchronous send() without checking the callback.
• Consumer-side loss: auto-commit, or committing before processing, and then crashing.
• Retention: data older than retention.ms is deleted before a slow consumer reads it.
• Compaction removes older values for a key, by design.

Key points to cover:

• The fix list is the reliability checklist: acks=all + min.insync.replicas=2 + replication factor 3 across AZs, no unclean elections, checked callbacks, commit after processing, and retention sized for the worst consumer outage.

Q11. When would you use a compacted topic instead of a regular one, and what are the trade-offs?

Short answer: Use log compaction (cleanup.policy=compact) when consumers need the latest state per key, not the full history:

• entity snapshots (customer profile, product price);
• Kafka Streams changelogs and KTables;
• CDC tables;
• configuration topics (Kafka Connect's own internal topics).

A new consumer can bootstrap the full current state by reading the topic from the start.

Trade-offs:

• History is lost: only the latest value per key is guaranteed.
• Deletes require tombstones (null values, retained for delete.retention.ms).
• Compaction isn't instant. Recent segments may still contain several values per key, so consumers must handle that.
• Records need keys.
• compact,delete combines compaction with time-based retention.

Q12. Kafka guarantees ordering, but when can that guarantee break?

Short answer: Ordering is guaranteed only within a single partition, for a single producer's successful writes. It can break when:

• Related messages go to different partitions: no key, the wrong key, or different producers choosing different keys.
• The partition count changes: hash(key) % partitions shifts, so new records for a key land in a different partition from the old ones.
• Producer retries without idempotence, with max.in.flight.requests.per.connection > 1: a failed batch retried after a later batch can reorder records. Idempotence (the default) prevents this.
• Consumer-side parallelism: processing records from one partition concurrently (thread pools, @Async) without per-key serialisation.
• Retries or DLQ-and-reprocess flows that re-inject old events later.
• Several producers writing the same key, with no coordination.

Follow-up questions this topic invites — and their answers

Q: What does "under-replicated partitions" mean, and why alert on it? A: Some followers are behind the leader, so you have fewer copies than configured. Another failure could then cause unavailability or data loss. Common causes are slow disks or networks, overloaded brokers, and a broker that's down.

Q: How do you choose retention.ms? A: Longer than your worst plausible consumer outage, plus the reprocessing window, balanced against disk cost. Tiered storage (KIP-405) makes long retention affordable by offloading old segments to object storage.

Q: What is MirrorMaker 2 used for? A: Replicating topics (and consumer-group offsets) between clusters, for disaster recovery, migrations, or active-active and hub-and-spoke topologies across regions.

Q: How do you reprocess events from yesterday? A: Reset the group's offsets to a timestamp (kafka-consumer-groups.sh --reset-offsets --to-datetime ..., with the group stopped), or run a separate consumer group from that point. Make sure the processing is idempotent first.