Spring Kafka — Error Handling, DLQs, Schemas & Operations — Interview Questions

How to use this lesson

Production Kafka code in Spring is mostly about what happens when things go wrong:

• poison messages;
• transient failures;
• redeliveries;
• rebalances;
• schema changes.

Show the exact Spring Kafka components (DefaultErrorHandler, DeadLetterPublishingRecoverer, @RetryableTopic), and the operational practices around them.

Q1. How does Spring Kafka handle error recovery and retries?

Short answer: When a @KafkaListener throws, the listener container's CommonErrorHandler decides what happens. The default is the DefaultErrorHandler:

• Blocking retries: the container seeks back to the failed record, and re-delivers it, according to a BackOff (FixedBackOff/ExponentialBackOffWithMaxRetries). The partition is blocked while it retries, which preserves ordering, but stalls the other records in that partition.
• After the retries are exhausted: the ConsumerRecordRecoverer runs, typically the DeadLetterPublishingRecoverer, which publishes to <topic>-dlt (or .DLT) with exception headers (the stack trace, the original topic, partition and offset). Then the offset is committed, and consumption continues.
• Not-retryable exceptions (deserialisation errors, validation failures, business rule violations) go straight to the DLT: addNotRetryableExceptions(...). DeserializationException is wrapped by the ErrorHandlingDeserializer, so poison pills don't loop forever.
• Batch listeners: throw BatchListenerFailedException(index), so only the failed record is retried or recovered.
• Non-blocking retries: @RetryableTopic (Q2) routes failures to retry topics with delays, so the main partition keeps flowing.

Learn it in depth → Retry & Backoff Strategies

Q2. How do you configure retry policies and backoff strategies in Spring Kafka?

Short answer:

• Blocking retries (they preserve ordering, and suit short transient blips):

@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    var recoverer = new DeadLetterPublishingRecoverer(template,
            (rec, ex) -> new TopicPartition(rec.topic() + ".DLT", rec.partition()));
    var backOff = new ExponentialBackOffWithMaxRetries(4);
    backOff.setInitialInterval(500); backOff.setMultiplier(2.0); backOff.setMaxInterval(10_000);
    var handler = new DefaultErrorHandler(recoverer, backOff);
    handler.addNotRetryableExceptions(ValidationException.class, JsonProcessingException.class);
    return handler;                       // Boot wires a single CommonErrorHandler bean into the container factory
}

• Non-blocking retries (for long delays, without stalling the partition; the order isn't preserved):

@RetryableTopic(attempts = "5",
        backoff = @Backoff(delay = 1000, multiplier = 3.0, maxDelay = 60_000),
        exclude = {ValidationException.class},
        dltStrategy = DltStrategy.FAIL_ON_ERROR,
        topicSuffixingStrategy = TopicSuffixingStrategy.SUFFIX_WITH_INDEX_VALUE)
@KafkaListener(topics = "orders.placed", groupId = "invoicing")
void on(OrderPlaced event) { invoices.create(event); }

@DltHandler
void dlt(OrderPlaced event, @Header(KafkaHeaders.EXCEPTION_MESSAGE) String error) { alerts.deadLetter(event, error); }

Choose blocking retries for quick transient errors where ordering matters, and non-blocking retry topics when downstream outages can last minutes, and per-key ordering can be relaxed. Always make processing idempotent, because retries mean duplicates.

Q3. How do you implement a dead-letter queue (DLT) in Spring Kafka?

Short answer:

• Use DeadLetterPublishingRecoverer in the DefaultErrorHandler (blocking), or @RetryableTopic/@DltHandler (non-blocking). Failed records go to the DLT with diagnostic headers: kafka_dlt-exception-message, -stacktrace, -original-topic, -original-offset, and so on.
• Create the DLT topics with the same partition count (or a custom resolver), and long retention.
• Operate them:
  • alert on DLT traffic;
  • build a DLT consumer or dashboard for triage;
  • offer a replay tool (fix the bug or data, then re-publish the records to the original topic, or seek);
  • redact personal data if the DLTs are widely readable.
• Deserialisation failures: use the ErrorHandlingDeserializer (the raw bytes go to the DLT, instead of an infinite loop).

Q4. What is KafkaTemplate used for?

Short answer: It's Spring's high-level producer API:

• send(topic, key, value) (and send(ProducerRecord)/send(Message<?>)) returns a CompletableFuture<SendResult> (Spring Kafka 3). Handle the completion (log failures, metrics), and avoid blocking get() per message on hot paths;
• transactions: executeInTransaction(...), or with a KafkaTransactionManager/@Transactional (consume-process-produce EOS);
• a default topic, and RoutingKafkaTemplate (different producer configurations per topic);
• ProducerListener, for callbacks;
• observation (Micrometer tracing headers).

Boot auto-configures it from spring.kafka.producer.* (serialisers, acks=all, idempotence, compression). With ReplyingKafkaTemplate, you get request-reply over Kafka (rarely a good idea).

Q5. How do you guarantee delivery between microservices with Kafka?

Short answer: End-to-end reliability needs every hop covered:

1. The producer side: no lost events. Use the transactional outbox (the state change and the event are committed in one database transaction; Debezium or a relay publishes the event). Producer settings: acks=all, idempotence, retries, delivery.timeout.ms. Handle send failures.
2. The broker side: durability. Replication factor 3, min.insync.replicas=2, unclean leader election disabled, and retention longer than the worst consumer downtime.
3. The consumer side: no lost processing. Commit offsets only after successful processing, retries plus a DLT (never drop silently), and idempotent handling (an inbox table) to absorb redeliveries.
4. Verification: reconciliation jobs (compare the source-of-truth counts with the consumers' state), lag and DLT alerts, and end-to-end tracing.

The result is effectively-once outcomes, with at-least-once delivery.

Learn it in depth → Outbox Pattern

Q6. How do you implement sagas using Kafka or RabbitMQ?

Short answer:

• Choreography: each service listens to domain events (OrderCreated → Payment; PaymentCompleted → Inventory; InventoryFailed → Payment refunds, and Order cancels), performs a local transaction plus an outbox event, and publishes the next event. Compensations react to failure events.
  • With Kafka: topics per aggregate or event type, keyed by saga or order ID (for ordering), with consumer groups per service.
  • With RabbitMQ: topic exchanges, with routing keys (order.created), and a queue per consumer service.
• Orchestration: a saga orchestrator persists the saga state (a state machine), and sends commands (reserve-inventory topics or queues), then consumes replies (inventory-replies), correlated by saga ID. Timeouts come from scheduled checks or delayed messages (a RabbitMQ delayed exchange, or Kafka retry topics). Or use frameworks: Axon, Eventuate Tram, Temporal (which handles durability without the broker plumbing).
• Essentials: outbox publishing, idempotent steps and compensations, a correlation ID in the headers, DLQs, and observability of the saga states.

Learn it in depth → Saga Pattern

Q7. What is a Schema Registry, and why do you need one?

Short answer: A central service (Confluent Schema Registry, Apicurio, AWS Glue Schema Registry) that stores versioned schemas (Avro, Protobuf, JSON Schema) per subject (usually per topic value or key), and enforces compatibility rules (BACKWARD, FORWARD, FULL, and the transitive variants) when new versions are registered. How it works:

• producers' serialisers register or look up the schema, and write a schema ID (a small header, or a magic byte plus the ID) with each message, not the whole schema;
• consumers' deserialisers fetch the schema by ID (cached), and deserialise, even as the schemas evolve.

Why you need it:

• a contract between independent producers and consumers;
• safe schema evolution (incompatible changes are rejected at registration, in CI or at produce time);
• compact messages;
• documentation and discoverability;
• data governance (lineage, and data contracts with rules).

Q8. How do you configure several Kafka listeners with different topics?

Short answer:

• Several @KafkaListener methods, each with its own topics (or topicPattern), groupId, concurrency, and even its own containerFactory (different deserialisers, error handlers, batch mode or ack mode):

@KafkaListener(topics = "orders.placed", groupId = "billing", concurrency = "6")
void billing(OrderPlaced e) { ... }

@KafkaListener(topics = {"payments.completed", "payments.failed"}, groupId = "order-status",
               containerFactory = "paymentsListenerFactory")
void paymentStatus(ConsumerRecord<String, PaymentEvent> rec) { ... }

@KafkaListener(topicPattern = "audit\\..*", groupId = "audit-archiver", batch = "true")
void archive(List<AuditEvent> events) { ... }

• Class-level @KafkaListener with @KafkaHandler methods, dispatching by payload type.
• Separate ConcurrentKafkaListenerContainerFactory beans for different configurations.
• autoStartup = "false" plus the KafkaListenerEndpointRegistry, to start, stop, pause or resume listeners at runtime.
• Keep the consumer groups separate for the independent concerns, so their offsets and scaling are independent.

Q9. How do you scale Spring Kafka consumers in a cloud-native deployment?

Short answer:

• The parallelism ceiling is partitions per topic per consumer group. Size the partitions for peak throughput.
• Within a pod: concurrency (listener container threads, each a consumer). Across pods: replicas. The total consumers across replicas × concurrency should be at most the number of partitions.
• Autoscale on consumer lag with KEDA (the Kafka scaler), rather than CPU. Set minimum and maximum replicas to fit the partition limits.
• Rebalance-friendly:
  • the cooperative sticky assignor;
  • static membership (group.instance.id from the pod name in StatefulSets, or stable IDs);
  • graceful shutdown (the container stops and commits within terminationGracePeriodSeconds);
  • avoid long processing that exceeds max.poll.interval.ms.
• Processing efficiency: batch listeners plus bulk database writes, virtual threads or asynchronous I/O inside the processing (with ordered, per-key handling, if needed), and back-pressure through pause and resume.
• Observe: lag per partition, the rebalance count, processing latency, and the DLT rate.

Q10. How do you test Kafka message flows in integration tests?

Short answer:

• Infrastructure: Testcontainers Kafka (@ServiceConnection), for real broker behaviour, or @EmbeddedKafka for speed.
• Flow tests:
  1. publish an input event with KafkaTemplate;
  2. await the outcome with Awaitility: a database row, an outgoing event consumed by a test consumer (KafkaTestUtils.getSingleRecord), or a mocked downstream call;
  3. assert the headers (trace, correlation), the keys (partitioning) and the schemas.
• Failure paths: send a poison message, and assert it lands on the DLT after the configured retries. Send duplicates, and assert that it's processed once. Test ordering per key.
• Contract tests for the event schemas (schema compatibility checks in CI, Spring Cloud Contract messaging, or Pact message pacts).
• Keep the tests deterministic: unique topics or groups per test, short backoffs in the test configuration, and auto.offset.reset=earliest.

Follow-up questions this topic invites — and their answers

Q: What's the danger of retrying forever with blocking retries? A: A single poison message blocks the whole partition indefinitely. Consumer lag grows, and all the later messages (possibly for other keys) are stuck. Always cap the retries, and route to a DLT.

Q: What does AckMode.MANUAL_IMMEDIATE do? A: The listener receives an Acknowledgment, and the offset is committed immediately when you call acknowledge(), which gives precise control (for example, acknowledging after an asynchronous completion). It carries the risk of forgetting to acknowledge, which causes redelivery after a rebalance.

Q: How do you replay DLT messages safely? A: Fix the root cause first. Then re-publish the DLT records to the original topic (a tool or admin endpoint that preserves keys and headers), or have a DLT consumer call the same idempotent handler. Monitor it, and cap the replay rate.

Q: Should event payloads contain full state, or only IDs? A: Both styles exist. Event-carried state transfer (full relevant state) lets consumers work without calling back, which is resilient and decoupled. Thin events (IDs only) force callbacks, which couples services and adds load. Include the data consumers commonly need, versioned.