Data Consistency, Sagas & Kafka Messaging — Interview Questions

How to use this lesson

Consistency is the senior microservices topic. Show that you:

• design for eventual consistency on purpose;
• publish events reliably (the outbox), and consume them idempotently;
• write compensations as business actions, not database rollbacks.

For Kafka, get consumer groups exactly right. It's a very common place to go wrong.

Q1. How do you address data consistency across microservices?

Short answer: Accept that there's no cross-service ACID transaction, and design for it:

1. Each service owns its data, and changes it only with local ACID transactions.
2. Sagas coordinate multi-service business processes. They're a sequence of local transactions, with compensating actions on failure.
3. The transactional outbox (or CDC with Debezium) guarantees that the state change and the event are published together. No "saved but event lost", and no "event sent but rollback".
4. Idempotent consumers (processed-event tables or natural keys), because delivery is at-least-once.
5. Semantic locks and state machines (PENDING → CONFIRMED/CANCELLED), so intermediate states are explicit and visible.
6. Reconciliation jobs and monitoring, to detect and repair drift.
7. Use 2PC/XA only when it's unavoidable, and within a single platform.

Learn it in depth → Eventual Consistency Design

Q2. Database per service or a shared database: which do you prefer, and why?

Short answer: Database per service (at least a schema per service with no cross-access) for real microservices:

• Loose coupling: schema changes are private.
• Independent deployment and scaling.
• Fault isolation.
• The right store for each job: Postgres for orders, Elasticsearch for search, Redis for sessions.
• Clear ownership.

The costs:

• There are no cross-service joins. Use API composition, or read models built from events (CQRS).
• Consistency needs sagas.
• More infrastructure to operate.
• Reporting needs a data pipeline or warehouse.

When a shared database is acceptable: as a transitional step while migrating from a monolith, for small systems owned by one team, or for read-only shared reference data. Even then, give each service its own schema and credentials, so coupling is visible and controlled.

Learn it in depth → Data Ownership Model

Q3. How do you handle distributed transactions in microservices? Explain the Saga pattern.

Short answer: A saga breaks one business transaction into a series of local transactions, one per service. Each step commits locally, and triggers the next through an event or command. If a step fails, the saga runs compensating transactions for the completed steps, in reverse order. The result is eventual consistency, without distributed locks.

It comes in two styles:

• Choreography: each service reacts to the others' events. For example, OrderCreated → Payment charges → PaymentCompleted → Inventory reserves. There's no central coordinator; it's simple for short flows, but hard to follow as it grows.
• Orchestration: a saga orchestrator (a state machine: a Camunda/Temporal workflow, or a Spring State Machine or custom service) sends commands and tracks the state. The flow is explicit, and timeouts and monitoring are easier; the cost is a central component.

Key points to cover:

• Sagas lack isolation, so other transactions can see intermediate states. Countermeasures:
  • semantic locks (a PENDING status);
  • commutative updates;
  • reordering steps so the pivot (the point of no return, like capturing a payment) comes as late as possible;
  • rereading values before acting.

Learn it in depth → Saga Pattern

Q4. If a step in your saga fails, how do you keep data consistent? Give a specific compensation example.

Short answer: A travel booking saga (flight → hotel → car):

1. Flight booked (FLIGHT_HELD).
2. Hotel booked (HOTEL_CONFIRMED).
3. Car rental fails (no availability).
4. The orchestrator runs the compensations in reverse: cancel the hotel booking (per the cancellation policy), then release or cancel the flight hold. The trip is marked FAILED, and the customer is notified. If money was already taken, the compensation is a refund.

An e-commerce order saga:

Key points to cover:

• Compensations are business operations, not undos. A refund isn't "deleting the payment". It's a new, audited transaction. The source says everything is rolled back to its initial state, but some effects (an email already sent) can't be undone, only compensated ("sorry, your order was cancelled").
• Compensations must be idempotent and retryable. They can't be allowed to fail permanently. Retry them with backoff, and escalate to manual intervention queues after a limit.
• Persist the saga state, so a crashed orchestrator resumes where it left off. Set timeouts for steps that never answer.

// Orchestrator step handling (simplified)
void onShipmentFailed(ShipmentFailed e) {
    Saga saga = sagas.load(e.sagaId());
    saga.markCompensating();
    commands.send(new VoidPayment(saga.orderId(), saga.paymentAuthId()));      // each command is idempotent (sagaId as key)
    commands.send(new ReleaseStock(saga.orderId()));
    commands.send(new CancelOrder(saga.orderId(), "SHIPPING_UNAVAILABLE"));
    sagas.save(saga);
}

Q5. How do you handle Kafka messages that several different services must consume, each with its own logic?

Short answer: Publish once to one topic. Give each service its own consumer group (group.id=notification-service, group.id=analytics-service, group.id=fraud-service).

• Every consumer group receives every message: that's the pub/sub fan-out.
• Within a group, the topic's partitions are divided among that service's instances: that's load balancing.
• Each service tracks its own offsets, so a slow or broken consumer doesn't affect the others, and can replay independently.

Common trap: the source puts all the services "as part of a consumer group". If they shared one group, each message would go to only one of them. Different services need different group IDs.

Key points to cover:

• Key by the entity (for example orderId), so each service sees that entity's events in order.
• Evolve the event schema backward-compatibly (a Schema Registry), because many consumers depend on it.
• Each consumer has its own retry and DLQ topics.

Q6. You need Kafka for real-time notifications in a social media application. How would you set it up?

Short answer:

1. Events: domain services publish PostLiked, CommentAdded, UserFollowed and so on to topics keyed by the recipient's user ID, to preserve per-user ordering. Use the transactional outbox for reliability.
2. The notification service (@KafkaListener, its own consumer group, concurrency ≈ its partition count):
   • deduplicates (by event ID);
   • applies user preferences and mutes;
   • aggregates ("Asha and 12 others liked your post") with a short window;
   • stores the notification (the inbox);
   • fans out to delivery channels.
3. Real-time delivery:
   • For online users, push over WebSockets or SSE from gateway nodes. Route to the node holding the user's connection through Redis pub/sub, or a per-node Kafka topic.
   • For offline users, send mobile push (FCM/APNs) or email through separate topics and workers, each with retries and a DLQ.
4. Configuration:
   • Producer: acks=all, idempotence.
   • Consumer: manual or batch commits after processing.
   • Error handling: DefaultErrorHandler + DLT.
   • Enough partitions for peak throughput.
   • JSON or Avro serialisers.
5. Scale and operate:
   • Scale consumers on consumer lag.
   • Rate-limit per user, to avoid spamming.
   • Metrics: end-to-end latency and delivery success.

spring:
  kafka:
    bootstrap-servers: ${KAFKA_BROKERS}
    producer:
      acks: all
      properties:
        enable.idempotence: true
    consumer:
      group-id: notification-service
      auto-offset-reset: earliest
      properties:
        spring.json.trusted.packages: "com.social.events"
    listener:
      ack-mode: record

Learn it in depth → Design a Notification Service

Q7. With User, Order and Payment services, how would you handle communication between them? Synchronous or asynchronous?

Short answer: Choose per interaction:

• Synchronous (REST or gRPC) when the caller needs the answer to continue, and the user is waiting. For example:

  • Order → User: validate the customer, or fetch the address. Better still, keep a local replica built from UserUpdated events, to avoid the runtime dependency.
  • Order → Payment: authorise, if checkout must confirm payment immediately.

  Wrap these calls in timeouts, a circuit breaker and retries (idempotent calls only, or with an idempotency key).

• Asynchronous (Kafka or RabbitMQ) for side effects and long workflows, where decoupling and resilience matter more than immediacy:

  • OrderPlaced → Inventory, Notifications, Analytics;
  • PaymentCompleted → Order confirms, Shipping starts.

  It's also how the saga's steps are chained.

Key points to cover:

• Rule of thumb: minimise the length of synchronous call chains (each hop multiplies the failure probability and adds latency), and use events for everything that can be eventually consistent.
• Asynchronous design needs a correlation ID, idempotency, ordering by key, DLQs, and user-visible pending states.

Q8. When several services must update shared data, how do you ensure consistency? Distributed transactions or eventual consistency?

Short answer:

• First, remove the "shared data". Give it one owner service. Others change it only through that owner's API or commands, and read it through replicated read models.

• Eventual consistency (the default):

  • Sagas.
  • Outbox + idempotent consumers.
  • Optimistic concurrency (versions) on the owner's data.
  • Explicit intermediate states.
  • Reconciliation.

  It scales, and tolerates failures. The price is temporary inconsistency, which the business must accept (and usually already does: "payment processing").

• Distributed transactions (2PC/XA):

  • Strong consistency, but a blocking protocol: the coordinator is a single point of failure, locks are held across the network, throughput is low.
  • Poor support in cloud databases and brokers (Kafka has no XA).
  • Use them only for rare cases inside a single controlled platform (for example two XA databases in one application).

• Where strong consistency is essential (an account balance), keep that invariant inside one service and one database transaction. Design the boundaries so it doesn't span services.

Learn it in depth → Outbox Pattern

Follow-up questions this topic invites — and their answers

Q: How does the transactional outbox work? A: In the same local transaction as the business change, insert the event into an outbox table. A relay then publishes the outbox rows to Kafka, and marks them sent. The relay is either a poller, or CDC with Debezium reading the write-ahead log. Delivery is at-least-once, so consumers deduplicate by event ID.

Q: Does Kafka's "exactly-once" remove the need for idempotent consumers? A: Only within Kafka (read → process → write to Kafka, with transactions and read_committed). Side effects in external systems (databases, emails, APIs) still need idempotency, or the outbox/inbox pattern.

Q: Choreography or orchestration: how do you choose? A: Choreography for short, simple flows with few participants, where maximum decoupling matters. Orchestration for long, complex flows that need timeouts, visibility, error handling and business-level monitoring. Many systems use orchestration for core flows, and events for peripheral reactions.

Q: How do you query data spread across services (for example, "orders with customer names")? A: With API composition (a BFF or gateway calls both services, and joins in memory) for simple cases, or CQRS read models: a query service subscribes to Order and User events, and maintains a denormalised view optimised for that query.