Event-Driven Architecture, Kafka at Scale, IoT & Real-Time Pipelines — Interview Questions

How to use this lesson

Event-driven designs are judged on a few points:

• reliability (no lost or duplicated effects: the outbox, idempotent consumers);
• ordering (partition keys);
• evolution (schemas);
• replay;
• operability (lag, DLQs).

Kafka concepts (partitions, consumer groups, delivery semantics) are covered in the Senior messaging chapter. Here they're applied to system design.

Q1. Design an event-driven architecture. How do you implement domain events?

Short answer:

• Event types:
  • domain events: business facts in the past tense (BookingConfirmed, PaymentCaptured), owned by one service;
  • integration events: the published, stable contract for other services;
  • commands (asking someone to do something): keep them distinct from events.
• The building blocks:
  • a broker (Kafka) with topics per domain (booking.events);
  • keys chosen for ordering (bookingId, hotelId);
  • a schema registry (Avro or Protobuf) with compatibility rules;
  • the outbox for reliable publishing;
  • idempotent consumers;
  • DLQs;
  • tracing propagated in headers.
• Choreography vs orchestration: choreography (services react to events) for simple flows; an orchestrator or saga (Temporal, Camunda, or a state machine) for long, multi-step processes that need visibility and compensation.
• Event design:
  • include an event ID, the type, a version, the occurrence time, the aggregate ID and enough data (event-carried state transfer) to avoid chatty call-backs;
  • don't expose internal models directly.
• Domain events in code: the aggregate records events when its state changes, and they're published after commit (Spring Data's @DomainEvents / AbstractAggregateRoot, then @TransactionalEventListener(phase = AFTER_COMMIT)), or written to the outbox in the same transaction.

@Entity
class Booking extends AbstractAggregateRoot<Booking> {
    Booking confirm() {
        this.status = CONFIRMED;
        registerEvent(new BookingConfirmed(id, hotelId, checkIn, checkOut));
        return this;
    }
}
// Published when bookingRepository.save(booking) is called; handle with
// @TransactionalEventListener(phase = AFTER_COMMIT) or persist to an outbox table.

Learn it in depth → Design a Message Queue System

Q2. What is the outbox pattern?

Short answer: It solves the dual-write problem: updating a database and publishing to Kafka cannot be atomic, so a crash between them either loses an event or publishes an event for a rolled-back change.

• In the same local transaction as the business change, insert the event into an outbox table (ID, aggregate type and ID, event type, payload, created at).
• A relay publishes the outbox rows to Kafka:
  • CDC (Debezium reading the database's transaction log, with the outbox event router): low latency, no polling load;
  • or a polling publisher (SELECT … FOR UPDATE SKIP LOCKED, publish, mark as sent).
• Delivery is at-least-once, so consumers must be idempotent (dedupe by event ID in a processed_events table, or naturally idempotent upserts).
• Clean up the published outbox rows (or partition the table by time).

The inbox pattern is the mirror image: the consumer stores the incoming event ID in the same transaction as its own update, which gives effectively-once processing.

Q3. What is an event replay strategy?

Short answer: Replay means reprocessing past events, to rebuild a read model, fix a consumer bug, backfill a new service, or recover from data corruption.

• Retention: keep the events long enough (a Kafka retention of days or weeks, tiered storage, compacted topics for the latest state), or archive them to object storage (S3, via Kafka Connect) for long-term replay.
• Mechanics:
  • reset the consumer group offsets (kafka-consumer-groups --reset-offsets --to-datetime …);
  • or start a new consumer group that builds a new version of the read model alongside the old one, then switch over (blue/green projections).
• Requirements:
  • consumers are idempotent and deterministic;
  • they don't repeat side effects (emails, payments) during a replay. Use a replay mode flag or separate handlers;
  • schema compatibility across old event versions (upcasters);
  • throttle the replay to protect downstream systems.
• Event sourcing takes this further: the event log is the source of truth, and the state is always rebuilt from it (with snapshots for speed).

Q4. What is log compaction?

Short answer: A Kafka topic setting (cleanup.policy=compact) that keeps at least the latest record for each key, and removes the older records with the same key in the background. It isn't time-based deletion.

• A record with a null value (a tombstone) marks the key as deleted; the tombstone itself is removed after delete.retention.ms.
• Uses: the latest-state topics: configuration, current prices or inventory per key, KTable changelogs (Kafka Streams state stores), __consumer_offsets, and CDC topics used to bootstrap new consumers.
• Guarantees: the order within a partition is kept; compaction happens only in closed log segments (the active segment isn't compacted), so recent duplicates are still visible; consumers still need to handle them.
• It can be combined as compact,delete, to also cap the retention by time.

Q5. How does IoT ingestion work, and how does Kafka improve it? Design an IoT real-time ingestion pipeline.

Short answer: Example: sensors in hotel rooms (temperature, occupancy, energy, door locks), or any device fleet.

• Device edge:
  • devices publish over MQTT (lightweight, handles unreliable networks, QoS levels);
  • to an MQTT broker or IoT hub (AWS IoT Core, HiveMQ, EMQX);
  • with device authentication (X.509 certificates per device);
  • an optional edge gateway buffers and aggregates data when offline.
• Bridge to Kafka: an MQTT-to-Kafka connector writes to Kafka topics (telemetry.raw), keyed by device ID (ordering per device, parallelism across devices).
• Why Kafka helps:
  • it absorbs bursts (a durable buffer, and backpressure);
  • fan-out to many consumers (alerts, storage, analytics) independently;
  • replay;
  • horizontal scale by partitions;
  • decouples the devices from the processing speed.
• Stream processing (Kafka Streams or Flink):
  • validation and enrichment (device to room or hotel metadata);
  • windowed aggregations (averages per minute);
  • anomaly and threshold detection, sent to an alerts topic;
  • dealing with late or out-of-order data (event time, watermarks).
• Storage:
  • hot data in a time-series database (TimescaleDB, InfluxDB) or ClickHouse, for dashboards;
  • raw data archived to object storage (Parquet) for analytics and ML;
  • the device state (the latest reading) in a compacted topic or Redis.
• Operations:
  • device provisioning and OTA firmware updates;
  • monitoring ingestion lag and dropped messages;
  • backpressure and rate limits per device;
  • data retention policies.

Q6. Design a scalable, highly available Kafka architecture.

Short answer:

• The cluster:
  • at least 3 brokers across 3 availability zones (rack awareness with broker.rack, so replicas spread across AZs);
  • KRaft controllers (ZooKeeper is removed in Kafka 4.0);
  • replication factor 3, min.insync.replicas=2, and producers with acks=all plus idempotence (the default in recent clients): the cluster survives one broker or AZ failure without losing acknowledged data;
  • unclean.leader.election.enable=false.
• Partitioning:
  • choose the partition count from the target throughput divided by the per-partition throughput, and the consumer parallelism;
  • leave room for growth (increasing it later changes the key-to-partition mapping);
  • watch for hot partitions (skewed keys).
• Clients:
  • producer batching (linger.ms, batch.size), compression (zstd or lz4);
  • consumers sized to the partitions;
  • cooperative sticky rebalancing; static membership (fewer rebalances).
• Multi-region:
  • MirrorMaker 2 or Cluster Linking (Confluent) for DR or aggregation, which is asynchronous (an RPO above zero);
  • or stretched clusters across nearby regions, with latency costs.
• Operations:
  • monitoring (under-replicated partitions, ISR shrinks, consumer lag, request latency, disk);
  • quotas per client;
  • tiered storage for long retention;
  • schema registry and ACLs, TLS, SASL;
  • Cruise Control for rebalancing partitions;
  • or use a managed service (MSK, Confluent Cloud).

Common trap: acks=all alone doesn't prevent data loss. With min.insync.replicas=1, "all" can mean just the leader. Pair acks=all with min.insync.replicas=2 and a replication factor of 3.

Q7. Design a real-time dashboard.

Short answer: Example: live bookings, revenue and occupancy per hotel.

• Ingest: business events (bookings, cancellations, payments) from Kafka (through the outbox or CDC).
• Process:
  • streaming aggregation (Kafka Streams or Flink) into windowed metrics per hotel, region, minute and hour;
  • or load the events into a real-time OLAP store (ClickHouse, Apache Pinot or Druid) that can aggregate billions of rows in sub-seconds.
• Serve:
  • a query API over the OLAP store or a pre-aggregated cache;
  • push updates to browsers with WebSockets or Server-Sent Events (or poll every few seconds, which is often enough);
  • Grafana or Superset for internal dashboards.
• Design points:
  • define the freshness SLA (seconds versus minutes);
  • use event time, not processing time;
  • handle late data;
  • pre-aggregate the heavy queries;
  • never query the OLTP primary for dashboards;
  • access control per tenant or hotel;
  • consistent metric definitions (a semantic layer).

Learn it in depth → Design a Real-Time Analytics Dashboard

Follow-up questions this topic invites — and their answers

Q: Choreography or orchestration for a booking saga? A: Orchestration usually wins for booking and payment: the steps, timeouts and compensations are explicit in one place and visible to operations. Choreography suits simple fan-out reactions (notifications, analytics).

Q: How do you choose a partition key? A: Pick the entity whose events must stay ordered (booking, account, device), with enough distinct values to spread the load evenly. Avoid low-cardinality or skewed keys.

Q: How do idempotent consumers work in practice? A: Store the processed event IDs (with a unique constraint) in the same transaction as the side effect, or make the operation naturally idempotent (upserts with versions), so a redelivered message has no additional effect.

Q: What are watermarks in stream processing? A: A watermark is the processor's estimate that no events older than time T are still to come. It lets windowed aggregations close and emit results while tolerating a bounded amount of late, out-of-order data.