Redis, Search, Time-Series, CDC & Transactional Data Modelling — Interview Questions

How to use this lesson

These questions come from real product domains (bookings, pricing, dashboards), and test whether you can pick the right store for each access pattern, and keep the stores in sync reliably. Always state the source of truth, and how derived stores are updated.

Q1. What are Redis's data structures, and what are they used for?

Short answer:

• Strings: values, counters (INCR), and cached JSON, with TTLs.
• Hashes: objects with fields (HSET user:42 name … plan …).
• Lists: queues and recent-items lists (LPUSH/BRPOP).
• Sets: unique membership, tags, "who liked this".
• Sorted sets (ZSET): members ordered by score. Leaderboards, priority queues, sliding-window rate limiting, time-ordered feeds.
• Streams: an append-only log with consumer groups. Lightweight event streaming and job queues.
• Bitmaps and bitfields: compact flags, and daily active users.
• HyperLogLog: approximate unique counts, in 12 KB.
• Geospatial: GEOADD/GEOSEARCH, nearby drivers or stores.
• JSON, search and vector (Redis Stack, and Redis 8): document storage, secondary indexes, and vector similarity.
• Pub/Sub: fire-and-forget messaging.

Learn it in depth → Redis Data Structures & Use Cases

Q2. How does Redis improve performance?

Short answer:

• In-memory storage, with efficient encodings, gives sub-millisecond reads and writes.
• A single-threaded command execution model (with I/O threads in Redis 6+) means no locking overhead, and atomic operations and Lua scripts.
• Rich server-side operations (increment, sorted-set ranking, set intersection) avoid round trips and application-side logic.
• Pipelining batches commands.

Uses:

• cache-aside caching of expensive database or API results;
• sessions;
• rate limiting;
• distributed locks (with care);
• leaderboards and counters;
• deduplication (idempotency keys with SET NX EX).

It reduces database load and latency dramatically. The trade-offs: memory cost, persistence (RDB snapshots or AOF) that's weaker than a database, and eviction policies that can drop data. Treat it as a cache or ephemeral store, unless it's configured and operated as a primary store.

Q3. What is Redis Cluster?

Short answer: Redis's built-in sharding plus high availability:

• the keyspace is divided into 16,384 hash slots (CRC16(key) mod 16384), distributed across master nodes, each with replicas for failover;
• clients are cluster-aware: they follow MOVED/ASK redirects, and cache the slot map;
• multi-key operations only work when the keys are in the same slot. Use hash tags ({user:42}:cart, {user:42}:profile);
• resharding moves slots online.

Alternatives: Redis Sentinel (HA for a single master, no sharding), managed services (ElastiCache, MemoryDB, Azure Cache), and proxies (Envoy, Twemproxy). The caveats: asynchronous replication (acknowledged writes can be lost on failover), and no cross-slot transactions.

Q4. What is a TTL, in general and in Redis?

Short answer: Time-to-live: an expiry after which data is automatically removed. In Redis:

• SET key value EX 300, EXPIRE, PEXPIRE, and per-field TTLs on hashes (Redis 7.4+);
• expiry works lazily (checked on access) plus actively (sampling in the background).

Uses: cache freshness, sessions, one-time tokens (OTPs), rate-limit windows, locks, and idempotency keys.

Best practices:

• add jitter to TTLs, to avoid mass expiry and stampedes;
• set TTLs on everything in a cache (unbounded keys leak memory);
• combine them with eviction policies (allkeys-lfu/allkeys-lru) and maxmemory.

Other stores have TTLs too: DynamoDB TTL, Cassandra USING TTL, MongoDB TTL indexes.

Q5. How do you avoid cache inconsistency?

Short answer: Inconsistency comes from races between database writes and cache updates, and from failed invalidations. The strategies:

• Cache-aside with delete-on-write: update the database, then delete the cache key (rather than writing the new value, which races badly). The next read repopulates it.
• Invalidate after commit: @TransactionalEventListener(AFTER_COMMIT), or better, CDC-driven invalidation (Debezium emits changes, and a consumer evicts the keys). That survives application crashes between the commit and the eviction.
• Delayed double delete: delete, then write the database, then delete again after a short delay, to handle a concurrent reader re-caching stale data.
• Versioning: store a version or timestamp with the cached value, and reject older writes (SET … if version newer, through Lua).
• TTL as a backstop, bounding the staleness.
• Single-writer principles, and avoiding caching of highly volatile or critical data (balances, stock).
• Local plus distributed caches: broadcast invalidations (Redis pub/sub or Kafka) to every instance's local cache.

Q6. What is a Bloom filter?

Short answer: A space-efficient probabilistic set that answers "possibly in the set" or "definitely not in the set". It's a bit array plus k hash functions:

• to add an element, set k bits;
• to query, check the k bits: all set means maybe present, and any unset means definitely absent.

It gives false positives, but never false negatives, and it can't delete elements (counting Bloom filters or cuckoo filters can). Uses:

• avoiding expensive lookups for keys that don't exist: LSM stores (RocksDB, Cassandra) skip SSTables, and caches block cache penetration from requests for non-existent IDs;
• "has this user already seen this item" feeds;
• a weak-password check against breached lists;
• deduplication at scale.

Redis provides BF.ADD/BF.EXISTS (RedisBloom, built into Redis 8), and Guava has BloomFilter.

Learn it in depth → Design a Bloom Filter

Q7. What is time-series data, and how do you handle and bucket it?

Short answer: Time-series data is measurements or events indexed by time: metrics, IoT sensor readings, prices, clickstreams, logs. Its characteristics:

• append-heavy, with high ingest rates;
• queries over time ranges, with aggregations (avg, max, percentiles per interval);
• recent data is hot;
• old data is downsampled or expired.

Bucketing strategies:

• Partition or bucket by time: a partition per day or hour, or in Cassandra, a key like (sensor_id, day), so partitions stay bounded, and queries hit few of them.
• Combine entity and time to avoid hot partitions: all writes for "now" would otherwise land in the same bucket.
• Downsample and roll up: keep raw data for 7 days, 1-minute aggregates for 90 days, and 1-hour aggregates for years (continuous aggregates in TimescaleDB, or recording rules in Prometheus).
• Retention: drop old partitions, or use TTLs.

Stores: TimescaleDB (Postgres), InfluxDB, Prometheus or Mimir (metrics), ClickHouse, Druid, Cassandra/ScyllaDB, and QuestDB.

Q8. Hot vs cold storage? How do you design an analytics database?

Short answer:

• Hot storage: fast, expensive media (SSD, memory, the primary OLTP database, Redis, Elasticsearch hot nodes) for recent and frequently accessed data, with low latency.
• Warm or cold storage: cheaper, slower tiers (object storage like S3 Standard-IA or Glacier, data lakes, archive tables) for older, rarely accessed data, with higher latency (and retrieval fees).
• Tiering policies: move data by age or access pattern (Elasticsearch ILM hot, warm and cold phases, and S3 lifecycle rules).
• Analytics database design:
  • separate it from OLTP;
  • ingest through CDC or ELT into a columnar warehouse or lakehouse (Snowflake, BigQuery, ClickHouse, Iceberg or Delta on S3);
  • model a star schema (fact tables such as bookings or payments, with dimensions like date, customer, product and region), and partition and cluster by date;
  • pre-aggregate hot dashboards;
  • add data-quality checks and lineage, handle late-arriving data, and mask personal data.

Q9. How does Elasticsearch indexing work? How does it scale? What is the refresh interval?

Short answer:

• Indexing:
  1. Documents (JSON) are analysed (tokenised, normalised) per the field mappings.
  2. They're written into Lucene segments (immutable inverted indexes, plus doc values for aggregations), through an in-memory buffer and a translog (for durability).
  3. Refresh makes new segments searchable.
  4. Flush commits the segments to disk.
  5. Merges combine small segments.
• Scaling:
  • an index is split into primary shards (the count is fixed at creation; use rollover or reindex to change it), each with replicas;
  • shards are distributed across data nodes;
  • queries fan out to the shards, and results are merged.
  • Scale with more nodes, replicas (for read throughput), index-per-time-period with ILM (logs), and routing keys.
  • Avoid too many small shards (heap overhead).
• Refresh interval: how often new data becomes visible to search. The default is 1 second, which makes search near-real-time. Increase it (for example to 30 seconds, or -1 during bulk loads) to improve indexing throughput. Lower values cost CPU and I/O through more segments.

Q10. What is Logstash?

Short answer: Logstash is the "L" in ELK: a data-processing pipeline that ingests from many inputs (Beats, Kafka, files, JDBC, HTTP), transforms the data with filters (grok parsing, mutate, date, geoip, enrichment), and outputs to Elasticsearch, Kafka, S3 and more. It's powerful but heavyweight (JVM-based). Many modern setups use Filebeat or Elastic Agent, Fluent Bit, Vector, or the OpenTelemetry Collector for collection, and Elasticsearch ingest pipelines for light transformations.

Q11. How do you sync MySQL or Postgres with Elasticsearch, or with Kafka? What are CDC and Debezium?

Short answer:

• Change Data Capture (CDC) reads the database's transaction log (MySQL binlog, Postgres WAL through logical decoding) and emits every committed insert, update and delete as an event:
  • no application changes;
  • captures all writes (including batch jobs and manual fixes);
  • ordered per row or key;
  • low overhead;
  • no dual-write inconsistency.
• Debezium is the leading open-source CDC platform. It runs as Kafka Connect source connectors (or Debezium Server or embedded), producing change events (with before and after images, plus metadata) to Kafka topics per table. It supports initial snapshots, schema history, and the outbox event router (which publishes curated domain events from an outbox table).
• The pipeline to Elasticsearch: database → Debezium → Kafka → an Elasticsearch sink connector, or a custom consumer that builds the denormalised search documents (joining related data), with idempotent upserts keyed by ID, and deletes on tombstones. Rebuild by replaying from snapshots.
• The alternatives:
  • application-level dual writes (unsafe without the outbox);
  • batch polling by an updated_at column (misses deletes, and adds lag);
  • Logstash's JDBC input (polling).

Q12. What is Avro vs Protobuf?

Short answer: Both are compact binary serialisation formats with schemas, used for events and RPC:

• Avro:
  • the schema is JSON, and the writer's schema is needed to read the data (usually through a Schema Registry ID in each Kafka message);
  • strong support for schema evolution with defaults;
  • dynamic, with no code generation required;
  • popular in the Kafka and Hadoop ecosystems (Confluent).
• Protobuf:
  • .proto IDL with numbered fields, and generated code in many languages;
  • very compact and fast;
  • evolution rules: never reuse field numbers, and add optional fields;
  • the standard for gRPC, also used with Kafka (through the registry).
• JSON Schema is a third option: human-readable, but larger and slower.

Choose by ecosystem and team: Avro for Kafka-centric data platforms, Protobuf for polyglot services and gRPC. Enforce compatibility modes (BACKWARD, FORWARD, FULL) in the Schema Registry.

Q13. How do you design an audit trail?

Short answer:

• Record who did what, when, to which entity, from where, and why:
  • the actor (user or service, and the impersonation chain);
  • the action;
  • entity type and ID;
  • before and after values (or a diff);
  • timestamp (UTC);
  • request or trace ID;
  • source IP or client;
  • reason or ticket.
• Capture mechanisms:
  • in-application audit events (domain events through the outbox);
  • Hibernate Envers (automatic entity versioning into _AUD tables);
  • CDC from the database log (it captures everything, even out-of-band changes);
  • database triggers (simple, but tightly coupled).
• Storage:
  • append-only;
  • tamper-evident (hash chaining, WORM storage such as S3 Object Lock, or a ledger database);
  • kept separately from operational data, with its own retention and access controls;
  • partitioned by time.
• Privacy: minimise the personal data stored, mask sensitive fields, and comply with retention and deletion rules.
• Querying: index by entity and time and by actor, and export to the SIEM for security monitoring.

Q14. How would you design a booking table, a rate table and a booking ledger? How do you ensure atomic booking?

Short answer: For example, a hotel:

• Inventory (availability) table: room_inventory(hotel_id, room_type_id, stay_date, total, booked, version), with the primary key (hotel_id, room_type_id, stay_date), and a check constraint booked <= total. There's one row per night, so a multi-night booking touches several rows.

• Rate table: rate(hotel_id, room_type_id, rate_plan_id, stay_date, currency, amount, valid_from, valid_to), versioned (rates change over time; keep history for audit and for price-at-booking). Rate plans carry the restrictions (min stay, non-refundable). Cache the hot rate lookups.

• Booking tables: booking(id, customer_id, status, total, currency, created_at, idempotency_key UNIQUE, version), plus booking_night(booking_id, stay_date, room_type_id, rate_amount), which snapshots the price at booking time.

• Booking ledger: an append-only, double-entry-style ledger of money movements and state changes: ledger_entry(id, booking_id, type (HOLD, CHARGE, REFUND, FEE), amount, currency, created_at, reference). The balances are derived from it, never updated in place, which gives auditable, reconcilable finance.

• Atomic booking: in one database transaction:

  1. conditionally increment booked for every night, in date order (a consistent lock order):

     Copy

     UPDATE room_inventory SET booked = booked + 1
     WHERE hotel_id = :h AND room_type_id = :r AND stay_date BETWEEN :in AND :out - 1 AND booked < total;
     

     If the rows updated ≠ the number of nights, roll back (sold out);

  2. insert the booking, the nights and the ledger HOLD entry;

  3. write the outbox event.

  Add idempotency keys to reject duplicate submissions. Use holds with a TTL during payment (released by a scheduler). Distributed or high-contention cases can add Redis pre-checks, or queue per hotel.

Learn it in depth → Design a Meeting Room / Calendar Booking System

Follow-up questions this topic invites — and their answers

Q: Are Redis distributed locks safe? A: A single-instance SET key token NX PX ttl, with a token-checked release (Lua), is fine for efficiency locks. For correctness-critical mutual exclusion, lease expiry during GC pauses or network delays can let two holders act at once. Use fencing tokens checked by the resource, or a consensus system (etcd or ZooKeeper), or database locks.

Q: What is cache penetration vs cache breakdown vs avalanche? A: Penetration: requests for keys that don't exist bypass the cache (fix with Bloom filters, or caching nulls briefly). Breakdown (stampede): a hot key expires, and many requests hit the database at once (fix with locks or single-flight, refresh-ahead). Avalanche: many keys expire together, or the cache goes down (fix with jittered TTLs, HA caches, and circuit breakers).

Q: How do you reindex Elasticsearch with zero downtime? A: Create a new index with the new mappings, backfill it (a snapshot plus replaying CDC from a recorded position), dual-write or keep consuming changes, then atomically switch an alias from the old index to the new one, and delete the old index later.

Q: Why store money as integers or DECIMAL, never floating point? A: Floating point can't represent most decimal fractions exactly, so sums drift. Use DECIMAL(19,4) in SQL and BigDecimal in Java, or integer minor units (paise or cents), together with the currency code.