Designing Caches & Rate Limiters — Interview Questions

How to use this lesson

Caching and rate limiting are the two most common "design X" building blocks. For caches, always cover:

• what to cache, and the keys;
• the pattern (cache-aside, read-through, write-through);
• TTL and invalidation;
• stampede protection;
• hot keys;
• what happens when the cache is down.

For rate limiters: the algorithm, where it runs, atomicity, the key (user, API key, tenant, IP), and the response (429, Retry-After).

Learn it in depth → Caching Strategies

Q1. Design a rate cache (for example, hotel rates and availability).

Short answer:

• Requirements: very high read volume (search), low latency (single-digit milliseconds), freshness within seconds after a change, and correctness at booking (re-validated against the source).
• Data layout in Redis:
  • key per rate:{hotelId}:{roomType}:{ratePlan} holding a hash of date → price and availability, or one key per night for fine-grained updates;
  • {hotelId} as a hash tag keeps a hotel's keys on the same Redis Cluster slot, so multi-key operations work.
• Population: write-through from the pricing and inventory pipeline: when prices or availability change, the pipeline writes the new values (a push model). The cache is a precomputed read model, not a lazy cache-aside, which avoids misses on the search path.
• Consistency: events carry a version or timestamp; the writer ignores older versions (a Lua script compares versions). A periodic full refresh heals drift.
• Local near-cache (Caffeine) in the search service for the hottest hotels, with short TTLs (a few seconds), or invalidated by pub/sub.
• At booking, always re-check the source of truth (inventory database and price quote).
• Capacity: estimate the keys (hotels × room types × rate plans × 365 dates), and the memory per entry; use compact encodings (Redis hashes with small fields, or binary formats).

Q2. Design a high-performance Redis cache.

Short answer:

• Topology:
  • Redis Cluster (sharded across 16,384 hash slots) for scale; replicas for reads and failover;
  • or a managed service (ElastiCache, MemoryStore);
  • Sentinel for HA without sharding.
• Data modelling:
  • pick compact structures (hashes for objects, sorted sets for rankings, HyperLogLog for counts);
  • avoid big keys (multi-megabyte values, huge collections) because they block the single-threaded command execution;
  • set a TTL on everything with a lifecycle.
• Client side:
  • connection pooling (Lettuce is non-blocking and thread-safe);
  • pipelining and MGET to cut round trips;
  • Lua scripts for atomic multi-step operations;
  • short timeouts, and a fallback to the database with a circuit breaker if Redis is down.
• Performance problems to prevent:
  • hot keys: local caching, key replication (key#1..N) with random reads;
  • cache stampede: request coalescing (single-flight), mutex refresh, probabilistic early expiration, stale-while-revalidate;
  • penetration (lookups for keys that don't exist): cache the negative results briefly, or use a Bloom filter;
  • avalanche (many keys expiring at once): TTL jitter.
• Memory: maxmemory with an eviction policy (allkeys-lru or allkeys-lfu for caches; volatile-* when mixing persistent data).
• Operations: monitor hit ratio, latency, evictions, memory fragmentation and slow log; don't run KEYS * in production (use SCAN).

Learn it in depth → Design a Distributed Cache

Q3. Design a cache invalidation strategy.

Short answer:

• TTL only: the simplest option; accept staleness up to the TTL. Good for data that changes rarely, or tolerates staleness.
• Delete on write (cache-aside): update the database, then delete the cache key (don't update it), so the next read reloads it. The race (a slow reader re-populating stale data) is mitigated by a short TTL, or a delayed second delete.
• Event-driven invalidation: CDC (Debezium reading the database log) or domain events publish changes, and consumers invalidate or update the keys. This is reliable because it follows committed changes, even from other writers.
• Versioned keys: include a version in the key (product:42:v17), so readers of the new version never see stale data; old keys expire.
• Write-through: the writer updates the database and cache together (a precomputed read model).
• Multi-level caches: invalidate local caches through pub/sub (Redis pub/sub, Kafka), or keep their TTLs very short.
• Choose by the staleness tolerance: prices at search allow seconds; account balances allow none (don't cache them, or validate them against the source).

Common trap: "update the database, then update the cache" in two steps, with concurrent writers, can leave the cache holding the older value permanently (if the writes interleave). Deleting the key, plus a TTL, is safer than setting it.

Q4. Design a cache-warming strategy.

Short answer:

• Why: after a deployment, a cache flush or a failover, a cold cache sends the whole load to the database, which can cause an outage.
• Techniques:
  • preload the hot set at startup or before switching traffic: the top-N keys by access frequency (tracked from analytics or LFU stats), run by a warming job, with rate limiting so the database isn't hammered;
  • precomputed read models (write-through) are warm by design;
  • gradual traffic shift to new instances (canary, slow start in the load balancer);
  • persistent caches (Redis with RDB/AOF, or replicas promoted on failover), so restarts aren't cold;
  • scheduled refresh of expensive keys before they expire (refresh-ahead);
  • in Spring: an ApplicationReadyEvent listener that warms the local caches before the readiness probe reports ready.

Q5. What is rate limiter design? Design a distributed rate limiter.

Short answer:

• The algorithms:
  • Token bucket: tokens refill at rate R up to a capacity B; each request takes a token. Allows bursts up to B, smooth on average. The most common.
  • Leaky bucket: a queue processed at a constant rate; smooths the output.
  • Fixed window counter: simple, but it allows 2× bursts at window boundaries.
  • Sliding window log: exact, but it stores every timestamp (memory-heavy).
  • Sliding window counter: weights the previous and current window counts. A good approximation with low memory.
• Distributed: the counters must be shared across instances. That means Redis, with an atomic Lua script (read, compute, update and set the TTL in one step). Key it by rl:{tenant}:{apiKey}:{window}.
• Where it runs:
  • at the API gateway (Spring Cloud Gateway's RequestRateLimiter uses a Redis token bucket), per client or tenant;
  • in the service, for finer-grained limits;
  • at the edge or WAF for IP-based abuse protection.
• The response: 429 Too Many Requests, with Retry-After, and RateLimit-* headers showing the limit and remaining quota.
• Resilience: if Redis is unavailable, fail open (allow, and log) or fail closed (deny), depending on the risk. A local in-memory limiter is a fallback. For very high scale, use local token buckets with periodic synchronisation of the global counts (approximate, but no Redis call per request).

-- token bucket in Redis (KEYS[1]=bucket, ARGV: capacity, refillPerMs, nowMs, cost)
local b = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local cap, rate, now, cost = tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3]), tonumber(ARGV[4])
local tokens = tonumber(b[1]) or cap
local ts = tonumber(b[2]) or now
tokens = math.min(cap, tokens + (now - ts) * rate)
local allowed = tokens >= cost
if allowed then tokens = tokens - cost end
redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', KEYS[1], math.ceil(cap / rate))
return allowed and 1 or 0

Learn it in depth → Design a Rate Limiter

Q6. Design an API throttling mechanism.

Short answer: Throttling is rate limiting plus policy:

• Tiers and quotas: the limits per plan (free: 100 per minute; enterprise: 10,000 per minute), plus daily or monthly quotas, stored in configuration and cached at the gateway.
• Several dimensions: per API key, per user, per tenant, per IP, per endpoint (expensive endpoints get lower limits), and global limits that protect the backend.
• Concurrency limits (maximum in-flight requests) for slow endpoints, separate from rate limits; adaptive limits (reduce them automatically when latency or error rates rise: load shedding).
• Prioritisation: critical traffic (bookings, payments) is served before non-critical traffic (analytics, bulk exports) under pressure.
• Communication: clear 429 responses, Retry-After, usage dashboards for clients, and warnings before quotas run out.
• Queueing instead of rejecting, for asynchronous workloads (bulk operations accepted with 202 Accepted, and processed at a controlled rate).

Q7. Follow-up: where would you put rate limiting in a microservices architecture?

Short answer: In layers:

1. the edge (CDN or WAF) for IP-level abuse and DDoS protection;
2. the API gateway for per-client and per-tenant quotas;
3. services, for resource protection (bulkheads, concurrency limits);
4. outbound clients, to respect partners' limits.

Each layer protects a different resource.

Follow-up questions this topic invites — and their answers

Q: Why is Lua used for Redis rate limiting? A: A Lua script runs atomically on the Redis server, so the read-compute-write sequence can't interleave with another client's, and it costs one round trip.

Q: What's a cache stampede, and how do you prevent it? A: Many concurrent requests miss the same expired key and all hit the database at once. Prevent it with request coalescing (one loader, others wait), a lock around the refresh, stale-while-revalidate, early probabilistic refresh, and TTL jitter.

Q: Should rate limits be exact? A: Usually not. An approximate limiter (sliding window counter, local buckets with sync) is enough for protection and much cheaper. Billing quotas need exact counting, usually done asynchronously from usage events.

Q: What does Caffeine add in front of Redis? A: A near cache in the JVM: nanosecond-to-microsecond reads for the hottest keys, fewer network round trips, and protection from Redis hot keys. The cost is extra staleness, managed with short TTLs or invalidation messages.