These questions test operational design: protecting systems from overload and dependency failures, and keeping latency predictable. Give the algorithm, the Java or Spring implementation option, and how you'd observe it in production.
Q1. How do you design rate-limited APIs?
Short answer:
Decide what to limit, and for whom: per API key or client, per user, per tenant, per IP (for anonymous traffic), per endpoint (expensive search vs cheap reads), with tiers by plan. Also consider concurrency limits (in-flight requests), and cost-based limits (GraphQL query cost, LLM tokens).
Choose the algorithm: token bucket (bursty but bounded), sliding window (smooth fairness), fixed window (simplest), or leaky bucket (constant output).
Enforce it in layers:
at the edge (API gateway, CDN or WAF: Spring Cloud Gateway RequestRateLimiter with Redis, AWS API Gateway usage plans, Envoy's global rate limit service);
in the service for fine-grained rules (Bucket4j, backed by Redis or Hazelcast, or Resilience4j RateLimiter for protecting outbound calls).
Distributed state: limits must hold across instances, so use a shared store (Redis) with atomic operations (Lua scripts), or approximate local limits (the global limit divided by the instance count) when some slack is acceptable.
Respond clearly:429 Too Many Requests, with Retry-After, and RateLimit-Limit/Remaining/Reset headers. Document the limits.
Operate it: metrics per key, alerts on abuse, allow-lists for internal traffic, and fail-open vs fail-closed decisions if Redis is unavailable.
Short answer: A bucket holds up to capacity B tokens, refilled at rate r tokens per second. Each request consumes one token (or a cost-weighted number). If there are no tokens, the request is rejected (or waits). The properties:
it allows bursts up to B, while enforcing a long-term average of r;
it's simple and memory-cheap: store tokens plus lastRefillTimestamp per key, and refill lazily on each request (tokens = min(B, tokens + elapsed × r)).
It's used by AWS API throttling, Spring Cloud Gateway's RedisRateLimiter, Bucket4j and Guava's RateLimiter (a smooth variant). The leaky bucket is its cousin: a queue drained at a constant rate, which smooths the output.
Short answer: It limits the number of requests in the last W seconds, avoiding the fixed-window boundary burst (with fixed one-minute windows, a client can send 2× the limit around a window edge):
Sliding log: store a timestamp for every request (for example, a Redis sorted set per key); remove the entries older than now − W; count them; allow if the count < limit. It's exact, but the memory is O(requests).
Sliding window counter (an approximation): keep the counts for the current and previous fixed windows, and estimate: count = current + previous × (overlap fraction). Memory is O(1) per key, it's accurate enough in practice, and it's used by Cloudflare-style limiters.
Implement it atomically in Redis (a Lua script: ZREMRANGEBYSCORE + ZCARD + ZADD + EXPIRE), to avoid races across instances.
Q4. How should you handle timeouts?
Short answer:
Set a timeout on every remote call. The defaults are often infinite, or very long. Configure:
connect timeouts and read/response timeouts separately (HTTP clients such as RestClient with JDK HttpClient or Apache HttpClient; WebClient with Netty responseTimeout);
connection-pool acquisition timeouts;
database statement and lock timeouts;
Kafkadelivery.timeout.ms/request.timeout.ms.
Size them from the dependency's latency distribution: slightly above its p99 (not the average), and within the caller's own budget.
Deadline propagation: pass the remaining time budget downstream (gRPC deadlines, or a custom header), so downstream calls don't keep working after the caller has given up. The downstream timeout must be shorter than the upstream one.
Combine them with retries (only the idempotent ones, within the overall deadline) and circuit breakers (Resilience4j TimeLimiter).
Observe: timeout counts and latency histograms per dependency. Treat timeouts as signals of saturation.
Q5. How do you design fallbacks? What is a circuit-breaker fallback strategy?
Short answer: A fallback is the degraded but acceptable response when a dependency fails, or its circuit is open. Design it per use case:
cached or stale data (last known prices, with a "may be outdated" flag);
defaults (generic recommendations when the personalisation service is down);
graceful feature removal (hide the reviews widget);
queue for later (accept an order as PENDING, and process the payment asynchronously);
an alternative provider (a secondary payment gateway or SMS vendor);
fail fast with a clear error, when there's no safe fallback (you must not fake a payment success).
The principles:
fallbacks must be cheap and reliable themselves (no calls to other fragile services);
they must be tested (chaos tests);
they must be observable (a fallback-rate metric, with alerts);
never silently hide correctness-critical failures.
Q6. How do you troubleshoot a cascading failure across services?
Short answer:
Stabilise first:
shed load (rate limits, disable non-critical features through flags);
open the circuits and fail fast to the unhealthy dependency;
scale out the healthy capacity;
stop the retry storms (reduce the retries, add backoff);
roll back any recent deployment.
Find the origin:
distributed traces and service dependency maps: the first service whose latency or errors rose, often a database, a third-party API or a saturated shared resource (connection pool, thread pool, Kafka partition);
look at the saturation metrics (pool active or pending, queue depths, CPU throttling, GC);
correlate with deployments or configuration changes, and traffic spikes.
Understand the propagation:
missing timeouts (threads pile up);
retries multiplied across layers;
shared thread pools (no bulkheads);
synchronous dependency chains;
health checks that include dependencies (restart loops).
Fix and prevent: timeouts everywhere, bulkheads, circuit breakers, retry budgets, asynchronous decoupling, load-shedding policies, and chaos tests that reproduce the scenario. Hold a postmortem.
Q7. How do you monitor and investigate latency spikes?
Short answer:
Measure the right thing:latency histograms and percentiles (p50, p95, p99, p99.9) per endpoint and dependency. Averages hide the spikes. Enable Micrometer percentile histograms (and OpenTelemetry exemplars).
Alert on SLO burn, not on single spikes.
Correlate the spikes with:
GC pauses (JFR, GC logs);
CPU throttling in containers;
safepoints and deoptimisations;
pool waits (Hikari pending, executor queue);
lock contention;
slow downstream spans (traces);
cold starts or new pods (JIT warm-up after scale-out or deploy);
noisy neighbours;
network retransmits and DNS latency;
database autovacuum or checkpoints;
cache expiry storms.
Tools: tracing with tail-based sampling that keeps the slow traces, continuous profiling (async-profiler or Pyroscope in wall-clock mode), JFR on demand, and dashboards overlaying deployments and GC.
Fix the identified cause (tune the GC or the heap, fix N+1 queries, add caching, increase the pool, jitter the TTLs, pre-warm), and verify with percentiles.
Q8. What cache eviction policies exist?
Short answer:
LRU (least recently used): evicts the entries not accessed longest. Good general-purpose recency.
LFU (least frequently used): keeps the popular items. It resists scans, but adapts slowly to shifts (it needs ageing).
W-TinyLFU (Caffeine): a frequency sketch plus a small LRU window. It's near-optimal hit rates for most workloads, and is Caffeine's default.
FIFO, random, and TTL or expiry-based (time-to-live, time-to-idle).
Size-based vs weight-based (by bytes).
Redis maxmemory-policy:allkeys-lru, allkeys-lfu, volatile-lru/lfu/ttl (only keys with a TTL), noeviction (writes fail when memory is full, which is suitable when Redis is a primary store).
Note that expiry (data becoming stale) and eviction (making room) are different concerns, and you usually need both.
Q9. How do you design high-performance caching?
Short answer:
Layers:
a CDN or HTTP cache (Cache-Control, ETag) for public content;
an in-process L1 (Caffeine: nanosecond access, size-bounded, a short TTL);
a distributed L2 (Redis or a Hazelcast cluster);
the database.
Patterns:
cache-aside (most common);
read-through or write-through (a library-managed loader);
write-behind (buffered writes, with a durability risk);
refresh-ahead for hot keys.
Data design:
cache DTOs or serialised projections, not entities;
use compact serialisation (Kryo or Protobuf vs JSON);
keep key cardinality under control;
avoid huge values (split them, or use hashes).
Consistency: evict on write after commit, CDC-driven invalidation, versioned keys, jittered TTLs, and local-cache invalidation broadcasts.
Protection: stampede control (Q10), negative caching (short TTLs for "not found" results, which prevent cache penetration), and Bloom filters for key existence.
Performance: pipelining or batching (MGET), connection pooling (Lettuce), co-locating caches with services (same availability zone), and sharding hot keys (replicate them to several keys, or keep them in local caches).
Measure: hit ratio, latency, evictions and memory. Watch for hot keys (redis-cli --hotkeys).
Q10. How do you prevent a cache stampede?
Short answer: A stampede (thundering herd, dog-piling) happens when a hot key expires or is evicted, and many concurrent requests miss at once, all hitting the database or the expensive computation. The mitigations:
Request coalescing (single-flight): only one loader per key computes, and the others wait for its result. Caffeine's get(key, loader) and AsyncCache do this per instance. Spring @Cacheable(sync = true) does it too.
A distributed lock or lease on the recomputation (Redis SET NX), with the others serving stale data or briefly waiting.
Stale-while-revalidate or soft TTL: serve the stale value while one request refreshes it in the background (refresh-ahead: Caffeine refreshAfterWrite).
Probabilistic early expiration (XFetch): each request has a small, rising chance of refreshing before the expiry.
Jittered TTLs, so many keys don't expire simultaneously (preventing an avalanche).
Pre-warming caches on deployment or scale-out, and on a schedule for known hot keys.
Rate-limiting the backend (bulkheads), as a last line of defence.
Q11. What is chaos engineering?
Short answer:Chaos engineering is disciplined experimentation on a system, to build confidence in its ability to withstand turbulent conditions, by injecting failures deliberately:
define the steady state (SLO metrics);
form a hypothesis ("if one availability zone is lost, checkout success stays above 99.5%");
inject faults: kill pods or instances, add latency or packet loss, fail dependencies, exhaust CPU or memory, cut a zone, throttle an API;
observe the outcome;
fix the weaknesses;
automate the experiment.
Tools: Chaos Monkey (Chaos Monkey for Spring Boot), Gremlin, LitmusChaos, Chaos Mesh, AWS Fault Injection Service, Toxiproxy (in tests), and Istio fault injection.
Practise it safely: start in staging, limit the blast radius, have abort conditions, run game days with the teams, and graduate to production with care. It validates timeouts, fallbacks, autoscaling, alerts and runbooks, which is exactly what fails during real incidents.
Follow-up questions this topic invites — and their answers
Q: Where should rate limiting be enforced: client, gateway or service?
A: Primarily at the gateway or edge (cheap rejection, global view), with per-service limits for expensive operations. Also on clients calling third-party APIs, so they respect the provider's quotas (outbound limiting with Resilience4j or Bucket4j).
Q: Why can retries make an outage worse?
A: Each layer's retries multiply the load on an already struggling dependency (3 retries at 3 layers means 27× the load). Use retry budgets, exponential backoff with jitter, circuit breakers, and retry at a single layer.
Q: What's a hot key, and how do you mitigate it in Redis?
A: A key receiving a disproportionate share of traffic (a celebrity profile, or a flash-sale item), which overloads a single shard. Add a local L1 cache in front, replicate the key under several suffixes and read one at random, or split the value.
Q: What is load shedding vs rate limiting?
A: Rate limiting enforces per-client quotas, even when the system is healthy. Load shedding drops or deprioritises work when the system is overloaded, whatever the client, to protect the core functions (prioritising checkout over recommendations).
