Load balancing, caching, CAP theorem, and scaling — revisited specifically as HLD-interview ammunition — plus the 5-step framework for structuring any design round.
Published September 22, 2026
Before walking through specific case studies, the underlying vocabulary needs to be automatic — you shouldn't be deriving what CAP theorem means mid-interview. This is that refresher, framed around what actually gets asked.
A load balancer distributes incoming requests across multiple servers. The interview-relevant part isn't "what is a load balancer" — it's which algorithm, and why: round-robin (simple, ignores server load), least-connections (routes to whichever server has fewest active connections — better under uneven request durations), and consistent hashing (routes based on a hash of the request key, e.g. user ID — critical when you need the same user's requests to keep landing on the same server, such as for in-memory session data or cache locality).
The interview-relevant caching question is rarely "what is a cache" — it's where in the stack, and what strategy for staying correct:
The follow-up that separates strong answers: how do you invalidate a cache when the underlying data changes? — TTL expiry (simple, tolerates staleness), explicit invalidation on write (more consistent, more code paths to get right), or a pub/sub invalidation message to all cache nodes in a distributed cache layer.
CAP theorem: in the presence of a network Partition, a distributed system must choose between Consistency (every read gets the most recent write) and Availability (every request gets a non-error response). It does not say "pick any 2 of 3" in normal operation — partition tolerance isn't optional for any real distributed system; the actual choice is CP vs AP, and only during an actual partition. Getting this precise distinction right is itself a strong interview signal, since "pick 2 of 3" is the most common CAP misstatement.
Vertical (scale up): bigger machine, more CPU/RAM. Simple, no architecture changes, but has a hard ceiling and a single point of failure. Horizontal (scale out): more machines. No hard ceiling, better fault tolerance, but requires the application to be designed for distribution in the first place (stateless services, a load balancer, data partitioning) — you can't horizontally scale a system that assumes all state lives in one process's memory without redesigning it first.
L4 (transport layer): routes based on IP + port only, doesn't inspect HTTP content —
faster, lower overhead, can't route based on URL path or headers
L7 (application layer): inspects the actual HTTP request (path, headers, cookies) —
enables path-based routing (/api/orders -> Order Service, /api/users -> User Service),
more overhead per request, but far more routing flexibility
Beyond the algorithm (round-robin, least-connections, consistent hashing), an interviewer probing deeper often wants L4 vs L7: an L7 load balancer is what makes it possible to route different URL paths to different backend services from a single entry point, which is exactly what an API Gateway does — L4 is faster but can only make routing decisions based on IP/port, not request content.
Health checks are what let a load balancer stop routing to an unhealthy instance automatically — this is precisely the mechanism the Health Checks lesson's readiness concept plugs into: the load balancer periodically polls each instance's readiness endpoint and removes any instance reporting unready from rotation, closing the loop between an instance self-reporting a problem and traffic actually stopping.
Sticky sessions (routing the same client's requests to the same backend instance, via a cookie) solve the same problem consistent hashing solves at the load-balancer layer specifically — needed when a service holds in-memory per-client state that isn't replicated elsewhere; the better long-term fix is usually making the service stateless (session data in a shared store like Redis) so sticky sessions aren't required at all, since sticky sessions reintroduce a soft dependency on a specific instance staying up.
Once a cache is in place, two follow-up questions come up almost every time:
Eviction policy — when the cache is full, what gets removed to make room for a new entry?
Thundering herd / cache stampede — a specific, real failure mode:
A popular cache key expires. In the same instant, 10,000 concurrent requests all miss
the cache simultaneously and all hammer the database trying to recompute the SAME value
-> the database gets a massive, avoidable spike of duplicate work
Standard mitigations: request coalescing (the first request to miss the cache recomputes the value; concurrent requests for the same key wait on that one in-flight computation instead of each independently hitting the database), or probabilistic early expiration (recompute slightly before actual expiry, staggered per-request, so not every client experiences the miss at the exact same instant).
Horizontal scaling's requirement that services be stateless (from the section above) is necessary but not sufficient on its own — two further pieces make it work in practice:
Auto-scaling: adding/removing instances automatically based on a live metric (CPU utilization, request queue depth, or a custom application metric) rather than manually provisioning for peak load year-round — this is what makes horizontal scaling cost-efficient, not just theoretically possible; provisioning permanently for 3x-peak capacity when peak only happens a few hours a day wastes real money the rest of the time.
Database scaling specifically (the harder half of horizontal scaling, since data has state by definition):
The most common failure mode isn't getting any single step wrong — it's skipping straight to step 3 without doing 1 and 2 first, which produces a design with no stated justification for any of its choices.
A design that jumps straight to "I'll use a load balancer and shard the database" without first stating why those choices fit this system's actual scale and consistency needs reads as pattern-matching, not reasoning. Explicitly stating "we need 10K writes/sec, eventual consistency is acceptable, p99 latency under 200ms" before drawing anything gives every later decision a stated justification an interviewer can follow and challenge.
Q: If CAP forces a CP-vs-AP choice only during a partition, what do you do the rest of the time? A: Outside of an actual partition, a well-designed distributed system can be both consistent and available — CAP is specifically a statement about behavior during a partition, not a permanent tax paid at all times.
Q: Is consistent hashing only useful for load balancing?
A: No — it's the same core technique behind distributed cache sharding and distributed hash tables generally: minimizing how many keys need to move when a node is added or removed (only a fraction of keys remap, not all of them, unlike naive hash % N sharding).
Q: When would you choose write-through over cache-aside despite the extra write cost? A: When staleness is unacceptable even for a short window — e.g. a cache backing a feature where a stale read causes a real user-facing correctness issue, not just a minor inconsistency, and the extra write latency is an acceptable tradeoff for that guarantee.
Q: What's a concrete failure if you skip the estimation step entirely? A: You risk proposing a single-database design for a workload that actually needs sharding (or the reverse — over-engineering a low-traffic system with unnecessary distributed complexity) — estimation is what tells you which regime you're actually designing for.
Q: Why would you ever choose L4 over L7 if L7 gives more routing flexibility? A: L4 has meaningfully lower latency and overhead since it doesn't parse the HTTP request at all — appropriate when routing needs are simple (a single backend pool, no path-based routing needed) and the extra performance matters more than L7's flexibility; many real architectures use an L4 balancer in front of an L7 layer for exactly this reason, layering the two.
Q: Can LRU and LFU be combined? A: Yes — hybrid policies (like LRU-K or ARC) exist specifically to get LFU's resistance to one-off access spikes without losing LRU's simplicity and adaptiveness to changing access patterns; most real cache systems (Redis included) offer several eviction policy choices precisely because no single policy is best for every workload.
Q: Does sharding make read replicas unnecessary? A: No — they solve different problems and are often combined: sharding distributes total data volume and write load across shards, while each individual shard can STILL have its own read replicas to scale read throughput within that shard further; large-scale systems commonly use both together, not one instead of the other.