Round-robin, least connections, consistent hashing — route traffic across servers.
Published September 21, 2026
Once a service runs on more than one server, something has to decide which server handles each request. That's the load balancer. It spreads traffic so that no server is overloaded while others sit idle, stops sending traffic to unhealthy servers, and gives clients one stable address, even as servers are added, removed or redeployed behind it.
Load balancers appear at several layers of a system: DNS spreading users across regions, an edge balancer in front of the web tier, internal balancers between microservices, and client-side balancing inside service meshes.
Layer 4 (transport) balancers route TCP/UDP connections using IP addresses and ports. They don't look inside the traffic.
Layer 7 (application) balancers terminate the connection and understand HTTP: path, host, headers, cookies.
/api/* → API service, /images/* → media service), TLS termination, per-request balancing, retries, header manipulation, rate limiting, compression, and a natural place for auth checks or a WAF.Typical setups combine them: an L4 balancer for raw throughput at the edge, and L7 proxies behind it for routing.
| Algorithm | How it picks | Good when | Weakness |
|---|---|---|---|
| Round robin | Next server in turn | Similar servers, similar requests | Ignores current load |
| Weighted round robin | In turn, proportional to weights | Servers of different sizes; canary releases (5% to the new version) | Still load-blind |
| Least connections | Server with the fewest active connections | Request durations vary a lot | Connections aren't a perfect proxy for load |
| Least response time / latency-aware | Fastest recent responses, fewest in flight | Heterogeneous or noisy servers | Needs good metrics |
| Power of two random choices | Pick 2 servers at random, send to the less loaded | Large fleets, distributed balancers | Slightly less optimal than a global view, but far cheaper |
| Hash-based (IP, user ID, key) | hash(key) chooses the server | Cache affinity, sharded state | Uneven if keys are skewed. Plain % N reshuffles when N changes |
| Consistent hashing | Key maps to a position on a hash ring | Caches and stateful routing where servers come and go | More complex. Needs virtual nodes for balance |
Power of two choices deserves a mention in interviews. Choosing the less loaded of two random servers avoids the herd behaviour where many balancers all pick "the least loaded server" at once and overload it, and it performs dramatically better than pure random.
/health every few seconds. After several failures, it removes the server; after several successes, it adds it back. Thresholds prevent flapping.Sticky sessions (routing a client to the same server via a cookie or IP hash) exist for applications that keep session state in memory. They cause uneven load and lose sessions when a server dies. The preferred design is stateless servers, with session data in a shared store or in signed tokens. See Horizontal vs Vertical Scaling.
Affinity is still legitimately useful for cache locality: routing the same key to the same server keeps its cache warm. That's done with consistent hashing, not cookies.
In microservice systems, the caller often balances by itself: it gets the list of healthy instances from service discovery (Consul, Kubernetes endpoints, or a service-mesh sidecar such as Envoy) and picks one per request. That removes a network hop and a central bottleneck. Service meshes add retries, timeouts and outlier detection uniformly across services.
Q: When would you choose an L4 balancer over L7? A: For non-HTTP protocols, for maximum throughput and minimal latency, or when you want TLS to pass straight through to the backends. Use L7 when you need path- or header-based routing, per-request balancing, TLS termination or HTTP-aware retries.
Q: Why can round robin perform poorly? A: It ignores how busy each server is. If some requests take 10 ms and others take 5 seconds, or servers differ in capacity, some servers pile up work while others idle. Least-connections, latency-aware or power-of-two-choices algorithms adapt to actual load.
Q: Why does gRPC often need L7 balancing? A: gRPC uses long-lived HTTP/2 connections that carry many requests. An L4 balancer assigns the whole connection to one server, so load stays unbalanced. An L7 proxy, or client-side balancing, distributes the individual requests.
Q: How do you deploy a new version without dropping requests? A: Take one instance at a time out of rotation (fail its readiness check), drain in-flight requests, update it, wait for it to pass health checks, and return it to rotation. Or shift weighted traffic gradually from the old version to the new one (canary or blue–green).
Q: What happens if the load balancer itself fails? A: Run it redundantly: a pair with a floating IP, multiple instances behind DNS or anycast, or a managed multi-zone balancer. Health-checked DNS or global load balancing handles the loss of an entire region.