Horizontal vs Vertical Scaling

Horizontal vs Vertical Scaling

When a system can't keep up (pages slow down, CPU sits at 100%, the database queue grows), you have two basic ways to add capacity:

• Vertical scaling (scale up): make the machine bigger.
• Horizontal scaling (scale out): add more machines and spread the work across them.

Real systems use both, often for different components. The skill is knowing which applies where, and what each one requires of your architecture.

Vertical scaling: a bigger machine

Move from 4 vCPUs and 16 GB of RAM to 64 vCPUs and 512 GB. Nothing about the software changes.

Advantages:

• No code or architecture changes. It's often just a restart onto a larger instance type.
• No distributed-systems problems: one database means real transactions, simple joins and no replication lag.
• Low latency between components that share a machine.

Limits:

• A hard ceiling. The largest available machine is only so large, and the cost of top-end hardware rises faster than its capacity.
• A single point of failure. A bigger machine fails just as completely.
• Resizing usually needs a restart and some downtime, unless there's a failover replica.
• Some bottlenecks don't improve. A single-threaded process, or lock contention, won't get faster with more cores.

It's worth stating plainly: vertical scaling goes much further than many people assume. A single modern database server with plenty of RAM and fast NVMe storage handles very large workloads. Scaling up first is often the correct, cheapest decision, especially for databases.

Horizontal scaling: more machines

Run N copies of the service behind a load balancer:

                 ┌──▶ app-1 ─┐
 clients ─▶ LB ──┼──▶ app-2 ─┼──▶ shared state: database · cache · object storage
                 └──▶ app-3 ─┘

Advantages:

• Capacity grows roughly linearly with instances, with no fixed ceiling for stateless tiers.
• Fault tolerance: one instance dying reduces capacity, not availability.
• Elasticity: add instances for the evening peak and remove them overnight (autoscaling).
• Rolling deployments without downtime: update instances one at a time.

Requirements and costs:

• The service must be stateless (see below).
• More moving parts: load balancing, health checks, service discovery, deployment tooling, observability across many instances.
• Shared dependencies become the bottleneck. Ten app servers all hit the same database. Scaling the app tier just moves the pressure.
• Stateful components are much harder to scale out: they need replication and sharding, with the consistency trade-offs that come with them.

Statelessness: the prerequisite for scaling out

A service is stateless if any instance can handle any request, because nothing needed for later requests is kept only in one instance's memory or local disk.

❌ Session stored in app-1's memory → the next request lands on app-2 → the user is "logged out"
❌ An uploaded file saved to app-1's local disk → app-2 can't serve it
❌ An in-memory rate-limit counter per instance → the real limit becomes N × the intended one

✅ Sessions in Redis, or self-contained signed tokens (JWT)
✅ Files in object storage (S3 / GCS / Azure Blob)
✅ Counters and locks in a shared store
✅ Background jobs in a queue that any worker can consume

Sticky sessions (the load balancer always routes a user to the same instance) are a workaround, not a fix. Load becomes uneven, and users lose their state when their instance dies or is replaced during a deployment.

Scaling each tier

The database order matters. Each step adds complexity:

• Read replicas bring replication lag and read-your-writes problems.
• Sharding brings cross-shard queries, rebalancing and distributed transactions.

Take each step only when the previous one runs out.

Autoscaling

Cloud platforms add and remove instances automatically, based on metrics:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: api }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: api }
  minReplicas: 3            # enough for redundancy even when idle
  maxReplicas: 30           # a ceiling protects the database and the budget
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 65 } }

Practical points:

• Scale on the metric that actually tracks load. CPU works for compute-bound services. Queue depth or requests per instance work better for others.
• New instances take time to start and warm up, so leave headroom (a target of 60–70%, not 95%).
• A maxReplicas limit prevents a traffic spike, or a bug, from overwhelming the database behind the app tier.

Choosing in practice

• Early-stage, modest traffic: one or two app instances (two for redundancy) and a well-sized database. Keep the app stateless from day one, because it costs almost nothing and keeps scaling out easy later.
• Growing traffic: scale the stateless tiers horizontally, add caching, scale the database up, then add read replicas.
• Very large scale: shard stateful systems, partition by region, and use specialized stores.

Follow-up questions this topic invites — and their answers

Q: Why is horizontal scaling usually preferred in system design? A: It has no fixed ceiling, provides fault tolerance through redundancy, and supports elastic, rolling operation. But it requires stateless services, and it pushes the hard problems down to the shared data tier. Vertical scaling is still often right for databases.

Q: What makes a service stateless, and how do you handle sessions? A: No instance keeps data that later requests depend on. Sessions go into a shared store such as Redis, or become signed tokens the client presents on each request. Files go to object storage, and shared counters to a central store.

Q: How do you scale a database that has hit its limits? A: In order: optimize queries and indexes, scale up the hardware, add caching, add read replicas for read-heavy load, then partition data by function or shard by key for write-heavy load. Each step adds consistency and operational complexity.

Q: Why are sticky sessions discouraged? A: They tie a user's state to one instance, which unbalances load, loses sessions when the instance fails or is redeployed, and hides the underlying statefulness problem.

Q: What metrics would you autoscale on? A: Whatever best tracks saturation for that service: CPU for compute-bound APIs, queue depth or message lag for workers, requests per second or concurrency per instance for I/O-bound services. Combine it with a minimum for redundancy and a maximum to protect downstream systems.