Distributed Systems Fundamentals — CAP, Consistency, Availability & SLOs — Interview Questions

How to use this lesson

These are the conceptual foundations interviewers use to judge whether your designs are principled. State the theory precisely (CAP is often misquoted), then connect it to a concrete decision you'd make, and its cost.

Q1. What's the difference between horizontal and vertical scaling?

Short answer:

• Vertical scaling (scale up): a bigger machine (more CPU, RAM, faster disks). It's simple (no code changes, no distribution problems), but it has hard ceilings, cost that grows non-linearly, a single point of failure, and often downtime to resize.
• Horizontal scaling (scale out): more machines or instances, behind load balancing or partitioning. It's near-linear capacity growth, and gives fault tolerance (losing one node isn't fatal), and elasticity (autoscaling). It requires: statelessness, or partitioned state; load balancing; and handling distributed-systems problems (consistency, coordination, network failures).

In practice: scale the stateless tier horizontally. Scale databases vertically first (it's often cheaper and simpler), then with read replicas, partitioning and sharding when you must.

Learn it in depth → Horizontal vs Vertical Scaling

Q2. What is the CAP theorem, and how does it affect distributed design?

Short answer: For a distributed data store, when a network partition (P) happens, you must choose between:

• Consistency (C): every read sees the latest write (linearisability);
• Availability (A): every request to a non-failed node gets a (non-error) response.

Partitions aren't optional in real networks, so the practical choice is CP vs AP during a partition:

• CP systems (etcd, ZooKeeper, HBase, Spanner, and a relational primary with synchronous replication) refuse or block some requests, to stay consistent;
• AP systems (Cassandra and DynamoDB with eventual consistency, DNS, shopping-cart-style designs) keep answering, possibly with stale or conflicting data, and reconcile later.

Common trap: the popular phrasing "pick any two of three" is misleading. You can't give up P in a distributed system. PACELC refines it: if Partitioned, choose A or C; Else (normal operation), choose Latency or Consistency. That explains why many systems trade consistency for latency even with no partition.

The design impact: choose per use case, not per system. Strong consistency for money, inventory and uniqueness constraints. Availability and eventual consistency for catalogues, feeds, recommendations and analytics.

Learn it in depth → CAP Theorem

Q3. Give a real-world scenario where you had to choose between consistency and availability.

Short answer: Use the STAR structure. A strong example: during festival sales on an e-commerce platform:

• Product browsing, search and prices shown are served from caches, replicas and the search index. We accepted eventual consistency (a price or stock badge could be seconds stale), so the site stayed available and fast, even when the primary database or a region was under stress. That's AP.
• Checkout and inventory reservation stayed strongly consistent (CP): an atomic conditional decrement on the primary, so there was no overselling. When the inventory database was unreachable, checkout failed fast with a friendly retry message, rather than accepting orders we couldn't fulfil.
• The result: high availability for 95% of the traffic, correct orders, and clear customer messaging. We monitored oversell count (0), checkout error rate, and staleness.

Key points to cover:

• It's rarely "the whole system is AP". Different operations get different guarantees.

Q4. What consistency models exist in distributed systems?

Short answer: From strongest to weakest:

• Linearisability (strong consistency): operations appear instantaneous at some point between their invocation and completion, in real-time order. It needs consensus (Raft or Paxos) or a single leader.
• Sequential consistency: everyone sees the same order of operations, respecting each client's program order, but not necessarily real time.
• Causal consistency: causally related operations (a reply after the post) are seen in order by everyone. Concurrent operations may differ.
• Session guarantees (practical, client-centric): read-your-writes, monotonic reads, monotonic writes, writes-follow-reads.
• Bounded staleness: reads lag by at most k versions or t seconds (Cosmos DB offers this).
• Eventual consistency: if updates stop, the replicas converge. There are no ordering guarantees in the meantime.

Database transaction isolation levels (read committed, repeatable read, serialisable) are a different axis: they concern concurrent transactions, not replica convergence. Strict serialisability combines serialisability with linearisability (Spanner).

Q5. How do you design a system for high availability and fault tolerance? What does 99.99% availability require?

Short answer:

• The principles:
  • eliminate single points of failure (redundancy at every layer: instances, zones, load balancers, databases with automatic failover, and brokers with replication);
  • isolate failures (bulkheads, circuit breakers, timeouts, cell or shard architectures);
  • degrade gracefully (fallbacks, feature shedding);
  • detect and recover automatically (health checks, self-healing orchestration, automated failover);
  • deploy safely (canaries, fast rollback);
  • and test failure (chaos engineering, game days).
• 99.99% ("four nines") allows about 52.6 minutes of downtime per year, and about 4.3 minutes per month. That means:
  • no manual steps in the recovery path: failover must be automatic, and detected within seconds;
  • multi-AZ everything (and often multi-region for the critical paths), with every dependency at least as available (serial dependencies multiply: 5 dependencies at 99.99% give about 99.95% overall), which pushes you toward async decoupling and redundancy;
  • zero-downtime deployments and migrations, with rapid rollback;
  • capacity headroom (N+1 or N+2), and load shedding;
  • SLO-based alerting and on-call practices, with fast incident response.
• Cost: each extra nine costs a lot more. Agree on the target with the business, per user journey.

Q6. What are SLIs, SLOs and SLAs? What's an error budget?

Short answer:

• SLI (Service Level Indicator): a measured metric of user-perceived reliability. For example, the proportion of successful requests, or the proportion of requests faster than 300 ms, measured at the load balancer or the client.

• SLO (Service Level Objective): the internal target for an SLI over a window. For example, "99.9% of checkout requests succeed over 28 days", or "99% under 300 ms". It drives engineering priorities.

• SLA (Service Level Agreement): a contractual promise to customers, with consequences (credits or penalties). It's usually looser than the SLO, so you have a buffer.

• Error budget: 1 − SLO, the allowed unreliability. At 99.9% over 28 days, that's about 40 minutes of failures. How it's used:

  • while there's budget left, ship features faster (take risks);
  • when it's exhausted (or burning fast), freeze risky releases, and prioritise reliability work;
  • alert on the burn rate (multi-window, multi-burn-rate alerts), not on raw thresholds.

  It aligns product and operations around one number.

Learn it in depth → Alerting Strategy

Q7. Two-phase commit vs sagas?

Short answer:

• Two-phase commit (2PC/XA):

  1. the coordinator asks every participant to prepare (vote yes or no, holding locks);
  2. if all vote yes, it sends commit, otherwise abort.

  It gives atomicity and strong consistency across resources. The drawbacks:

  • it's blocking: if the coordinator fails after the prepare phase, participants hold locks indefinitely (in-doubt transactions);
  • latency from several round trips;
  • it reduces availability (every participant must be up);
  • it's poorly supported by modern infrastructure (Kafka, many NoSQL and cloud databases);
  • it creates tight coupling.

  It fits within one trust boundary, with XA-capable resources (two databases in one application server), rarely across microservices.

• Sagas: a sequence of local transactions, with compensating actions, coordinated by events (choreography) or an orchestrator. Non-blocking, highly available, loosely coupled, and it works with any technology. The costs:

  • no isolation (intermediate states are visible, so use semantic locks or pending states);
  • eventual consistency;
  • compensations must be designed and idempotent;
  • more complex reasoning and testing.

• In microservices, use sagas (plus the outbox and idempotency). Keep invariants that need atomicity inside a single service's database.

Learn it in depth → Two-Phase Commit

Q8. What is distributed locking, and when is it (un)safe?

Short answer: A mutual-exclusion lock shared by processes on different machines, used to ensure that only one instance performs an action at a time (a scheduled job, a resource refresh, a leader-only task). Implementations:

• database locks: SELECT … FOR UPDATE, advisory locks (pg_try_advisory_lock), or a lock table (ShedLock uses this for @Scheduled jobs);
• Redis: SET key token NX PX ttl, with a token-checked release (Lua), and Redisson's RLock (a watchdog renewal);
• consensus systems: etcd leases, ZooKeeper ephemeral sequential nodes, Consul sessions. These give stronger guarantees;
• Kubernetes Lease objects, for leader election.

The pitfalls:

• Leases can expire while the holder is paused (GC pauses, network delays, VM stalls), so two processes believe they hold the lock. For correctness-critical operations, use fencing tokens: a monotonically increasing number issued with each lock grant, checked by the protected resource, which rejects older tokens.
• The Redlock debates (Kleppmann vs antirez): don't rely on a timing-based lock for safety.
• Prefer designs that don't need locks: idempotent operations, single-writer partitioning (Kafka partition ownership), optimistic concurrency, or database constraints.

Follow-up questions this topic invites — and their answers

Q: Is a single relational database CP or AP? A: A single node isn't a distributed system in the CAP sense. With replicas: synchronous replication with failover that refuses writes during partitions is CP-ish, and asynchronous replicas serving reads are AP-ish for those reads (stale data). It depends on how you configure and use it.

Q: What's the difference between availability and durability? A: Availability means the system responds to requests now. Durability means committed data survives failures. S3, for example, offers 11 nines of durability, but lower availability SLAs. You can lose availability temporarily without losing data, and vice versa.

Q: How do you compute a composite SLO for dependent services? A: For serial hard dependencies, multiply the availabilities (for example 0.999 × 0.999 ≈ 0.998). Redundant parallel paths improve it: 1 − (1 − a)ⁿ. That's why critical paths should minimise synchronous hard dependencies.

Q: What is a "cell-based architecture"? A: Splitting the whole stack into independent cells (each serving a subset of customers), so failures and bad deployments affect only one cell's blast radius. It's used by large SaaS providers and AWS services.