The specific failure-mode questions a strong HLD design must survive being asked — single points of failure, a dependency going down, a network partition, a hot key — and how to answer each without hand-waving. Includes a worked e-commerce checkout walkthrough: payment outage, last-unit race, duplicate submission, degraded notifications, and saga compensation.
Published September 23, 2026
Every HLD interview eventually asks some version of "what happens when X fails?" This is a distinct, learnable skill from designing the happy path — a design that only works when nothing goes wrong isn't actually a design, it's a diagram. This lesson works through the standard failure questions directly.
Every box in your diagram that appears exactly ONCE is a candidate SPOF.
The fix is always the same shape: replication + failover, not a single instance of anything critical.
A strong answer names the SPOF explicitly (rather than waiting to be asked) and states the mitigation: a single load balancer instance becomes an active-passive or active-active pair; a single database instance gets a replica with automatic failover; a single service instance gets deployed as multiple instances behind a load balancer. The interviewer is checking whether you actually audit your own diagram for this, not whether you can recite "add redundancy" when prompted.
This is directly the territory of Circuit Breaker Pattern and Timeout Strategy from the Resilience course: does the calling service hang forever (no timeout — the worst answer), fail fast and return an error (acceptable), or degrade gracefully by serving a fallback/cached response (best, where the dependency isn't essential to the core function — see Health Checks' essential-vs-non-essential framing)? Naming which of these three your design does, and why, is the actual answer — "we'd add retries" alone is incomplete without also naming the timeout and fallback behavior.
Partition: Node A can't reach Node B, but both are still up and serving requests to
different clients — the CP-vs-AP choice from CAP theorem becomes concrete here:
CP: one side stops accepting writes until the partition heals (consistency over uptime)
AP: both sides keep accepting writes, reconciled/merged after the partition heals
(uptime over consistency — requires a conflict-resolution strategy for the merge)
This is where HLD Fundamentals Refresher's CAP theorem discussion gets applied concretely: a strong answer states which side of CP/AP your specific system needs for its specific data (a bank balance needs CP; a "like count" can tolerate AP with eventual reconciliation) rather than giving one blanket answer for the whole system — different data within the same system can reasonably make different choices.
A celebrity's profile getting 1000x normal read traffic, all hashing to the SAME shard
→ that one shard/node gets overwhelmed while every other shard sits idle
A sharding or partitioning scheme (by user ID, by hash) can still fail under skewed access patterns — one key receiving disproportionate traffic overwhelms its single shard regardless of how well-distributed the overall key space is. The standard mitigations: an additional caching layer in front of the hot key (absorbing read traffic before it hits the shard), or detecting and splitting a hot key's data across multiple replicas specifically for that key. Naming this as a distinct failure mode from "uneven overall load" is a genuine differentiator in an interview.
The four questions above are generic. Interviewers usually make them concrete against your design. Here is the same audit applied to a typical checkout: an Order service that calls Payment synchronously, then publishes an event that Inventory and Notification consume asynchronously. Each scenario follows the same answer shape: what goes wrong → what the user sees → the mechanism that handles it.
What goes wrong: Order calls Payment and gets a timeout or connection error.
What the user should see: a clear "payment couldn't be processed, you haven't been charged, please try again" message. Never a spinner that hangs forever, and never an order that looks placed but isn't paid.
Mechanisms:
PAYMENT_PENDING or PAYMENT_FAILED, not PLACED. The order's state machine must never jump to a "placed" state before payment is confirmed.There's one subtlety interviewers look for. A timeout doesn't mean the payment failed. The request may have reached the gateway and succeeded after your client gave up. That's why the retry below must be idempotent, and why a reconciliation job that compares your records with the payment provider's is part of any serious payment design.
What goes wrong: both requests read "stock = 1", both decide there's enough, both decrement. You've sold two of something you had one of.
This is a classic check-then-act race condition. The read and the write are separate steps, and another request slips in between them.
Mechanisms (pick one and say why):
-- Option A: make the check and the write a single atomic statement
UPDATE inventory
SET available = available - 1
WHERE product_id = :id AND available >= 1;
-- 1 row updated → reserved; 0 rows → out of stock. No window for a race.
WHERE version = :readVersion. The loser sees 0 rows updated and retries or reports "sold out".SELECT ... FOR UPDATE): correct, but it holds a row lock during the transaction and hurts throughput on hot items. It's worth mentioning mainly to explain why you'd prefer the first two.What the user sees: the loser gets a clear "just sold out" message at checkout, not after paying. That's why reservation should happen before or as part of confirming the order.
What goes wrong: the user taps "Place order", the response is lost on a bad mobile connection, and the app (or the user) retries. Without protection you create two orders and charge twice.
Mechanism: an idempotency key.
Client generates a unique key per checkout attempt (e.g. a UUID) and sends it as a header:
POST /orders Idempotency-Key: 7f3c...e21
Server:
1. Look up the key.
- Not seen → process the order, store (key → result) atomically with the order
- Seen, done → return the stored result; do NOT process again
- Seen, running → return 409 / "still processing"
2. Keep keys for a window long enough to cover realistic retries (e.g. 24 hours)
The key must be stored in the same transaction as the order it created. Otherwise a crash between "create order" and "remember the key" reopens the duplicate window. The same key should be forwarded to the payment provider, most of which accept an idempotency key precisely so a retried charge isn't applied twice.
What goes wrong: emails are slow or failing.
The right answer is that checkout doesn't care. Sending the confirmation email is a side effect of a placed order, not part of placing it. Because Notification consumes the OrderPlaced event asynchronously, it being down means messages wait in the topic and emails go out late. No order fails.
This is also a design-review point to raise proactively. If the diagram showed Order calling Notification synchronously, a notification outage would block checkouts, which is exactly the kind of coupling to remove. The general rule is that only dependencies whose answer you need right now belong on the synchronous path.
What goes wrong: the customer has been charged, but Inventory, reacting asynchronously, discovers it can't reserve the item (stock was corrected, a warehouse went offline). Payment and Inventory each own their own database, so there's no single transaction you can roll back across both.
Mechanism: a saga with compensating actions. A saga is a sequence of local transactions, one per service. If a later step fails, earlier steps are undone by new actions, not rolled back.
Happy path: Order: PENDING ─▶ Payment: CHARGED ─▶ Inventory: RESERVED ─▶ Order: CONFIRMED
Failure at inventory:
Inventory: RESERVATION_FAILED
│ emits InventoryReservationFailed
▼
Payment: REFUND issued ← compensating action for "charge"
│ emits PaymentRefunded
▼
Order: CANCELLED ← compensating action for "create"
│
▼
Customer notified: "item unavailable, refund on its way"
Points that make this answer strong:
Q: Should you proactively bring up failure scenarios, or wait for the interviewer to ask? A: Proactively naming at least the most obvious SPOF and one dependency-failure behavior, briefly, before being asked is a strong signal — it shows the failure-mode audit is a genuine habit, not a reactive answer produced only under prompting; going into exhaustive detail unprompted on every possible failure, though, can eat time better spent on the core design.
Q: How deep should a failure-scenario answer go before it's 'enough' for an interview? A: Enough to name the specific failure, the specific user-visible consequence, and the specific mitigation mechanism (not just 'we'd add monitoring') — 'the payment service could time out, so we'd set an aggressive timeout with a circuit breaker and return a clear retry-safe error to the client' is a complete answer; 'we'd handle errors' is not.
Q: Do all four failure questions (SPOF, dependency-down, partition, hot key) apply to every system? A: Hot-key and partition questions matter most for systems with real horizontal data distribution (sharded databases, distributed caches); a simpler system without sharding may not have a meaningful hot-key answer — naming which failure modes are actually relevant to YOUR specific design (rather than reciting all four regardless of fit) is itself part of a strong answer.
Q: How does failure-scenario thinking change the original high-level design, not just add caveats to it? A: A design reviewed against these questions often changes concretely — e.g. discovering a SPOF in the diagramming step (Architecture Diagramming) should lead back to revising the diagram itself, not just verbally noting the gap; treating failure analysis as feeding back into the design (not a separate bolt-on discussion at the end) is the stronger interview pattern.
Q: Why not just use a distributed transaction (two-phase commit) across Payment and Inventory instead of a saga? A: Two-phase commit holds locks in every participant until the coordinator decides, so one slow or failed service blocks the others. External systems like a payment provider don't take part in your 2PC at all. Sagas accept temporary inconsistency (charged but not yet reserved) in exchange for services that stay independent and available. That's the right trade for most business workflows, as long as every step has a well-defined compensation.
Q: How long should an idempotency key be remembered? A: At least as long as a client could realistically retry the same request: typically hours to a day for checkouts. Keep it longer if clients queue retries offline. Expire keys afterwards so the table doesn't grow forever, and document the window, because a retry after expiry is treated as a brand-new request.
Q: If the payment call times out, should Order retry it automatically?
A: Only if the retry carries the same idempotency key, so a payment that actually succeeded isn't charged twice. Use a small number of retries with exponential backoff and jitter, stop when the circuit breaker is open, and fall back to marking the order PAYMENT_PENDING for the reconciliation job to settle. That's safer than guessing.