SLO-burn-rate alerting over fixed thresholds, avoiding alert fatigue, symptom vs cause-based alerts, on-call/runbook linkage, chaos engineering, and the SLI/SLO/SLA relationship precisely.
Published September 23, 2026
SLO: 99.9% of requests succeed over a 30-day window (allows a 43-minute "error budget" per month)
Alert condition: error budget being consumed at a rate that would EXHAUST it before the window ends
— e.g. burning the entire monthly budget in 1 hour = alert IMMEDIATELY (severe)
— e.g. burning it at a rate that would exhaust it in exactly 30 days = no alert needed (on pace, acceptable)
An arbitrary fixed threshold ("alert if error rate > 1%") doesn't account for how much error budget is actually being consumed relative to what's acceptable — a brief 5% error spike lasting 30 seconds is a non-event against a monthly SLO; the same 5% sustained for an hour could burn a meaningful fraction of the entire month's budget. Burn-rate alerting computes the rate at which the error budget is being consumed and alerts based on whether that rate, if sustained, would exhaust the budget too early — a fundamentally more meaningful signal than a fixed percentage threshold.
Every alert that fires should be actionable — something a human receiving it can and should actually do right now — or it shouldn't fire at all. An alert that fires regularly for a condition nobody actually acts on (because it's known-benign, or because there's genuinely nothing to do about it) trains the on-call rotation to ignore alerts, which is dangerous precisely because it degrades response to the alerts that do matter — alert fatigue is a real, measurable cause of missed genuine incidents, not a minor annoyance.
Symptom-based (user-facing impact): "checkout error rate exceeds 1%" — alerts on what users are actually experiencing, regardless of root cause. Cause-based (internal metric anomaly): "connection pool utilization exceeds 90%" — alerts on an internal signal that might lead to user impact. Symptom-based alerts are generally preferred as the primary page-someone-now trigger (they directly reflect whether users are actually affected); cause-based alerts are valuable as supporting diagnostic signals once a symptom-based alert has already fired, helping localize the root cause quickly — but alerting primarily on internal causes risks paging someone for conditions that never actually impact users, contributing directly to alert fatigue.
Every alert that pages someone should link directly to a runbook — a document describing likely causes and first diagnostic/remediation steps for that specific alert — rather than leaving the on-call engineer to reconstruct context from scratch at 3am. Escalation policies (if the primary on-call doesn't acknowledge within N minutes, page a secondary; if still unacknowledged, page a manager) ensure a critical alert doesn't go unaddressed simply because one person is unreachable.
Deliberately, in a controlled way: kill a random instance, inject artificial latency into a dependency,
partition part of the network — then verify the system's resilience claims (circuit breakers, retries,
bulkheads, failover) actually hold under real failure conditions, not just in theory.
Every resilience pattern covered earlier in this course (circuit breakers, bulkheads, retries) is a claim about how the system behaves under failure — chaos engineering is how that claim gets actually verified rather than assumed. A circuit breaker that's misconfigured, or a fallback that itself has a bug, might never be discovered until a genuine production failure — deliberately, safely inducing failure conditions in a controlled environment (or carefully in production, for mature practices) surfaces these gaps proactively, on the team's own schedule, rather than during a real incident.
The relationship is a hierarchy: SLI is what you measure, SLO is the internal bar you hold that measurement to, SLA is what you've promised externally (and is deliberately set less strict than the SLO, so there's room to notice and fix a problem internally before it becomes an SLA breach with real business consequences).
Q: Why is burn-rate alerting harder to implement than a fixed threshold, and is it always worth the complexity? A: It requires tracking error budget consumption over the SLO's actual time window and computing a rate, more involved than a simple point-in-time threshold check — for a mature service with a well-established SLO, the reduced false-positive/false-negative rate is generally worth it; for a simpler or less critical service, a fixed threshold might be a reasonable, lower-effort starting point.
Q: How would you decide whether a specific chaos experiment is safe to run in production vs staging only? A: Start in staging to validate the experiment itself doesn't have unintended consequences, then graduate carefully-scoped, well-understood experiments to production with safeguards (a fast abort mechanism, running during low-traffic windows, limiting blast radius to a small percentage of traffic/instances) — running an untested chaos experiment directly in production is itself a way to cause the very incident you're trying to proactively surface in a controlled way.
Q: Can a service have an SLO stricter than its own dependencies' SLOs? A: Only with careful design — a service's own achievable SLO is fundamentally bounded by its dependencies' reliability (see Why Microservices Fail's chatty-chain availability math) unless it adds resilience patterns (caching, fallbacks, circuit breakers) that let it maintain its own SLO despite a dependency occasionally missing its own, which is a real, common reason to add resilience patterns beyond just 'best practice.'
Q: Why might an alert be both symptom-based AND cause-based simultaneously? A: A single alert condition can sometimes serve both roles if the internal metric IS effectively user-facing (e.g. 'checkout success rate' is both the direct user-facing symptom and, from the payment service's perspective, its own primary cause-level metric) — the distinction matters more when the internal metric and user impact are genuinely one step removed from each other, not when they're effectively the same signal viewed from two levels.