Setting CloudWatch alarm thresholds that mean something, a log-aggregation strategy across many instances/services, building a minimal RED-method on-call dashboard, and defining 'on-call ready' as a real launch gate.
Published September 23, 2026
This lesson is the practical, AWS-flavored application of the observability principles from Metrics & Monitoring and Alerting Strategy — the concrete checklist for getting a service genuinely production-monitored, not just theoretically instrumented.
Alarm: CPUUtilization > 80% for 5 consecutive minutes → notify on-call
Alarm: 5xxErrorRate > 1% over a 5-minute window → page on-call immediately
Alarm: p99 Latency > 2000ms for 3 consecutive periods → notify on-call
CloudWatch alarms watch a specific metric against a threshold over a defined evaluation window, triggering a notification/page when breached. The genuinely important design decision — directly echoing Alerting Strategy's actionability principle — is setting thresholds that represent a REAL problem worth waking someone up for, not an arbitrary round number: an 80% CPU threshold held for a brief legitimate traffic spike shouldn't page anyone if the service handles it fine; the threshold and evaluation window together need to represent 'this is genuinely degraded, not just momentarily busy.'
With multiple instances (or multiple services) each producing their own logs, an on-call engineer investigating an issue can't reasonably SSH into each instance individually checking local log files — this is the exact problem Centralized Logging's correlation-ID-based aggregation pipeline solves, and "monitoring in production" concretely means that pipeline is ACTUALLY set up and working (logs flowing into CloudWatch Logs, or a dedicated aggregation stack like the ELK/EFK stack from Centralized Logging) before the service is genuinely considered production-ready — not a nice-to-have added later.
Dashboard panels:
Rate: requests/sec, by endpoint
Errors: error rate %, by status code
Duration: p50/p95/p99 latency
A minimal, genuinely useful on-call dashboard doesn't need to be exhaustive — covering the three RED-method metrics (Metrics & Monitoring) for the service's key endpoints is enough to answer the first, most urgent on-call question during an incident: "is this service actually degraded right now, and in what way (more errors? slower? both?)." A dashboard that's too comprehensive (dozens of panels, buried key signals) is genuinely WORSE for incident response than a small, focused one — the goal is fast orientation under pressure, not completeness.
A service shouldn't ship to production without an explicit, deliberate answer to: does it have alarms tied to meaningful thresholds? Is its logging aggregated and searchable? Does a dashboard exist? Is there a runbook (Alerting Strategy) for its most likely failure modes? Is someone actually ON the on-call rotation for it? Treating "on-call ready" as an EXPLICIT checklist/gate before launch — rather than an implicit assumption that monitoring will somehow get added later — is what prevents the common, painful pattern of a service running unmonitored in production until its first real incident reveals the gap the hard way.
Q: How would you tune an alarm threshold if you don't yet have historical data for a brand-new service? A: Start with a conservative, best-guess threshold based on similar existing services' known-good baselines, ship it, then TUNE it based on actual observed behavior over the first few weeks — treating the initial threshold as a starting hypothesis to refine, not a permanent, precisely-correct value chosen once and never revisited.
Q: Should every service have its own dedicated dashboard, or is a shared platform-wide dashboard sufficient? A: Both, serving different purposes — a shared platform-wide dashboard gives a quick top-level health check across everything; a per-service RED dashboard is what an on-call engineer actually drills into once an issue is localized to a specific service, and skipping the per-service view in favor of only a shared one leaves exactly the investigative detail an incident response actually needs.
Q: Does 'on-call ready' as a launch gate ever conflict with shipping speed? A: There's a real, honest tension, and it's worth naming explicitly rather than pretending it doesn't exist — the standard resolution is scoping the on-call-ready bar to what's TRULY minimal (alarms on the handful of metrics that actually matter, not an exhaustive suite) so it's a fast, lightweight gate rather than a heavyweight process that meaningfully slows every launch.
Q: How does this lesson's guidance relate to Health Checks' liveness/readiness concepts? A: They're complementary layers of the same overall goal — liveness/readiness (Health Checks) is about the SERVICE'S OWN automated self-reporting and recovery; the monitoring/alerting/dashboard setup covered here is about HUMAN visibility and response when automated recovery isn't sufficient on its own — a fully production-ready service needs both working together, not one substituting for the other.