RED and USE methods, Micrometer/Prometheus/Grafana in practice, pull vs push scraping, why percentiles beat averages for latency, and load testing before shipping vs monitoring after.
Published September 23, 2026
For request-driven services (most application services): Rate (requests per second), Errors (error rate — the fraction of requests failing), Duration (latency distribution — how long requests take). These three, tracked per service/endpoint, cover the majority of "is this service healthy" questions directly and are the standard starting dashboard for any service.
For infrastructure/resources (a different angle, complementary to RED): Utilization (how busy is the resource — CPU%, memory%), Saturation (how much queued/waiting work exists beyond what's being actively serviced — a queue depth, waiting thread count), Errors (resource-level errors, e.g. disk I/O errors). USE fits infrastructure/resource-level monitoring (a database, a message queue, a disk) where RED's "requests" framing doesn't map as directly.
@Timed(value = "order.processing.time")
public Order processOrder(OrderRequest request) { ... }
// or programmatically:
MeterRegistry registry;
Counter ordersCreated = registry.counter("orders.created");
ordersCreated.increment();
Micrometer is Spring Boot's metrics facade — application code instruments against Micrometer's vendor-neutral API (the same vendor-neutrality principle as OpenTelemetry for tracing), and Micrometer exports to whichever backend is configured — commonly Prometheus, an open-source metrics storage/query system that's become close to a de facto standard for this role.
Prometheus Server → periodically SCRAPES → GET /actuator/prometheus on each service instance
Prometheus pulls metrics — it periodically scrapes each service's /actuator/prometheus endpoint, rather than services pushing metrics out to a central collector. This inverts the more intuitive "push metrics somewhere" mental model, and has real practical benefits: Prometheus itself controls scrape timing and can detect a service that's stopped responding to scrapes (implicitly, its absence signals a problem) — a push-based model instead requires services to reliably know where to push and handle push failures themselves. Push-based alternatives (StatsD, and some cloud-native metrics pipelines) exist and fit differently-shaped infrastructure, particularly short-lived/ephemeral jobs that might not exist long enough to be reliably scraped.
Grafana queries Prometheus (or other backends) and renders the results as dashboards — a service health dashboard typically combines RED metrics (request rate, error rate, latency percentiles) with resource metrics (CPU, memory, connection pool state from Connection Pooling) into one visual view, the standard "first thing to check" during an incident or a routine health review.
Average latency: 120ms (looks fine!)
p50: 80ms (half of requests are fast)
p95: 200ms (5% of requests are noticeably slower)
p99: 3500ms (1% of requests are DRAMATICALLY slower — a real problem hiding behind a fine-looking average)
An average can look perfectly healthy while a meaningful fraction of real users experience terrible latency — a small number of very slow outlier requests barely move the average (especially at high request volume, where they're diluted among many fast ones) but represent genuine, real user pain. p50/p95/p99 (percentile latencies — "99% of requests complete faster than this value") expose the tail directly, which is exactly why Timeout Strategy's guidance to set timeouts from observed p99 (not average) matters — the tail is where real problems concentrate, and averages actively hide them.
Load testing simulates expected (and peak) traffic against a system before it ships, specifically to validate capacity assumptions and find the actual breaking point under controlled conditions — distinct from production monitoring, which observes genuine real traffic after the fact. Load testing answers "will this hold up under Black-Friday-level load" proactively; monitoring answers "is it currently holding up" reactively — both are necessary, and skipping load testing in favor of "we'll just watch monitoring in production" means discovering capacity limits during a real, live traffic spike rather than in a controlled test beforehand.
Q: Why does Prometheus's pull model make it easier to detect a fully-down service than a push model would? A: A scrape that simply fails (connection refused, timeout) is itself a clear, unambiguous signal the target is unreachable — in a push model, a service that's completely down can't push anything at all, and distinguishing 'silent because down' from 'silent because it just has nothing to report right now' requires additional explicit heartbeat logic that pull-based scraping gets for free.
Q: How would you decide between p95 and p99 as the primary SLO target for a service? A: Depends on the acceptable pain threshold and traffic volume — at very high request volume, even p99 represents a large absolute number of genuinely slow requests (1% of a million daily requests is 10,000 bad experiences), which sometimes argues for tracking p99.9 instead for latency-critical, high-volume services, while p95 might be sufficient for less critical or lower-volume ones.
Q: Is RED method sufficient on its own for a service that also has significant background/async processing, not just request-response? A: Not fully — RED's 'requests' framing fits synchronous request-response cleanly; background job processing needs its own metrics shape (queue depth, job processing rate, job failure rate — closer to USE's saturation/error framing), which is why a service doing both typically needs both RED (for its API) and a job-specific metric set, not RED alone covering everything.
Q: How does load testing avoid accidentally causing a real production incident while testing? A: Load tests are typically run against a staging/pre-production environment sized similarly to production, or against production itself but during low-traffic windows with careful monitoring and an immediate abort capability if real user traffic starts showing impact — running an uncontrolled load test directly against a live production system with no safeguards is itself a common, avoidable cause of self-inflicted outages.