Building the monitoring platform itself — metric ingestion, a write-heavy time-series database with downsampling, and evaluating thousands of alert rules efficiently without re-scanning all data per rule.
Published September 23, 2026
Metrics & Monitoring and Alerting Strategy covered USING a monitoring platform (Prometheus, CloudWatch). This case designs that platform itself — metric ingestion, storage, and alert evaluation at scale.
Design a monitoring and alerting system that ingests metrics from thousands of service instances, stores them efficiently for both recent and historical querying, and evaluates alert rules against them without falling behind as the number of monitored services and alert rules both grow.
Functional: ingest metric data points continuously from many sources; store time-series data queryable by time range; evaluate alert rules against incoming/recent data and trigger notifications on breach; support dashboarding. Non-functional: handle very high write volume (metrics arrive constantly from every monitored instance); support long retention without unbounded storage growth; evaluate potentially thousands of alert rules with acceptable latency, not degrading linearly as rule count grows.
Write-heavy, append-mostly: a metric data point is essentially never updated
after being written — pure inserts, at very high volume
Downsampling for long retention: keep full-resolution data for a short recent
window, progressively coarser resolution for older data (directly the same
strategy as Distributed Logging & Metrics Pipeline's tiered storage)
A time-series database is optimized specifically around this write-heavy, append-only access pattern — using storage engines suited to sequential writes (an LSM-tree-style design, from Data Warehouse / Analytics Storage Layer later in this chapter, fits this profile well) rather than a general-purpose OLTP database optimized for point updates. Downsampling (storing full-resolution data only briefly, then progressively coarser aggregates as data ages) is what keeps long-term storage bounded — nobody needs per-second granularity for a metric from 8 months ago, but the AGGREGATE trend over that period is still valuable to retain cheaply.
WRONG: for each of 10,000 alert rules, independently query/scan the relevant
metric data every evaluation cycle — massive REPEATED read load
RIGHT: as metric data arrives (or on a shared evaluation pass), match it against
ALL relevant rules in one pass, indexed by which metric/label combinations
each rule actually cares about
Naively evaluating each alert rule as an independent query against the full metric store doesn't scale — at high rule counts, this means massively redundant read load, much of it re-scanning largely overlapping data. A more scalable design indexes alert rules by the SPECIFIC metric streams they depend on (similar in spirit to Search Engine's inverted index, but mapping metric-name→interested-rules instead of term→document) — as new data arrives for a given metric, only the rules actually watching THAT metric are evaluated against it, rather than every rule re-scanning everything.
Q: How does this system avoid becoming a bottleneck itself under a traffic spike that generates MORE metrics (ironically, right when monitoring matters most)? A: The ingestion path needs to be horizontally scalable independently of the query/alerting path (similar separation to Distributed Logging & Metrics Pipeline's collection-vs-processing split), and ideally buffered through a message queue absorbing bursts — a monitoring system that falls over under exactly the load spike it's supposed to be observing is a real, damaging failure mode worth designing against explicitly.
Q: Should alert evaluation happen on every single incoming data point, or on a periodic schedule? A: Often a hybrid — critical, low-latency alerts might evaluate on each relevant incoming data point; less urgent rules can batch-evaluate on a periodic cycle (every 30-60 seconds) — trading evaluation freshness for reduced evaluation overhead, tuned per rule's actual urgency, similar to Alerting Strategy's symptom-vs-cause-based urgency distinction.
Q: How does downsampling interact with alert rules that need recent, full-resolution data? A: Downsampling should only apply to data past a defined AGE threshold — alert evaluation always operates against the still-full-resolution recent window, and only historical dashboard queries/trend analysis touch the downsampled, coarser older data; conflating the two would risk alert rules missing a genuine short-lived spike that got smoothed away by premature downsampling.
Q: Is there a risk in indexing alert rules by metric dependency, similar to any indexing strategy? A: Yes — the index itself needs to be kept in sync as rules are added/modified/removed, and a rule with a broad or poorly-scoped dependency (matching a huge number of metric streams) undermines the indexing benefit, similar to a hot-key problem in a database index — well-scoped rule definitions are part of what makes this indexing strategy actually effective at scale.