Designing the observability pipeline itself, at platform scale — log/metric collection agents, buffering and backpressure, the hot/warm/cold storage tiers, and why the pipeline must degrade gracefully rather than ever blocking the services it observes.
Published September 23, 2026
Centralized Logging, Distributed Tracing, and Metrics & Monitoring covered USING this kind of infrastructure from an application developer's perspective. This lesson designs the pipeline itself, at platform scale — the system that collects, transports, and stores observability data from thousands of service instances.
Design a pipeline that collects logs, metrics, and traces from thousands of service instances across a platform, transports them reliably, and makes them queryable — without ever meaningfully impacting the performance of the services being observed.
Functional: collect structured logs, metrics, and traces from every service instance; make recent data queryable with low latency; retain data for a defined period, tiered by age. Non-functional: the pipeline must never materially slow down or block the services generating the data (observability is a side effect, never a critical-path dependency); handle massive data volume (far exceeding the actual application traffic it's observing); survive partial pipeline failures without losing data disproportionately.
[Application] → writes to a LOCAL AGENT (e.g. Fluentd, a metrics sidecar)
│ buffers locally, batches, compresses
▼
[Aggregation Layer] (e.g. Kafka — see Message Queue System)
│
▼
[Processing/Indexing] → [Storage tiers]
Application code should NEVER make a direct, synchronous network call to a remote logging/metrics backend for every single log line or metric point — that would tie application request latency to the observability pipeline's availability and latency, precisely backwards from the goal (observability should never be a critical-path dependency, directly echoing Health Checks' essential-vs-non-essential framing, applied here to the pipeline's OWN role). Instead, a local agent on each host buffers and batches data, sending it asynchronously — the application only ever writes to a fast local buffer, never blocks on the network.
A message queue (Message Queue System) sits between collection agents and the processing/storage layer specifically to absorb bursts — if processing/indexing temporarily falls behind (a traffic spike, a downstream outage), the queue buffers the backlog rather than data being dropped or, worse, agents blocking and backing up into the applications they're observing. This is the same backpressure-absorption role a queue plays in many designs in this course (video transcoding, distributed task scheduling) — decoupling producers from consumers in both time and load.
Hot (last few hours/days): fast, expensive storage — optimized for interactive queries
during active incident investigation
Warm (last few weeks): cheaper, somewhat slower — less frequent access
Cold (long-term, compliance): cheapest object storage — rarely queried, retained for audit/history
Given the sheer volume (observability data routinely exceeds the actual application traffic volume it's observing, sometimes by a large multiple), storing everything in the fastest tier indefinitely is prohibitively expensive — a tiered retention policy, automatically moving data to cheaper storage as it ages (and eventually deleting or archiving it per compliance requirements), is standard and necessary. This is a direct extension of Back-of-Envelope Estimation's storage math applied specifically to observability data's own, often-underestimated volume.
Beyond application-level trace sampling (Distributed Tracing), the pipeline itself often downsamples metrics as they age — keeping full-resolution data (every second) only for the hot tier, and aggregating to coarser resolution (every minute, every hour) for warm/cold tiers, since a 6-month-old incident investigation rarely needs per-second granularity. This trades historical query precision for a large storage cost reduction, applied progressively as data ages rather than uniformly.
Q: What happens if the local agent's buffer fills up faster than it can send data (a sustained high-volume burst)? A: The agent needs an explicit policy: drop the oldest buffered data (favoring recency), drop new data (favoring not losing history already buffered), or apply backpressure to the application itself as an absolute last resort — dropping is generally preferred over ever blocking the application, since observability data loss is recoverable/acceptable in a way that degrading the actual service is not.
Q: Why not just write logs directly to the storage/indexing system, skipping the aggregation queue? A: Without the queue absorbing bursts, a spike in log volume (e.g. every service logging heavily during a platform-wide incident — precisely when observability matters MOST) could overwhelm the indexing layer directly, causing exactly the data loss or backpressure the pipeline needs to avoid at exactly the worst possible time to lose it.
Q: How does this pipeline itself get monitored — who watches the watcher? A: The pipeline needs its own basic, independent health signals (agent uptime, queue lag, indexing throughput) — often deliberately kept SIMPLE and separate from the main pipeline (sometimes literally a different, smaller monitoring path) specifically so that a failure in the primary observability pipeline doesn't also blind the team to that very failure.
Q: Does every service need the SAME retention policy, or can it vary? A: It commonly varies — audit-relevant logs (security events, payment transitions — see Payment — Requirements' auditability point) often need longer, compliance-driven retention than routine debug logs, which argues for retention policy being configurable per log category/source rather than one uniform platform-wide setting.