Ingesting high-volume events and serving near-real-time dashboards via stream processing and windowed aggregation, and why pre-aggregation is what keeps dashboard queries fast without scanning raw events.
Published September 23, 2026
Design a system that ingests high-volume events (page views, clicks, transactions) and serves near-real-time aggregate dashboards ("views per minute," "conversion rate over the last hour") — updated continuously as new events arrive, not on a slow overnight batch schedule.
Functional: ingest a high-volume event stream; compute aggregate metrics (counts, sums, averages) over sliding/tumbling time windows; serve dashboard queries with low latency. Non-functional: dashboard data should reflect events from at most a few seconds to a couple of minutes ago (near-real-time, not literally instant); handle high event throughput without falling behind; dashboard QUERIES must stay fast regardless of how much raw event history has accumulated.
[Event Producers] → [Message Queue] (Message Queue System) → [Stream Processor]
(e.g. Kafka Streams) → computes windowed aggregates continuously
Tumbling window: non-overlapping fixed windows — "count per exact 1-minute bucket"
Sliding window: overlapping windows — "count over the trailing 5 minutes, updated every second"
A stream processor (Kafka Streams, Flink, or similar) consumes the event stream continuously and maintains RUNNING aggregates over time windows, rather than periodically re-scanning raw events in batch. This is the architectural answer to "near-real-time": aggregates update incrementally as each new event arrives, with the aggregation state itself maintained in the stream processor (often backed by a local, fast key-value store per processing node), not recomputed from scratch.
Batch processing (Batch Processing System, later in this chapter) periodically processes a full chunk of accumulated data — simpler to reason about, easier to reprocess correctly if something goes wrong, but introduces real LATENCY (results reflect data as of the last batch run, potentially hours old). Stream processing trades that simplicity for near-real-time freshness, at the cost of genuinely harder correctness concerns (handling out-of-order events, exactly-once aggregation semantics under retries) — the right choice depends on whether the dashboard's actual USE CASE needs near-real-time freshness (an operations dashboard during an active incident) or can tolerate batch latency (a daily business-metrics report).
WRONG: dashboard query scans millions/billions of raw events every time it's viewed
— gets slower and slower as event volume grows, and repeats the same computation
for every viewer
RIGHT: the stream processor maintains PRE-COMPUTED aggregate values (already summed/
counted per time bucket), and the dashboard query just READS those small,
already-aggregated numbers
This is the same principle as Search Engine's inverted index doing the expensive work OFFLINE, ahead of query time — pre-aggregation means the expensive computation (scanning and summing potentially billions of raw events) happens ONCE, continuously, in the stream processor, and the dashboard's actual READ path only ever touches small, already-computed aggregate values (stored in a fast key-value or time-series store) — keeping dashboard query latency roughly constant regardless of how much raw event volume has accumulated underneath.
Q: How do you handle events that arrive OUT OF ORDER (e.g. a delayed event that should have counted in an earlier window)? A: Stream processors typically support a configurable 'watermark' — a policy for how long to keep a window open accepting late-arriving events before finalizing it, trading a bit of additional latency for correctness against realistic network/processing delays; events arriving after the watermark has passed are either dropped or routed to a separate late-data handling path, a genuine, explicit design decision rather than an accident.
Q: Does pre-aggregation limit what questions the dashboard can answer later? A: Yes, meaningfully — pre-aggregating specifically by 'count per minute per category' means you can't later ask a genuinely different aggregation (like 'count per user' if user wasn't part of the pre-aggregation key) without reprocessing the raw event stream again; this is why raw events are typically still retained (per Distributed Logging & Metrics Pipeline's tiered storage) even though the dashboard itself only reads pre-aggregated values.
Q: How would this design change if 'real-time' genuinely meant sub-second, not a few seconds? A: Sub-second freshness pushes harder on every part of the pipeline — smaller, more frequent windows, lower-latency message queue configuration (tuning producer batching/acks trade-offs from Messaging Technology Choices toward latency over throughput), and a push-based (WebSocket) dashboard update mechanism rather than the dashboard client periodically polling for the latest aggregate.
Q: Is stream processing infrastructure worth the complexity for a lower-traffic system? A: Often not — for a system with modest event volume where a few minutes of batch latency is genuinely acceptable, a simpler periodic batch job computing the same aggregates is meaningfully easier to build, debug, and reason about; stream processing's complexity is worth paying specifically when the near-real-time freshness requirement is a genuine, stated business need, not a default best-practice to reach for regardless of actual need.