Design a Metrics Collection Library

Three metric types, one registry

interface Metric { String name(); }

class Counter implements Metric {
    AtomicLong value = new AtomicLong(0);
    void increment() { value.incrementAndGet(); } // only ever goes UP
}

class Gauge implements Metric {
    Supplier<Double> valueSupplier; // reads a CURRENT value on demand, can go up or down
    double value() { return valueSupplier.get(); }
}

class Histogram implements Metric {
    List<Double> samples = new CopyOnWriteArrayList<>(); // or a more compact summary structure at scale
    void record(double value) { samples.add(value); }
    double percentile(double p) { /* sorted samples, interpolate at percentile p */ }
}

class MetricsRegistry {
    Map<String, Metric> metrics = new ConcurrentHashMap<>();
    void register(Metric metric) { metrics.put(metric.name(), metric); }
}

These three types map directly onto real metric semantics from Metrics & Monitoring: Counter (monotonically increasing — a request count, never decrements), Gauge (a current point-in-time value that can go up or down — active connection count, queue depth), Histogram (a DISTRIBUTION of observed values — request LATENCY specifically, where you care about the shape, not just an average, directly connecting to Metrics & Monitoring's percentile-vs-average discussion).

Sampling for histograms at high volume

class ReservoirSamplingHistogram implements Metric {
    double[] reservoir = new double[1000]; // fixed-size sample, NOT every single value
    int count = 0;

    void record(double value) {
        count++;
        if (count <= reservoir.length) {
            reservoir[count - 1] = value;
        } else {
            int j = ThreadLocalRandom.current().nextInt(count);
            if (j < reservoir.length) reservoir[j] = value; // reservoir sampling: replace with decreasing probability
        }
    }
}

Recording EVERY single value for a high-volume histogram (millions of requests/sec) is both a memory problem (storing every value) and a computation problem (percentile calculation over an ever-growing list). Reservoir sampling maintains a FIXED-SIZE representative sample — new values replace existing reservoir entries with DECREASING probability as more values arrive, which is what keeps the sample statistically representative of the full stream's distribution even though only a small, bounded subset is actually stored. This is the practical answer to "discuss sampling for histograms at high volume vs recording every single value" — a small, well-chosen sample gives a statistically sound percentile estimate at a fraction of the memory/compute cost of tracking everything.

Mapping directly onto Micrometer's actual design

Spring Boot's Micrometer (already introduced in Metrics & Monitoring) exposes exactly this same three-type model — Counter, Gauge, and Timer/DistributionSummary (Micrometer's histogram-equivalents) — registered against a MeterRegistry that plays the identical role as this exercise's MetricsRegistry. Building this from scratch is what turns @Timed and meterRegistry.counter(...) from magic annotations into a concrete, understood mechanism.

Tags (dimensions) and the cardinality trap

Real metrics aren't just http.requests = 1523. They carry tags: http.requests{method=GET, uri=/orders/{id}, status=200}. Tags let you slice one metric many ways (error rate per endpoint, latency per region). In the registry, the tags become part of the metric's identity:

record MeterId(String name, SortedMap<String, String> tags) {
    static MeterId of(String name, String... kv) {
        if (kv.length % 2 != 0) throw new IllegalArgumentException("tags must be key/value pairs");
        SortedMap<String, String> t = new TreeMap<>();                  // sorted: {a=1,b=2} == {b=2,a=1}
        for (int i = 0; i < kv.length; i += 2) t.put(kv[i], kv[i + 1]);
        return new MeterId(name, Collections.unmodifiableSortedMap(t));
    }
}

final class MetricsRegistry {
    private final ConcurrentMap<MeterId, Counter> counters = new ConcurrentHashMap<>();
    private final int maxSeriesPerName;
    private final ConcurrentMap<String, AtomicInteger> seriesCount = new ConcurrentHashMap<>();

    MetricsRegistry(int maxSeriesPerName) { this.maxSeriesPerName = maxSeriesPerName; }

    Counter counter(String name, String... tags) {
        MeterId id = MeterId.of(name, tags);
        Counter existing = counters.get(id);
        if (existing != null) return existing;                               // fast path: no allocation
        if (seriesCount.computeIfAbsent(name, n -> new AtomicInteger()).incrementAndGet() > maxSeriesPerName) {
            seriesCount.get(name).decrementAndGet();
            return Counter.NOOP;                                             // refuse to explode memory
        }
        return counters.computeIfAbsent(id, k -> new Counter());
    }
}

Every unique combination of tag values is a separate time series, stored and exported separately. That's the cardinality trap:

• uri=/orders/{id} (a route template) gives a handful of series.
• uri=/orders/98231 (the raw path) gives one series per order, and unbounded memory in your app and your monitoring system.
• Tagging by userId, orderId, email or full URL has the same problem.

Rules of thumb:

• Tag values must come from a small, bounded set: HTTP methods, status classes (2xx, 5xx), route templates, regions, outcome (success/failure).
• Put high-cardinality details in logs or traces, not metrics.
• Guard at the library level, as above, with a per-name series limit that turns into a no-op and a warning, rather than a memory leak. Micrometer offers the same protection through MeterFilter.maximumAllowableTags.

Fast counters under contention

A single AtomicLong updated by dozens of threads becomes a contention point, because every increment competes for the same cache line. LongAdder spreads increments across internal cells, and sums them only when read:

final class Counter {
    static final Counter NOOP = new Counter() { @Override void increment(long n) {} };
    private final LongAdder adder = new LongAdder();
    void increment() { increment(1); }
    void increment(long n) { adder.add(n); }
    long count() { return adder.sum(); }      // reads are rare (once per scrape), writes are constant
}

That's the right trade-off for metrics: writes happen on every request, and reads only when the exporter scrapes, perhaps every 15 seconds.

Histograms with fixed buckets

Reservoir sampling estimates percentiles in one process, but percentiles can't be averaged across instances: the mean of each server's p99 isn't the fleet's p99. The approach Prometheus popularized uses fixed bucket boundaries with cumulative counts, which can be summed across instances, and percentiles are then estimated from the combined buckets:

final class BucketHistogram {
    private final double[] bounds;                 // e.g. 5, 10, 25, 50, 100, 250, 500, 1000 ms
    private final LongAdder[] counts;              // counts[i] = observations <= bounds[i]; last = +Inf
    private final DoubleAdder sum = new DoubleAdder();

    BucketHistogram(double... bounds) {
        this.bounds = bounds.clone();
        this.counts = new LongAdder[bounds.length + 1];
        for (int i = 0; i < counts.length; i++) counts[i] = new LongAdder();
    }

    void record(double value) {
        int i = Arrays.binarySearch(bounds, value);
        int bucket = i >= 0 ? i : -i - 1;          // the first bound >= value
        counts[bucket].increment();
        sum.add(value);
    }
}

The trade-off: accuracy depends on the bucket choice, so choose boundaries around your SLOs (if the target is 300 ms, have a bucket boundary at 300). Recording is O(log buckets), and memory is fixed regardless of traffic.

Follow-up questions this topic invites — and their answers

Q: Why can't a Gauge simply be implemented the same way as a Counter, with increment/decrement methods? A: It COULD be, but the Supplier<Double>-based approach shown (reading a current value ON DEMAND rather than maintaining running state) is often preferable for values that are already tracked elsewhere (e.g. connectionPool.getActiveCount()) — it avoids DUPLICATING state that already exists in the thing being measured, reading it fresh each time the metric is scraped instead.

Q: Does reservoir sampling bias the percentile estimate in any way? A: A correctly-implemented reservoir sampling algorithm (as shown, with the replacement probability decreasing appropriately as count grows) gives an UNBIASED uniform random sample of the full stream — the key correctness property is that every value seen so far has an EQUAL probability of being in the final reservoir, which the specific replace-with-probability-length/count formula guarantees.

Q: How would you export these in-memory metrics to an external system like Prometheus? A: A separate EXPORTER component would periodically read all registered metrics' current state (iterating the MetricsRegistry) and format them per Prometheus's expected text exposition format at a scrape endpoint (/actuator/prometheus, from Metrics & Monitoring) — the collection/storage layer (this exercise) and the export/format layer are cleanly separable concerns.

Q: Should Counter use a plain long with synchronization, or AtomicLong as shown? A: AtomicLong is the correct choice for a counter incremented from many concurrent threads — it provides lock-free, thread-safe increments via compare-and-swap (the same underlying mechanism discussed in this course's concurrency-focused lessons), meaningfully cheaper under contention than a synchronized block around a plain long.

Q: Why is tagging metrics with user IDs a bad idea? A: Each distinct tag combination creates a separate time series. User or order IDs create an unbounded number of series, which exhausts memory in the application and the monitoring backend, and makes queries slow and expensive. Use logs or traces for per-entity detail.

Q: Why use LongAdder instead of AtomicLong for counters? A: Under heavy concurrent increments, AtomicLong threads contend on one memory location. LongAdder spreads updates across cells and sums them on read, which is much faster when writes vastly outnumber reads, as they do for metrics.

Q: Why can't you average p99 latencies across servers? A: Percentiles aren't additive. The fleet-wide p99 depends on the full distribution, not on each server's summary. Bucketed histograms solve this, because bucket counts can be summed across servers before the percentile is estimated.