Observability, Logging, Alerting & Audit Systems — Interview Questions

How to use this lesson

Split these questions into two kinds:

• using observability (what to measure and alert on);
• building the platform (ingestion pipelines, storage, retention, cost).

Mention SLOs, correlation between signals (trace IDs in logs), cardinality, and cost control. Those are the senior signals.

Q1. What is observability? How do you monitor a distributed system?

Short answer:

• Observability is the ability to understand a system's internal state from its external outputs, including failure modes you didn't predict. Monitoring watches for known conditions (dashboards and alerts); observability lets you ask new questions during an incident.
• The signals:
  • metrics (cheap aggregates over time: RED/USE, business KPIs);
  • logs (detailed, structured events);
  • traces (the path of one request across services, with timings);
  • increasingly profiles (continuous profiling), and events (deploys, config changes).
• Monitoring a distributed system:
  • instrument everything the same way: OpenTelemetry or Micrometer; trace context propagation across HTTP, gRPC and Kafka (the W3C traceparent header);
  • correlate the signals: the trace ID in every log line, and exemplars linking metrics to traces;
  • golden signals per service (latency, traffic, errors, saturation), plus dependency metrics (database, cache, broker, external APIs), consumer lag, and business metrics (bookings per minute);
  • SLOs per user journey (for example, 99.9% of searches under 500 ms), with alerts on error-budget burn;
  • synthetic monitoring (probes of key journeys) and real-user monitoring;
  • service maps from traces; deployment markers on dashboards.
• The stack examples: Prometheus plus Grafana (or Mimir and Thanos for scale), Loki or ELK/OpenSearch for logs, Tempo or Jaeger for traces, and the OpenTelemetry Collector. Commercial options include Datadog, New Relic and Dynatrace.

Learn it in depth → Metrics & Monitoring

Q2. What is an alerting strategy? Design an alerting system (a monitoring and alert system).

Short answer:

• Strategy (what to alert on):
  • alert on symptoms that affect users (SLO burn rate, error rate, latency, failed bookings), not on every cause (CPU at 80%). Causes go on dashboards;
  • every page must be actionable, urgent, and linked to a runbook; otherwise it's a ticket or dashboard item;
  • multi-window, multi-burn-rate SLO alerts (a fast burn pages immediately; a slow burn creates a ticket);
  • severity levels and routing (page, chat, ticket); deduplication and grouping; maintenance-window silencing;
  • review alert quality regularly: delete noisy alerts, tune the thresholds, and track pages per on-call shift.
• System design (for building it):
  1. Collection: metrics scraped or pushed (Prometheus, the OTel Collector), plus logs and events.
  2. Evaluation: rule evaluation on time-series data (PromQL rules every 15–60 seconds), with for: durations to avoid flapping.
  3. Alert manager: dedupe, group related alerts, inhibit dependent alerts (for example, suppress service alerts when the whole cluster is down), silence, route by team or severity, escalate if not acknowledged.
  4. Notification channels: PagerDuty, Opsgenie, Slack, SMS, email, with retries.
  5. Incident tooling: acknowledgement, incident channels, runbooks, and a status page.
• Scale and reliability: shard the evaluation by tenant or metric; make the alerting pipeline more available than the systems it monitors (a separate failure domain); add dead man's switch alerts that fire if the monitoring itself stops.

# Prometheus: page when 1h error-budget burn rate > 14.4x for a 99.9% SLO
- alert: CheckoutErrorBudgetBurnFast
  expr: |
    sum(rate(http_server_requests_seconds_count{uri="/bookings",status=~"5.."}[1h]))
      / sum(rate(http_server_requests_seconds_count{uri="/bookings"}[1h])) > (14.4 * 0.001)
  for: 2m
  labels: { severity: page, team: booking }
  annotations: { runbook: "https://runbooks/booking/errors" }

Learn it in depth → Design a Monitoring & Alerting System

Q3. Design a centralized logging platform (a logging pipeline with ELK).

Short answer:

• Producers: services log structured JSON to stdout (Logback with a JSON encoder), with the trace ID, span ID, service, version, environment, and tenant. No personal data or secrets (mask them at source).
• Collection: node agents (Fluent Bit, Filebeat, or the OTel Collector as a DaemonSet in Kubernetes) tail the container logs and add metadata (pod, namespace).
• Buffer: Kafka between the agents and the indexers absorbs spikes and lets you replay if indexing falls behind.
• Processing: Logstash or ingest pipelines (or Vector, or OTel processors) parse, enrich and redact, and route (drop debug logs in production; sample noisy logs).
• Storage and search:
  • Elasticsearch or OpenSearch, with index lifecycle management (hot → warm → cold → delete; daily indices or data streams; rollover by size);
  • or Loki (indexes only labels, much cheaper, but grep-like querying);
  • archive to S3 for long-term compliance.
• Access: Kibana or Grafana; RBAC per team; audit of who searched what, for sensitive data.
• Cost controls: log levels, sampling, short retention for debug-level logs, rate limits per service, and avoiding high-cardinality fields that aren't needed.

Learn it in depth → Centralized Logging

Q4. Design a metrics dashboard (and an API analytics tracking system).

Short answer:

• A metrics dashboard (operational):
  • Prometheus-compatible metrics from Micrometer, stored in Prometheus (short-term) plus Thanos, Mimir or VictoriaMetrics (long-term, global view, downsampling);
  • Grafana dashboards per service (RED, JVM, pools, dependencies) and per journey (SLOs), provisioned as code, with deployment annotations;
  • cardinality control: never use user IDs or raw URLs as labels.
• API analytics tracking (product and business):
  • the gateway or a filter emits an access event per request (API key or client, tenant, endpoint template, status, latency, bytes, timestamp) asynchronously to Kafka, never blocking the request;
  • a stream processor aggregates per minute, hour and day (per client, endpoint, status);
  • store it in a columnar/OLAP database (ClickHouse, BigQuery, Druid) for flexible queries;
  • dashboards for usage, error rates per client, latency percentiles, quota usage and billing, and deprecated-API usage (who still calls v1);
  • privacy: hash or strip personal data and IP addresses, according to policy.

Q5. How do you ensure audit logging? Design an audit logging system.

Short answer:

• What an audit log is: a record of who did what, to which resource, when, from where, and with what result, for security, compliance (SOX, PCI DSS, GDPR, HIPAA) and dispute resolution. It's different from debug logs: it's complete, immutable, and retained for a long time.
• Capture:
  • business-level audit events (BOOKING_CANCELLED by user X, reason Y), emitted from the application layer (an AOP aspect, or explicit calls), in the same transaction through an outbox, so the events can't be lost;
  • entity history where needed (Hibernate Envers, or CDC with before and after values);
  • authentication and admin actions (logins, role changes, configuration changes).
• The event schema: event ID, timestamp (UTC), actor (user or service, and the impersonator if any), action, resource type and ID, before and after (or a diff), request and trace ID, source IP and user agent, outcome, and tenant.
• Storage and integrity:
  • append-only storage (no updates or deletes through the application);
  • tamper evidence: hash chaining (each record includes the hash of the previous one) or signed batches; WORM storage (S3 Object Lock);
  • a separate access-controlled store;
  • retention as the regulations require.
• Access: a search UI for auditors, with its own access being audited; exports for investigations.
• Privacy: minimise the personal data in audit records; pseudonymise it where possible, and handle erasure requests consistently with the legal retention duties.

Learn it in depth → Design an Audit Logging Framework

Q6. Follow-up: how do you control observability costs?

Short answer:

• Metrics: limit label cardinality; drop unused metrics; downsample the long-term storage.
• Logs: use the right levels, sample the high-volume success logs, keep short retention for verbose logs, and use cheaper tiers (Loki, S3).
• Traces: tail-based sampling (keep all errors and slow traces, and a small percentage of the rest).
• Review usage per team (showback).

Follow-up questions this topic invites — and their answers

Q: What's the difference between an SLI, an SLO and an SLA? A: An SLI is a measured indicator (the fraction of requests under 300 ms). An SLO is the internal target for it (99.9% over 30 days). An SLA is the external contract, with penalties, usually looser than the SLO.

Q: What is an error budget? A: The allowed unreliability (1 − SLO). If it's being spent too fast, the team prioritises reliability work over features; if there's plenty left, it can take more release risk.

Q: Why is high cardinality a problem for metrics? A: Each unique label combination is a separate time series. User IDs or raw paths can create millions of series, exploding memory, storage and query cost. Put high-cardinality detail in logs or traces instead.

Q: Head-based or tail-based trace sampling? A: Head-based decides at the start (cheap, but can miss rare errors). Tail-based decides after the trace completes (keeps errors and slow requests), but needs a collector that buffers spans.