Versioned config with environment-specific overrides, an audit trail, push vs pull delivery and avoiding a thundering herd on restart, and secret rotation without a full service redeploy.
Published September 23, 2026
Design a Config Management Client (LLD) designed the CLIENT library consuming configuration; Kubernetes' Config & Secrets covers a specific platform's built-in mechanism. This case designs the CENTRALIZED SERVICE itself — the system a config client actually talks to, at organization scale.
Design a centralized service that stores versioned configuration and encrypted secrets, serves them to potentially thousands of service instances across many environments, and supports secret rotation without requiring affected services to be redeployed.
Functional: store configuration values with environment-specific overrides (a db.host that differs between staging and production); version every change with a full audit trail; store secrets encrypted, never in plaintext at rest; support rotating a secret's value.
Non-functional: serve potentially thousands of concurrent client instances without becoming a bottleneck; changes should propagate to clients within a bounded, acceptable delay; the service itself must be highly available (a huge number of services depend on it to even start up).
Pull: each client periodically POLLS the config service for updates
— simple, but a large fleet all polling on the SAME interval creates
synchronized load spikes, and especially dangerous right after the
CONFIG SERVICE ITSELF restarts, when every client's cache might be
stale simultaneously and all poll at once ("thundering herd")
Push: the config service NOTIFIES clients of changes (via a long-lived
connection or a pub/sub mechanism) — avoids synchronized polling, but
requires the service to maintain many open connections
This is the exact thundering-herd concern named explicitly in the requirements: if every client independently polls on a fixed interval, a config-service restart or a widespread client restart (a platform-wide deployment) can synchronize thousands of clients into hammering the config service simultaneously right as it's recovering — the worst possible timing. Mitigations directly parallel earlier lessons: JITTERED poll intervals (Design a Retry Mechanism with Backoff's jitter reasoning, applied to polling instead of retries) spread out the load; a PUSH-based model avoids the problem more fundamentally by never having clients poll on a shared schedule at all.
WRONG: rotating a database password requires updating it in the config
service AND redeploying every service that uses it, to pick up the change
— slow, disruptive, and a real risk of downtime during the redeploy window
RIGHT: services subscribe (push) or periodically re-fetch (pull) secret
values at RUNTIME, so a rotation just means: config service updates the
value, EVERY subscribed client picks it up within the propagation window,
no redeploy needed
This is the direct payoff of the config client design covered in Design a Config Management Client's Observer-pattern approach — a service that reads its secrets ONCE at startup and never again cannot support this at all (rotation genuinely requires a redeploy for such a service); a service using a live-updating config client picks up a rotated secret automatically. Secret rotation being possible WITHOUT redeployment is a genuine, valuable capability this system needs to be explicitly designed to support, not an incidental side effect.
config_key: "db.host"
base value: "db.internal"
staging override: "db-staging.internal"
production override: "db-prod.internal"
history: [ {value, changedBy, changedAt}, ... ] — full audit trail, append-only
Storing a BASE value plus explicit per-environment OVERRIDES (rather than fully independent config sets per environment) means most config values are defined once and inherited everywhere, with only genuinely environment-specific values needing an explicit override — reducing duplication and the risk of environments silently drifting apart on values that were never meant to differ. The audit trail (append-only history of every change, who made it, when) directly mirrors Payment — Requirements' auditability principle, applied to configuration changes — critical for tracing back "when did this value change, and who changed it" during an incident investigation.
Q: What happens to a service that starts up and CANNOT reach the config service at all (not just a stale value, a total outage)? A: This is a genuinely critical availability concern, since the config service is a dependency for essentially EVERY other service starting up — a common mitigation is a LOCAL FALLBACK CACHE (persisted to local disk, not just in-memory) that a service can boot from if the config service is entirely unreachable, trading freshness for the ability to start at all during a config-service outage.
Q: How is encryption of secrets at rest different from just base64-encoding, per Kubernetes' Secret limitation covered in Config & Secrets? A: Genuine encryption uses an actual cipher with a securely-managed key (often via a dedicated key-management service), making the stored value cryptographically protected, not merely encoded — this is exactly the gap Kubernetes' base64-only default has, and a purpose-built centralized secrets system should get this right by design, not require a separate opt-in cluster setting.
Q: Should configuration and secrets be stored in the SAME system, or kept separate? A: Many real systems DO combine them (as this design does) for a single operational surface, but apply stricter access controls and encryption specifically to secret VALUES — the underlying versioning/audit/environment-override mechanics are genuinely shared infrastructure, even though secrets need meaningfully stronger protection than ordinary config values.
Q: How would you test that secret rotation actually works correctly before relying on it during a real incident? A: Regularly, deliberately ROTATING non-critical secrets on a routine schedule (not just when a rotation is urgently needed) is the standard way to build genuine confidence the mechanism works — an untested rotation path that's only ever exercised during a real emergency is a real risk, similar in spirit to chaos engineering's philosophy of proactively testing failure/recovery paths rather than discovering gaps during an actual incident.