Design a Notification Service

Design a Notification Service — the 45-minute interview walkthrough

This lesson is the interview version of the design: what to say, in what order, and which decisions to defend, so you can deliver it within a normal HLD interview. The full reference design, with capacity numbers, API and data model in detail, is the case study Design a Notification Service in this same chapter.

A notification service is the one place every other service calls when it needs to tell a user something ("your order shipped", "your OTP is 481233", "Asha liked your post"). It decides whether to send (user preferences, rate limits), how (push, email, SMS), when (now, or batched), and makes sure the message arrives once, even though every provider in the chain can fail.

Minutes 0–5: clarify requirements

Ask before drawing anything. The answers change the design:

• Channels? Push (iOS APNs, Android FCM), email, SMS. In-app inbox too?
• Volume and spikes? Assume ~10M notifications/day (~115/s average), with spikes of 50–100× when a marketing campaign or a breaking event fans out to millions.
• Latency by type? An OTP or security alert must arrive in seconds. "Order shipped" within a minute is fine. Marketing can take hours.
• Guarantees? "At least once, deduplicated" is the realistic target. Exactly-once across external providers isn't achievable.
• User control? Per-channel and per-type opt-out, quiet hours, and legal unsubscribe for marketing email.

State the priorities you'll optimize for: no lost critical notifications, no duplicates the user notices, and marketing volume must never delay OTPs.

Minutes 5–15: the high-level design

 Order / Payment / Social services
          │  POST /notifications  { userId, type, data, idempotencyKey, priority }
          ▼
 ┌────────────────────┐   validates, applies preferences + rate limits,
 │  Notification API  │   renders the template, stores a record
 └─────────┬──────────┘
           │ publish by priority
   ┌───────┼────────────────┐
   ▼       ▼                ▼
 [critical] [transactional] [bulk]        ← separate queues/topics per priority
   │          │                │
   ▼          ▼                ▼
 Channel workers: push-worker │ email-worker │ sms-worker
           │
           ▼
 APNs / FCM · SES / SendGrid · Twilio      ← external providers (can fail, can rate-limit you)
           │
           ▼
 Delivery status + provider callbacks → status store → analytics

Walk the interviewer through one request end to end: the caller sends an event with an idempotency key. The API checks preferences, renders the text, saves a record as PENDING, and enqueues it on the right priority queue. A channel worker picks it up, calls the provider, and records SENT or schedules a retry.

Why a queue in the middle? It absorbs spikes (a campaign of 5 million emails becomes a backlog, not an outage). It decouples callers from slow providers (the Order service's request returns immediately). And it lets each channel scale its workers independently.

Minutes 15–35: the deep dives interviewers pick

1. Duplicates and retries

Duplicates come from two places: producers retrying (the Order service didn't get a response and calls again) and workers retrying (the provider timed out but may actually have sent). The answer has two layers:

• An idempotency key per logical notification, supplied by the caller (for example order-123:shipped). The API stores it with a unique constraint, so a repeated request returns the existing record instead of creating another.
• The worker marks the send before acknowledging the queue message, and on retry checks the stored status first.

Admit the limit: if a provider accepts a message and then times out, you can't know whether it was delivered. You retry, and occasionally the user gets two. The job is to make that rare and harmless, not impossible.

2. Priority isolation

If OTPs and a 5-million-email campaign share one queue, the OTP waits behind the campaign. Separate queues (or topics) per priority, with their own worker pools, guarantee critical traffic a fast lane. Bulk sends are also throttled deliberately so they don't hit provider rate limits and get your whole account throttled, OTPs included.

3. Provider failures

Every external provider will have outages and rate limits:

• Retries with exponential backoff and jitter, up to a limit, then a dead-letter queue for inspection.
• A circuit breaker per provider, so a failing provider is skipped quickly instead of tying up workers.
• A secondary provider for critical channels (a second SMS vendor for OTPs), switched to automatically when the breaker opens.

4. Preferences and rate limiting

Preferences are read on every send, so they live in a fast store (a cache in front of the database). Add per-user rate limits ("no more than 3 marketing pushes a day") and quiet hours. Transactional and security messages usually bypass quiet hours; marketing never does.

Minutes 35–45: trade-offs and wrap-up

Summarize the choices and what you'd do next:

• At-least-once + idempotency instead of chasing exactly-once.
• Priority queues protect critical messages at the cost of more infrastructure.
• Templates rendered in the service (not by each caller) keep branding and localization consistent, but make the service a shared dependency, so it must be highly available.
• Next steps with more time: per-provider delivery analytics (callback webhooks), digesting (turn 20 "someone liked your post" pushes into one), multi-region delivery.

Common mistakes in interviews

• Calling providers synchronously from the API request, which makes every caller as slow and fragile as the slowest provider.
• One queue for everything, so a marketing blast delays password resets.
• Claiming exactly-once delivery, which interviewers will immediately probe.
• Forgetting user preferences and unsubscribe, which are legal requirements for marketing email in many countries.

Follow-up questions this topic invites — and their answers

Q: How do you send one campaign to 50 million users without melting anything? A: Don't create 50 million requests up front. A campaign job pages through the audience in chunks and enqueues them onto the bulk queue at a controlled rate that stays under provider limits. Workers scale on queue depth. The campaign takes an hour, which is fine for marketing, and critical queues stay untouched.

Q: Push or pull for the in-app notification inbox? A: Store inbox items per user (newest first, capped), and deliver in real time over an existing WebSocket or long-poll connection when the user is online. When they're offline, the item waits in the inbox and a push notification brings them back. The inbox is the source of truth, and pushes are just nudges.

Q: How do you know a notification was actually delivered? A: Providers report asynchronously: email bounces and opens, SMS delivery receipts, APNs/FCM feedback about invalid tokens. Expose webhook endpoints for these callbacks and update the status record. Remove invalid device tokens so you stop paying to send to dead devices.

Q: How would you support multiple regions? A: Run the pipeline in each region, route users to their home region, and keep preferences replicated. Pay attention to data residency (some countries require user data to stay in-region) and to provider choice per region: SMS vendors differ in coverage and cost by country.