Design a Notification Service

Design a notification service that can deliver push notifications, SMS, and emails at scale. Notifications may be triggered by system events (payment confirmed) or marketing campaigns (10M users simultaneously). Delivery must be tracked.

Capacity Estimation

• Rate: 100M notifications/day = 1,157 notifications/sec average
• Peak (marketing blast to 10M users): 10M / 60 seconds = 166K/sec — must queue and fan out
• Delivery records: 100M × 365 × 3 bytes status = ~110 GB/year
• Templates: small, fit in cache

Architecture

[Event Sources]
  Payment Service → payment.confirmed event
  Marketing Tool → campaign.triggered event
           │
           ▼
    [Notification API Service]
      - validates request
      - checks user preferences
      - resolves template
      - publishes to channel-specific Kafka topic

           ↓
  ┌─────────────────────────────────────┐
  │         Kafka Topics                │
  │  notifications.push                 │
  │  notifications.email                │
  │  notifications.sms                  │
  └─────────────────────────────────────┘
           ↓
  [Channel Workers] (scale independently)
    Push Worker → APNs / FCM
    Email Worker → SendGrid
    SMS Worker → Twilio
           ↓
  [Delivery Tracker] → updates delivery_status in DB

Implementation focus: NotificationDispatcher class hierarchy

interface NotificationChannel { void send(Notification notification); }

class EmailChannel implements NotificationChannel {
    public void send(Notification n) { /* SMTP send */ }
}
class SmsChannel implements NotificationChannel {
    public void send(Notification n) { /* SMS gateway send */ }
}
class PushChannel implements NotificationChannel {
    public void send(Notification n) { /* APNs/FCM send */ }
}

class NotificationDispatcher {
    private final Map<ChannelType, NotificationChannel> channels;
    NotificationDispatcher(Map<ChannelType, NotificationChannel> channels) { this.channels = channels; }

    void dispatch(Notification notification) {
        NotificationChannel channel = channels.get(notification.getChannelType());
        channel.send(notification); // Strategy pattern — channel selected at dispatch time, no branching per channel
    }
}

This is Strategy Pattern applied per channel (see Strategy Pattern): NotificationDispatcher never branches on channel type internally — it looks up the right NotificationChannel implementation and delegates. Adding a new channel (e.g. Slack) means writing one new class and registering it in the map, with zero changes to NotificationDispatcher itself — the same OCP argument made throughout the pattern lessons.

Retry queue with exponential backoff, as a class

class RetryableNotificationQueue {
    private final DelayQueue<DelayedNotification> queue = new DelayQueue<>();
    private static final int MAX_RETRIES = 5;

    void enqueue(Notification notification, int attempt) {
        long delayMs = (long) (1000 * Math.pow(2, attempt)); // exponential: 1s, 2s, 4s, 8s, 16s...
        queue.put(new DelayedNotification(notification, attempt, delayMs));
    }

    void processNext() throws InterruptedException {
        DelayedNotification item = queue.take(); // blocks until an item's delay has elapsed — see BlockingQueue family
        try {
            dispatcher.dispatch(item.notification);
        } catch (DeliveryException e) {
            if (item.attempt < MAX_RETRIES) enqueue(item.notification, item.attempt + 1);
            else deadLetterQueue.add(item.notification); // give up — route to a dead-letter queue for manual inspection
        }
    }
}

DelayQueue (a BlockingQueue variant not covered in Concurrent Utilities & Coordination's family list, specifically designed for "not ready until time T" semantics) is the natural fit here — it only releases an item via take() once its delay has elapsed, which is exactly the shape exponential backoff needs, without any manual timer/scheduling code.

Template design for notification content across channels

abstract class NotificationTemplate {
    final String render(NotificationContext ctx) { // Template Method — fixed sequence, customizable steps
        String subject = renderSubject(ctx);
        String body = renderBody(ctx);
        return formatForChannel(subject, body);
    }
    protected abstract String renderSubject(NotificationContext ctx);
    protected abstract String renderBody(NotificationContext ctx);
    protected abstract String formatForChannel(String subject, String body);
}

class EmailTemplate extends NotificationTemplate {
    protected String renderSubject(NotificationContext ctx) { return "Your order " + ctx.orderId() + " shipped"; }
    protected String renderBody(NotificationContext ctx) { return "<html>...</html>"; }
    protected String formatForChannel(String subject, String body) { return subject + "\n\n" + body; }
}

class SmsTemplate extends NotificationTemplate {
    protected String renderSubject(NotificationContext ctx) { return null; } // SMS has no subject
    protected String renderBody(NotificationContext ctx) { return "Order " + ctx.orderId() + " shipped!"; } // must be terse
    protected String formatForChannel(String subject, String body) { return body; } // subject ignored entirely
}

Template Method (see Template Method) fits precisely because content generation for every channel follows the same fixed sequence (build subject, build body, format for the channel) while each channel customizes individual steps — an SMS template overriding renderSubject to return null and formatForChannel to ignore it entirely is a clean way to express "this channel doesn't have this concept" without special-casing it in the shared render() method.

Priority handling: keeping a HIGH-priority notification from queuing behind a 10M-user campaign

notifications.push.high     -- payment confirmations, security alerts: separate topic,
                                its own dedicated worker pool, never contends with campaign traffic
notifications.push.default  -- normal transactional notifications
notifications.push.bulk     -- marketing campaigns: separate topic, own worker pool,
                                explicitly throttled to protect provider rate limits

A single shared queue for all push notifications means a 10M-user marketing campaign enqueued just before a time-critical payment-confirmation notification would force that payment notification to wait behind millions of lower-priority messages — directly violating the push delivery latency requirement for the notification that matters most. The fix is priority-segregated topics/queues with SEPARATE worker pools per priority tier, not a single shared queue with an in-memory priority sort (which doesn't actually solve the problem once a worker has already pulled a large batch of low-priority messages) — this is the same principle as Alerting Strategy's symptom-based-alert urgency applied to notification delivery infrastructure.

Notification batching / digest to avoid spam

Sending every individual low-priority event (e.g. "someone liked your post") as its own immediate push notification risks notification fatigue — a direct analog of Alerting Strategy's alert-fatigue problem, applied to end users instead of on-call engineers. A digest/batching layer accumulates low-priority notifications over a short window (e.g. 15-30 minutes) and sends a single combined notification ("5 people liked your post") rather than 5 separate pushes — this requires the notification service to distinguish notification categories upfront (which are safe to batch vs which must be delivered immediately, like a security alert or a payment confirmation) and apply different delivery timing per category, not a single uniform delivery policy for every notification type.

Unsubscribe and compliance handling

Every marketing/campaign notification (email specifically, per CAN-SPAM and similar regulations) must include a functioning unsubscribe mechanism, and an unsubscribe request must be honored immediately and durably — checked against notification_prefs (already in the schema) BEFORE a campaign send, not after. A campaign that ignores a very recent opt-out (due to using a stale, pre-fetched recipient list — see Scaling for Campaign Blasts' 'pre-compute recipient lists' step) is a real compliance risk; the recipient-list pre-computation step needs to re-check preferences as close to send-time as is practical, not rely purely on a list resolved hours earlier.

API Design

// Trigger single notification
POST /api/v1/notifications
Body:
{
  "userId": "user-123",
  "template": "payment_confirmed",
  "data": { "amount": "$49.99", "orderId": "ORD-789" },
  "channels": ["PUSH", "EMAIL"],
  "priority": "HIGH"
}

// Trigger campaign (batch)
POST /api/v1/campaigns
Body:
{
  "segmentId": "premium_users",
  "template": "new_feature_announcement",
  "scheduledAt": "2025-09-20T09:00:00Z"
}

// Check delivery status
GET /api/v1/notifications/{notificationId}/status

Database Design

Notifications (Cassandra — write-heavy delivery logs)

notification_log
  notification_id  UUID    PK
  user_id          UUID
  channel          ENUM    (PUSH, EMAIL, SMS)
  template_id      VARCHAR
  status           ENUM    (PENDING, SENT, DELIVERED, FAILED, CLICKED)
  created_at       TIMESTAMP
  delivered_at     TIMESTAMP

User preferences (PostgreSQL)

notification_prefs
  user_id          UUID    PK
  channel          VARCHAR
  enabled          BOOLEAN
  updated_at       TIMESTAMP

Templates (Redis cache backed by DB)

template:{templateId}  →  { subject, body_html, body_text }

Scaling for Campaign Blasts

Sending to 10M users in <60 seconds requires:

1. Pre-compute recipient lists (async, before scheduled time)
2. Parallel Kafka partitions: 100 partitions × 100K messages/partition = 10M messages fanned out
3. Worker fleet: 100 push workers × 1,600 notifications/sec each = 160K push/sec
4. APNs/FCM limits: Both support batch send APIs (~1000 tokens per batch call)

Campaign trigger → segment resolver → writes 10M user IDs to Kafka
                                              │
                               100 Kafka partitions
                                              │
                               100 Push Worker instances
                                              │
                               10M APNs/FCM calls over 60 seconds

Multi-region delivery and provider failover

At genuine global scale, the notification pipeline itself should run active-active across multiple regions (Design Global Content Delivery) — a user's notification is processed and sent from whichever region is nearest their associated provider endpoint, reducing delivery latency and avoiding a single region's outage blocking notification delivery entirely. Each channel worker should also support PROVIDER FAILOVER as a first-class concern, not an afterthought: if the primary SMS provider (Twilio) is degraded or down, the SMS worker fails over to a SECONDARY provider (Vonage) automatically — directly extending the Failure Scenarios section's SMS-provider-failure discussion from a manual mitigation into an automated, designed-for capability.

Deduplication at scale: when multiple upstream producers might retry the same event

Event producer (e.g. Order Service) publishes 'OrderShipped' -> a network blip means
the producer isn't sure its publish succeeded -> producer RETRIES the same event
  -> Notification Service must not send the SAME notification twice to the user

This is the same idempotent-consumer problem from Message Queue System and Payment — Idempotency Implementation, applied specifically to notification delivery: an upstream producer's own retry (after an ambiguous publish outcome) can result in the Notification Service receiving the logically-same event more than once. A DEDUPLICATION KEY (derived from the event's own natural identity — e.g. orderId + eventType, not a fresh UUID per publish attempt) checked against a short-lived dedup cache (Redis, with a TTL matching the realistic retry window) before actually dispatching is what prevents the user from seeing the same 'your order shipped' notification twice, without requiring every upstream producer to independently solve this problem.

Delivery analytics

Beyond the existing per-notification delivery_status tracking (Database Design), delivery ANALYTICS aggregates this into actionable signals: delivery success rate per channel (a dropping push-notification success rate might indicate expiring/invalid tokens accumulating, per the Bottlenecks section), delivery latency percentiles per channel (Metrics & Monitoring's percentile-over-average reasoning, applied here), and click-through rate per template (feeding back into which notification content/templates are actually effective). This is naturally implemented as its own downstream consumer of the same notification_log stream (Event-Driven Architecture Patterns' event-carried-state-transfer pattern) — a dedicated analytics pipeline reading delivery events, rather than burdening the hot dispatch path with analytics computation directly.

Priority bypass: why a security alert can't sit behind a marketing blast

This is the exact scenario the priority-segregated topics (added earlier in this case's Architecture section) exist to solve concretely: a critical security alert and a 10M-user marketing campaign must never share the same queue/worker pool, specifically because a campaign enqueued moments before a critical alert would otherwise force that alert to wait behind millions of lower-priority messages — the separate notifications.push.high topic with its own dedicated, never-shared worker pool is what guarantees a security alert's delivery latency stays bounded regardless of what else is happening on the bulk/marketing path at that exact moment.

• At-least-once vs exactly-once: Kafka guarantees at-least-once. De-duplicate at the client (track notification_id) to avoid showing the same notification twice.
• Fire-and-forget vs delivery tracking: tracking requires storing delivery records (~100M rows/day) — adds cost and latency. Make it opt-in per notification type.
• Push vs in-app notification: push reaches offline users but can't be revoked; in-app is revocable but requires an open session.

• APNs/FCM rate limits: enforce per-app-per-token limits by distributing across multiple APNs connections
• Invalid tokens: ~5% of push tokens are stale. Purge invalid tokens from DB based on APNs/FCM feedback service
• Email reputation: high bounce/spam rates lower sender score. Use dedicated IPs and honour unsubscribes immediately

• APNs down: messages buffered in Kafka. Workers retry with exponential backoff. Campaign delayed, not lost.
• SMS provider failure: fallback to secondary SMS provider (Vonage → Twilio).
• Worker crash: Kafka consumer group rebalances — unprocessed messages reassigned to healthy workers.