Loading…
Loading…
Design a notification service that can deliver push notifications, SMS, and emails at scale. Notifications may be triggered by system events (payment confirmed) or marketing campaigns (10M users simultaneously). Delivery must be tracked.
[Event Sources]
Payment Service → payment.confirmed event
Marketing Tool → campaign.triggered event
│
▼
[Notification API Service]
- validates request
- checks user preferences
- resolves template
- publishes to channel-specific Kafka topic
↓
┌─────────────────────────────────────┐
│ Kafka Topics │
│ notifications.push │
│ notifications.email │
│ notifications.sms │
└─────────────────────────────────────┘
↓
[Channel Workers] (scale independently)
Push Worker → APNs / FCM
Email Worker → SendGrid
SMS Worker → Twilio
↓
[Delivery Tracker] → updates delivery_status in DB
interface NotificationChannel { void send(Notification notification); }
class EmailChannel implements NotificationChannel {
public void send(Notification n) { /* SMTP send */ }
}
class SmsChannel implements NotificationChannel {
public void send(Notification n) { /* SMS gateway send */ }
}
class PushChannel implements NotificationChannel {
public void send(Notification n) { /* APNs/FCM send */ }
}
class NotificationDispatcher {
private final Map<ChannelType, NotificationChannel> channels;
NotificationDispatcher(Map<ChannelType, NotificationChannel> channels) { this.channels = channels; }
void dispatch(Notification notification) {
NotificationChannel channel = channels.get(notification.getChannelType());
channel.send(notification); // Strategy pattern — channel selected at dispatch time, no branching per channel
}
}
This is Strategy Pattern applied per channel (see Strategy Pattern): NotificationDispatcher never branches on channel type internally — it looks up the right NotificationChannel implementation and delegates. Adding a new channel (e.g. Slack) means writing one new class and registering it in the map, with zero changes to NotificationDispatcher itself — the same OCP argument made throughout the pattern lessons.
class RetryableNotificationQueue {
private final DelayQueue<DelayedNotification> queue = new DelayQueue<>();
private static final int MAX_RETRIES = 5;
void enqueue(Notification notification, int attempt) {
long delayMs = (long) (1000 * Math.pow(2, attempt)); // exponential: 1s, 2s, 4s, 8s, 16s...
queue.put(new DelayedNotification(notification, attempt, delayMs));
}
void processNext() throws InterruptedException {
DelayedNotification item = queue.take(); // blocks until an item's delay has elapsed — see BlockingQueue family
try {
dispatcher.dispatch(item.notification);
} catch (DeliveryException e) {
if (item.attempt < MAX_RETRIES) enqueue(item.notification, item.attempt + 1);
else deadLetterQueue.add(item.notification); // give up — route to a dead-letter queue for manual inspection
}
}
}
DelayQueue (a BlockingQueue variant not covered in Concurrent Utilities & Coordination's family list, specifically designed for "not ready until time T" semantics) is the natural fit here — it only releases an item via take() once its delay has elapsed, which is exactly the shape exponential backoff needs, without any manual timer/scheduling code.
abstract class NotificationTemplate {
final String render(NotificationContext ctx) { // Template Method — fixed sequence, customizable steps
String subject = renderSubject(ctx);
String body = renderBody(ctx);
return formatForChannel(subject, body);
}
protected abstract String renderSubject(NotificationContext ctx);
protected abstract String renderBody(NotificationContext ctx);
protected abstract String formatForChannel(String subject, String body);
}
class EmailTemplate extends NotificationTemplate {
protected String renderSubject(NotificationContext ctx) { return "Your order " + ctx.orderId() + " shipped"; }
protected String renderBody(NotificationContext ctx) { return "<html>...</html>"; }
protected String formatForChannel(String subject, String body) { return subject + "\n\n" + body; }
}
class SmsTemplate extends NotificationTemplate {
protected String renderSubject(NotificationContext ctx) { return null; } // SMS has no subject
protected String renderBody(NotificationContext ctx) { return "Order " + ctx.orderId() + " shipped!"; } // must be terse
protected String formatForChannel(String subject, String body) { return body; } // subject ignored entirely
}
Template Method (see Template Method) fits precisely because content generation for every channel follows the same fixed sequence (build subject, build body, format for the channel) while each channel customizes individual steps — an SMS template overriding renderSubject to return null and formatForChannel to ignore it entirely is a clean way to express "this channel doesn't have this concept" without special-casing it in the shared render() method.
notifications.push.high -- payment confirmations, security alerts: separate topic,
its own dedicated worker pool, never contends with campaign traffic
notifications.push.default -- normal transactional notifications
notifications.push.bulk -- marketing campaigns: separate topic, own worker pool,
explicitly throttled to protect provider rate limits
A single shared queue for all push notifications means a 10M-user marketing campaign enqueued just before a time-critical payment-confirmation notification would force that payment notification to wait behind millions of lower-priority messages — directly violating the push delivery latency requirement for the notification that matters most. The fix is priority-segregated topics/queues with SEPARATE worker pools per priority tier, not a single shared queue with an in-memory priority sort (which doesn't actually solve the problem once a worker has already pulled a large batch of low-priority messages) — this is the same principle as Alerting Strategy's symptom-based-alert urgency applied to notification delivery infrastructure.
Sending every individual low-priority event (e.g. "someone liked your post") as its own immediate push notification risks notification fatigue — a direct analog of Alerting Strategy's alert-fatigue problem, applied to end users instead of on-call engineers. A digest/batching layer accumulates low-priority notifications over a short window (e.g. 15-30 minutes) and sends a single combined notification ("5 people liked your post") rather than 5 separate pushes — this requires the notification service to distinguish notification categories upfront (which are safe to batch vs which must be delivered immediately, like a security alert or a payment confirmation) and apply different delivery timing per category, not a single uniform delivery policy for every notification type.
Every marketing/campaign notification (email specifically, per CAN-SPAM and similar regulations) must include a functioning unsubscribe mechanism, and an unsubscribe request must be honored immediately and durably — checked against notification_prefs (already in the schema) BEFORE a campaign send, not after. A campaign that ignores a very recent opt-out (due to using a stale, pre-fetched recipient list — see Scaling for Campaign Blasts' 'pre-compute recipient lists' step) is a real compliance risk; the recipient-list pre-computation step needs to re-check preferences as close to send-time as is practical, not rely purely on a list resolved hours earlier.
// Trigger single notification
POST /api/v1/notifications
Body:
{
"userId": "user-123",
"template": "payment_confirmed",
"data": { "amount": "$49.99", "orderId": "ORD-789" },
"channels": ["PUSH", "EMAIL"],
"priority": "HIGH"
}
// Trigger campaign (batch)
POST /api/v1/campaigns
Body:
{
"segmentId": "premium_users",
"template": "new_feature_announcement",
"scheduledAt": "2025-09-20T09:00:00Z"
}
// Check delivery status
GET /api/v1/notifications/{notificationId}/status
Notifications (Cassandra — write-heavy delivery logs)
notification_log
notification_id UUID PK
user_id UUID
channel ENUM (PUSH, EMAIL, SMS)
template_id VARCHAR
status ENUM (PENDING, SENT, DELIVERED, FAILED, CLICKED)
created_at TIMESTAMP
delivered_at TIMESTAMP
User preferences (PostgreSQL)
notification_prefs
user_id UUID PK
channel VARCHAR
enabled BOOLEAN
updated_at TIMESTAMP
Templates (Redis cache backed by DB)
template:{templateId} → { subject, body_html, body_text }
Sending to 10M users in <60 seconds requires:
Campaign trigger → segment resolver → writes 10M user IDs to Kafka
│
100 Kafka partitions
│
100 Push Worker instances
│
10M APNs/FCM calls over 60 seconds
At genuine global scale, the notification pipeline itself should run active-active across multiple regions (Design Global Content Delivery) — a user's notification is processed and sent from whichever region is nearest their associated provider endpoint, reducing delivery latency and avoiding a single region's outage blocking notification delivery entirely. Each channel worker should also support PROVIDER FAILOVER as a first-class concern, not an afterthought: if the primary SMS provider (Twilio) is degraded or down, the SMS worker fails over to a SECONDARY provider (Vonage) automatically — directly extending the Failure Scenarios section's SMS-provider-failure discussion from a manual mitigation into an automated, designed-for capability.
Event producer (e.g. Order Service) publishes 'OrderShipped' -> a network blip means
the producer isn't sure its publish succeeded -> producer RETRIES the same event
-> Notification Service must not send the SAME notification twice to the user
This is the same idempotent-consumer problem from Message Queue System and Payment — Idempotency Implementation, applied specifically to notification delivery: an upstream producer's own retry (after an ambiguous publish outcome) can result in the Notification Service receiving the logically-same event more than once. A DEDUPLICATION KEY (derived from the event's own natural identity — e.g. orderId + eventType, not a fresh UUID per publish attempt) checked against a short-lived dedup cache (Redis, with a TTL matching the realistic retry window) before actually dispatching is what prevents the user from seeing the same 'your order shipped' notification twice, without requiring every upstream producer to independently solve this problem.
Beyond the existing per-notification delivery_status tracking (Database Design), delivery ANALYTICS aggregates this into actionable signals: delivery success rate per channel (a dropping push-notification success rate might indicate expiring/invalid tokens accumulating, per the Bottlenecks section), delivery latency percentiles per channel (Metrics & Monitoring's percentile-over-average reasoning, applied here), and click-through rate per template (feeding back into which notification content/templates are actually effective). This is naturally implemented as its own downstream consumer of the same notification_log stream (Event-Driven Architecture Patterns' event-carried-state-transfer pattern) — a dedicated analytics pipeline reading delivery events, rather than burdening the hot dispatch path with analytics computation directly.
This is the exact scenario the priority-segregated topics (added earlier in this case's Architecture section) exist to solve concretely: a critical security alert and a 10M-user marketing campaign must never share the same queue/worker pool, specifically because a campaign enqueued moments before a critical alert would otherwise force that alert to wait behind millions of lower-priority messages — the separate notifications.push.high topic with its own dedicated, never-shared worker pool is what guarantees a security alert's delivery latency stays bounded regardless of what else is happening on the bulk/marketing path at that exact moment.