Design WhatsApp

Design a messaging app like WhatsApp. Users can send text, images, and videos to individuals or groups. Messages must be delivered in order and receipts (sent/delivered/read) must be tracked.

Capacity Estimation

• Message rate: 100B messages/day = 1.16M messages/sec
• Message size: avg 100 bytes text. 100B × 100 bytes = 10 TB/day text storage
• Media: ~20% messages have media. Stored on S3; metadata only in DB
• Active connections: 500M concurrent WebSocket connections

Architecture

[Client A] ──── WebSocket ────▶ [Chat Server]
                                      │
                              [Message Queue (Kafka)]
                                      │
                            ┌─────────┴──────────┐
                     [Delivery Worker]    [Push Notification Service]
                            │                         │
                    [Client B online?]         [APNs / FCM]
                         Yes │
                    [Client B WebSocket]

Each chat server maintains WebSocket connections. A user might connect to any server — a routing layer (via consistent hashing on userId) directs messages to the right server.

Real-time delivery: WebSocket vs long-polling vs Server-Sent Events

• WebSocket (what this design uses): a persistent, full-duplex connection — server can push to the client the instant a message arrives, client can send with no per-message connection setup. Best latency, but requires holding one open connection per online user (real but manageable cost at scale with connection-oriented load balancing).
• Long-polling: client sends a request that the server holds open until a message arrives (or a timeout), then the client immediately re-requests. Works through any standard HTTP infrastructure with no special protocol support, but adds request-response overhead per message and slightly higher latency than a truly persistent connection.
• Server-Sent Events (SSE): server-to-client push over a standard HTTP connection, but one-directional only — the client would still need a separate channel (regular HTTP POST) to send messages. Simpler than WebSocket where only server-to-client push is needed, but doesn't fit a bidirectional chat protocol as cleanly.

WebSocket wins here specifically because chat is genuinely bidirectional and latency-sensitive — the other two are better fits for one-directional or latency-tolerant push (e.g. SSE for a live dashboard, long-polling as a WebSocket-unsupported fallback).

Message delivery semantics: at-least-once vs exactly-once

Chat systems in practice target at-least-once delivery, not exactly-once — a message might be delivered twice under retry (e.g. the client's ack was lost even though the server delivered successfully), which is why every message needs a client-generated unique ID: the receiving client de-duplicates by that ID, turning at-least-once delivery (cheap, achievable with retries) into effectively-exactly-once observed behavior (the hard part is pushed to idempotent de-duplication at the edge, not to the delivery guarantee itself, which is far simpler to build reliably).

API Design

WebSocket messages (not HTTP):

// Send message
{ "type": "MESSAGE", "to": "userId", "content": "Hello", "clientMsgId": "uuid" }

// Delivery receipt (from server)
{ "type": "DELIVERED", "msgId": "...", "to": "userId" }

// Read receipt
{ "type": "READ", "msgId": "...", "by": "userId" }

REST API for non-real-time:

GET /messages/{conversationId}?before=cursor&limit=50
POST /media/upload → presigned S3 URL
GET /users/{userId}/presence

Database Design

Messages (Cassandra — write-heavy, time-series)

messages
  conversation_id UUID    (PK partition key)
  message_id      TIMEUUID (PK clustering key, newest first)
  sender_id       UUID
  content         TEXT
  media_url       TEXT    nullable
  status          ENUM    (SENT, DELIVERED, READ)
  created_at      TIMESTAMP

Why TIMEUUID for clustering? Guarantees ordering by time AND uniqueness — no two messages have the same ID even if created at the same millisecond.

Conversation metadata (PostgreSQL)

conversations: id, type (1:1 | GROUP), created_at
participants: conversation_id, user_id, joined_at

Presence (Redis, TTL-based)

SETEX presence:userId 30 "ONLINE"  # refreshed every 20s by heartbeat

Scaling

WebSocket connection scaling

500M concurrent WebSockets cannot live on one server. Each chat server handles ~50K connections. We need ~10,000 chat servers.

A routing service (Redis pub/sub or a service mesh) tracks which server each user is connected to:

REDIS: user:{userId}:server → "chatserver-047"

When Server A needs to deliver to a user on Server B, it publishes to Redis pub/sub channel for that user, and Server B picks it up.

Message ordering

Use Cassandra TIMEUUID + per-conversation sequence numbers. The client reorders on display.

Fan-out strategy for group chats

• Fan-out-on-write: when a message is sent to a group, immediately write it into every member's message queue/inbox. Fast reads (each user's inbox is precomputed), but a message to a 500-person group means 500 writes — expensive for very large groups.
• Fan-out-on-read: store the message once against the group; each member's client queries the group's message log directly when reading. Cheap writes regardless of group size, but reads require merging across the group's shared log rather than a precomputed per-user inbox.

Most chat systems fan out on write for typical group sizes (tens to low hundreds of members, where the per-message write cost is trivial) and fall back toward fan-out-on-read behavior for very large broadcast-style channels — the same push-vs-pull tradeoff as a news feed system (see the News Feed System case for the same pattern at a different scale).

• WebSocket vs long-polling: WebSocket is bidirectional and low-latency. Long-polling is simpler but wasteful.
• Cassandra vs PostgreSQL for messages: Cassandra scales writes horizontally; PostgreSQL is easier but harder to shard for 100B msgs/day.
• Fan-out vs pull for group messages: Fan-out to each member (1000 copies) is expensive for large groups. For 1000-member groups, fetch-on-read is more practical.

• Hot conversations — rate-limit message sending per conversation
• Media upload — use presigned S3 URLs; don't proxy media through chat servers
• Notification fan-out — for large groups, batch and de-duplicate push notifications

• Chat server crash: client reconnects via WebSocket to any server. Messages queued in Kafka until delivery.
• Kafka lag: delivery delayed but not lost — Kafka retains messages for 7 days.
• DB unavailable: writes buffered in Kafka, processed when DB recovers.