Design WhatsApp

Design a messaging app like WhatsApp. Users can send text, images, and videos to individuals or groups. Messages must be delivered in order and receipts (sent/delivered/read) must be tracked.

Capacity Estimation

• Message rate: 100B messages/day = 1.16M messages/sec
• Message size: avg 100 bytes text. 100B × 100 bytes = 10 TB/day text storage
• Media: ~20% messages have media. Stored on S3; metadata only in DB
• Active connections: 500M concurrent WebSocket connections

Architecture

[Client A] ──── WebSocket ────▶ [Chat Server]
                                      │
                              [Message Queue (Kafka)]
                                      │
                            ┌─────────┴──────────┐
                     [Delivery Worker]    [Push Notification Service]
                            │                         │
                    [Client B online?]         [APNs / FCM]
                         Yes │
                    [Client B WebSocket]

Each chat server maintains WebSocket connections. A user might connect to any server — a routing layer (via consistent hashing on userId) directs messages to the right server.

Real-time delivery: WebSocket vs long-polling vs Server-Sent Events

• WebSocket (what this design uses): a persistent, full-duplex connection — server can push to the client the instant a message arrives, client can send with no per-message connection setup. Best latency, but requires holding one open connection per online user (real but manageable cost at scale with connection-oriented load balancing).
• Long-polling: client sends a request that the server holds open until a message arrives (or a timeout), then the client immediately re-requests. Works through any standard HTTP infrastructure with no special protocol support, but adds request-response overhead per message and slightly higher latency than a truly persistent connection.
• Server-Sent Events (SSE): server-to-client push over a standard HTTP connection, but one-directional only — the client would still need a separate channel (regular HTTP POST) to send messages. Simpler than WebSocket where only server-to-client push is needed, but doesn't fit a bidirectional chat protocol as cleanly.

WebSocket wins here specifically because chat is genuinely bidirectional and latency-sensitive — the other two are better fits for one-directional or latency-tolerant push (e.g. SSE for a live dashboard, long-polling as a WebSocket-unsupported fallback).

Message delivery semantics: at-least-once vs exactly-once

Chat systems in practice target at-least-once delivery, not exactly-once — a message might be delivered twice under retry (e.g. the client's ack was lost even though the server delivered successfully), which is why every message needs a client-generated unique ID: the receiving client de-duplicates by that ID, turning at-least-once delivery (cheap, achievable with retries) into effectively-exactly-once observed behavior (the hard part is pushed to idempotent de-duplication at the edge, not to the delivery guarantee itself, which is far simpler to build reliably).

API Design

WebSocket messages (not HTTP):

// Send message
{ "type": "MESSAGE", "to": "userId", "content": "Hello", "clientMsgId": "uuid" }

// Delivery receipt (from server)
{ "type": "DELIVERED", "msgId": "...", "to": "userId" }

// Read receipt
{ "type": "READ", "msgId": "...", "by": "userId" }

REST API for non-real-time:

GET /messages/{conversationId}?before=cursor&limit=50
POST /media/upload → presigned S3 URL
GET /users/{userId}/presence

Database Design

Messages (Cassandra — write-heavy, time-series)

messages
  conversation_id UUID    (PK partition key)
  message_id      TIMEUUID (PK clustering key, newest first)
  sender_id       UUID
  content         TEXT
  media_url       TEXT    nullable
  status          ENUM    (SENT, DELIVERED, READ)
  created_at      TIMESTAMP

Why TIMEUUID for clustering? Guarantees ordering by time AND uniqueness — no two messages have the same ID even if created at the same millisecond.

Conversation metadata (PostgreSQL)

conversations: id, type (1:1 | GROUP), created_at
participants: conversation_id, user_id, joined_at

Presence (Redis, TTL-based)

SETEX presence:userId 30 "ONLINE"  # refreshed every 20s by heartbeat

Scaling

WebSocket connection scaling

500M concurrent WebSockets cannot live on one server. Each chat server handles ~50K connections. We need ~10,000 chat servers.

A routing service (Redis pub/sub or a service mesh) tracks which server each user is connected to:

REDIS: user:{userId}:server → "chatserver-047"

When Server A needs to deliver to a user on Server B, it publishes to Redis pub/sub channel for that user, and Server B picks it up.

Message ordering

Use Cassandra TIMEUUID + per-conversation sequence numbers. The client reorders on display.

Fan-out strategy for group chats

• Fan-out-on-write: when a message is sent to a group, immediately write it into every member's message queue/inbox. Fast reads (each user's inbox is precomputed), but a message to a 500-person group means 500 writes — expensive for very large groups.
• Fan-out-on-read: store the message once against the group; each member's client queries the group's message log directly when reading. Cheap writes regardless of group size, but reads require merging across the group's shared log rather than a precomputed per-user inbox.

Most chat systems fan out on write for typical group sizes (tens to low hundreds of members, where the per-message write cost is trivial) and fall back toward fan-out-on-read behavior for very large broadcast-style channels — the same push-vs-pull tradeoff as a news feed system (see the News Feed System case for the same pattern at a different scale).

• WebSocket vs long-polling: WebSocket is bidirectional and low-latency. Long-polling is simpler but wasteful.
• Cassandra vs PostgreSQL for messages: Cassandra scales writes horizontally; PostgreSQL is easier but harder to shard for 100B msgs/day.
• Fan-out vs pull for group messages: Fan-out to each member (1000 copies) is expensive for large groups. For 1000-member groups, fetch-on-read is more practical.

• Hot conversations — rate-limit message sending per conversation
• Media upload — use presigned S3 URLs; don't proxy media through chat servers
• Notification fan-out — for large groups, batch and de-duplicate push notifications

• Chat server crash: client reconnects via WebSocket to any server. Messages queued in Kafka until delivery.
• Kafka lag: delivery delayed but not lost — Kafka retains messages for 7 days.
• DB unavailable: writes buffered in Kafka, processed when DB recovers.

Follow-up questions this case invites — and their answers

Q: How are messages delivered to a user who is offline? A: The message is stored durably, in a per-recipient queue or inbox table, as soon as the server accepts it, and the sender sees a single tick. A push notification wakes the recipient's device. When it reconnects, it fetches pending messages and acknowledges them, and the server marks them delivered (two ticks) and eventually deletes them from the server-side queue.

Q: How do you guarantee message ordering within a chat? A: Assign each message a sequence number per conversation. That's easy when one server or partition owns the conversation. Clients sort by that sequence and detect gaps, which trigger a re-fetch. Global ordering across all chats isn't needed, and would be very expensive.

Q: How does the server know which gateway server a recipient is connected to? A: A session registry, a fast key-value store, maps each user to the gateway holding their persistent connection. It's updated on connect and disconnect, with heartbeats to expire stale entries. The sender's gateway looks up the recipient's gateway and forwards the message through an internal messaging layer.

Q: How does end-to-end encryption change the server design? A: The server only relays and stores ciphertext, so it can't read, search or moderate message content. The server distributes public keys (pre-keys) so senders can set up encrypted sessions. Group messages are encrypted for each member, or with a shared sender key. Features such as server-side search must move to the client.

Q: How do group messages scale for large groups? A: The sender uploads the message once, and the server fans it out to each member's inbox. Media is uploaded once to blob storage, and only a reference plus a key are sent. For very large groups, fan-out is done asynchronously by workers, and delivery receipts are aggregated rather than shown individually.

Q: How would you implement typing indicators and "online" status cheaply? A: Treat them as ephemeral signals. Send them over the existing connection only to users currently viewing that chat, never store them, and throttle them (for example one "typing" event every few seconds). Last-seen timestamps are updated in a cache on disconnect, not on every message.