A Dropbox/Google Drive-style case study: chunking and dedup, separating metadata from blob storage, delta sync, and last-write-wins vs versioning for offline conflicts.
Published September 23, 2026
A Dropbox/Google Drive-style problem: upload, download, sync across multiple devices, sharing, and versioning — the interesting parts are almost entirely about sync and conflict handling, not the storage itself.
Upload and download files, sync changes across a user's devices automatically, share files/folders with other users, and maintain version history for recovery from accidental changes or conflicts.
Uploading (or syncing) a large file as one atomic unit is fragile — a dropped connection midway means restarting from zero, and even a tiny one-byte edit would otherwise require re-uploading the entire file. The standard fix: split every file into fixed-size chunks (e.g. 4MB each) before upload, tracking each chunk independently.
file.pdf (18MB) → [chunk1: 4MB][chunk2: 4MB][chunk3: 4MB][chunk4: 4MB][chunk5: 2MB]
each chunk identified by SHA-256(chunk content)
Deduplication falls out of content-hashing almost for free: if two chunks (from the same file, a different file, or even a different user's file) hash to the same value, they're byte-for-byte identical — the storage backend only needs to store that chunk's content once, with multiple files simply referencing the same chunk hash. This is why cloud storage providers can offer far more effective storage than the sum of every user's raw files would suggest: common files (OS files, popular documents, shared media) frequently produce identical chunks across many unrelated users.
[Metadata Service] [Blob Storage]
- file tree (folders, filenames) - actual chunk bytes, keyed by content hash
- permissions - simple key-value: hash -> bytes
- chunk-hash list per file version - horizontally scalable, no knowledge of "files" at all
- version history
The metadata service (file tree, permissions, which chunk hashes compose which version of which file) is a relatively small, relational-shaped problem — well-suited to a strongly-consistent database. Blob storage (the actual chunk bytes) is a much larger-volume, simpler-shaped problem — a pure content-addressed key-value store, trivially horizontally scalable, since chunks are immutable and identified purely by their hash (no update-in-place ever needed, only new chunks being added). Separating these lets each be scaled and operated independently, matching its own actual access pattern rather than forcing one storage system to handle both shapes well.
A lightweight watcher on each device monitors the local file system for changes (OS-level file-system-event APIs, not polling). On a detected change, the client re-chunks the modified file and compares the new chunk hashes against the previously-synced version's chunk list — only chunks that actually changed get uploaded (delta sync), not the whole file. A one-line edit to a large document might change only one or two 4MB chunks out of dozens — delta sync means only those changed chunks cross the network, not the entire file every time.
The hardest real problem in this system: Device A and Device B both go offline, both edit the same file, then both reconnect. Two strategies:
Most production systems default to versioning specifically because silent data loss (last-write-wins) is a far worse failure mode for a storage product than asking the user to manually reconcile two conflicting copies — losing data invisibly erodes trust in a way that an extra, mildly annoying file never does.
Once a change syncs to the server, other devices need to learn about it to trigger their own sync pull. Two approaches: long-polling (a device holds a request open, the server responds when a change is detected — see the WebSocket vs long-polling vs SSE tradeoffs in Design WhatsApp for the general comparison) or a push notification waking the device to trigger a pull. Either way, the actual content transfer still happens via a separate delta-sync pull — the notification's only job is telling a device "something changed, go check," not delivering the changed data itself.
Q: Why hash chunks instead of hashing the whole file for deduplication? A: Whole-file hashing only deduplicates when two files are 100% byte-identical — chunk-level hashing catches the far more common case of two mostly similar files (a document with one paragraph edited, two versions of a codebase) sharing most of their chunks, deduplicating the unchanged portions even when the files as a whole differ.
Q: What happens if two different files legitimately produce the same chunk hash by coincidence (a hash collision)? A: With SHA-256, the probability is astronomically small enough to be considered negligible in practice for this use case — production systems generally accept this risk rather than engineering around it, though a maximally paranoid design could verify byte-equality on hash match before actually deduplicating, at the cost of extra I/O on every dedup check.
Q: How does versioning interact with storage cost, given chunks are already deduplicated? A: Efficiently — a new version of a file whose edit only changed a couple of chunks only needs to store those new chunk hashes; the unchanged chunks are already in blob storage and get referenced by both the old and new version's metadata, not duplicated. Version history is comparatively cheap precisely because of the same content-addressed dedup that makes cross-user storage efficient.
Q: Why not just re-upload the whole file on every sync and skip delta sync's complexity entirely? A: For large files edited frequently (a shared spreadsheet, a large design file), whole-file re-upload on every small edit would dominate both bandwidth and sync latency — delta sync's added complexity (chunk comparison, partial upload) is directly justified by how much smaller a typical incremental change is compared to the whole file, which is the entire reason this pattern exists rather than being over-engineering for its own sake.