System Design Case Study

How does Dropbox sync file changes across millions of devices within seconds?

?? Design a file sync engine: millions of devices, block-level dedup, delta sync, conflict resolution
Concepts Involved
Conflict Resolution CDC Message Queues Consistency Blob Storage

Problem Statement

How does a cloud storage platform sync file changes across millions of devices within seconds, deduplicating content at the block level and tracking per-file block maps for efficient delta sync?

Core challenge: User edits a 1GB file on laptop. Only 4KB changed. How do you sync just the changed blocks (not the whole file) to phone, tablet, and web · within seconds · while deduplicating identical blocks across 700M+ users?
700M+
registered users
billions of files
4KB
block size
content-defined chunking
<5s
sync latency
change ? all devices
~60%
dedup savings
block-level deduplication

Architecture · Block-Level Sync Engine

Content-defined chunking ? hash blocks ? upload only new blocks ? notify other devices

CLIENT LAYER · Detect, Chunk, Hash, Upload Delta FS Watcher inotify (Linux) FSEvents (macOS) ReadDirChanges (Win) Detects file modify instant notification Chunking Engine Rabin fingerprint Content-defined 4KB blocks Boundaries based on content not fixed offsets Insert 1 byte ? 1 block shifts Hash Blocks SHA-256 each block Content-addressed Send hash list to server Server returns "need" list Typically 1-5% are new Upload Delta (only NEW blocks) Edit 4KB in 1GB file ? upload 1 block Massive bandwidth savings Parallel upload (4 concurrent streams) Resumable: track last successful block 700M users | 4KB blocks | <5s sync SERVER LAYER · Metadata + Block Store + Notifications Metadata Service file ? [block_hash_1, hash_2, ...] Version history per file Permissions & sharing metadata MySQL sharded by user_id File = ordered list of block hashes (tiny) Conflict: divergent block maps ? copy Block Store (S3/Magic Pocket) Content-addressed: key = SHA-256 Dedup across ALL users globally Same block stored ONCE, referenced N times Reference counting for garbage collection ~60% storage savings globally 3· replication across AZs Notification Service Long-poll / WebSocket channel "file X changed" ? all devices Per-user notification queue Fallback: periodic poll (5 min) Triggers sync on other devices Instant push for online devices SYNC LAYER · Other Devices Receive & Reconstruct Receive Notification "file X changed" via WebSocket/long-poll Instant for online devices Queued for offline Pull New Block Map Fetch updated file metadata Compare with local block map Identify missing blocks Diff: old hashes vs new hashes Download Missing Only blocks not cached locally Skip blocks already present Parallel download streams LAN sync if same network Reconstruct File Assemble blocks in order from block map sequence File ready locally <5s total sync time CONTENT-DEFINED CHUNKING vs FIXED-SIZE CHUNKING Fixed-size: insert 1 byte at start ? ALL block boundaries shift ? re-upload ENTIRE file Content-defined (Rabin): insert 1 byte ? only 1 block boundary shifts ? upload 1 block (4KB) DEDUPLICATION Same PDF shared by 1M users = stored ONCE in block store (referenced 1M times) ~60% storage savings globally | 99% dedup for popular files 700M users | 4KB blocks | <5s sync | 99% dedup for popular files Block deleted only when reference count = 0 (garbage collection)
StepWhat HappensKey Technology
1. Detect changeFile system watcher detects modified fileinotify (Linux), FSEvents (macOS), ReadDirectoryChanges (Windows)
2. Chunk fileSplit file into 4KB blocks using content-defined chunking (Rabin fingerprint)Rolling hash · boundaries shift with content, not fixed offsets
3. Hash blocksSHA-256 each block ? check which are new (not in server's block store)Client sends hash list ? server returns "need" list
4. Upload deltaUpload only NEW blocks (not already on server)Typically 1-5% of file for small edits ? massive bandwidth savings
5. Update metadataUpdate file's block map: [block_hash_1, block_hash_2, ...]Metadata DB (file ? ordered list of block hashes)
6. Notify devicesPush notification to all user's other devices: "file X changed"Long-poll / WebSocket notification channel
7. Pull deltaOther devices fetch new block map, download only missing blocks, reconstruct fileSame dedup: if block already cached locally, skip download
Content-defined chunking: Unlike fixed-size blocks, Rabin fingerprint creates boundaries based on content. Inserting 1 byte at the start of a file only changes 1 block boundary · not all blocks. This means small edits = small uploads regardless of edit position.
Deduplication: Same block content across ANY user = stored once. A popular PDF shared by 1M users = stored once, referenced 1M times. Reference counting tracks when blocks can be garbage-collected. Saves ~60% storage globally.
Conflict resolution: Two devices edit same file offline ? both upload ? conflict detected (divergent block maps). Resolution: keep both versions as "file.txt" and "file (conflicted copy).txt". User manually merges. For collaborative docs: use OT/CRDT instead.
Real-world: Dropbox · Magic Pocket (custom block store, exabytes). Google Drive · similar chunking for large files. OneDrive · differential sync with BITS protocol. rsync · rolling checksum algorithm (inspiration for all sync engines).

Resilience & Edge Cases

FailureImpactRecovery
Upload interrupted mid-filePartial blocks uploadedResumable upload: track last successful block. Resume from there on reconnect.
Conflict (offline edits)Two versions of same fileCreate "conflicted copy" with device name + timestamp. User resolves.
Block store corruptionFile can't be reconstructedChecksum verification on every read. Replicate blocks across AZs (3· redundancy).
Notification service downOther devices don't know about changesPeriodic full-sync poll (every 5 min) as fallback. Catch up on reconnect.
Large file (10GB+)Chunking takes time, upload slowBackground chunking. Prioritize small files. Parallel block uploads (4 concurrent).

Interview Cheat Sheet

The 7 things to say for file sync design

1. Content-defined chunking (Rabin fingerprint) · boundaries based on content, not fixed offsets
2. Block-level dedup · SHA-256 per block, upload only new blocks (~60% storage savings)
3. Delta sync · edit 4KB in 1GB file ? upload only 1 block, not entire file
4. File = ordered list of block hashes · metadata is tiny, blocks are content-addressed
5. Notification channel (long-poll/WS) · push "file changed" to other devices instantly
6. Conflict = divergent block maps · create "conflicted copy", user resolves manually
7. Reference counting for GC · block deleted only when zero files reference it