Cloud Storage

The Problem

A user has a 2GB folder synced across three devices: a laptop, a phone, and a desktop. They edit a 100MB presentation on the laptop. The system must sync that change to the other two devices. Uploading the entire 100MB file every time a single slide changes is wasteful. If two devices edit the same file simultaneously, the system must detect and resolve the conflict without losing either person's work.

Cloud storage is a deceptively complex system. The user sees a folder. Behind it is a distributed architecture handling deduplication, chunking, delta synchronization, conflict detection, and metadata management across millions of concurrent users.

High-Level Architecture

The architecture separates two fundamentally different concerns. The metadata service tracks the file tree: which files exist, their sizes, version numbers, and block hashes. The block storage service stores the actual file data as content-addressed chunks. This separation is the foundation that makes deduplication and delta sync possible.

Block-Level Chunking

Files are not stored as single objects. Each file is split into fixed-size blocks, typically 4MB each. Each block is hashed using SHA-256 to produce a content fingerprint. The file's metadata record stores the ordered list of block hashes that compose it.

This chunking approach enables three capabilities:

• Parallel upload/download. Multiple blocks transfer simultaneously instead of waiting for a sequential file transfer.
• Resumable transfers. If a connection drops mid-upload, only the incomplete block needs to be re-sent, not the entire file.
• Content-addressed storage. Blocks are stored by their hash, not by filename. This is the foundation for deduplication.

Block-Level Deduplication

If two users upload the same 4MB block, only one copy is stored. The system checks: does a block with this SHA-256 hash already exist in storage? If yes, skip the upload and just reference the existing block. If no, upload the block and register its hash.

The impact at scale is enormous. Consider a company where 500 employees all have the same 50MB onboarding PDF in their synced folders. Without deduplication, that is 25GB of storage. With block-level dedup, it is 50MB. The other 499 copies are just metadata pointers to the same blocks.

Deduplication also applies within a single user's files. If you copy a folder, the blocks already exist. The system creates new metadata entries pointing to existing blocks. The copy is instant from a storage perspective.

Delta Sync

A user edits one slide in a 100MB presentation. The file is composed of 25 blocks (at 4MB each). Only the block containing the modified slide has changed. Delta sync identifies which blocks changed and uploads only those.

Instead of uploading 100MB, the client uploads 4MB. That is a 96% reduction in bandwidth. For users on slow connections or mobile networks, this difference is the reason the product is usable at all.

The mechanism relies on the rsync-style algorithm. The client computes a rolling checksum across the modified file to identify which block boundaries shifted. For each block, it computes the SHA-256 hash and compares it against the previously stored hashes. Only blocks with new hashes are uploaded.

Sync Strategy Comparison

Block-level delta is the sweet spot for cloud storage. It captures most of the savings of byte-level delta with significantly less computational overhead. Computing a binary diff of a 100MB file is expensive. Comparing 25 block hashes is cheap.

Conflict Resolution

Two users edit the same file on different devices while offline. Both devices come online and attempt to sync. The metadata service detects a conflict: both edits are based on the same parent version, but they produce different block hashes.

Three strategies handle this:

• Last-writer-wins. The most recent edit (by timestamp) becomes the canonical version. The other edit is discarded. Simple but lossy. Used for low-value or auto-generated files.
• Conflict copy. The system keeps both versions. One becomes "presentation.pptx" and the other becomes "presentation (conflict copy).pptx." The user decides which to keep. This is Dropbox's default approach.
• Merge. For structured documents (like text files with line-based content), the system attempts an automatic three-way merge using the common ancestor version. Git uses this approach. It works well for code but poorly for binary files.

The conflict copy approach is the safest default. It never loses data. The cost is user inconvenience: someone must manually reconcile the two versions. For most cloud storage products, this trade-off is acceptable because conflicts are rare. Most files are edited by one person at a time.

Notification and Propagation

When the laptop syncs a change, the desktop and phone need to know. The notification service uses a pub/sub pattern. Each device subscribes to a channel for its user's file tree. When the metadata service records a new version, it publishes a notification. Each subscribed device pulls the updated metadata and downloads any new blocks.

For devices that are offline, the notification queues until the device reconnects. On reconnection, the sync agent pulls all pending changes and applies them in version order.

Further Reading

• Dropbox Architecture Deep Dive: Building a High-Frequency File Storage System at Scale (Chengchang Yu). Detailed walkthrough of Dropbox's chunking, sync, and metadata architecture.
• Design a Cloud Storage System Like Dropbox: A Step-by-Step Guide (System Design Handbook). Estimation, API design, and component breakdown.
• The rsync algorithm: Rolling checksum (rsync.samba.org). Original technical report on the rolling checksum that powers delta sync.
• Rsync Algorithm in System Design (GeeksforGeeks). Accessible explanation of the two-level checksum system and block matching.