BitTorrent: the protocol that actually works, explained

BitTorrent is a peer-to-peer (P2P) protocol designed to distribute large files efficiently. It has become one of the most successful P2P protocols ever shipped and is used for distributing official Linux ISOs, game updates, and other content. Its design ideas, such as swarming, tit-for-tat, and distributed hash tables (DHTs), have influenced many modern distributed systems.

The Problem BitTorrent Solves

In traditional file distribution models, a single server sends a large file to 1000 people, which means the server must handle 1000 uploads. This can cause the file to become unavailable due to its own popularity. BitTorrent flips this model by allowing everyone downloading the file to also upload pieces they have to other peers. As a result, the more popular and healthy a torrent is, the faster it gets.

How It Works

File Splitting and Hashing

File Pieces: The file is divided into small chunks called pieces. Each piece is hashed, and these hashes are stored in the .torrent file or magnet link. This ensures that each piece can be verified upon arrival, preventing corruption and tampering (content addressing).

The Swarm

Swarm: A swarm consists of all peers currently sharing a torrent. Seeds have 100% of the pieces and are not downloading but only uploading. Leechers are still downloading and uploading pieces they already have.
Download Strategy: You download different pieces from multiple peers simultaneously, often out of order (Rarest-First strategy). This ensures that scarce pieces are copied before common ones, keeping the swarm healthy.

Finding Peers

Trackers: Traditionally, a tracker is used to coordinate peer connections. It provides a centralized point for peers to discover each other.
DHT and Peer Exchange: Modern BitTorrent implementations use DHTs (distributed hash tables) where peers form their own lookup network without needing a central server. Magnet links combined with DHT enable fully serverless swarms.

Incentive Mechanism

Tit-for-Tat: Clients upload preferentially to peers who have uploaded back, discouraging freeloaders and promoting fair sharing. This game-theory-based mechanism generally works well.
Optimistic Unchoking: Newcomers are given a chance by being unchoked more often than they would be otherwise.

Concepts That Transfer Elsewhere

BitTorrent introduces several concepts that have influenced other distributed systems:

Content Addressing: Verifying data by hash rather than trusting the source.
Swarming: Consumers also act as distributors, improving overall efficiency.
DHTs: Serverless lookup mechanisms used in IPFS and blockchain node discovery.
Incentive Design: Protocols that assume selfish behavior tend to outlast those assuming virtuous behavior.

By understanding BitTorrent's design principles, you can appreciate its role in modern distributed systems and protocols.