Erasure coding and durability

Copies against erasure codes: how cloud storage reaches eleven nines for less space.

On the roadmap:Advanced · step 3: Storage engines

Intermediateabout 10 to 15 min6 experiments

Experiments

New here? A few short steps show what happens, one at a time.

Settings

How the data is protected

Copies3

Chance a disk fails in a year2.0%

Hours to replace a lost piece24 h

Objects stored1,000,000,000

Space used per byte

3.00x

Failures survived

Nines of durability

12.8

Pieces read to repair

3 copies: stores 3.00 times the data, survives 2 lost pieces, and a rebuild reads 1 piece. With each piece having a 0.006% chance of failing before repair, an object is lost with probability 1.7e-13 a day, about 12.8 nines. Across 1,000,000,000 objects that is one every 16 years.

Seven ways to store the same data (with your failure and repair settings)

Scheme	Space	Survives	Nines	Rebuild reads	Objects lost
1 copies	1.00x	0	4.3	1	20202148 per year
2 copies	2.00x	1	8.5	1	1118 per year
3 copies	3.00x	2	12.8	1	one every 16 years
4 copies	4.00x	3	17+	1	one every 291,936 years
6+3 erasure code	1.50x	3	14.9	6	one every 2,317 years
10+4 erasure code	1.40x	4	17+	10	one every 2,635,724 years
17+3 erasure code	1.18x	3	13.3	17	one every 60 years

Assumes pieces fail independently and one at a time is repaired within the window. Real failures are correlated (a rack, a power feed), so real systems also spread pieces across racks and zones.

Where the pieces are

Each copy is the whole file. Any one is enough.

In plain words

Storing a file once loses it when a disk fails. Storing three copies is safe but triples the cost. An erasure code cuts the file into k pieces, calculates m more, and stores all k plus m, so any k of them rebuild the file. It survives m failures for a fraction of the space of copying.

Why it matters

Object stores promise durability of eleven nines or more, and could not afford to if they used full copies of petabytes. Erasure coding is how cloud storage gets that safety at a low cost, and it shapes how failures are repaired, which is where the real trade-off is.

How real systems do it

The simulation is a simplified model. Here is what production systems add or do.

Amazon S3 states a design durability of 99.999999999% (eleven nines) for objects, and stores them redundantly across at least three availability zones. Google Cloud Storage and Azure Blob Storage make similar claims.
Facebook's f4 warm blob store used Reed-Solomon coding (10 data and 4 parity pieces) to cut space from 3.6 times to 2.1 times for older photos and videos, compared with their hot store. Windows Azure Storage published local reconstruction codes to cut the repair reads.
HDFS 3 supports erasure coding as an option, and Ceph offers erasure-coded pools next to replicated ones. Reed-Solomon is the most used family of codes.
Real failures are correlated: a rack, a power feed or a software bug can take several pieces at once. Systems place pieces in different racks and zones, so the independence this simulator assumes is something the design has to earn.