How big data evolved, and the papers behind it

Why a new wave kept appearing

Each generation of data tools answered a limit of the previous one. Understanding the problem behind each wave makes the tools easier to place.

Image: Trower, NASA · Public domain · Source · Wikimedia Commons: Columbia Supercomputer - NASA Advanced Supercomputing Facility

A timeline

100%

Figure SDA-BDP-01-1 · A timeline · © forcsguy.com

1. Before: relational databases and data warehouses

Data fit on one big server. SQL and data warehouses served reports. Limits: scaling up got very expensive, and unstructured data (web logs, clicks) did not fit neatly.

2. Web scale (around 2003 to 2006)

Search companies had far more data than any single machine could hold. Google described how it coped, and open-source projects copied the ideas.

• GFS (Google File System): a distributed file system on commodity machines that expects failures.
• MapReduce: a model to process huge datasets in parallel.
• Hadoop (HDFS plus MapReduce) brought both to everyone.

3. NoSQL (about 2006 to 2010)

Web applications needed massive scale and high availability that relational systems struggled to give.

• Bigtable: a sparse, sorted, wide-column store; inspired HBase and Cassandra's data model.
• Dynamo: an always-writable key-value store with consistent hashing and eventual consistency; inspired Cassandra, Riak and DynamoDB.

4. Faster and friendlier processing (2010 onward)

• Dremel: interactive SQL over huge datasets using columnar storage and tree-shaped execution; the idea behind BigQuery and later engines, and the Parquet format.
• Spark (resilient distributed datasets): keeps data in memory between steps, much faster than disk-based MapReduce for iterative work, with friendly APIs.

5. Streaming

Waiting for nightly batches was too slow. Kafka made a durable, replayable log the backbone of event pipelines, and engines such as Flink and the Dataflow model handled continuous computation with proper treatment of event time, windows and late data.

6. The cloud

Cloud providers separated storage (cheap object storage) from compute (elastic clusters or serverless queries). Managed warehouses (BigQuery, Redshift, Snowflake and others) removed cluster management.

7. The lakehouse

Data lakes were cheap but unreliable (no transactions, messy files). Table formats (Delta Lake, Apache Iceberg, Apache Hudi) add ACID transactions, schema evolution and time travel on top of files in object storage, so one copy of data can serve both analytics and machine learning.

100%

Figure SDA-BDP-01-2 · Key dates, in order · © forcsguy.com

The classic papers

These are the research papers most often cited. Search the title to find each; most are freely available.

Paper	Authors (and year)	The idea
The Google File System	Ghemawat, Gobioff, Leung (2003)	Large-file storage across commodity machines, built to tolerate failures
MapReduce: Simplified Data Processing on Large Clusters	Dean, Ghemawat (2004)	A simple map and reduce model with automatic parallelism and fault tolerance
Bigtable: A Distributed Storage System for Structured Data	Chang and others (2006)	A sorted, sparse, multi-dimensional map for structured data at scale
Dynamo: Amazon's Highly Available Key-value Store	DeCandia and others (2007)	Consistent hashing, quorums, vector clocks and eventual consistency
Dremel: Interactive Analysis of Web-Scale Datasets	Melnik and others (2010)	Columnar nested data and tree execution for fast SQL
Kafka: a Distributed Messaging System for Log Processing	Kreps, Narkhede, Rao (2011)	A partitioned, replicated log as the backbone for event data
Resilient Distributed Datasets (Spark)	Zaharia and others (2012)	In-memory, fault-tolerant parallel computing
Spanner: Google's Globally-Distributed Database	Corbett and others (2012)	Strongly consistent, globally distributed SQL using synchronised clocks
The Dataflow Model	Akidau and others (2015)	A unified model for batch and streaming with event time and windows
Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores	Armbrust and others (2020)	Transactional tables on object storage: the lakehouse idea

A good companion is Jay Kreps's essay "The Log: What every software engineer should know about real-time data's unifying abstraction" (2013).

What to take away

• Scale-out on cheap machines (GFS, MapReduce) made big data practical.
• Availability versus consistency choices (Dynamo, Bigtable, Spanner) shaped databases.
• Columnar storage (Dremel, Parquet) made analytics fast.
• Logs and streams (Kafka, dataflow) made data continuous.
• Separating storage from compute and open table formats define today's platforms.

Real-world reading