The system design library
Search and filter every curated system design video, article, repository and paper in one place.
58 resources
Mining of Massive Datasets
Large-scale data processing: MapReduce, similarity search, recommendation and streams. The authors offer the whole book as a free download.
BooksAdvancedFree from the authorBy Jure Leskovec, Anand Rajaraman, Jeff Ullman · Stanford University
30x Performance Improvements on MySQLStreamer at Yelp
Making 30x performance improvements on Yelp’s MySQLStreamer Abrar Sheikh, Software Engineer Feb 5, 2018 Introduction MySQLStreamer is an important application in Yelp’s Data Pipeline infrastructure. It’s responsible for streaming high-vo…
Case StudiesIntermediateRead hereBy Yelp engineers · Yelp
Airflow at Grab
This blog post focuses on the lessons we learned while building our batch data pipeline.
Case StudiesIntermediateRead hereBy Grab engineers · Grab
Analytics Pipeline at Lyft
A real-world engineering write-up from Lyft on big Data.
PapersIntermediateBy Lyft
Apache Hadoop 3.3.5
Hardware failure is the norm rather than the exception. An HDFS instance may consist of hundreds or thousands of server machines, each storing part of the file system’s data. The fact that there are a huge number of components and that e…
Case StudiesIntermediateRead hereBy Apache engineers · Apache
Apache Spark Documentation
Overview and guides for Spark's batch and streaming engine.
DocumentationIntermediateBy Apache Spark · Apache Software Foundation
AWS Big Data Blog
AWS posts and announcements on analytics and data topics, such as Redshift, EMR and Apache Iceberg tables.
BlogsIntermediateBy Amazon Web Services engineers · Amazon Web Services
AWS DevOps Blog
AWS CodeDeploy now offers a purpose-built RESTART deployment mode that reapplies the last successful revision to EC2 and on-premises fleets. It keeps restarts inside CodeDeploy with the same batch sizing, health checks, alarm monitoring,…
BlogsIntermediateBy Amazon Web Services engineers · Amazon Web Services
Big Data Analytics and ML Techniques at LinkedIn
A real-world engineering write-up from LinkedIn on big Data.
PapersIntermediateBy LinkedIn
Big Data Processing (2 parts) at Spotify
In this part we’ll take a closer look at Scio, including basic concepts, its unique features, and concrete use cases here at Spotify.
Case StudiesIntermediateRead hereBy Spotify engineers · Spotify
Big Data Processing at Uber
A real-world engineering write-up from Uber on big Data.
PapersIntermediateBy Uber
Black-Box Auditing: Verifying End-to-End Replication Integrity between MySQL and Redshift at Yelp
Black-Box Auditing: Verifying End-to-End Replication Integrity between MySQL and Redshift Jacob Park, Software Engineering Intern Apr 12, 2018 Since Yelp introduced its real-time streaming data infrastructure, “Data Pipeline”, it has...
Case StudiesIntermediateRead hereBy Yelp engineers · Yelp
CaffeOnSpark: Distributed Deep Learning on Big Data Clusters at Yahoo
yahoohadoop: By Andy Feng(@afeng76), Jun Shi and Mridul Jain (@mridul_jain), Yahoo Big ML Team Introduction Deep learning (DL) is a critical capability required by Yahoo product teams (ex. Flickr, Image Search) to gain intelligence from …
Case StudiesIntermediateBy Yahoo engineers · Yahoo
Cloud Big Data Design Patterns - Lynn Langit
A real-world engineering write-up from Lynnlangit on designing Large-Scale Systems.
BlogsIntermediateBy Lynnlangit engineers · Lynnlangit
Curalate Engineering Blog
A blog about developing at Curalate. How we handle big data architecture, design for the consumer web, and help our customers get the most out of their imagery.
BlogsIntermediateBy Curalate engineers · Curalate
Data Infrastructure at LinkedIn
Shirshanka Das describes LinkedIn’s Big Data Infrastructure and its evolution through the years, including details on the motivation and architecture of Gobblin, Pinot and WhereHows.
Case StudiesIntermediateBy LinkedIn engineers · LinkedIn
Data Orchestration Service at Spotify
TL;DR Within Spotify, we run 20,000 batch data pipelines defined in 1,000+ repositories, owned by 300+ teams — daily. The majority of our pipelines rely on two tools: (for the Python folks) and (for the Java folks). In 2019, the data orc…
Case StudiesIntermediateRead hereBy Spotify engineers · Spotify
Data Pipeline Monitoring System at LinkedIn
Monitoring big data pipelines often equates to waiting for a long-running batch job to complete and observing the status of the execution. The status can result in “Failed” or “Successful” or even “Incomplete.” From there, it’s the team’…
Case StudiesIntermediateRead hereBy LinkedIn engineers · LinkedIn
Data Platform at Netflix
A real-world engineering write-up from Netflix on big Data.
YouTubeIntermediateBy Netflix
Deduplication at Segment
The single requirement of all data pipelines is that they cannot lose data. Data can usually be delayed or re-ordered–but never dropped.
Case StudiesIntermediateRead hereBy Segment engineers · Segment
Designing Data-Intensive Applications
The standard book on data systems: storage engines, replication, partitioning, transactions, batch and stream processing.
BooksIntermediateSold by publisherBy Martin Kleppmann · O'Reilly
Dremel: Interactive Analysis of Web-Scale Datasets (paper)
A columnar, tree-structured query engine that scans trillions of rows interactively; the basis of BigQuery-style analytics.
PapersAdvancedRead hereBy Melnik et al. · Google