Library / Case Studies
Accelerator: Fast Data Processing Framework at eBay
The Expertmaker Accelerator is a well-proven data processing framework that provides fast data access, parallel execution, and automatic organization of source code, input data, and results. It can be used for daily data analysis tasks as well as operating as a live recommendation system with hundreds of thousands…
By eBay engineers · eBay · Added 2026-10-12
All content belongs to eBay engineers / eBay. We only link to it and give credit. Report a problem
Learn this case study
About 37 min left. Your work is saved as you go.
What you are about to study
eBay explains a real system. You should know the basics: caching, queues and replication.
Ideas you will meet
- Machine Learning Systems
- Recommendations, ranking, feature stores and serving models at scale.
What we already know about this
Our own lessons on these ideas. Read them here before you start, so it makes sense.
Core Concepts: How Things Actually Work · 7 minDurability: replication and erasure codingHow storage systems survive disk failures without storing three full copies of everything.Read here
Why durability is hard
Disks fail every day in a large fleet. Durability is the probability data is not lost over time, and cloud object storage advertises numbers like eleven nines. How?
Image: Mk2010 · CC BY 4.0 · Source · Wikimedia Commons: Hard disk drive platter, Samsung MP0402HTechnique 1: replication
Keep N full copies on different machines, racks and zones. Simple and fast:
- Any copy can serve a read.
- Recovery is just copying a replica.
- Cost: 3 copies mean 3x storage.
Technique 2: erasure coding
Split an object into k data pieces, compute m parity pieces (using maths such as Reed-Solomon), and store all k + m pieces on different machines. Any k of the k + m pieces can rebuild the original.
100%Figure SDA-CORE-14-1 · Technique 2: erasure coding · © forcsguy.com
With (6, 3) you store 9 pieces for 6 pieces of data: 1.5x overhead, and you can lose any 3 disks. That beats 3x replication on both counts.
100%Figure SDA-CORE-14-2 · Surviving two failures and rebuilding · © forcsguy.com
The trade-offs
Replication Erasure coding Storage overhead high (3x) low (about 1.2x to 1.5x) Read from one copy, fast read several pieces, fast if all data pieces present Repair copy one replica read many pieces and recompute Small updates easy costly (re-encode) Best for hot, small, frequently changed data cold, large, rarely changed objects Making it real
- Placement: put pieces in different failure domains (disk, server, rack, zone) so one event cannot take many.
- Detection and repair: monitor continuously, and rebuild lost pieces before more fail. The faster the repair, the higher the durability.
- Checksums: verify data on read and in background scrubbing to catch silent corruption.
- Tiering: hot data replicated; older data converted to erasure coding.
- Versioning and backups protect against human error, which replication does not (a delete replicates too).
More on this site
Related reading
- Uber Schemaless: A Scalable MySQL DatastoreUber Engineering · UberCase Studies
- How Discord Stores Trillions of MessagesBo Ingram · DiscordBlogs
- Jordan Has No Life: Systems Design Concepts SeriesJordan · Jordan Has No LifeYouTube
- Inside the Magic PocketDropboxCase Studies
Show 6 moreShow fewer
- Kafka Source & DocsApache Software Foundation · apacheGitHub
- Black-Box Auditing: Verifying End-to-End Replication Integrity between MySQL and Redshift at YelpYelp engineers · YelpCase Studies
- Apache CassandraApache Software Foundation · GitHubGitHub
- Types of Replication - SQL ServerMicrosoft engineers · MicrosoftCase Studies
- Concept && CodingShrayansh Jain · Concept && CodingYouTube
- Accelerator: Fast Data Processing Framework at eBayeBay engineers · eBayCase Studies
Go deeper on this
A path through videos, articles, other case studies and code on the same ideas as Accelerator: Fast Data Processing Framework at eBay.
ReadEngineers explain it in their own words.
Blogs
Adobe Research
Look into the future of innovation at Adobe Research
AdobeIntermediate
Blogs
Amazon Science
Technical deep-dives and perspectives from our scientists who are pushing boundaries across multiple scientific disciplines while solving real-world challenges. Explore how our teams combine research excellence, world...
AmazonIntermediate
WatchHow others solve the same problem, on video.
Videos belong to their creators and play from YouTube. More on YouTube (opens in a new tab)
See it in another companyThe same ideas, a different system.
Case Studies · Read here
Aroma: Using ML for Code Recommendation at Facebook
Thousands of engineers write the code to create our apps, which serve billions of people worldwide. This is no trivial task—our services have grown so diverse and complex that the codebase contains…
FacebookIntermediate
Case Studies · Read here
Concourse: Generating Personalized Content Notifications in Near-Real-Time at LinkedIn
Social media are computer-mediated platforms that facilitate creation and sharing of information, ideas, career interests, and other forms of expression via virtual communities and networks. In the last few years, not...
LinkedInIntermediate
Keep learning
Loading community…