Library / Case Studies
DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second - High Scalability -
I remember the excitement of when Twitter first opened up their firehose. As an early adopter o...
By High Scalability engineers · High Scalability · Added 2026-10-12
All content belongs to High Scalability engineers / High Scalability. We only link to it and give credit. Report a problem
Learn this case study
About 37 min left. Your work is saved as you go.
What you are about to study
High Scalability explains a real system. You should know the basics: caching, queues and replication.
Ideas you will meet
- Distributed Systems
- Consensus, time, failure and the fundamentals of scale-out.
What we already know about this
Our own lessons on these ideas. Read them here before you start, so it makes sense.
Core Concepts: How Things Actually Work · 6 minLatency, throughput and bandwidthThree numbers that are often confused, and why a fast network can still feel slow.Read here
Three different questions
- Latency: how long does one request take from start to finish? Measured in milliseconds.
- Bandwidth: how much could the pipe carry per second? A capacity, measured in bits per second.
- Throughput: how much actually flows per second, measured in requests or bytes per second. It is at most the bandwidth, and often much less.
A motorway analogy: bandwidth is the number of lanes, latency is the time one car needs to drive the length, and throughput is how many cars arrive per hour. Widening the road (more lanes) lets more cars through but does not make any single car arrive sooner.
Why it matters in design
Many speed problems are really latency problems, and more bandwidth will not fix them:
Symptom Likely cause Lever Page makes 80 small requests and feels slow Round-trip latency adds up Fewer requests, bundling, HTTP/2, caching Users far away complain Distance CDN, regional servers Big downloads are slow Throughput or bandwidth Compression, parallel transfers, bigger pipe System is slow only at busy times Throughput limit Scale out, queue, shed load 
Image: Syced · CC0 · Source · Wikimedia Commons: Peabird Ethernet switch in 2003Where the time goes
100%Figure SDA-CORE-01-1 · A latency budget for one page · © forcsguy.com
Tail latency
Averages hide the worst experiences. If 1% of requests take 3 seconds, a page that makes 50 calls will hit one of those slow calls most of the time. Track percentiles (p95, p99), and design to keep the slowest requests under control with timeouts and hedged requests.
Latency budget
Break a target (say 300 ms for a page) into a budget: network 80 ms, application 60 ms, database 50 ms, rendering 80 ms, slack 30 ms. When the total exceeds the target you know where to cut.
Real-world reading
Related reading
- Optimizing Web Servers for High Throughput and Low Latency at DropboxDropbox engineers · DropboxCase Studies
- Natural Language Processing and Content Analysis (2 parts) at Condé NastCondé Nast engineers · Condé NastCase Studies
- Finding Kafka throughput limit in infrastructure at DropboxDropbox engineers · DropboxCase Studies
- Scaling Gradient Boosted Trees for Click-Through-Rate Prediction at YelpYelp engineers · YelpCase Studies
Show 6 moreShow fewer
- Machine Learning (2 parts) at Condé NastCondé Nast engineers · Condé NastCase Studies
- DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second - High Scalability -High Scalability engineers · High ScalabilityCase Studies
- Deploying Brotli for Static Content at DropboxDropbox engineers · DropboxCase Studies
- How Twitter Handles 3,000 Images Per Second - High Scalability -High Scalability engineers · High ScalabilityCase Studies
- ESPN's Architecture at Scale - Operating at 100,000 Duh Nuh Nuhs Per Second - High Scalability -High Scalability engineers · High ScalabilityCase Studies
- Architecture NotesArchitecture Notes · architecturenotes.coPodcasts & Newsletters
Go deeper on this
A path through videos, articles, other case studies and code on the same ideas as DataSift Architecture: Realtime Datamining at 120,000 Tweets Per Second - High Scalability -.
Warm upGet the vocabulary first.
Books · Free from the author
Concurrent and Distributed Systems (Cambridge lecture notes)
Eight lectures on time, broadcast, replication, consensus and transactions, with notes, slides and video. Free from the author under CC BY-SA.
University of CambridgeIntermediate
Books · Free from the author
Computer Networks: A Systems Approach
How networks are built and why: protocols, routing, congestion control and the internet's architecture. The authors publish it free online under CC BY 4.0.
Systems ApproachIntermediate
WatchHow others solve the same problem, on video.
Videos belong to their creators and play from YouTube. More on YouTube (opens in a new tab)
ReadEngineers explain it in their own words.
Blogs · Free from the author · Read here
Scalable Web Architecture and Distributed Systems
A full chapter on the building blocks of large websites: availability, performance, reliability, scalability, caches, proxies, indexes, load balancers and queues. Free to read here under CC BY 3.0.
The Architecture of Open Source ApplicationsBeginner40 min
Papers · Read here
Facebook’s photo storage
This paper describes Haystack, an object storage system optimized for Facebook’s Photos application. Facebook currently stores over 260 billion images, which translates to over 20 petabytes of data.
USENIXIntermediate
See it in another companyThe same ideas, a different system.
Case Studies · Read here
Aroma: Using ML for Code Recommendation at Facebook
Thousands of engineers write the code to create our apps, which serve billions of people worldwide. This is no trivial task—our services have grown so diverse and complex that the codebase contains…
FacebookIntermediate
Case Studies · Read here
Brooklin: Distributed Service for Near Real-Time Data Streaming at LinkedIn
Brooklin—a distributed service for streaming data in near real-time and at scale—has been running in production at LinkedIn since 2016, powering thousands of data streams and over 2 trillion messages per day. Today, w...
LinkedInIntermediate
Read the codeSee how it looks when it is real.
GitHub
Engineering Blogs
A huge directory of company engineering blogs to learn from real production systems.
kilimchoiIntermediate
GitHub
Awesome Scalability
Curated reading list of articles on the architecture of scalable, reliable systems.
binhnguyennusIntermediate
Keep learning
Loading community…