Library / Case Studies
MemSQL and Kinesis for Real-time Insights at Disney
We’ve made the very difficult decision to cancel all future O’Reilly in-person conferences. Instead, we’ll continue to invest in and grow O’Reilly online learning, supporting the 5,000 companies and 2.5 million people who count on our experts to help them stay ahead in all facets of business and technology.
By Disney engineers · Disney · Added 2026-10-12
All content belongs to Disney engineers / Disney. We only link to it and give credit. Report a problem
Learn this case study
About 37 min left. Your work is saved as you go.
What you are about to study
Disney explains a real system. You should know the basics: caching, queues and replication.
Ideas you will meet
- WebSockets
- Persistent bidirectional connections for real-time features.
- Distributed Systems
- Consensus, time, failure and the fundamentals of scale-out.
What we already know about this
Our own lessons on these ideas. Read them here before you start, so it makes sense.
Core Concepts: How Things Actually Work · 7 minPolling, long polling, SSE and WebSocketsFour ways to get live updates to a browser, and how to choose between them.Read here
The problem
HTTP is request and response: the client asks. How does the server tell a browser something now?
The four options
Technique How Good Weak Short polling Client asks every N seconds Trivial, works everywhere Wasteful, delayed by N Long polling Client asks; the server holds the request until there is news (or a timeout), then the client asks again Near real time, plain HTTP A held request per client, awkward at scale Server-sent events (SSE) One long HTTP response streams events Simple, auto-reconnect, text over HTTP One-way only WebSockets HTTP upgrades to a two-way connection Low latency both ways Stateful connections, more infrastructure 100%Figure SDA-CORE-08-1 · The four options · © forcsguy.com
Before: polling: keep asking
100%After: WebSocket: server pushes
100%What changed
- No wasted requests when nothing has changed
- Updates arrive the moment they happen
- The server must hold many open connections
Choosing
- Updates are rare or delays are fine: polling.
- The server pushes, the client mostly listens (feeds, scores, notifications, progress): SSE.
- Both sides send often (chat, games, collaborative editing): WebSockets.
Running persistent connections
Connections are long-lived, so capacity is about how many connections a server can hold, not requests per second.
- Put connection handling in servers that are separate from your stateless API.
- Use heartbeats to detect dead connections, and reconnect with backoff and jitter.
- Track which server holds which user so messages can be routed (a presence or session store), and use a pub/sub system between servers.
- When a client reconnects, let it ask for what it missed (events after id X), because connections drop.
- Authenticate on connect and again when tokens expire.
More on this site
Related reading
- RFC 6455: The WebSocket ProtocolIETFDocumentation
- MDN: WebSockets APIMDN Contributors · MozillaDocumentation
- Uber's Real-Time Push PlatformUber Engineering · UberCase Studies
- How Discord Scaled Elixir to 5,000,000 Concurrent UsersDiscordCase Studies
Show 6 moreShow fewer
- The WhatsApp Architecture Facebook Bought for $19BHigh ScalabilityCase Studies
- Erlang at Facebook (slides)Eugene Letuchy · FacebookCase Studies
- Slack EngineeringSlack engineers · SlackBlogs
- Real-time MessagingSlack Engineering · SlackCase Studies
- How Uber Scales Their Real-time Market Platform - High Scalability -High Scalability engineers · High ScalabilityCase Studies
- Real-Time Predictions at DoorDashDoorDash engineers · DoorDashCase Studies
Go deeper on this
A path through videos, articles, other case studies and code on the same ideas as MemSQL and Kinesis for Real-time Insights at Disney.
Warm upGet the vocabulary first.
Books · Free from the author
Concurrent and Distributed Systems (Cambridge lecture notes)
Eight lectures on time, broadcast, replication, consensus and transactions, with notes, slides and video. Free from the author under CC BY-SA.
University of CambridgeIntermediate
Books · Free from the author
High Performance Browser Networking
How TCP, TLS, HTTP/2, WebSocket and mobile networks affect web performance; free to read.
O'ReillyIntermediate
WatchHow others solve the same problem, on video.
Videos belong to their creators and play from YouTube. More on YouTube (opens in a new tab)
ReadEngineers explain it in their own words.
Blogs
Slack Engineering
How Slack builds messaging, search, job queues and infrastructure.
SlackIntermediate
Blogs · Free from the author · Read here
Scalable Web Architecture and Distributed Systems
A full chapter on the building blocks of large websites: availability, performance, reliability, scalability, caches, proxies, indexes, load balancers and queues. Free to read here under CC BY 3.0.
The Architecture of Open Source ApplicationsBeginner40 min
See it in another companyThe same ideas, a different system.
Case Studies · Read here
Brooklin: Distributed Service for Near Real-Time Data Streaming at LinkedIn
Brooklin—a distributed service for streaming data in near real-time and at scale—has been running in production at LinkedIn since 2016, powering thousands of data streams and over 2 trillion messages per day. Today, w...
LinkedInIntermediate
Case Studies · Read here
Concourse: Generating Personalized Content Notifications in Near-Real-Time at LinkedIn
Social media are computer-mediated platforms that facilitate creation and sharing of information, ideas, career interests, and other forms of expression via virtual communities and networks. In the last few years, not...
LinkedInIntermediate
Read the codeSee how it looks when it is real.
GitHub
Engineering Blogs
A huge directory of company engineering blogs to learn from real production systems.
kilimchoiIntermediate
GitHub
Awesome Scalability
Curated reading list of articles on the architecture of scalable, reliable systems.
binhnguyennusIntermediate
Keep learning
Loading community…