Resilience
Timeouts, retries, circuit breakers, bulkheads and load shedding.
All topics · 2 lessons · 68 resources
Learn Resilience on System Design Atlas
- 01Designing for failure: reliabilityTimeouts, retries, circuit breakers, rate limiting and redundancy: how systems stay up when parts of them break.System Design Foundations · 12 min read · start here
- 02Resilience: timeouts, retries, bulkheads, backpressure and circuit breakersHow to stop one slow or failing dependency from taking everything down: the patterns, how they fit together and where each one goes wrong.Advanced Distributed Systems · 13 min read
Curated Resilience resources
Books
- Security EngineeringHow to build systems that stay secure when things go wrong: threat models, protocols, access control and failures. The author publishes the book free online.University of Cambridge
- Site Reliability EngineeringHow Google runs production systems: SLOs, error budgets, monitoring and incident response. Free to read online from Google.Google
- Building Secure and Reliable SystemsDesigning, implementing and operating systems that are both secure and reliable.O'Reilly / Google
- Release It! (2nd edition)Michael Nygard on stability patterns such as timeouts, bulkheads and circuit breakers.Pragmatic Bookshelf
- The Site Reliability WorkbookThe practical companion to the SRE book, with worked examples of SLOs, alerting and incident response.O'Reilly / Google
Case Studies
- 10 Tips for Building Resilient Payment SystemsPractical rules from Shopify's payments team: idempotency, retries, reconciliation and more.Shopify
- Automating Datacenter Operations at DropboxHow Dropbox's Pirlo system automates switch provisioning and server validation in its data centres, raising first-pass success and engineer output.Dropbox
- Autoscaling AWS Step Functions Activities at YelpAutoscaling AWS Step Functions Activities Devesh Chourasiya, Jingjing Gu Jun 26, 2019 In an ongoing effort to break down our monolithic applications into microservices here at Yelp, we’ve migrated several...Yelp
- Autoscaling Bigtable Clusters based on CPU Load at SpotifyAt Spotify we are moving to managed infrastructure solutions to focus more on our core competencies. In the databases space that means moving to .Spotify
- Autoscaling Jenkins at TrivagoI love to take complex and tedious processes and automate the pain out of them until they are reduced to three or four steps!As part of the release team ...Trivago
- Autoscaling Pub-Sub Consumers at Spotifyis responsible for delivering hundreds of billions of events every day. Most of the events are generated as a response to a user action, such as playing a song, following an artist or clicking on an ad. All in all, more than 300 different types of events are being collected from Spotify clients.Spotify
- Building Reliable Social Infrastructure for Google - Marc Alvidrez, Senior Manager at GoogleBook your room for $225 single or double plus tax or call (800) 255-9925 or (408) 988-8411 and reference USENIX Association or Billing ID #32992. Room rate includes WiFi and complimentary shuttle to the Hyatt Regency Santa Clara.Google
- Bulkheads: Partition and Tolerate Failure in One PartHow ship bulkheads inspire system designs that confine a failure to one part, and what that isolation costs in efficiency.Skife
- Circuit Breaker at ShopifyLearn to predict how your application will behave in times of failure and how to configure every parameter for your Semian circuit breaker.Shopify
- Circuit Breaking in Distributed SystemsMatt Heath discusses how circuit breakers and other similar patterns can be used to increase reliability in distributed systems such as Go-based microservice platforms.InfoQ
- Clusterman: Autoscaling Mesos Clusters at YelpAutoscaling Mesos Clusters with Clusterman David R. Morrison, Compute Infra Tech Lead Feb 19, 2019 Here at Yelp, we host a lot of servers in the cloud. In order to...Yelp
- Designing a Microservices Architecture for FailureThis article introduces the most common techniques and patterns to build and operate a highly available microservices architecture.Risingstack
- Determinism (4 parts) in League of Legends ServerThe price of determinism is eternal vigilance. I’m Rick Hoskinson, an engineer on the League of Legends Core Gameplay Initiative, and this is the final article in the “Determinism” series. Previously in this series…Riotgames
- ELB for Automatic Failover at GoSquaredCombine ELB health checks with Auto Scaling groups to identify failing instances and cycle them out for automatic failover.GoSquared
- Eliminate the Database for Higher Availability at American ExpressFor critical platforms, high availability is a top priority. Learn strategies for reducing complexity and thus increasing availability by eliminating or reducing database dependency.American Express
- Enforce Timeout: A Reliability Methodology at DoorDashWhat would happen if we removed statement timeouts in our Postgresql databases?” That’s one of the questions asked in a management meeting.DoorDash
- Ensuring Resilience to Disaster at QuoraQuora's mission is to share and grow the world's knowledge, and we're committed to making sure Quora is a knowledge repository that will be around forever. To this end, we perform daily backups of all of our ground-truth datastores—MySQL, HBase, Redis, and feed services—and replicate them to mult...Quora
- Failover with Redis Sentinel at VintedWhy Vinted chose Redis Sentinel over Redis Cluster for failover, with notes on failover timing, migration steps and monitoring.Vinted
- GitHub Availability Report: June 2023A real incident write-up, useful as an example of how to explain causes and follow-ups.GitHub
- High Availability Cloud Infrastructure at TwilioA real-world engineering write-up from Twilio on high Availability Cloud Infrastructure at Twilio.Twilio
- High-availability SaaS Infrastructure at FreeAgentThe cloud is commoditising web application hosting but at FreeAgent we continue to build and manage our own infrastructure, using hand-picked servers, switches and elbow grease. Why we do this is a quFreeAgent
- Improved Production Stability with Circuit Breakers at HerokuFun fact: the Heroku API consumes more endpoints than it serves. Our availability is heavily dependent on the availability of the services we interact...Heroku
- Lessons in Resilience at SoundCloudBuilding and operating services distributed across a network is hard. Failures are inevitable. The way forward is having resiliency as a key part of design decisions. This post talks about two key aspects of resiliency when doing RPC at scale - the circuit breaker pattern, and its power combined with client-side…SoundCloud
- Multi-Clustering: Improving Resiliency and Stability of a Large-scale Monolithic API Service at LinkedInHow do you increase the resiliency and stability of a monolithic API service that is used by three different platforms, serving 500+ million members, developed by over 400 engineers, deployed three times per day, and consuming almost 300 downstream services?LinkedIn
- MySQL High Availability at GitHubA real-world engineering write-up from GitHub on failover.GitHub
- NodeJS High Availability at YahooDo you have a NodeJS application serving production traffic? Ever wonder how Yahoo might handle this? If yes, at some point or other you must have thought it would be nice to have a clean and simple solution for high availability. Read below for a detailed overview of how you can achieve this just like we did here…Yahoo
- Operate Kubernetes Reliably at StripeWe recently built a distributed cron job scheduling system on top of Kubernetes, an exciting new platform for container orchestration. Kubernetes is very popular right now and makes a lot of exciting promises: one of the most exciting is that engineers don’t need to know or care what machines their applications run on.Stripe
- Operations (11 parts) at LinkedInEditor’s Note: This is part of the series “Every Day Is Monday in Operations”. All posts in the series will link back to this introduction and every post will be linked in the index below as they are published.LinkedIn
- Practical NoSQL Resilience Design Pattern for the Enterprise at eBayBuilding applications resilient to infrastructure failure is essential for systems that run in a distributed environment, such as NoSQL databases.eBay
- Protector: Circuit Breaker for Time Series Databases at TrivagoAt trivago we store a subset of our realtime metric data in InfluxDB and we are quite impressed by the load it can handle. Despite all the joy, we had to learn some lessons the hard way. It is p...Trivago
- Resilience Engineering with Project Waterbear at LinkedInFounder & CEO, AskNio | Voice AI that puts your best frontline knowledge into every customer and employee conversationLinkedIn
- Resiliency and Disaster Recovery with Kafka at eBayHigh availability, resilience and scalability have become the standard when building user-facing applications. Cloud computing has been a contributing factor to that.eBay
- Site Resiliency at ExpediaSahar Samiei and Willie Wheeler share Expedia’s resiliency journey, starting with resiliency as an afterthought and progressing toward resiliency as a first-class concern. They talk about the importance of partnering with the teams experiencing operational struggles, and equipping them with the data to make the…Expedia
- Stability Patterns Applied in a RESTful ArchitectureA walk through stability patterns such as timeouts, circuit breakers and bulkheads applied to a RESTful service.JavaWorld
- Testing for Disaster Recovery Failover TestingHow can we increase our confidence in the system being working after failing over? We want to confidently declare that it should work, instead of typing the finger-crossed emoji. In this talk, we share our experiment with setting up testing for the DR environment. The biggest question here is whether we would…USENIX
- The Evolution of Global Traffic Routing and FailoverA SREcon16 talk from Microsoft on how service design shapes global traffic routing and failover, with lessons from running Bing.com.USENIX
- Troubleshooting a Connection Timeout Issue with tcp_tw_recycle Enabled at eBayAvailability and stability are very important for eBayeBay
Blogs
- Avoiding fallback in distributed systemsWhy rarely used fallback paths often fail when needed, and what Amazon does instead.Amazon Web Services
- Avoiding insurmountable queue backlogsWhy queues can accumulate work faster than it can be done, and design choices that keep recovery fast.Amazon Web Services
- AWS Architecture BlogReference architectures and design lessons for building on AWS.Amazon Web Services
- Challenges with distributed systemsA tour of why distributed systems are hard: partial failure, unreliable networks, queues and retries.Amazon Web Services
- Circuit BreakerThe pattern that stops calls to a failing dependency and tests recovery gradually.Martin Fowler
- Ensuring rollback safety during deploymentsHow to make sure data and messages written by a new version can still be read after rolling back.Amazon Web Services
- GitHub EngineeringArchitecture, scaling and incident write-ups from GitHub.GitHub
- Google SRE Book: Addressing Cascading FailuresHow overload spreads between services and the defences: load shedding, deadlines, and graceful degradation.Google
- Google SRE Book: Embracing RiskWhy 100 percent reliability is the wrong goal and how error budgets balance reliability against speed.Google
- Google SRE Book: Handling OverloadClient-side throttling, per-client quotas and criticality levels for staying up when demand exceeds capacity.Google
- Implementing health checksKinds of health checks, how load balancers use them, and how a careless check can take down a whole fleet.Amazon Web Services
- Making retries safe with idempotent APIsHow idempotency tokens let clients retry requests without causing duplicate effects.Amazon Web Services
- Marc Brooker's BlogAn AWS distinguished engineer's short essays on distributed systems, databases and performance.brooker.co.za
- PagerDuty BlogWelcome to PagerDuty’s Engineering Blog! Here you’ll find curated, technical content from our internal engineering team.PagerDuty
- Shopify EngineeringScaling a very large commerce platform: sharding, resilience, flash sales and payments.Shopify
- Static stability using Availability ZonesDesigning so a system keeps working through a failure without needing to change anything at that moment.Amazon Web Services
- The Network is ReliableA collection of real network-partition incidents showing why systems must plan for them.aphyr.com
- Timeouts, retries, and backoff with jitterWhy retries can make an outage worse, and how timeouts, capped retries and random jitter prevent retry storms.Amazon Web Services
- Using load shedding to avoid overloadHow to reject excess work early and fairly so a service stays healthy under a surge.Amazon Web Services
- Workload isolation using shuffle shardingA technique that assigns each customer a random small subset of servers, so one bad actor affects very few others.Amazon Web Services
Documentation
- Chaos MonkeyThe tool that randomly terminates instances in production to prove services survive failures.Netflix
- Circuit Breaker patternStates, thresholds and implementation advice for the circuit breaker.Microsoft Azure
- Cloud Design PatternsA catalogue of design patterns for cloud systems with guidance on when to use each.Microsoft Azure