High-availability SaaS Infrastructure at FreeAgent

Ideas you will meet

Resilience

Timeouts, retries, circuit breakers, bulkheads and load shedding.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

• Advanced Distributed Systems · 13 minResilience: timeouts, retries, bulkheads, backpressure and circuit breakersHow to stop one slow or failing dependency from taking everything down: the patterns, how they fit together and where each one goes wrong.Read hereClose

  The core problem

  In a system of many services, slowness is more dangerous than failure. A dependency that fails fast is easy to handle. One that responds in 30 seconds makes callers hold threads, connections and memory until they too stop responding: a cascading failure. Resilience patterns limit how far a problem can spread.

  The toolbox

  100%

  Figure SDA-ADS-06-1 · Resilience layers around a call · © forcsguy.com

  Timeouts

  Every network call needs a timeout, including connect, read and overall. Pick from measured latency (for example slightly above the 99th percentile), not a guess, and make the caller's budget smaller than its own caller's so work is abandoned in an orderly way. Propagate deadlines across calls.

  Retries

  Retry only transient failures and only idempotent operations (or use idempotency keys). Use exponential backoff with jitter (random spread) so clients do not retry in lockstep, cap the attempts, and apply a retry budget (for example no more than 10 percent extra traffic). Retry at one layer, not at every layer, or attempts multiply.

  Circuit breaker

  A breaker watches failures or slow calls. Closed: normal. After too many failures it opens: calls fail immediately (or use the fallback) so the dependency can recover and callers stop wasting resources. After a cool-down it goes half-open: a few trial calls decide whether to close again.

  100%

  Figure SDA-ADS-06-2 · Circuit breaker states · © forcsguy.com

  Bulkheads

  Partition resources so one dependency cannot consume them all: separate thread pools or connection pools per dependency, separate queues per tenant or per priority, or separate instances for critical paths. A slow recommendation service then cannot starve checkout.

  Backpressure and load shedding

  If producers outpace consumers, work piles up. Without limits the buffer grows until memory is exhausted or latency becomes huge. Options:

  • Bounded queues that reject or block when full.
  • Signalling upstream to slow down (credit-based flow in streaming systems, TCP windows, reactive streams).
  • Load shedding: deliberately drop low-priority work early (return 429 or 503 quickly) to protect the rest.
  • Rate limiting at the edge (see the rate-limiter lesson).

  Failing fast and visibly under overload is better than accepting everything and failing slowly.

  Fallbacks and graceful degradation

  Return something useful when a dependency fails: cached data, a default, a simpler algorithm, or hide a non-essential section. Decide which features are essential before an incident, and test the degraded mode.

  How they interact

  • A timeout is the minimum; without it nothing else works.
  • Retries inside a breaker inside a bulkhead is the usual nesting.
  • Everything needs metrics: breaker state, rejected calls, queue depth, timeout counts. Alert on them.
  • Test it: inject latency and failures (chaos experiments) to confirm the patterns work before production does it for you.

  Common mistakes

  Retries with no backoff; timeouts longer than the caller's; unbounded queues; a shared thread pool for everything; breakers that never close; fallbacks that call the same failing service; and testing only the happy path.

  Real-world reading

  Related reading

  • Timeouts, retries, and backoff with jitterAmazon Builders' Library · Amazon Web ServicesBlogs
  • Using load shedding to avoid overloadAmazon Builders' Library · Amazon Web ServicesBlogs
  • Circuit BreakerMartin FowlerBlogs
  • Circuit Breaker patternMicrosoft · Microsoft AzureDocumentation

  Show 6 moreShow fewer

  • Google SRE Book: Addressing Cascading FailuresGoogle SRE team · GoogleBlogs
  • Chaos Monkey (source)Netflix · GitHubGitHub
  • Bulkheads: Partition and Tolerate Failure in One PartSkife engineers · SkifeCase Studies
  • Protector: Circuit Breaker for Time Series Databases at TrivagoTrivago engineers · TrivagoCase Studies
  • Autoscaling Jenkins at TrivagoTrivago engineers · TrivagoCase Studies
  • Troubleshooting a Connection Timeout Issue with tcp_tw_recycle Enabled at eBayeBay engineers · eBayCase Studies

  Open this lesson with its quiz and exercise

• Interview Preparation · 9 minTrade-off and numbers cheat sheetA one-page reference: latency numbers, availability maths, capacity formulas and the standard trade-offs, for quick revision.Read hereClose

  Round numbers worth remembering

  These are approximate orders of magnitude, enough for estimates (exact figures vary by hardware and year).

  Operation	Rough time
  CPU cache or memory read	nanoseconds to about 100 ns
  Read 1 MB sequentially from memory	tens of microseconds
  SSD random read	around 100 microseconds
  Round trip within a data centre	around 0.5 ms
  Read 1 MB sequentially from SSD	around 1 ms
  Disk seek (hard disk)	around 10 ms
  Round trip, same continent	tens of ms
  Round trip, across oceans	100 ms and above

  Take-aways: memory is far faster than disk, disk faster than network across regions; avoid unneeded round trips; sequential access beats random.

  Powers and sizes

  Unit	Approximate size
  1 KB	a thousand bytes (a short text)
  1 MB	a million bytes (a photo, a small document)
  1 GB	a billion bytes
  1 TB	a thousand GB
  1 PB	a thousand TB
  Seconds per day	about 86,400 (use about 100,000 for quick maths)
  Seconds per month	about 2.6 million
  Seconds per year	about 31.5 million

  Quick capacity formulas

  • Requests per second (average) = requests per day / 86,400. Peak = average times a peak factor (2 to 5).
  • Storage = records per day times bytes per record times days retained (times replication factor).
  • Bandwidth = requests per second times average response size.
  • Servers = peak requests per second / requests per second one server handles, plus headroom.
  • Cache size = hot working set (often 20 percent of data serves 80 percent of reads).
  • Connections (for chat or streaming) = concurrent users; memory per connection times that gives a first estimate.

  Example: 10 million users, each posting 2 items a day of 1 KB: 20 million items a day, about 230 writes per second, 20 GB a day, about 7 TB a year before replication.

  Availability maths

  Availability	Downtime per year	Per month (30 days)
  99%	about 3.65 days	about 7.2 hours
  99.9%	about 8.8 hours	about 43 minutes
  99.99%	about 53 minutes	about 4.3 minutes
  99.999%	about 5.3 minutes	about 26 seconds

  Components in series multiply (two 99.9% parts in a chain give about 99.8%); components in parallel (redundant) improve availability: failure needs both to fail. This is why redundancy across zones helps and long dependency chains hurt.

  The standard trade-offs

  100%

  Figure SDA-INT-03-1 · Trade-offs to name in every design · © forcsguy.com

  Choice	Option A	Option B
  Data model	Relational: constraints, transactions	NoSQL: flexible, easier horizontal scale
  Scaling	Vertical: simple, limited	Horizontal: scalable, more complex
  Replication	Synchronous: safer, slower	Asynchronous: faster, may lose data on failover
  Feeds	Fan-out on write: fast reads, heavy writes	Fan-out on read: cheap writes, slow reads
  Delivery	At-least-once plus idempotency	At-most-once for loss-tolerant data
  Caching	Cache-aside: simple, risk of staleness	Write-through: fresher, slower writes
  Consistency	Strong: easy to reason about	Eventual: available and fast
  Communication	Synchronous calls: simple	Asynchronous messages: resilient, harder to trace
  Architecture	Monolith: simple to start	Microservices: independent scaling and teams, more operations
  ID generation	Sequential: sortable, coordination	Random (UUID): no coordination, poor locality

  Patterns to reach for

  Problem	Pattern
  Reads overwhelm the database	Cache, read replicas, CDN
  Writes overwhelm one database	Sharding, queues to smooth bursts, LSM-based stores
  Hot key	Replicate, split the key, local caching
  Slow external dependency	Timeout, bulkhead, circuit breaker, queue
  Duplicate deliveries	Idempotency keys, deduplication
  Cross-service transaction	Saga with compensation
  Need history or audit	Event log, event sourcing
  Unique "does this exist" checks at scale	Bloom filter in front of storage
  Even spread when nodes change	Consistent hashing
  Overload	Rate limit, load shed, backpressure

  Final checks before you finish an answer

  • Requirements and scale stated; numbers used.
  • Main flow and data model clear.
  • Bottlenecks and single points of failure named, with mitigations.
  • Failures, consistency and security discussed.
  • Monitoring and next steps mentioned.
  • Trade-offs spoken aloud.

  Real-world reading

  Related reading

  • MySQL High Availability at GitHubGitHub Engineering · GitHubCase Studies
  • Mitigating MySQL Replication Lag and Reducing Read Load at GithubGithub engineers · GithubCase Studies
  • Read Consistency with Database Replicas at ShopifyShopify engineers · ShopifyCase Studies
  • Uber Schemaless: A Scalable MySQL DatastoreUber Engineering · UberCase Studies

  Show 6 moreShow fewer

  • Apache CassandraApache Software Foundation · GitHubGitHub
  • Jordan Has No Life: Systems Design Concepts SeriesJordan · Jordan Has No LifeYouTube
  • Black-Box Auditing: Verifying End-to-End Replication Integrity between MySQL and Redshift at YelpYelp engineers · YelpCase Studies
  • Cassandra architecture overviewApache Cassandra · Apache Software FoundationDocumentation
  • Eliminate the Database for Higher Availability at American ExpressAmerican Express engineers · American ExpressCase Studies
  • High-availability SaaS Infrastructure at FreeAgentFreeAgent engineers · FreeAgentCase Studies

  Open this lesson with its quiz and exercise