Library / Blogs

Julia Evans' Blog

Clear, friendly explanations of networking, Linux, debugging and how things work.

BeginnerHTTP and QUIC Observability

By Julia Evans · jvns.ca · Added 2026-10-12

Open original source

All content belongs to Julia Evans / jvns.ca. We only link to it and give credit. Report a problem

Learn this article

About 37 min left. Your work is saved as you go.

What you are about to study

jvns.ca wrote this to explain how they thought about a problem. You can follow it without prior background.

Ideas you will meet

HTTP and QUIC: HTTP versions, QUIC, caching headers and web performance.
Observability: Metrics, logs, traces, SLOs and alerting.

What we already know about this

Our own lessons on these ideas. Read them here before you start, so it makes sense.

System Design Foundations · 9 minObservability and running a systemMetrics, logs and traces, how to define 'healthy' with SLOs, and how to ship changes without breaking things.Read here

You cannot fix what you cannot see

A system you cannot observe is a system you cannot trust. Observability is the ability to understand what is happening inside from the outside. Three kinds of signal work together.

Signal	What it is	Best for
Metrics	Numbers over time (requests per second, error rate, CPU)	Dashboards, alerts, trends
Logs	Timestamped records of events	Details of what happened
Traces	The path of one request through many services	Finding where time is spent

Figure SDA-FND-11-1 · From alert to root cause · © forcsguy.com

What to measure

A simple checklist, the four golden signals:

Latency: how long requests take (use percentiles).
Traffic: how much demand there is.
Errors: how many requests fail.
Saturation: how full the busiest resource is (CPU, memory, connections, disk, queue depth).

Averages hide pain. If the average is 100 ms but p99 is 4 seconds, one user in a hundred is waiting four seconds, and with many requests per page that is most of your users sometimes.

Defining healthy: SLIs, SLOs, error budgets

An SLI (indicator) is a measurement, like "fraction of requests answered successfully in under 300 ms".
An SLO (objective) is the target, like "99.9% over 30 days".
The error budget is what is left over: 0.1% of requests may fail. While there is budget, ship features; when it runs out, focus on reliability.

Alert on symptoms users feel (error rate, latency) more than on causes (one machine's CPU). Page a human only for things that need action now.

Ship changes safely

Most outages come from changes. Reduce the risk:

Small, frequent releases are easier to understand and undo than big ones.
Canary release: send a few percent of traffic to the new version and watch the metrics before going wider.
Feature flags: deploy code turned off, then enable it gradually, and switch it off instantly if needed.
Blue-green: keep the old version running so you can switch back in seconds.
Backwards-compatible changes: database and API changes should work with both old and new code during rollout.
Fast rollback must be a rehearsed, boring button.

When things go wrong

Have an incident process: someone leads, someone communicates, mitigate first and investigate later. Afterwards write a blameless postmortem that asks "how did our system and process allow this?" rather than "who did this?". That is how reliability actually improves.

A worked example: error budgets and burn rate

Your service has an SLO of 99.9% of requests succeed over 30 days, and it serves 10 million requests a month.

Error budget: 0.1% of 10,000,000 = 10,000 failed requests may happen in the month before you break the promise.
Burn rate is how fast you spend it. A burn rate of 1 uses exactly the budget over 30 days. If 5% of requests are failing now, the error rate is 50 times the allowed 0.1%, a burn rate of 50: the whole month's budget is gone in 30 / 50 = 0.6 days, about 14 hours.
Alert on burn rate, not on a single bad minute. A common pattern watches two windows together: page someone if the burn rate over the last hour is above about 14 and the last 5 minutes agree (a fast burn: 2% of the monthly budget gone in an hour). Open a ticket for a slower burn, say rate above 6 over 6 hours. Two windows avoid paging for blips and avoid missing slow leaks.

Why this beats "alert when error rate is above X": it ties alerts to what users were promised, it pages only when the budget is in danger, and it gives the team a rule for trade-offs: when the budget is healthy, ship faster; when it is nearly spent, slow down and fix reliability.

Figure SDA-FND-11-2 · Alerting on how fast the error budget burns · © forcsguy.com

Finding the cause with all three signals

A worked investigation shows why you want metrics, traces and logs together.

A metric alerts: the checkout success rate dropped and p99 latency rose. Metrics tell you that something is wrong and how bad, cheaply and over time.
Narrow it down by dimension: break the metric down by region, version and endpoint. Only the new version in one region is affected, a strong hint it is a deploy.
A trace shows where the time goes: one slow checkout request shows 4.8 of its 5 seconds in a call to the payment provider's tokenisation endpoint.
Logs give the detail: filter logs by the trace id from that request and read the exact error: a timeout talking to a new tokenisation URL.
Fix and verify: roll back the version, and watch the metric recover.

Figure SDA-FND-11-3 · From an alert to a root cause · © forcsguy.com

The glue is a correlation id (trace id) carried on every request and written into every log line, so you can jump from one signal to the others.

Mistakes people make

Alerting on causes instead of symptoms. "CPU is 85%" may be fine; "users are seeing errors" never is. Page on symptoms users feel, and use cause metrics for diagnosis.
Averaging percentiles. The mean of two hosts' p99 is meaningless; merge distributions instead (see the t-digest lesson).
High-cardinality labels. A label such as user_id on a metric creates a new time series per user, which can multiply storage and cost by millions. Put such detail in logs or traces.
Logs without structure or ids. Free text is hard to query. Log JSON with a trace id, service and version.
Dashboards no one owns. A wall of graphs nobody reads is noise. Build a small dashboard per service around the golden signals: latency, traffic, errors and saturation.
No runbook. An alert that does not say what to check first wastes the first ten minutes of every incident.

Try it yourself

Run any test in the System Builder and read its charts as if they were production graphs: which signal moves first, and which part would you look at next? Build the search metrics lab to practise percentile maths, and read the t-digest lesson for how real systems summarise latency at scale.

Real-world reading

Monitoring versus observability

Monitoring watches known questions ("is the error rate high?"). Observability is the ability to ask new questions of a running system from the data it emits, which matters when failures are unexpected. It rests on three signal types, often called the pillars:

Signal	What it is	Good for
Metrics	Numbers over time (counts, rates, durations)	Dashboards, alerts, trends; cheap to store
Logs	Timestamped event records	Detail about what happened in one place
Traces	A request's path across services	Finding where time and errors come from

Together with context (deploy markers, config changes) they let you move from "something is wrong" to "this is why". (Collection architecture is in the metrics monitoring lesson; instrumentation in the previous lesson.)

What to measure

Two handy checklists:

The four golden signals (for services): latency (separate successful from failed), traffic (demand), errors (rate of failures) and saturation (how full the most constrained resource is).
RED for request-driven services (Rate, Errors, Duration) and USE for resources (Utilisation, Saturation, Errors).

Use percentiles (p50, p95, p99) rather than averages for latency, because averages hide the slow tail that users feel (see the latency lesson). Keep label cardinality low (do not label metrics with user ids).

SLIs, SLOs and SLAs

Figure SDA-MSV-06-1 · From measurement to promise · © forcsguy.com

SLI (indicator): a ratio of good events to valid events, such as successful requests / valid requests or requests under 300 ms / all requests.
SLO (objective): a target for the SLI over a window. Few, meaningful SLOs beat many.
SLA (agreement): the contractual promise and what happens if missed; keep it looser than the SLO so you have room to react.
Error budget = 1 - SLO. With 99.9 percent over 30 days you may be "down" about 43 minutes. Common reference points: 99 percent ≈ 7.2 hours a month, 99.9 percent ≈ 43 minutes, 99.99 percent ≈ 4.3 minutes.

Using the budget: while budget remains, ship features and take risks; when it is nearly spent, prioritise reliability work. This turns arguments between "ship faster" and "be safer" into a shared, measurable policy.

Alerting that deserves a human

Alert fatigue is a reliability problem: people ignore noisy pages.

Alert on symptoms, not causes: "users see errors" rather than "one pod restarted". Keep cause metrics for dashboards and diagnosis.
Every page must be urgent, actionable and with a runbook; if not, make it a ticket or delete it.
Use burn-rate alerts: page when the error budget is being consumed much faster than allowed (for example 2 percent of a 30-day budget in an hour), ticket for slow burns. Use two windows (short and long) to be fast but not flappy.
Group and deduplicate related alerts; route by ownership; add severity.
Review alerts after incidents and prune them regularly.

Figure SDA-MSV-06-2 · From signal to action · © forcsguy.com

The tools, in one table

Need	Common tools
Metrics collection and alert rules	Prometheus (pull model), cloud metrics, Datadog
Dashboards	Grafana, vendor dashboards
Logs	ELK or OpenSearch stack, Loki, Splunk, cloud logging
Traces	Jaeger, Zipkin, Tempo, vendor APM
Instrumentation standard	OpenTelemetry
Incident paging	PagerDuty, Opsgenie and similar

Dashboards that help

One overview per service showing the golden signals and SLO burn; drill-down panels for dependencies and resources; deploy and config markers on graphs; and consistent layout across services so anyone on call can read any of them.

Incidents and learning

Have on-call rotations with clear escalation, runbooks for known failures, and blameless post-incident reviews that produce concrete follow-ups. Practise with game days. Reliability is a process as much as a technology: measure, set targets, spend the budget wisely, and learn from every failure.

Real-world reading

The problem

A rate limiter restricts how many requests a client may make in a period. It protects a service from abuse, accidental floods and runaway scripts, keeps costs predictable and shares capacity fairly.

Requirements

Limit by user, API key or IP, with different limits per rule or plan.
Add very little latency to each request.
Work across many servers, not just one.
Tell rejected clients clearly what happened.
If the limiter itself fails, the API should not go down with it.

Heavy traffic on a road: without limits, demand beyond capacity makes everyone slower Image: Digitura · CC0 · Source · Wikimedia Commons: Gulf, traffic jam 1970s

Where to put it

Figure SDA-CSD-01-1 · Rate limiter in front of the service · © forcsguy.com

The usual home is the API gateway or a middleware layer, because it sees every request early and rejects cheaply. Building it into each service duplicates effort, and a client-side limiter cannot be trusted.

Here is the same service without a limiter, and with one in front. Switch on Simulate and raise the traffic to see what each version does when a script floods it.

Before: Nothing in front

After: Limiter at the gateway

What changed

The abusive client is stopped at the gateway, not at the service
Counters sit in one shared store, so every server sees the same count
Blocked requests cost almost nothing to answer

Algorithms

Algorithm	Idea	Good	Bad
Token bucket	A bucket refills at a steady rate; each request takes a token	Allows bursts, tiny memory	Two numbers to tune
Leaky bucket	Requests join a queue processed at a fixed rate	Smooth output rate	Bursts fill the queue, old requests delayed
Fixed window counter	Count per minute, reset at the boundary	Simplest	Burst across a boundary doubles the limit
Sliding window log	Keep a timestamp per request, count those within the window	Exact	Memory per request
Sliding window counter	Weight the previous window's count by overlap	Cheap and close to exact	Approximate

Token bucket is a strong default: it is cheap, and bursts are usually fine for real clients. The sliding window counter is a good choice when you need smoother accuracy.

Token bucket in a few lines

For each key store (tokens, last_refill_time). On a request:

Add elapsed x refill_rate tokens, up to the bucket size.
If at least one token is available, take it and allow the request.
Otherwise reject.

The algorithms in pictures

Token bucket. Tokens drip into a bucket of fixed size; every request must take one.

Figure SDA-CSD-01-2 · Token bucket decision · © forcsguy.com

A walk through time (bucket of 4, refilled to full each minute):

Figure SDA-CSD-01-3 · Token bucket over a minute · © forcsguy.com

Leaky bucket. Requests wait in a bounded queue and leave at a fixed rate.

Figure SDA-CSD-01-4 · Leaky bucket · © forcsguy.com

Fixed window counter and its boundary problem. One counter per clock window is simple, but two bursts on either side of the boundary can pass double the limit.

Figure SDA-CSD-01-5 · Fixed window boundary burst · © forcsguy.com

Sliding window log. Keep a timestamp for every accepted request and count those inside the last window.

Figure SDA-CSD-01-6 · Sliding window log (2 per minute) · © forcsguy.com

Sliding window counter. Blend the previous window's count with the current one, weighted by how much of the previous window still overlaps the last minute.

Making it work across servers

Counters must be shared, otherwise each server enforces its own limit and a client gets N times the allowance. A fast in-memory store such as Redis works well.

Use an atomic operation (INCR with expiry, or a small Lua script) so the check and the update happen together. A separate read then write lets two servers both slip through.
Put the key in a predictable shape: rate:{user}:{rule}:{window}.
Partition keys across several Redis nodes for scale (consistent hashing, covered in lesson 2).
To save round trips, some systems keep local counters and sync periodically, accepting small overshoot.

Why counters must be shared when there are several limiter servers:

The system around the algorithm

The race condition when two servers read the same counter at once, and the fix:

Responses and visibility

Return HTTP 429. Include headers so clients can behave: how many requests are allowed, how many remain, and when they reset, plus Retry-After. Log rejections and watch them, since a sudden rise can mean an attack, a bug in a client or a limit set too low.

Going deeper

Two races hide in "read, check, write". If two requests read the counter at 3 at the same moment, both write 4 and one request goes uncounted. A lock fixes it but slows every request; the usual answers are an atomic script that does the read, check and increment in one step inside Redis, or a sorted set that stores one timestamp per request so each request is a single atomic add.

Many limiter servers need shared state. If each server counted only the traffic it sees, a client could send half its requests to each and double its quota. Sticky routing (always the same limiter for a client) helps but breaks when servers change and uneven load builds up. A central store such as Redis, or periodic sync with eventual consistency, is the common fix. Across regions, accept a little over-admission in exchange for not paying a cross-ocean round trip on every request.

Where the rules live. Keep rules as data on disk or in config (for example "login: 5 per minute", "marketing messages: 5 per day"), have workers refresh them into a cache, and let the limiter read them from there, so tightening a rule never needs a deploy.

Measure it. Watch two things: rules that are too strict (many valid users rejected) and an algorithm that cannot absorb a flash-sale burst. In the second case a token bucket, which allows short bursts, is the usual swap.

Other layers and the client's side. The same idea works lower down, for example per IP address in the network layer rather than per user in HTTP. Well-behaved clients cache responses, respect the limit headers, handle the rejection gracefully, and back off before retrying.

An approximate sliding-window counter blends the previous window's count (weighted by how much of it still overlaps) with the current one. It assumes traffic was even in the previous window; published tests by one large CDN found only a tiny fraction of requests were misjudged, so for most uses the saving in memory is worth it.

Failure and edge cases

Limiter store down: usually fail open (let requests through, alert loudly) so you do not turn a limiter outage into an API outage. For security-sensitive endpoints (login attempts) you may prefer failing closed.
Clock differences between servers: base windows on the store's clock, not each server's.
Hard versus soft limits: some rules throttle (delay), others reject.
Many identities: also limit by IP for anonymous traffic, and watch for one client rotating keys.

A worked example: setting the limits

Say your public API has a free plan and a paid plan. You decide a free client may make 60 requests per minute with bursts up to 10, and a paid client 600 per minute.

Token bucket for the free plan: capacity 10, refill 1 token per second. A client idle for 10 seconds can fire 10 requests at once (the burst) and then settles to 1 per second. After a 429, it must wait about a second for the next token.
How much traffic is that? If 50,000 free clients all use their full allowance, that is 3,000,000 requests per minute, or 50,000 per second. You almost never see that, because most clients use a fraction. Sizing from the allowance gives the worst case; sizing from observed use gives the normal case. Plan the limiter and the servers behind it for the first, and alert on the second.
Counter memory: a token bucket needs two numbers per client (tokens, last refill time), about 16 bytes plus key overhead, say 100 bytes. For 10 million active clients that is about 1 GB, comfortably within one Redis node. A sliding window log would store a timestamp per request, up to 60 per client, which is 5 to 10 times more, one reason counters and buckets are the usual choice.

Walking through a rejected request

A client sends its 61st request in a minute.
The edge gateway extracts the key (API key, falling back to IP address) and asks the limiter for that key.
The limiter reads the bucket from the shared store, sees no tokens, and answers "no" without touching the application servers.
The gateway returns HTTP 429 with a Retry-After header and, if you publish them, headers such as X-RateLimit-Limit and X-RateLimit-Remaining, so well-behaved clients can slow themselves down.
The rejection is counted in metrics. A rapid rise from one key suggests abuse; a rise from many keys suggests your limit is too tight or a new client release is misbehaving.

The cost of refusing is tiny compared with the cost of doing the work, which is the whole point: the limiter's decision must be much cheaper than the request it protects.

When the limiter itself fails

A limiter is in the path of every request, so decide in advance what happens when its shared store is slow or down:

Fail open (let requests through): users keep working, but you lose protection exactly when something odd is happening. Good for most public APIs, usually with a local fallback limit per server.
Fail closed (reject everything): safe for anything where overload is worse than refusal, such as an expensive or sensitive operation.
Keep a small local, in-memory limit on each server as a backstop, so even with the shared store down a single server cannot be flooded.

Mistakes people make

Limiting only by IP address. Many users share one address (offices, mobile carriers), and attackers rotate addresses. Prefer an API key or user id, with IP as a fallback.
Using the wrong clock. Limiter servers whose clocks drift apart count windows differently. Take time from the shared store, not each server.
Race conditions on counters. Reading a counter, adding one and writing it back lets two requests slip through together. Use an atomic increment or a small server-side script.
No visibility. Without rejection metrics you cannot tell abuse from a limit set too low.
Rejecting without guidance. Send Retry-After so clients can back off properly instead of retrying immediately and making things worse.

Try it yourself

Open the System Builder challenge A noisy client. One abusive client sends 97% of the traffic. Try to survive it by adding servers (you cannot, within 12), then switch on the rate limiter and watch the real users' errors disappear. Then build the sliding window rate limiter lab and see the boundary burst for yourself. You can also open the noisy-client design on the free canvas and place the rate limiter where you think it belongs.

More on this site

Go deeper on this

A path through videos, articles, other case studies and code on the same ideas as Julia Evans' Blog.

1
Warm upGet the vocabulary first.
- Courses
  DevOps Roadmap
  A visual path through DevOps tools and ideas.
  roadmap.shBeginner
  Study this Original (opens in a new tab)
- Documentation
  Prometheus overview
  The pull-based metrics system and its data model of labelled time series.
  CNCF / PrometheusBeginner
  Study this Original (opens in a new tab)
WatchHow others solve the same problem, on video.
Videos belong to their creators and play from YouTube. More on YouTube (opens in a new tab)
3
ReadEngineers explain it in their own words.
- Blogs
  Google SRE Book: Postmortem Culture
  How to learn from incidents with blameless reviews that produce concrete follow-ups.
  GoogleBeginner
  Study this Original (opens in a new tab)
- Blogs
  Cloudflare Blog
  Deep technical posts on networking, DNS, CDN, security and edge computing from a company that runs a huge global network.
  CloudflareIntermediate
  Study this Original (opens in a new tab)
4
See it in another companyThe same ideas, a different system.
- Case Studies · Read here
  Improve Zipkin Traces using Kubernetes Pod Metadata at SoundCloud
  We revisited our distributed tracing setup and incorporated Kubernetes pod metadata into it, significantly enhancing our engineers’ ability to troubleshoot problems that cut across microservices.
  SoundCloudIntermediate
  Study this Original (opens in a new tab)
- Case Studies · Read here
  Monitor DNS systems at Stripe
  The secret life of DNS packets: investigating complex networks.
  StripeIntermediate
  Study this Original (opens in a new tab)
5
Read the codeSee how it looks when it is real.
- GitHub
  Grafana
  The dashboard and visualisation tool for metrics, logs and traces.
  GitHubBeginner
  Study this Original (opens in a new tab)
- GitHub
  Jaeger
  A distributed tracing system modelled on Dapper.
  GitHubIntermediate
  Study this Original (opens in a new tab)

Keep learning

Loading community…

Cookie preferences

Julia Evans' Blog

Learn this article

What you are about to study

You cannot fix what you cannot see

What to measure

Defining healthy: SLIs, SLOs, error budgets

Ship changes safely

When things go wrong

A worked example: error budgets and burn rate

Finding the cause with all three signals

Mistakes people make

Try it yourself

Real-world reading

Monitoring versus observability

What to measure

SLIs, SLOs and SLAs

Alerting that deserves a human

The tools, in one table

Dashboards that help

Incidents and learning

Real-world reading

The problem

Where to put it

Algorithms

Token bucket in a few lines

The algorithms in pictures

Making it work across servers

The system around the algorithm

Responses and visibility

Going deeper

Failure and edge cases

A worked example: setting the limits

Walking through a rejected request

When the limiter itself fails

Mistakes people make

Try it yourself

More on this site

Go deeper on this

Warm upGet the vocabulary first.

WatchHow others solve the same problem, on video.

ReadEngineers explain it in their own words.

See it in another companyThe same ideas, a different system.

Read the codeSee how it looks when it is real.

Keep learning