Observability: metrics, SLIs, SLOs and alerting

Monitoring versus observability

Monitoring watches known questions ("is the error rate high?"). Observability is the ability to ask new questions of a running system from the data it emits, which matters when failures are unexpected. It rests on three signal types, often called the pillars:

Signal	What it is	Good for
Metrics	Numbers over time (counts, rates, durations)	Dashboards, alerts, trends; cheap to store
Logs	Timestamped event records	Detail about what happened in one place
Traces	A request's path across services	Finding where time and errors come from

Together with context (deploy markers, config changes) they let you move from "something is wrong" to "this is why". (Collection architecture is in the metrics monitoring lesson; instrumentation in the previous lesson.)

What to measure

Two handy checklists:

• The four golden signals (for services): latency (separate successful from failed), traffic (demand), errors (rate of failures) and saturation (how full the most constrained resource is).
• RED for request-driven services (Rate, Errors, Duration) and USE for resources (Utilisation, Saturation, Errors).

Use percentiles (p50, p95, p99) rather than averages for latency, because averages hide the slow tail that users feel (see the latency lesson). Keep label cardinality low (do not label metrics with user ids).

SLIs, SLOs and SLAs

100%

Figure SDA-MSV-06-1 · From measurement to promise · © forcsguy.com

• SLI (indicator): a ratio of good events to valid events, such as successful requests / valid requests or requests under 300 ms / all requests.
• SLO (objective): a target for the SLI over a window. Few, meaningful SLOs beat many.
• SLA (agreement): the contractual promise and what happens if missed; keep it looser than the SLO so you have room to react.
• Error budget = 1 - SLO. With 99.9 percent over 30 days you may be "down" about 43 minutes. Common reference points: 99 percent ≈ 7.2 hours a month, 99.9 percent ≈ 43 minutes, 99.99 percent ≈ 4.3 minutes.

Using the budget: while budget remains, ship features and take risks; when it is nearly spent, prioritise reliability work. This turns arguments between "ship faster" and "be safer" into a shared, measurable policy.

Alerting that deserves a human

Alert fatigue is a reliability problem: people ignore noisy pages.

• Alert on symptoms, not causes: "users see errors" rather than "one pod restarted". Keep cause metrics for dashboards and diagnosis.
• Every page must be urgent, actionable and with a runbook; if not, make it a ticket or delete it.
• Use burn-rate alerts: page when the error budget is being consumed much faster than allowed (for example 2 percent of a 30-day budget in an hour), ticket for slow burns. Use two windows (short and long) to be fast but not flappy.
• Group and deduplicate related alerts; route by ownership; add severity.
• Review alerts after incidents and prune them regularly.

100%

Figure SDA-MSV-06-2 · From signal to action · © forcsguy.com

The tools, in one table

Need	Common tools
Metrics collection and alert rules	Prometheus (pull model), cloud metrics, Datadog
Dashboards	Grafana, vendor dashboards
Logs	ELK or OpenSearch stack, Loki, Splunk, cloud logging
Traces	Jaeger, Zipkin, Tempo, vendor APM
Instrumentation standard	OpenTelemetry
Incident paging	PagerDuty, Opsgenie and similar

Dashboards that help

One overview per service showing the golden signals and SLO burn; drill-down panels for dependencies and resources; deploy and config markers on graphs; and consistent layout across services so anyone on call can read any of them.

Incidents and learning

Have on-call rotations with clear escalation, runbooks for known failures, and blameless post-incident reviews that produce concrete follow-ups. Practise with game days. Reliability is a process as much as a technology: measure, set targets, spend the budget wisely, and learn from every failure.

Real-world reading