Design a metrics and alerting system

The problem

Engineers need to see how systems behave (CPU, request rate, errors, latency) and be told when something is wrong. The system gathers metrics, stores them, lets people chart and query them, and sends alerts.

Requirements

• Huge, steady write volume; reads are mostly recent data.
• Fast queries for dashboards and alert rules.
• Reliable alerting that stays up when everything else is down.
• Affordable retention (weeks of detail, a year or more of summaries).

Data model

A metric point is name + labels + timestamp + value:

http_requests_total{service="checkout", status="500"}  1717000000  42

Each unique combination of name and labels is a time series. This is what makes slicing by service or status easy, and what makes cardinality matter: a label like user_id creates a series per user and can explode storage.

Architecture

100%

Figure SDA-RWS-09-1 · Metrics and alerting pipeline · © forcsguy.com

Collection: pull or push

• Pull: the collector scrapes an endpoint on each service on a schedule (as Prometheus does). Easy to see which targets are down, and the collector controls the rate.
• Push: services send metrics out (good for short-lived jobs and across firewalls). Needs rate limits and authentication.
• Many systems combine both, with agents on each host batching and forwarding data.

100%

Figure SDA-RWS-09-2 · Pulling metrics from whatever is currently running · © forcsguy.com

Ingest and storage

A queue (such as Kafka) in front absorbs bursts and lets several consumers read the same stream. Writers batch points into the time-series database, which:

• stores data in time-ordered blocks and compresses it heavily (timestamps and values change little, so delta encoding works very well),
• keeps recent data in memory for fast queries,
• supports retention and downsampling.

Downsampling and retention

Keep raw data for a short time (say 7 to 15 days), then roll up to 1-minute and 1-hour aggregates (average, max, count) kept for months or years. Old raw data is deleted or archived. This is what makes long retention affordable.

Querying

Dashboards ask things like "95th percentile latency per service over the last 6 hours". The query service reads relevant series, aggregates them and returns results. Cache common dashboard queries and limit query cost so one heavy dashboard cannot hurt others.

Image: Oleg Alexandrov · Public domain · Source · Wikimedia Commons: Residential smoke detector

Alerting

• Rules are queries with a condition and a duration, for example "error rate above 2% for 5 minutes".
• The evaluator runs rules on a schedule and creates alerts.
• The alert manager groups related alerts, deduplicates repeats, applies silences (planned maintenance) and routes by severity and team (page for urgent, ticket or chat for the rest).
• Alert on symptoms users feel (errors, latency), include a link to a dashboard and a runbook, and review noisy alerts regularly.

Scale

• Shard the time-series database by series (a hash of name and labels) or by time, and replicate for availability.
• Limit cardinality per metric and per tenant.
• Pre-aggregate at the source where a raw per-request metric is not needed.
• Choose the database for the workload: a purpose-built time-series store is usually better than a general SQL database for large-scale metrics.

Real-world reading