Shipping safely and handling a large outage

How strong teams ship software

Big companies deploy thousands of times a day and still stay up. The habits are surprisingly consistent.

• Small teams own services end to end: "you build it, you run it". The people who write the code carry the pager, so they care about reliability.
• Small, frequent changes: each change is easier to review, test, understand and undo than a large release.
• Automation: a CI/CD pipeline builds, runs tests, scans for vulnerabilities and deploys without manual steps.
• Safe rollout: canary releases, feature flags, blue-green switches, automatic rollback on bad metrics (see the deployment lesson).
• Observability: metrics, logs and traces, with alerts on user-visible symptoms.
• Testing in layers: unit, integration and end-to-end tests, plus load and failure testing.
• Standard platforms and templates ("golden paths") so teams do not rebuild logging, deployment or monitoring each time.

100%

Figure SDA-RWS-15-1 · CI/CD with canary and rollback · © forcsguy.com

When things break anyway

Outages will happen. What separates good teams is how they respond.

100%

Figure SDA-RWS-15-2 · Incident response lifecycle · © forcsguy.com

1. Detect

Good alerts fire on symptoms users feel (errors, latency), quickly, and point to a dashboard and a runbook.

2. Triage and roles

Declare an incident and a severity. Assign clear roles so people do not all do the same thing:

• Incident commander: coordinates and decides; does not debug.
• Communications lead: updates stakeholders and customers.
• Operations or subject experts: investigate and apply fixes.
• A scribe keeps a timeline.

3. Mitigate first

Stop the bleeding before finding the cause. Typical moves, roughly from safest to most disruptive:

• Roll back the last change, or turn off a feature flag.
• Fail over to a healthy region or replica.
• Shed load or degrade non-essential features.
• Scale up or add capacity.

The question is "what is the fastest safe way to reduce user harm?", not "why did this happen?".

4. Communicate

Update a status page and internal channels early and at a promised cadence, even when the news is "still investigating". Silence breeds panic and duplicate effort. State the impact, what you know, and when the next update comes.

5. Resolve and verify

Confirm recovery with metrics, not hope. Watch for relapse and clean up temporary measures.

6. Learn: the blameless postmortem

Within a few days write a postmortem: timeline, impact, causes (usually several), what went well, what did not, and action items with owners and dates. Ask "how did our system and process allow this?" instead of "who did this?". People who fear blame hide details, and the same failure returns.

Reducing the next outage

• Eliminate single points of failure and practise failover.
• Game days and chaos testing: break things on purpose in controlled ways.
• Load shedding, timeouts and circuit breakers so a local problem stays local (see the reliability lesson).
• Capacity planning and error budgets so reliability work is scheduled, not forgotten.
• Keep runbooks current and rehearse them. An on-call engineer at 3 a.m. should not be discovering the system for the first time.

Real-world reading