Rollback runbook

Decision tree (5 minutes, not 50)

Defect confirmed in prod, introduced by release N?
 ├─ Fixable by lever (flag/config/content/dispatcher)? ──▶ pull lever, verify, done
 ├─ Narrow defect, fix obvious & small? ──▶ hotfix path (if stage-verifiable quickly)
 └─ Broad/unknown blast radius, or fix uncertain ──▶ ROLLBACK now, diagnose calmly after

Named decision maker per release (rollback strategy); default under uncertainty is rollback — you can always re-deploy forward.

Code rollback execution

1. Announce: incident channel, authors told to pause publishing
2. Pipeline: deploy previous artefact (vN−1) — PUBLISH tier first (users), rolling with verification
3. AUTHOR: deploy vN−1 after publish is stable
4. Dispatcher: revert config if release N changed it; flush only affected trees
5. Verify exactly like a deploy: bundles, health, canaries, error.log, dashboards
6. Annotate dashboards; stand down; schedule the post-incident review

The content-since-deploy question

Authors kept publishing between deploy and rollback. Code rollback does not touch content — pages authored against release N's features may now render degraded. Inventory: what was published since deploy (audit log / replication history)? Communicate specific pages to content ops. This is why content-structure changes ship expand/contract (strategy) — so vN−1 still reads content authored under vN.

When rollback isn't clean

Release included an irreversible migration or model change: forward-fix is usually the real option; restore-from-backup is the last resort with data loss to authored content since backup — that trade is a business decision, made by the incident commander with content ops at the table (backup & restore).

After

Every rollback gets a blameless review: which gate should have caught it, and is the artefact/lever/verification gap now a ticket? Rollbacks that teach nothing repeat.