Hotfix process
Purpose: Ship an urgent production fix fast without abandoning the discipline that keeps releases safe.
Who this page is for
| Audience | Why it matters to you |
|---|---|
| All engineers | You will be in one eventually |
| Tech leads | Decision owners |
When a hotfix is justified
Sev1/sev2 production defect where waiting for the next release cycle costs more than an expedited change: broken purchase/lead flows, data exposure, widespread rendering failure, security patch. "The button is the wrong blue" waits.
The expedited path (same rails, fewer stops)
1. Branch from the DEPLOYED tag (release/2.14), never from main
2. Minimal diff — fix only; refactors ride the next release
3. Review: one senior reviewer minimum, even at 2am (a second pair of eyes is the last defence standing)
4. Pipeline: build v2.14.1 → deploy stage → targeted smoke of the defect + standard smoke
5. Prod deploy via the normal runbook (rolling, verified) — expedited ≠ unverified
6. BACKPORT to main immediately — an unbackported hotfix resurfaces next release, at the same hour
7. Post-incident note: why it escaped, which gate would have caught itEmergency levers that are NOT hotfixes (often faster)
| Lever | When |
|---|---|
| Config flag off | The broken feature is flag-gated (this is why flags exist) |
| OSGi config change via config pipeline | Endpoint/timeout/toggle issue |
| Dispatcher rule | Block an abused endpoint while fixing properly |
| Content fix | The "bug" is content — authors fix, publish, done |
| Rollback | Defect shipped in the latest release and is broad → rollback beats forward-fixing under pressure |
Check the lever list before cutting a hotfix branch; the fastest code change is none.
Anti-patterns
Console edits as "temporary" fixes (they drift — config management), hotfixing from main (ships unreleased features accidentally), skipping stage entirely (the 5-minute smoke has caught enough second disasters to pay for itself forever).