Logging & alerting readiness
Purpose: Verify a release is observable before it ships: the questions you must be able to answer at 3am.
Who this page is for
| Audience | Why it matters to you |
|---|---|
| All engineers | You build it, you make it observable |
| Ops engineers | Consumers of the answers |
The 3am test
For each new feature, the on-call engineer (who didn't build it) must be able to answer from dashboards/logs alone:
| Question | Requires |
|---|---|
| Is it working right now? | Health check / synthetic probe / rate metric |
| When did it break? | Time-series of its success/error rate |
| What exactly failed? | Correlated, levelled logs with context (IDs not payloads) |
| Who is affected? | Scope signal (all pages? one integration? one publisher?) |
| How do I stop the bleeding? | Runbook entry: fallback/flag/flush/rollback |
If any cell is empty, the feature isn't done.
Release-time checklist
- [ ] New integrations emit success/failure/latency metrics (or at minimum structured log lines a monitor parses)
- [ ] New Sling Health Checks tagged and visible at /system/health (monitoring endpoints)
- [ ] Log levels correct for prod (your packages at info/warn; debug config ready to enable, not enabled)
- [ ] Alerts defined WITH thresholds and routed (an unrouted alert is a diary entry)
- [ ] Dashboards updated — the release's key metric visible next to platform basics
- [ ] Runbook entry per new failure mode: symptom → check → action
- [ ] Correlation IDs propagate through new outbound calls
Alert quality bar
Every alert must be actionable (a human does something), urgent (or it's a ticket, not an alert), and documented (runbook link in the alert text). Alert fatigue is a security and reliability incident in slow motion — prune relentlessly. Deeper practice: Monitoring & alerting.