Replication queue blocked
Who this page is for
| Audience | Why it matters to you |
|---|---|
| All engineers | Content-not-updating incidents |
| Platform engineers | Prevention |
Recognise it
Authors report "published but nothing happens" — for everything, not one page (one page = different lane: that page's ACLs/references). Author → replication agents console: queue depth growing, oldest item stuck, retries climbing.
Diagnose the stuck head item
The queue is ordered: one poisoned item blocks everything behind it. Open the agent's log (linked from the agent page):
| Log pattern | Cause | Action |
|---|---|---|
| Connection refused / timeout | Publish down or unreachable (network, TLS cert expired on receiver) | Fix transport; Test Connection button confirms |
| 401/403 from publish | Transport user credentials/permissions broken (rotation? hardening change?) | Fix transport user; verify on publish |
| 500 from publish on one payload | Publish-side import failing (constraint violation, disk full, corrupt payload) | Publish error.log has the paired stack trace |
| OversizedItem / OOM patterns | Someone activated a giant tree/asset batch | See bulk note below |
Actions on the queue itself: retry the head after fixing transport; remove the head item only when diagnosed (record what it was — it needs re-publishing later); clearing the whole queue is a last resort that requires re-activation reconciliation afterwards.
Bulk-activation self-inflicted jams
Tree activation of 10k pages during business hours: queue technically healthy, throughput just saturated (and the dispatcher invalidation storm arrives when it drains — caching). Batch big activations off-peak, in chunks, and warn ops.
Prevention
- Alert on queue blocked >5min and depth > threshold (monitoring) — authors should never be the alerting system.
- Transport TLS certs and users in the rotation calendar (secrets rotation discipline applies).
- Runbook entry with the table above; the 3am fix is transport 90% of the time.