Incident Alert Routing: From Pager Storms to Calm On-Call
Severity tiers, escalation chains, and rotation design — make sure the right person gets woken, in the right order, for the right reason.
Picture the worst page you have ever received. Not the scariest — the most chaotic. The database hiccups at 2am and within ninety seconds forty alerts fire: the database is down, the API is down, the website is down, the queue is backing up. Five people get woken. Nobody knows who owns the incident. Two people start fixing the same thing in different ways. That is a pager storm, and it is not caused by bad monitoring — your checks all correctly reported symptoms. What failed was routing: the rules deciding which alert reaches which human, in what order, and what happens if that human does not answer.
Monitoring answers "what broke." Routing answers "who acts, and how" — severity tiers that mean something, escalation chains instead of contact lists, rotations humans can survive, and dedup rules that turn forty alerts into one incident.
Start with severity, not with tools
Every routing decision flows from one question: how bad is this? If your team cannot answer that in under ten seconds, your routing will be guesswork. Define four tiers and write down what each one means in user impact, not technical terms:
SEV1 — users cannot use the thing. The site is down, checkout is broken, data is at risk. Page immediately, any hour. The whole point of on-call exists for this tier.
SEV2 — degraded, but usable. Response times doubled, a secondary feature is broken, one region is struggling. Page during business hours; at night, a loud push notification that does not require instant action.
SEV3 — needs attention soon. Disk at 80%, certificate expires in two weeks, error rate creeping up. A ticket or chat message. Nobody gets woken.
SEV4 — informational. Deploys completed, routine threshold crossings, FYI-level events. A log entry or a quiet channel message.
Two rules make tiers work. First, the first responder can re-grade: whoever gets the page may upgrade or downgrade the severity within the first minutes, because the initial classification is a guess. Second, severity decides everything downstream — which channel fires, who gets paged, how fast escalation moves. If severity does not drive the routing, it is decoration.
Build escalation chains, not contact lists
A contact list says "these five people care about the database." An escalation chain says "page Ana first; if she does not acknowledge in five minutes, page Ben; after fifteen unacknowledged minutes, page the team lead; after thirty, page everyone." The difference matters at 3am, when the primary is in a dead zone and the list just sits there.
A working chain has three properties. Time-boxed steps: each level gets a fixed window — five minutes for SEV1, longer for lower tiers. Acknowledgement stops the chain: the moment a human confirms "I'm on it," paging stops, which is why every alert needs a one-tap acknowledge path from the lock screen. No silent drops: if nobody acknowledges, the chain keeps climbing — to a manager, a secondary team, a loud group channel — until someone responds. An unacknowledged SEV1 must never simply expire.
Write the chain down where the team can see it, and test it. A quarterly "page drill" — trigger a test alert and watch the chain fire — catches the classic failures: the ex-employee still on the rotation, the phone number that changed, the new hire who was never added.
Route by service, not by person
"Page Dave, he knows the database" is routing by person, and it breaks the moment Dave is on vacation, asleep, or burned out. Route by service and component instead: every alert carries a service tag, and a routing table maps each service to the team or rotation that owns it.
This requires an ownership registry — a simple, boring document that says the payments API belongs to the platform rotation, the blog belongs to the content rotation, the shared database belongs to the infra rotation. It does not need to be fancy; a wiki page counts. What matters is that it is current: review it whenever teams change, and make "who owns this new service?" part of the launch checklist for anything you ship.
Routing by service also fixes the bus-factor problem structurally. When alerts go to a rotation instead of a hero, knowledge spreads because different people handle different incidents. The hero still helps — as the escalation target, not as the single point of failure.
Design rotations humans can survive
On-call is a human-factors problem wearing a technical costume. Weekly rotations are the sweet spot for small teams: long enough for context to build, short enough that nobody dreads the month. Rotate on a weekday morning, never Friday evening.
Make handoffs explicit. A five-minute handoff note — "here's what's flaky, here's what's mid-migration, here's the runbook I updated" — prevents the incoming person from discovering context at 2am. If your team is distributed across time zones, consider follow-the-sun: each region covers its own daylight hours, and nobody gets woken at night. It only works if handoffs are disciplined, but when they are, it is the kindest schedule in operations.
Cap the load. Track how many pages each person gets per rotation. If someone is getting woken three times a week, the problem is not their stamina — it is your thresholds or your reliability, and the fix belongs in the system, not in the person. And compensate the burden openly: on-call is real work, and treating it as invisible labor is how teams quietly lose their best operators.
Suppress the storm: dedup and grouping
Remember the forty alerts from one database hiccup? Routing cannot fix that with chains alone — it needs suppression. Three mechanisms do most of the work:
Group related alerts into one incident. When five checks fail within a minute on the same host or service, it is almost certainly one event — open a single incident and attach the rest as context. Configure grouping before you need it.
Suppress downstream noise. If the database is down, you do not need separate pages for the API, the website, and the queue. Dependency-aware suppression holds dependents' alerts until the dependency recovers — this single rule eliminates most pager storms.
Flap detection and cooldowns. A check that flips between up and down every two minutes should alert once, then go quiet for a cooldown period — not page you twelve times. And maintenance windows: tell the system about planned deploys so it stays silent while you work instead of punishing you for touching the site.
Give the incident a structure
Once an incident is open and a human is on it, routing's job is done — but a lightweight structure keeps the response from dissolving into chaos. Three roles, even for a team of two:
Incident commander — owns the response, makes decisions, and is explicitly not the person typing commands. The commander's job is coordination: what do we know, what are we trying next, who is doing what.
Comms lead — writes the status updates, talks to users, keeps the status page current. During a real outage this is a full-time job; separating it from the technical response is what keeps updates flowing.
Scribe — keeps the timeline: what happened when, what was tried, what worked. The timeline becomes the postmortem's raw material, and writing it down in real time beats reconstructing it from memory the next day.
Link runbooks directly from the alert. The 3am version of a human is not clever; a runbook that opens with "symptoms → three most likely causes in order → commands to check each" turns panic into procedure.
Close the loop: let incidents improve routing
Routing rules rot as services change owners and thresholds drift. The fix is a monthly review asking two questions: which pages were unnecessary, and which incidents were slow to reach the right person? Every unnecessary page gets its rule tuned, demoted, or deleted; every slow-to-route incident gets its ownership or chain fixed.
Feed postmortems back into the routing table, not just into code fixes. "The alert went to the wrong rotation" is a routing bug with the same status as a software bug — file it, fix it, verify it. Teams that do this visibly converge: fewer pages per month, faster acknowledgement, calmer nights. That is the actual metric of routing quality — not how sophisticated your rules are, but how rarely the pager ruins someone's sleep for no reason.