Having one person on-call isn’t enough. What happens if your on-call engineer sleeps through their alert? What happens if their phone’s battery dies without them knowing, or if they get an alert at a really inconvenient time, like when stuck on a bus or in traffic? It will happen.

You need a backup! One or more people, waiting in the wings, ready to spring into action if your primary on-call is criminally negligent unable to perform his or her duties to the best of their abilities at any given time.

These backups don’t need to be AS “on-call” as your primary engineer^[1], but what they lose in readiness they make up in numbers. It often makes sense to have multiple backups.

In PagerDuty we group the currently-on-call engineer and all of his or her backups into an “escalation policy”, which sets the order in which we alert (email/phone/SMS) people, and the delays in between alerts. There are a lot of ways to organize these escalation policies, but I’ll go over some patterns used by many of our customers (both big and small).

Primary and Secondary

You of course already have some sort of “primary” on-call engineer, and this glorious position is hopefully determined by a calendar that rotates between the people in your ops team in a fair way^[2].

Many of our customers will supplement this rotation with an (unimaginatively-named) “secondary” rotation. This secondary rotation is usually setup to shadow the primary rotation by being configured identically to the primary rotation, but offset by a certain amount of time so it’s impossible to be both primary on-call and secondary on-call at the same time. For example, if your primary rotation contains “Alex, Bob, and Charlie”, then your secondary rotation can contain “Bob, Charlie, and Alex”, in that order.

Over 25% of the (many) on-call calendars we manage here at PagerDuty have either the word “primary” or “secondary” in them, so this is a very commonly-used pattern throughout our customer base.

First-tier and Last-tier Support

If your company is big enough or your operations tasks plentiful enough, you may also have a separate first-tier support team that handles all the basic operations tasks before even your “primary” on-call engineer. This front-line team is usually trained in handling all of the repetitive and annoying small problems that crop up often and have clear resolution procedures. (You know, that stuff that you should seriously just fix already, but hey, you’re busy.) These first-tier support teams are often shared amongst multiple engineering teams, and are placed first in an escalation policy.

So who is placed last in the escalation policy? Who is the last-tier support team? Management, of course!

Your escalation policy shouldn’t just end with your primary or secondary ops teams, but should escalate up to their manager’s phone, and then maybe even their manager’s manager’s phone, assuming nobody acknowledges the alert in time.

This has two purposes: (1) management is ultimately responsible for these important systems and is logically who should be informed if any major problems fall through the cracks, and (2) your ops team will be less likely to “miss” an automated PagerDuty phone call if they know it just means that their boss will phoning them up personally in a few minutes and asking some very pointed questions.

Example

So, putting this all together, a (very complete) Escalation Policy example for a hypothetical “Database Ops” team might look something like this:

1. Assign the incident to the user who is on-call in the First-Tier Ops Team schedule
2. Assign the incident to the user who is on-call in the Primary DBA schedule
3. Assign the incident to the user who is on-call in the Secondary DBA schedule
4. Assign the incident to Dilbert Adams (team lead)
5. Assign the incident to Pointi Haredboss (dev manager)

With timeouts of maybe 10-30 minutes in between escalations, depending on your needs. Note that any individuals (managers) put directly in the escalation policy are essentially on-call all the time, as last lines of defense, so should try to keep their cell phones handy and charged at all times.

[1] They don’t need to carry around a mobile broadband device, for instance, and can feel less guilty about doing things like binge drinking while on duty occasionally partaking in activities that might slightly decrease their on-call abilities.

[2] The primary engineer doesn’t have to be determined by a rotation, and you can instead have some poor soul be primary on-call every minute of every day for his whole tenure at your company, but this tenureship may be short due to burnout.

This post originally appeared on the PagerDuty blog on March 8, 2012