“Critical” Versus “Urgent”

Let’s say a staging environment is set up to closely mirror production environment in terms of configuration, data, etc. It’s a very important part of the deployment pipeline and effectively business-critical during the work day for all engineers who need it. It’s constantly monitored to ensure that it’s working properly, mirroring production as best it can, and available for deploys.

Then something goes wrong, and you start seeing “CRITICAL” or “ERROR” all over the place. Oh… I forgot to mention that it’s 3am where you live. Is it really “critical” in that moment? Well, technically it is. The environment is still busted. But do you want to fix it now? Is it urgent?

What Does “Critical” Even Mean, Anyway?

PagerDuty’s incident response model was originally based on the assumption that there is an issue and you need someone NOW: we’ll call the primary on-call repeatedly, we’ll escalate if they are unavailable or indisposed, but come hell or high water: we will get someone immediately and reliably – guaranteed. That’s how we give you the peace-of-mind to sleep at night. If it’s critical, you can count on us: we’re gonna getcha getcha getcha.

But what does “critical” mean? Who defines it? Some of our customers use the ITIL Framework to define incident priority. (In fact, it used to be called “incident severity” until ITIL v3 came around.) But the majority of our customers look to their monitoring tools to define a severity/priority: what threshold has been breached? How many times has it failed in the past 60 seconds? Is it mostly dead or all dead?

It came down to asking one main question: do I need to do something about this right now? Or paraphrasing: is it urgent?  

It May Be “Critical”, But Is It “Urgent”?

Having a range of alert responses is incredibly beneficial, because it often means you have advanced notice about something starting to trend “badly” before it actually gets “bad.” Just because something isn’t urgent doesn’t mean it isn’t important, and having all of your incidents, big and small, in PagerDuty gives you valuable analytics about trouble that could be brewing.

Unfortunately, some of our customers have found that turning every Warning into a PagerDuty incident can be a recipe for sleepless nights, miserable on-calls, and burnout. So now, with Incident Urgencies, you can avoid being woken up for things you know don’t need urgent attention, and you can figure out what’s important now and what can wait.

Only Get Woken Up If It’s “Urgent”

If you’re still getting woken up for stuff that isn’t urgent, use the following runbook:

1. Snooze it until morning! Then take care of it the next day.
2. Start using our Incident Urgencies features to define those events as low-urgency or depending on time-of-day.

And as always, send us some feedback at support@pagerduty.com! Any day, any time. Just don’t be upset if it doesn’t wake us up!