We’ve been carefully reviewing your feature requests to try to understand how best to improve PagerDuty.  One feature request came up far more often than the rest: make it easier to integrate PagerDuty with monitoring tools.  We’ve taken this request to heart and have begun reworking PagerDuty so that we will soon be able to support API integration with monitoring systems like Nagios.

Before we can release an API for PagerDuty, though, we need to correct some over-simplifications in PagerDuty’s design.  Up until now, PD required you to create a new alarm for each kind of problem that your monitoring systems are capable of detecting.  Unfortunately, this doesn’t work so well if you’re using a monitoring tool like Nagios that can track thousands of conditions at once.

So, for the past few weeks, we’ve been busy re-designing PD so that it can handle multiple open incidents from a single monitoring service.  We’re just about ready to roll out this new-and-improved version of PagerDuty, but before we do, we’d like to give you the chance to familiarize yourself with the system, and let us know if there’s any way we can make the new system even better prior to launch.

How do I try it out?

Glad you asked!  For at least the next week, we’re going to run a preview of the new PagerDuty system.  To log in, visit:

http://<your-subdomain>.pd-staging.com

and use your normal PagerDuty email and password.

All of your data has been migrated from your PagerDuty account, so you can see exactly how the system will look once we update the software on our production servers.  The preview release is fully functional, so please feel free to kick-the-tires and have it dispatch a few alerts for you.  Don’t worry — nothing you do in your preview account will have any impact to your production environment.  Of course, all SMS and phone calls made from the preview environment will be free of charge.

In order to maintain backward compatibility, we’ve configured all existing alarms to only support one active incident at once.  To remove this restriction, simply:

1. Click the “Services” tab
2. Select one of your existing alarms
3. Click “Edit this service” on the right side of the screen
4. Switch the incident creation mode to “Open a new incident for each trigger email”
5. Click “Save Changes”

The big change: Multi-Incident Support

PagerDuty is now capable of tracking multiple open concurrent incidents.  Put another way, your monitoring system can tell PagerDuty about 100 simultaneous and independent problems without you needing to create 100 PagerDuty alarms, as is the case now.

PagerDuty now uses “incidents” rather than “alarms” as the main object.  Your support team will be acknowledging, escalating, and resolving incidents, instead of the alarms that they work with now.  Incidents in PagerDuty are similar to tickets in a bug tracking system: they are created when a problem is detected, and are resolved or closed when the problem is fixed.

Since PagerDuty can now handle hundreds of open incidents at once, we’ve tried to carefully design PagerDuty’s interface to make it easy to work with large collections of incidents.  The new Incidents and Dashboard tabs feature tables that let you see all of the open incidents assigned to you at a glance.  You can also easily triage your incidents straight from these pages using the controls located at the top of the table.

One of the biggest advantages to PagerDuty’s existing single-incident design is that it can’t generate alert storms.  Even if Nagios sends hundreds of emails to PagerDuty at once, you’ll only receive one set of phone calls and SMS messages.  We’ve been careful to preserve this feature in the new version of the product.  PagerDuty will intelligently bundle multiple incidents into a single set of notifications so that you aren’t overwhelmed with alerts.

Other changes

We’ve made a few of other small changes to support the new multi-incident functionality.

First, we’ve renamed “alarms” to “services”.  Alarms/services are now used only to represent an integration point between PagerDuty and your monitoring services.  Currently, PagerDuty only has one type of service: the simple email-triggered mechanism you used in the previous version of PagerDuty.  In the coming weeks, we will be adding support for API-driven services so that we can offer even closer integration with products like Nagios.

For similar reasons, we’ve renamed “alarm groups” to “escalation policies”.  We think the new name better captures the use of these objects.

Finally, we’ve renamed incident “suppression” to “acknowledge”.  As before, this feature is used to temporarily prevent an incident from generating alerts.  We thought the word “acknowledge” better captured the purpose of the feature: “stop bothering me about this problem for now… I’m working on it!”.

What’s next

Next up is support for a PagerDuty API.  Once we’ve deployed PagerDuty multi-incident to production and ensured that everyone is comfortable with the new system, we’ll announce our plans for the API.  Stay tuned for more info!