Blog

If an asteroid strikes PagerDuty

by Baskar Puvanathasan August 10, 2011 | 3 min read

Updated on 9/21: We have replaced Twitter with our status page as a communication method.

At PagerDuty we strive for 100% uptime, and it is a major focus of ours.  But what happens if we fail?  Who watches the watchers?  Should a cataclysmic event take out PagerDuty, we want to make sure that you are aware that a critical piece of your operations infrastructure is compromised.

Such an outage happened recently due to the major AWS cross-AZ failure in US-EAST.  We are moving cross-cloud very soon and will be following up this posting with a post-mortem of the event.

We are implementing a new (completely separate and external) opt-in system for those interested in getting notified when PagerDuty is experiencing large-scale issues.  While this is being constructed, we have a temporary (but equally functional) mechanism to alert you of such catastrophic events.  We have a PagerDuty Status page customers can subscribe to get updates on incidents we may experience.

Again, it is an opt-in program, but we’d recommend that you participate in it.  We would tweet from this account very rarely, and only during such times that PagerDuty is having major problems delivering notifications (and hopefully never again). These tweets will never be automated; it would come directly from the PagerDuty Ops team. We also ask that you enable mobile notifications for your Twitter account so that tweets from @pagerdutyops reach you instantly — we’re including the simple setup instructions below.

What you’ll need to get started

You’ll need a Twitter account — please create one from here, if you don’t have one already. Follow @pagerdutyops from your Twitter account.

Setting up Twitter to send mobile notifications

Here is the short instructions on how to setup Twitter to send you a text message, when we tweet from @pagerdutyops.

  1. In Twitter, click on Settings under your profile.
  2. Click on the Mobile tab.
  3. Select your Country, enter your Phone Number and select your Mobile Carrier.
  4. Click the Start button.
  5. Twitter will now ask you to verify your mobile number. Follow the instructions on this page — this may require you to send a text message to Twitter with a special code.
  6. Once verified, you’ll be taken to the Text messaging page. Select Tweets from people you’ve enabled for mobile notifications, if not selected already.
  7. Click on the Save button to save your changes.
  8. Navigate to @pagerdutyops. Click on the Follow button, if not following already.
  9. Click on the small mobile icon besides the Following button to enable mobile notifications from @pagerdutyops.

And you should be all set to (hopefully, again, never) receive external notifications to learn about when and if PagerDuty is having failures in delivering notifications.