Since we’ve launched Single Sign-On we’ve received a lot of good feedback from our customers about having to remember one less password. While SSO doesn’t impact PagerDuty’s core operations performance management feature set, planning and implementing the new feature took a lot of careful consideration.

Coming Up with A Plan

After deciding to offer single sign-on support for PagerDuty, we knew we needed something that can integrate well with our Rails application. We researched possible solutions, but determined that OneLogin’s Ruby SAML Toolkit was the best solution for us. It is trusted by other companies in our space and had all the capabilities we envisioned needing.

To ensure this would work, we scrounged up a working prototype with a simple implementation. One endpoint to initiate authentication from our end and another to consume the response from identity providers.

The proof of concept worked, allowing us to start building the feature we wanted.

OneLogin’s Ruby SAML Toolkit

To make sure using OneLogin’s Ruby SAML toolkit didn’t introduce vulnerabilities to our systems we began with a mixture of manual and automated tests. One scenario we tested was when a SAML response from an identity provider is captured, and subsequently tampered with (e.g. changed email address) what would happen? The toolkit already has a few check-in points in place for this scenario and thus forbids authentication.

Even with the toolkit’s robust functionality, we did make a couple modifications to make the toolkit fit our needs and give it the PagerDuty stamp of approval. For example. we’ve modified SAML endpoints to always use SSL and made sure certificate validity periods are honored.

What about Mobile and SAML?

In order to support SSO for our mobile applications, we used SAML in combination with OAuth where our iOS and Android applications obtain an authentication token after successful SAML authentication.

Learning from Customer Previews

Before any major release, we give a few of our customers the chance to preview our upcoming feature with a test drive. This not only allows us to find bugs and dive deeper into our customers’ use cases, but (and arguably more importantly) allows us to learn to monitor our new feature effectively.

Refining Monitoring for Actionable Alerts

We monitor our Single Sign-On feature primarily using Sumo Logic. Setting up Sumo Logic’s monitoring for our SAML and OAuth endpoints during our customer preview set the stage for us to learn about what we should monitor.

In our initial set up, we learned that our alerts were not detailed enough (or actionable). We found that we often would receive an alert reading “Invalid SAML response,” which caused our on-calls to spend more time investigating what went wrong before being able to fix the issue. As a result, we started looking for events and set specific thresholds to create alerts with meaningful information (e.g. SAML response invalid, XML could not validate, assertion invalid due to date/time restrictions) without revealing any sensitive data about the authentication attempt.

For our team to work more effectively, we created dashboards and search queries for both the number of successful authentications and failures. Depending on the number of errors we can start to determine what’s going on and if action is required.

Our customer preview allowed us to learn a lot about authentication behavior to ensure we knew what each event meant to fine tune our alerting. If something unexpected occurs we can make sure we can act quickly for our customers and deliver the reliability our customers depend on.

Accounting for Clock Drift

Specifically during our preview we were confronted by Clock Drift. We learned that authentication between PagerDuty and an identity provider can fail due to clock skew between servers. An identity providers server may be ahead of our server’s time causing the authentication to fail because the server times don’t match.

Luckily, OneLogin’s Ruby SAML Toolkit has a built in feature that helps authenticate identity, accounting for clock drift. After this discovery we were able to set a certain value after some passes in our testing environment to account for the time differences between servers.

Setting Up SSO for your PagerDuty Account

We’ve partnered with Okta, OneLogin and Ping Identity to make it really easy to turn on SSO for your PagerDuty account. But you can also implement SSO for any SAML 2.0 capable identity provider, including Active Directory Domain Services (Via ADFS, which integrates with Active Directory Domain Services, using it as an identity provider). We’ve also put a guide together to help you set up SSO with Google Apps.

Has this feature benefited your team? Let us know in the comments.