Datadog Puts DevSecOps in Action With PagerDuty

Datadog provides a monitoring platform that enables teams to ensure that their cloud applications provide the best possible user experience. To help achieve this, Datadog also embraces DevOps, working in an agile manner to constantly innovate and rapidly deliver new features and enhancements to their customers.

This DevOps approach means that Datadog engineers make frequent updates to the infrastructure, which can cause alarm for its security team if the engineers fail to work closely with them. To avoid miscommunication and ensure that releases have security built in, Datadog involves its security team during development. In fact, Datadog is at the forefront of a growing trend called DevSecOps.

In August 2016, the company adopted PagerDuty as an integral component of its digital operations management. Today, in addition to using PagerDuty to support engineering teams with business continuity and disaster recovery (BC/DR), Datadog uses PagerDuty to notify its information security team of events that require an immediate response.

Embracing a New Approach to Scale Security

Datadog is an agile, operations-focused organization with hundreds of engineers distributed around the globe. In many organizations, information security teams are typically siloed from the rest of the development teams, which can often delay production releases due to validation processes during security code reviews. But Datadog knew this approach had to go. “Security wasn’t going to work if it was outside of the development organization, just trying to swoop in when things go sideways,” explained Andrew Becherer, Datadog’s Chief Security Officer.

Because it views security as another aspect of quality, Datadog embeds its security operations and development functions into the organization as a whole. “It behooves security to use the same tools, use the same methods, and bring the same types of technologies to bear in solving the problems faced by the rest of the development organization,” Becherer shared.

By extension, when it comes to vulnerability management, Datadog’s security team tracks issues in much the same way that its developer teams track issues, and security alerting and response follow a similar workflow as other teams within Datadog. “It’s uncommon that security teams at other companies are using PagerDuty in the capacity in which we’re using it,” said Becherer.

Validating Code Changes on Amazon Web Services (AWS)

Datadog is completely cloud-based, leveraging a range of AWS services to run code. With over 15 AWS accounts to manage everything from staging to production, keeping track of authorized changes to code can become quite complex. To validate changes, Datadog leverages ChatOps, integrating Slack, Duo Security, and PagerDuty. When a developer makes a potentially dangerous change to AWS, the security team sends a Slack message to the developer to validate the action. The developer confirms the code push via Slack and through two-factor authentication from Duo. If the developer does not reply in a timely manner or does not confirm the code change, PagerDuty sends an alert to the security team to escalate the response. If a change exceeds a certain threshold of risk, then either the security team is notified immediately via PagerDuty or automated AWS configuration management logic reverts the change to a trusted state.

In short, developers are constantly making changes to each AWS instance, but it’s up to the security team to determine whether or not changes are authorized across tens of billions of API calls per year on AWS.

“It behooves security to use the same tools, use the same methods, and bring the same types of technologies to bear in solving the problems faced by the rest of the development organization.”

– Andrew Becherer, Chief Security Officer, Datadog

Enabling Agile Security and Development With PagerDuty

Like other companies, Datadog’s security organization is deeply concerned about the time it takes to remediate security vulnerabilities. Rather than having to parse through audit logs to understand what happened, Datadog connects its developer teams to security as quickly as possible. By using PagerDuty, Datadog has reduced the overall time required to resolve such issues.

PagerDuty also provides visibility into security incidents to developers and engineers so that they can get immediate feedback on their actions if the security team deems them risky. “Developers are trying to solve a problem and they make a change [to address that problem],” explained Becherer. “You want to provide feedback as quickly as possible in that moment because they’re going to move on to something completely different [right after making that change].”

For example, in one recent security event, PagerDuty quickly escalated a security issue that occurred when a Datadog sales rep was preparing to give a demo. “Because of PagerDuty, we were able to connect a security engineer with our engineers within minutes,” recounted Becherer. “That’s solid gold. That’s where we have to be.”