Infrastructure as Code

As infrastructure stacks grow increasingly more complex and involve an ever-growing number of services and systems, teams have looked to abstract configuration to its own layer of code. This concept of configuring infrastructure as code is gaining traction throughout the industry for a variety of reasons.

For starters, it’s fast. When your infrastructure is all defined as code you can run a script to deploy a series of virtual servers; launch things like containers, databases, and load balancers; and configure any cloud service you might need—such as PagerDuty. Writing the configuration in code also helps keep the settings consistent, reduce the chance of introducing errors, and mitigate deviations between deployments.

Think of the last time a single engineer in your organization was the largest source of knowledge about a certain part of your deployment process. And now, think of how frustrating it was when that engineer left with that knowledge and the rest of the team had to scramble to figure out the missing pieces. When your infrastructure is defined as code, it is already documented for the whole team to see. Engineers can look at the code in a single place and read how the services and processes are configured. This minimizes the risk of losing valuable system knowledge. Of course, the configurations could be documented in a wiki. But we all know that trying to find the right information in a wiki can be a lot of extra work.

All of these benefits of configuring infrastructure as code point to the main reason for this strategy and that is increased efficiency. Knowing that the infrastructure is configured as expected gives engineers the confidence that it can be automatically deployed without any trouble. Then, the engineers can focus on building rather than configuring.

Tools such as Terraform from HashiCorp have emerged as one of the leading ways to declaratively configure technology stacks. And, PagerDuty is among the most popular services that is being being configured by Terraform users. PagerDuty is one of the top 20 Terraform providers, and in this post, I’ll describe how you can use it to configure your PagerDuty account.

Terraform

Terraform is an open-source software tool from HashiCorp that allows you to build, change, and version your infrastructure through code. It allows you to configure your infrastructure and services using a declarative programming language called HashiCorp Configuration Language (HCL).

As organizations adopted more external services, Terraform has grown with the trends to also support configuring a variety of applications. Today, Terraform is a configuration engine that interacts with over 200 providers to manage most of the pieces of your infrastructure. There are providers for everything from AWS to MySQL to Datadog.

The general idea of using Terraform with PagerDuty is to use it as the one source for creating, editing, and deleting resources in your PagerDuty account. With teams using Terraform as their tool of choice for defining other pieces of their infrastructure, it’s only natural that they would also want to configure their PagerDuty accounts in order to keep all of their infrastructure defined in a single location.

Terraform allows teams to manage their infrastructure in a safe, readable, and easily repeatable way. While the HCL code used in Terraform was developed by HashiCorp, engineers will recognize that it looks a lot like JSON. For example, to create a team in PagerDuty, the Terraform configuration would look something like this:

resource "pagerduty_team" "eng_seattle" {
    name = “Engineering (Seattle)"
    description = "All engineers in Seattle office"
}

Then, to add team members to that team, a user and team membership resources would be created:

resource "pagerduty_user" "susan_developer" {
    name = "Susan Developer"
    email = "susan@email.com"
}

resource "pagerduty_team_membership" "susan_team_membership" {
    user_id = pagerduty_user.susan_developer.id
    team_id = pagerduty_team.eng_seattle.id
}

Once all the configuration is set, the Terraform code can then be checked into a version control system so that a history of changes can be recorded in case there needs to be a roll-back in the deployed definitions.

Each time that Terraform runs it defines the state of your service settings. In the case of PagerDuty, if resource definitions for things such as users or teams are removed from the files, Terraform will think that you want to delete them in PagerDuty. So, be sure to have a clear understanding across the organization that Terraform is the source of truth for PagerDuty settings, and not the PagerDuty web interface.

Before we get too much further into showing code, it’s important to point out that there are three Terraform products from HashiCorp–Open Source, Cloud, and Enterprise. To get started with and experiment with Terraform this article will use the open source product as the example. Check out the Installing Terraform article from HashiCorp if you’re brand new to the product.

PagerDuty Terraform Provider

PagerDuty’s Terraform provider began as a community success story. The project was started by Alexander Hellbom, a DevOps Engineer in Sweden. Alexander’s company defined nearly all of its infrastructure configurations in Terraform. When they adopted PagerDuty, Alexander discovered that there wasn’t a PagerDuty provider and set out building one on his own. The reception he received from the Terraform community was so positive and supportive that he has continued to maintain the project to this day. Alexander continues to be involved as a maintainer, but PagerDuty has begun to take a more active role.

The PagerDuty provider supports a wide array of PagerDuty resources, from Add ons and Escalation Policies to Maintenance Windows and Schedules. Head over the the PagerDuty Provider documentation to see a full list of resources that are supported.

Example

Before creating any Terraform files you’ll first need to generate a REST API Key in PagerDuty. The key that you generate will be used as the value for token below.

To get started defining PagerDuty settings with Terraform, create a directory where you want to store your Terraform (.tf) files. In that directory create a new file with the .tf extension. For learning purposes, you can create a provider block for ‘pagerduty’. That block initializes the PagerDuty provider, setting the value of token using the API key generated above, and will look something like this:

provider "pagerduty" {
    token = "your_pagerduty_api_key"
}

Now, in production, you may be a little squeamish about checking API keys into code repositories, and rightfully so. The provider also supports reading the access token from your environment variables. To do this, set an environment variable named PAGERDUTY_TOKEN with the value of your API key. When the PagerDuty Terraform Provider finds this environment variable it will initialize the PagerDuty provider using that variable, and you no longer need to use a provider block in your code.

With the provider set, you’ll now use resource blocks to define the specific PagerDuty objects you want to create and manage. Building off of the example above, where we defined Susan Developer as a user and added her to the “Engineering (Seattle)” team, we could now create an Escalation Policy that included her team with the code below.

To reference the values defined for Susan’s user and team you provide an expression that takes the type of the resource, the name you gave it, and then the field you want to reference. For example, to get the team ID for the eng_seattle team you would write pagerduty_team.eng_seattle.id. Using that syntax, we can now define the escalate_seattle_eng Escalation Policy for Susan’s team (referenced as pagerduty_team.eng_seattle) and using Susan as a target (referenced as pagerduty_user.susan_developer).

resource "pagerduty_escalation_policy" "escalate_seattle_eng" {
    name = "Seattle Engineering Escalation Policy"
    num_loops = 2

    teams = [pagerduty_team.eng_seattle.id]

    rule {
        escalation_delay_in_minutes = 10
        target {
            type = "user"
            id = pagerduty_user.susan_developer.id
        }
    }
}

Terraform Plan and Apply

Before running any other Terraform commands, you’ll need to initialize Terraform by running terraform init. This will check the Terraform files in your current directory and install any providers that are referenced. In this case it will install the latest version of provider.pagerduty. With Terraform initialized, it’s time to test out our definitions.

With Terraform initialized, it’s time to verify the definitions that you made. To do this, run terraform plan. This creates an execution plan that determines which actions need to be taken to get to the state defined in your configuration files. This is a really nice way to make sure that the changes you defined actually create the desired outcome, and prevents you from making any unwanted changes.

When you are satisfied with the state described in the execution plan, it’s time to run terraform apply to actually execute those changes. Terraform will show you the output of the plan again, and then ask for you to type the word ‘yes’ to confirm that you want the planned changes to occur. After you confirm, and the plan is executed, Terraform will provide a message communicating the result of each change that was carried out.

In addition to tackling the technical complexity of provisioning systems, HashiCorp also recommends some practices for addressing the organizational complexity of maintaining infrastructure as your operation scales. Similar to how when an application grows it is broken up into microservices that are each focused on a specific thing, they’re built and managed by a single team, and can usually be developed in parallel with other services—as long as their API doesn’t change. In a similar way, Terraform configurations can be broken up into smaller configurations and delegated out to different teams. Each configuration uses something called output variables to publish information and remote state resources to access output from other configurations.

Closing

For more information on what the PagerDuty Terraform Provider has to offer, checkout the PagerDuty Provider documentation on HashiCorp’s Terraform site. To ask questions, post issues, or submit contributions to the PagerDuty Provider project, head over to the Terraform Provider PagerDuty repository on GitHub. Also, to learn more about Terraform, have a listen to the “Talking Terraform with HashiCorp” episode of the Page It to the Limit podcast where we sat down with Paul Hinze, Senior Director of Engineering for Terraform, and Robbie Th’ng, Director of Product for Terraform.