Australian Bank Supercharges Deployments and Automates Compliance Measures with PagerDuty

As this large Australian financial institution expanded offerings and added new customers, the DevOps team felt pressure to keep up. The team is responsible for supporting 7 non-production environments and 65 production applications, and receives daily changes from vendors and developers equating to over 500 deployments per month. They realized that in order to quickly push more products and maintain the excellent customer experience they are known for, they needed to expand their use of automation.

The team was already using Rundeck Community in a limited capacity to execute single commands. The Platform Engineering team recognized the value they experienced from runbook automation and decided to expand their use. They upgraded to PagerDuty Runbook Automation Self-Hosted to take advantage of the Enterprise Support and smart workflows.

PagerDuty enabled the bank to overhaul its release processes, and strengthen its security and compliance posture. The Platform Applications Manager explained, “As a bank, it’s critical that we have Enterprise support on the applications that we depend on. PagerDuty Runbook Automation Self-Hosted has become more and more the lifeblood of what we do.”

One-Click Deployments Through Runbook Automation Self-Hosted

Before implementing Runbook Automation Self-Hosted, the deployment process was lengthy and largely manual. Now they have turned daily deployments and major releases into one-click operations.

For daily deployments, the agile delivery teams hand application updates to DevOps to package for deployment into production. Under the bank’s change control practices and separation of duties, DevOps does not touch production. They build deployment automation in staging, test it, and pass it to the Operations team to execute— all via PagerDuty. The Platform Applications Manager shared, “The beauty of using Runbook Automation Self-Hosted is that to the Operations team, the deployment is identical no matter which platform it comes from. We do 500-600 deployments per month. Those kinds of numbers are not possible without good quality automation.”

Major releases are scheduled every 3 months, taking the bank offline for maintenance. Before PagerDuty, it took up to 40 minutes to shut down the systems by logging into servers and stopping applications one-by-one. Now, they can do this with the click of a button, the exercise takes 85% less time, and they avoid potential mistakes from manual processes.

The bank has experienced significant time savings by automating deployments, saving them about 30 minutes per deployment and approximately 250 hours of engineering time per month. “PagerDuty is the control plane of the ability to meet our business deliveries,” said the Platform Applications Manager.

Self-Service in a Secure Environment

Runbook Automation Self-Hosted has strengthened the security of the bank’s systems, as secrets such as login credentials do not need to be widely shared. Access to workflows are controlled by user roles, and all actions are logged by PagerDuty in addition to server logs.

With improved access control and authentication management, DevOps implemented self-service operations. The QA team was granted self-service access to common testing tasks. Before, the QA team would require data from DevOps during the testing process. When a request surfaced, someone in DevOps would stop what they were doing, log into service, find the data or run the scripts needed to gather that information, and send it back to the QA team. The Platform Applications Manager explained, “The safe, self-service capability in Runbook Automation Self-Hosted prevented our testing team from waiting 30 minutes to 4 hours for a team member to have capacity to execute a task. I estimate that this saves us around 20-30 hours a month of unnecessary delays to our testing schedules.”

“I estimate that Runbook Automation Self-Hosted saves us around 20-30 hours a month of unnecessary delays to our testing schedules.”

– Platform Applications Manager, Australian Bank

Automating Compliance in a Highly Regulated Industry

The proven repeatability of running workflows in PagerDuty has streamlined the change request process. Depending on the risk, changes to production need to go through the bank’s change request board. However, if there is an existing automation in PagerDuty that has been approved and run previously, this lowers the risk rating because there’s far less uncertainty. A manager is able to make these approvals faster and with more confidence, further streamlining deployments.

Audits are routine in the highly regulated FinServ industry and the bank uses PagerDuty to automate parts of the process. Every runbook automation job captures the necessary information for audits into the system used to track production applications. During the yearly audit, they can pull up every production deployment, show what changes were made and by whom, what time the changes were made, and what changes it was related to. “PagerDuty has helped immensely with compliance reporting because we can demonstrate consistent processes. Audits used to take two to three weeks of manually gathering information, finding change records, and deployment records—by using Runbook Automation Self-Hosted, we can provide this data in under a day,” said the Platform Applications Manager.

What’s Next?

The bank will continue modernizing its technology stack and is planning a major transformation from traditional data centers to the cloud. The Platform Engineering team sees Runbook Automation Self-Hosted as the main mechanism for deploying applications to the cloud, accelerating their speed-to-market.

Learn how PagerDuty Runbook Automation Self-Hosted can help you automate and delegate business and IT processes, contact your account manager or request a demo.