PagerDuty Runbook Automation Self-Hosted Delivers Results for ResultsCX

ResultsCX has a simple, yet critical mission: helping its customers take care of their customers.

As a global contact center, ResultsCX is responsible for trillions of interactions each year—whether over email, chat, SMS or telephone. If you’ve ever called a bank or a hotel, you’ve probably called a company like ResultsCX.

Call volume is directly related to the company’s revenue. Every missed interaction is not only damaging to the bottom line, but to the reputations of both ResultsCX and its customers. Jamie Vernon, Senior Vice President, IT Infrastructure and Operations explained, “We can do the full suite of whatever any company might need to help take care of their customers. It’s my team’s job to deliver those interactions to some 20,000 people every day, ensuring we keep our applications, services and networks running around the clock.”

Interruptions, Escalations, and Repetitive Tasks

Unplanned work like service impacts were a challenge for the infrastructure engineers at ResultsCX. Most instances required specialists, and it often took too much time to respond. For example, during a network failover it took up to 30 minutes to identify the right engineering specialist and get them on a conference bridge, plus another 10 minutes for them to plan next steps and ensure no further impact.

These types of disruptions caused other valuable work to be pushed aside or after hours, resulting in undue stress and strain on personal lives. “We’d have an outage we weren’t expecting, and whatever we were working on got dropped,” said Vernon.

Daily maintenance tasks were also time consuming. For example, one engineer would spend two hours a day on telecom maintenance. This job was repetitive and boring, and was flagged as something that could be handed off to a junior resource. However, hiring more staff was a challenge given the tight labor market. This drove ResultsCX to evaluate alternative solutions. “Our leadership at the time was more likely to pay for software than it was another person,” Vernon said.

ResultsCX Looks to Automation for Greater Scale, Speed (and Sanity)

Looking to improve overall IT uptime and create a better experience for engineers, ResultsCX turned to PagerDuty Runbook Automation Self-Hosted.

ResultsCX already used PagerDuty Incident Response, and the team selected Runbook Automation Self-Hosted for its tight integration. “PagerDuty understands how to create dependencies, define them, take in events and turn them into actual information. So, the logical next step was using Runbook Automation Self-Hosted to turn that information into action,” said Vernon.

Now, the team can attach a button in the mobile app to “run a diagnostic” for any technical service—for example, a network switch, firewall, or server. And, with the click of a button, the right people get detailed information about what went wrong and how to drive restoration. “PagerDuty Event Intelligence and Runbook Automation Self-Hosted allow us to more quickly identify what has failed, why it failed, and which team needs to drive repair.”

R.I.T.A. Joins the Team

IT leadership made an unconventional move to drive adoption of this new technology—when introducing Runbook Automation Self-Hosted, they gave their new “employee” a name: R.I.T.A., or “ResultsCX IT Automation.”

“Instead of talking about scripts, we decided to look at automation as a junior employee,” Vernon said. “I couldn’t hire a $100,000/year engineer, but I could ‘hire’ a junior and train up.”

R.I.T.A. got right to work with its automation directive, proving its value from basic diagnostics, informing teams about issues and driving service restoration, to automating maintenance and service tasks. Now, if there’s a network failover, anyone in a contact center can respond with the push of a button and resolve the issue in two minutes. Vernon explained, “Not only is it the restoration of a revenue stream, but it can be done by any resource on my team.”

Daily maintenance tasks, including telecom maintenance is another part of R.I.T.A.’s role. Now, all it takes is one click, and a report is ready for review 20 minutes later. This has freed up time for engineers to focus on more valuable and creative work.

“We’ve not only lowered execution times, but we’ve democratized that ability to anyone, not just those from the specialized skill set,” Vernon said. “Having these predefined automations—200 already—built and blessed by senior engineers means tasks not only take far less time and effort, but now anyone can do it.”

Exceeding Customer Expectations

With PagerDuty, ResultsCX has seen better overall IT uptime and exceeded SLAs—enabling agents to be productive and ensuring customer satisfaction. This is a result of:

• Faster diagnosis and resolution. Resolution time for service impacts like network failovers has been reduced from 40 minutes to 2 minutes.
• Increased capacity for high value work. Automating telecom maintenance now requires one click, freeing up two hours per day. With over 200 automations, teams have significantly increased their innovation capacity.
• Improved business agility. Anybody on the team can run automation jobs instead of escalating to specialists.
• Better engineering experience. Employees spend less time dealing with unplanned work and maintenance tasks, and can focus on more meaningful projects.

“Our overall IT uptime calculation is really simple: how many minutes of agent productivity did we deliver today?” Vernon said. “The industry standard is 99.95%, and R.I.T.A. has helped us exceed that. And if I can point to our uptime as a reason why customers should trust us, well now we are talking about revenue growth and expansion.”

Automating the Future

Things have gone so well that R.I.T.A. got a “promotion” to handle more tasks, earning the trust of its colleagues and instilling more confidence in execution. “We’re more trusting in our ability to articulate a process and document it and make it executable. We’re more confident in R.I.T.A.,” said Vernon.

The team plans to automate more over time, and is exploring how to further enrich diagnostic reports, drive self-service and self-repair to support the help desk team, and even automate client onboarding.

Learn more from Jamie in his Summit ‘22 presentation Cultural Adoption of IT Automation and his thought leadership blog.

To find out how PagerDuty Runbook Automation Self-Hosted can help you automate and delegate business and IT processes, contact your account manager or request a demo.