Customer impacting incidents increased by 43% during the past year- each incident costs nearly $800,000

90% of IT leaders reported that outages or disruption have reduced customer trust in their organization

Digital incidents are on the rise, lasting longer and costing more. However, many organizations have yet to fully automate incident response to deal with the deluge. A PagerDuty study of 500 IT leaders and decision-makers responsible for IT operations, detailed the gap between those that have fully automated incident response and those still relying on manual processes. Organizations, now more than ever, must enter their automation era.Incidents are on the rise: Digital incidents are fast becoming a fact of life. Over half (59%) of IT leaders say that customer-impacting incidents have increased, growing by an average of 43% in the last 12 months. These incidents were driven by increased complexity, the rapid expansion of digital services, and insufficient investment in IT infrastructure maintenance.The costs of incidents rack up: Thinking about the last major customer-facing outage, respondents said it cost their organization $1.5m in lost revenue/sales. IT leaders estimated the true cost of downtime to be $4,537 per minute. When you consider that the average resolution time takes 175 minutes, then each customer-impacting digital incident can cost $793,957. Organizations saw an average of 25 high-priority/priority incidents in the last 12 months, meaning customer-facing outages can cost $19.8m per year.The toll on trust: Nine-in-ten IT leaders reported that outages or disruption have reduced customer trust in their organization. The most important strategies for protecting trust were the protection of sensitive data and personally identifiable information (PII), quickly restoring services, and real-time updates to customers. This will require investment, but 69% of IT leaders say the board and management are failing to invest in protecting customer trust when outages occur.Toilsome tasks hinder incident response: Those on the incident response front lines are being bogged down by toil. More than 70% of IT leaders report that remediation, mobilizing responders, collaboration between teams, and internal communications with stakeholders are yet to be fully automated. Fixing common or recurring problems and external communications – such as updating status pages and sending customer emails – wastes the most time in the incident response process.Wasteful processes take their toll: The lack of fully automated processes means teams deal with a lot of waste – actions or steps that add little to no value – that can carry a significant cost. Organizations on average spent $1.9 million on incident responders in the last 12 months. But with these responders spending 38% of their time dealing with manual incident response processes, then the cost of toil can reach $700,000 per year. 

Barriers to automation: The primary barriers to automating end-to-end incident response are a lack of alignment across IT (24%), budget constraints (22%), insufficient talent/expertise (16%), inadequate data management practices (16%) and a lack of alignment across executive leadership (12%). But automation must become a priority, especially with IT leaders saying they’re under pressure to reduce the cost of IT operations (56%), improve efficiency and productivity (55%), improve the user experience (47%), leverage AI quickly (47%), and achieve cost savings (43%). 

Transform your operations

With the world relying on digital services and systems, uptime has never been more crucial. With incidents happening more often, and becoming more costly, it’s clear organizations can no longer rely on manual processes in incident response. Doing so will negatively impact revenue, reputation and trust, and make it harder to attract and retain talent.

Despite these hardships, 91% of respondents believe their IT team is effective at managing day-to-day operations and resolving technical issues promptly. Clearly there is room for improvement, but to do more with less, we can’t expect teams to work faster and harder. Encouragingly, 86% of IT leaders say their organization is making strides towards fully automating the end-to-end incident response process.

This is where the PagerDuty Operations Cloud can help accelerate critical work across the enterprise. Powered by AI and automation, the PagerDuty Operations Cloud helps incident responders by automating everything from mobilizing teams and incident diagnostics, to updating status pages and stakeholder communications. By scaling teams with automation, they are empowered to increase innovation velocity, grow revenue, cut costs, and reduce the risk of operational failure.

Read the full report here.