Blog

Top Skills for an Incident Commander

by Rachael Byrne June 8, 2017 | 6 min read

Credit: NASA

Organizations need many incident commanders to provide a high level of service to their customers while avoiding on-call load. Many shy away from becoming an incident commander because they assume only senior technical leads can be one. However, soft skills are actually more important, and with a well-defined process like the one outlined in this post, your team can train multiple people to be successful in leading and driving coordination during customer-impacting outages.

I’m a ScrumMaster at PagerDuty, not an engineer, and I recently became an Incident Commander. I learned first hand that you don’t need to be a senior engineer to serve as Incident Commander.

Anyone Can Be An Incident Commander

Effective incident response requires an Incident Commander to serve as the decision maker and to provide clear coordination. It’s a demanding role that requires a unique set of skills. As incidents can happen at any time, organizations need a sufficient number of Incident Commanders to reduce on-call load and avoid burnout. It’s therefore important to develop an Incident Commander training process that is both welcoming and effective.

While high level knowledge of how your organization’s services interact with each other is important for an Incident Commander to have, you by no means need to be highly technical to lead an incident response. The Incident Commander should remain focused on coordinating the response with key soft skills, not performing any technical remediation or investigation tasks themselves. The Incident Commander needs to actively listen to identified symptoms and proposed actions to decide on the best course of action and should not let any technical knowledge bias your chosen approach.

Anyone can be an Incident Commander, regardless of rank or technical expertise! PagerDuty’s open-sourced incident response documentation is a great starting point to formalize your own processes, but written documentation, as clear as it may be, isn’t enough to fully train a new Incident Commander. Proper training requires hands-on practice. At PagerDuty, we have developed a supportive training program that can get even the most junior team members comfortable with leading a major incident response.

Here are some of its key tenets:

Process: Our incident response process helps us quickly resolve issues that arise and keep our systems working within their SLA. This is a structured process with clear roles and communication rules. A good Incident Commander has a knack for process and can bring structure to a chaotic situation.
Directive Communication: An Incident Commander should be comfortable with directive communication. It is your responsibility to delegate tasks, so you can’t be afraid to tell people what to do, regardless of their day-to-day rank.
Time Management: Timeboxing is critical during an incident response. When you delegate a task, tell them when you will check back in with them for an update. You want to keep information flowing and maintain a cadence of regular updates to everyone on the call.
Listening: While structure and directiveness are important, you also need the ability to listen to expert feedback and flexibility to modify plans on-the-fly. You facilitate the gathering of information and proposals from everyone on the call. Ask for the impact of every recommendation, and then make a decision.

Let’s contextualize these and discuss exactly how we put process, directive communication, time management, and listening into practice.

Our Incident Commanders host regular office hours open to all employees that are interested in incident response. Here, prospective Incident Commanders who have decided to begin their training can ask questions and learn about the process. This is an opportunity to explain the need for more Incident Commanders, and help everyone give it a try and learn.

If you’re interested in seeing how we train our staff to get ready for incident response, be sure to register for our free webinars:

After we kickstart the process with office hours, we then get the trainees on the Incident Commander shadow schedule. Shadowing the Incident Commander helps the trainee get a feel for what it’s like to be on-call. They also get woken up or interrupted, whenever the Incident Commander gets paged. While shadowing, the trainee joins the incident call to listen in. It is important for the trainee to remain a silent observer, holding any questions until the end, to avoid distracting from the response. Incident Commanders then spend time answering trainees’ questions to help them learn, increase their comfort, and make them feel supported by the community.

After the trainee listens to a few calls, encourage them to jump in as scribe, documenting the timeline of an incident as it progresses. We’ve learned that for longer incidents, frequent handoffs make a huge difference for maintaining an effective response and avoiding burnout. Serving as scribe is a great way for someone to start helping before they are fully ready to be an Incident Commander. Starting to get involved with the incident response in this way also helps the trainee get even more familiar with the process, increasing their confidence.

After listening to and scribing for a few incident calls, encourage the trainee to take the plunge and get on the main schedule. They can reverse shadow, leading the call with the support of a backup Incident Commander. Assure them the backup will be there to help throughout the response. It is important to let the new Incident Commander take point on the call, so they can earn credibility and further build their confidence. Privately message them with any tips and reminders, as well as with encouragement.

After a new Incident Commander leads their first incident response, celebrate them! Leading an effective Incident Response directly impacts the success of your business and the happiness of your customer’s, and is also key in maintaining the morale of the team. By creating an Incident Response community that supports each other, you’ll be able to welcome more Incident Commanders and reduce on-call load for all.