Guest blog post by Ron Vidal, Rob Schnepp, and Chris Hawley of Blackrock 3 Partners LLC. Blackrock 3 Partners are experts in Incident Management, combining decades of experience in the fire service, law enforcement and anti-terrorism managing large-scale public safety emergencies with decades of experience in managing web operations, critical infrastructure, capital markets and M&A activities for international broadband network operators and high performance computing companies. 

It’s a quiet, sunny day, like so many other quiet, sunny Peacetime days…

Then, somebody sees flames and smoke pouring out of a building… Wartime… 

Call 911… Help is on the way!

In seconds, the 911 Communications Center dispatches fire engines, ambulances, rescue companies and Incident Commanders to the location of the emergency. In about 4 minutes, the first 25 trained firefighters arrive on scene with their specialized vehicles, tools and skills. The Incident Commander establishes command, sizes-up the situation, sets tactical objectives, begins operations, evaluates resources needed and organizes the effective resolution of the emergency.

Does this sound like how your DevOps team responds to high severity incidents?

If you have Operations, then you will also have Emergency Operations. Like an alarm sounding at a fire station, immediately launching firefighters into action, DevOps teams must respond with the same level of urgency to resolve their emergency. In both cases, the clock is ticking.  The problem is unlikely to get better until the right resources are dispatched and respond at the right time, working under a leader making the right decisions, all within an organizational framework.

The Shift from Peacetime to Wartime

Peacetime is the mode of operation that occurs during the normal day-to-day activities of any IT organization. Developers write code. Operations keep the infrastructure running. Business as usual. In other words, a perfect Peacetime day.

Here’s what a typical Peacetime organization chart looks like:

Wartime is the mode of operation that occurs when systems are NOT normal. Operations has declared a SEV level event and initiated an Incident conference bridge. On-call subject matter expert engineers are querying alerts and looking at performance data. Customers are out of service. Business is NOT ususal. We have a serious problem and  and it needs to be fixed, right now. In other words, a Wartime Incident.

In the Fire Department, the shift from Peacetime to Wartime occurs when an emergency is reported. Wartime is different. People behave differently. Their language and method of communications is different. Conversations are typically much shorter, more direct and aimed at problem solving on a compressed timeframe. To the uninitiated it sounds abrupt. It sounds choppy. It sounds sterile. And it should.

As 60 Minutes reported in its March 17, 2013 interview with Jack Dorsey, “Young Jack was intrigued by the messages he heard coming out of the St. Louis emergency dispatch center. At home he listened to it all on a police scanner. And he was struck by the fact that everyone talked in short bursts of sound – a system of communication that later inspired him to invent Twitter.”

In Wartime, the Incident Commander is thinking faster than the emergency is unfolding. To do that, communications must be direct, crisp and clean, like “Voice Twitter”.

Understanding Wartime Communication

The Wartime organizational chart will look different than the Peacetime organizational chart. In fact, the CEO (Peacetime leader) is exactly the wrong person to lead the emergency (Wartime) response, because someone still has the run the business and the unaffected parts of the organization. Roles and responsibilities, chain of command, and the assignment of tasks in Wartime will be very different than they are in Peacetime.

Here’s what a Wartime organization looks like:

Here’s a comparison of how a Fire Department and DevOps respond to an emergency:

In short, solving Wartime problems requires a Wartime mentality, and a defined process for incident management. Fire Departments have developed a system that has been in use for over 40 years and has managed tens of millions of Wartime incidents. Without a doubt, DevOps teams are emergency response organizations much like Fire Departments.

When an emergency is reported, it is a clear signal to all that the organization has shifted from Peacetime to Wartime. All responders must understand and accept the sense of urgency and accountability that comes with the shift to Wartime and perform their assigned tasks accordingly.

Every emergency (Wartime) response absolutely depends on robust communications for the entire incident, from alert notifications to dispatch to dedicated tactical channels to escalation to resource accountability. Peacetime has a clutter of multiple communications systems that distract Wartime responders. Wartime demands clear communications for every incident and every emergency responder, and centralized incident management system will help cut through the noise. Each element of the emergency (Wartime) response depends on effective, reliable, communications. Without rock solid communications, Incident Management fails.

So, next time you see a fire engine speeding to an emergency Code 3 (lights and siren), just remember that they are in Wartime and using the same tools and systems that you can use to manage your DevOps fires.

For more information on Blackrock 3 Partners LLC, please visit www.blackrock3.com.