The big advantage of configuration management tools like Chef, Puppet, and Ansible is that they turn your data center into “scripted” infrastructure. Instead of wasting hours of your life provisioning and configuring each server manually, you can use a configuration tool that does the dirty work for you.

However, those tools aren’t designed to automate incident management. And that begs the following questions: Why are you handling incident management manually when the rest of your IT operations are scripted? Isn’t it time to integrate incident management into your scripted infrastructure routine? Yes! By taking a scripted infrastructure approach to incident management, you can scale monitoring and alert management just as well as the rest of your operations.

The Problem

First, let’s discuss why a scripted infrastructure approach to incident management is so crucial.

I’ll begin by saying that you shouldn’t be too hard on yourself if you’re still doing incident management manually. You’re not a bad admin, just a victim of circumstance. Until recently, automated incident management solutions haven’t been as readily available as with other spaces, with infrastructure management tools like Chef.

Incident management requirements also haven’t always been as complex as they are today. Your data center ten years ago probably comprised, at most, a few dozen on-premise servers. You could handle incident management there manually.

But today, infrastructure is larger and more complex than ever because of demands around scalability and faster product innovation. You have on-premise bare-metal servers. You have local virtual servers. You have cloud servers, containers, and mobile devices. And with the IoT revolution now in full swing, it’s a safe bet that you soon may have to add refrigerators and microwaves and parking meters to the mix, too.

If you want to do incident management effectively on all of these devices, your strategy needs to scale and eliminate repetitive, manual tasks wherever possible. To accomplish that, you need next-generation incident management solutions that can be automated and scripted in the same way that you automate the configuration of your burgeoning data center infrastructure.

The Solution

Now, let’s talk specifics. To handle incident management effectively in the age of scripted infrastructure, your incident management tools should:

• Route alerts to the right people, every time, automatically. If there’s a manual step anywhere in notifying the right people of an issue, your process is broken.
• Escalate incidents automatically. Here again, you can’t wait on a human to manually reassign an issue when people forget to take action, especially if you have a huge infrastructure. Your software needs to be smart enough to do that for you, just as Chef and Puppet are smart enough to configure your servers automatically.
• Manage alert behavior at scale. Part of what makes infrastructure-scripting tools so handy is that they’re good at making the most efficient use of existing resources. They know where in the cloud to place your virtual servers without asking you, for example. In the same way, your incident management tools should be able to group, suppress, and route alerts to the right services and teams automatically, reducing noise as well as response times.
• Integrate with ChatOps so that your team can collaborate on incident response without siloing the communications process off from incident management work. Plus, through chatbots, ChatOps can help automate certain response tasks.
• Support all of your monitoring needs. Chances are that you have multiple monitoring systems in place, like AWS Cloudwatch, Nagios, and Pingdom. To make your incident management truly scalable and automated, these tools need to work together without manual intervention. An incident management strategy that automates alerts from all sources except one is just as problematic as a Puppet infrastructure that configures all of your infrastructure apart from one type of server which you provision by hand. Centralization of all your tools within a solution that enables you to turn events into automated workflows is key.
• Be up 100% of the time. That may seem obvious, but I make this point as a reminder of why it’s a bad idea to rely solely on on-premises notifications. I loved Nagios as much as the next guy when it was 2002 and the cloud was still just a thing in the sky. But today, if you rely solely on an old-guard tool like Nagios running locally to deliver your alerts, you run the risk of having your incident management system itself go down if there’s a problem with your infrastructure. Using Nagios is good and well, but you should feed its alerts, along with the ones coming from the rest of your monitoring systems, to a centralized cloud-based incident management solution, which will not be affected by problems on your infrastructure.

If you’re used to working solely with legacy alerting and monitoring systems, the demands on this list may seem like fantasies, but they’re not. Incident management software that automates rapid response workflows around all your event data, just as effectively as scripted infrastructure can automate your data center, is here now. And to be far more productive and effective at your job, now’s the time to take advantage of it.