Zenoss 3 Integration Guide

Zenoss 3 is a very popular open source network, server and application monitoring system used by thousands of IT organizations. It provides one of the best event management systems available in any open source monitoring system. The plug-in architecture of Zenoss allows it to be extensible by anyone.PagerDuty extends Zenoss’s functionality by providing on-call scheduling, alerts and incidents tracking through the PagerDuty API. You are notified of the most critical Zenoss events by PagerDuty so you are able to take immediate action.This guide describes how to integrate your Zenoss installation with PagerDuty using cURL. You might need to slightly alter these instructions depending on your exact Linux distribution and your Zenoss configuration.  If you are having trouble completing the installation, please contact us.

Looking for Zenoss 5? Go here

Looking for Zenoss 4? Go here

Setting up the PagerDuty / Zenoss 3 integration

In PagerDuty

1. From the Configuration menu, select Services. 

2. On your Services page:

   If you are creating a new service for your integration, click +Add New Service.

   If you are adding your integration to an existing service, click the name of the service you want to add the integration to. Then click the Integrations tab and click the +New Integration button.




3. Select your app from the Integration Type menu and enter an Integration Name.

   If you are creating a new service for your integration, in General Settings, enter a Name for your new service. Then, in Incident Settings, specify the Escalation Policy, Notification Urgency, and Incident Behavior for your new service.

4. Click the Add Service or Add Integration button to save your new integration. You will be redirected to the Integrations page for your service.
   

5. Copy the Integration Key for your new integration:

In Zenoss:

1. Install cURL and its dependencies on the machine running Zenoss (if not installed already):For Debian, Ubuntu, and other Debian-derived systems:

   sudo aptitude install curl

   RHEL, Fedora, CentOS, and other Redhat-derived systems:

   sudo yum install curl

2. Log into your Zenoss Console

3. Click on the Events tab

4. Click on the Event Manager sub-tab

5. Click on the Commands link on the left panel

6. Add a new command by entering the name “PagerDuty” and then click the Add button

7. Click on the “PagerDuty” command to view the command details

8. Enable the command by selecting “True” for the label titled “Enabled”

9. Copy and paste the following into the field labeled “Command”:

   curl -H "Content-Type: application/json" -X POST -d '{"service_key": "YOUR-SERVICE-KEY", "incident_key": "${evt/evid}", "event_type": "trigger", "description": "${evt/device}: ${evt/summary}", "details": { "device": "${evt/device}", "ipAddress": "${evt/ipAddress}", "severity": "${evt/severity}", "summary": "${evt/summary}", "message": "${evt/message}", "evid": "${evt/evid}"} }' https://events.pagerduty.com/generic/2010-04-15/create_event.json

   NOTE: PagerDuty does not accept invalid JSON including unescaped newline characters. Please verify within your Zenoss configuration that no Zenoss variables contain invalid JSON data. 

10. Replace YOUR–SERVICE–KEY with the key from PagerDuty

11. Copy and paste the following into the field labeled “Clear Command”:

    curl -H "Content-Type: application/json" -X POST -d '{"service_key": "YOUR-SERVICE-KEY", "incident_key": "${evt/evid}", "event_type": "resolve"}' https://events.pagerduty.com/generic/2010-04-15/create_event.json

12. Replace YOUR–SERVICE–KEY with the key from PagerDuty

13. Click on the “Save” button at the bottom to save the changes

FAQ

What if a Zenoss event happens while my network is down?

At the moment, if the event command fails to submit notifications to PagerDuty, it will be registered as another event in Zenoss. The repeat time can be set in command settings such that notification would be retried. Note: PagerDuty will deduplicate events based on the event id, thus only one PagerDuty incident would be created even if Zenoss submits events multiple times.You should also configure an external ping check service to monitor your site’s external connectivity. Of course, you can use PagerDuty to forward alerts from these services as well.

How do I setup Zenoss to work with multiple PagerDuty integrations?

This is easy to do with the current integration, as a Zenoss integration in PagerDuty is directly mapped to an “Event Command” in Zenoss.In order to setup multiple integrations, just duplicate the existing event command rename it (i.e. pagerduty_database, pagerduty_network, etc.) and update the command with the appropriate integration key. Don’t forget to set filters on the command such that only the desired events are sent to PagerDuty.