Nagios Integration Guide – Agentless

Nagios is one of the leading providers of open source and enterprise-grade IT infrastructure monitoring tools. Used by hundreds of thousands of users worldwide, Nagios allows it's users to monitor their entire IT infrastructure, spot problems before they occur, detect security breaches and plan/budget for IT upgrades. By integrating PagerDuty into your existing Nagios monitoring solution, you can have Nagios alerts go directly to the the person on-call in your PagerDuty schedule. The guide below describes how to integrate your Nagios 2, 3 or 4 installation with PagerDuty using a simple Perl-based plugin. Note that you must be logged in as root to complete the installation. You might need to slightly alter these instructions depending on your exact Linux distribution, Nagios configuration and Nagios version. Please contact our support team if you have any trouble completing the integration. Note: This guide has been replaced by the Nagios Two-Way Integration Guide that uses the PagerDuty Agent. Please try following the steps in the new integration guide first, and use this one only if you have a system which the PagerDuty Agent can not be installed on. Also be aware that the two-way integration script from the new integration guide can not be used with the Perl-based integration as-is; the two-way integration is only supported when using the new integration.

In PagerDuty

1. 1. 1. Navigate to Services → Service Directory.
      2. On your Service Directory page:
      3. If you are creating a new service for your integration, click +New Service.
      4. If you are adding your integration to an existing service, click the name of the service you want to add the integration to. Then click the Integrations tab and click the Add a new integration link.
      5. Select your app from the Integration Type menu and enter an Integration Name.
      6. If you are creating a new service for your integration, in General Settings, enter a Name for your new service. Then, configure the Incidents Settings and Incident Behavior for your new service.
      7. Click the Add Service or Add Integration button to save your new integration. You will be redirected to the page of your service.
      8. Copy the Integration Key for your new integration:

On Your Nagios Server

This guide includes steps for Debian-based (i.e. Ubuntu) and RHEL-based (i.e. CentOS, Fedora) Linux distributions. You do not need to execute all commands in this guide, only the ones for your type of system. Note that all commands provided are intended to be run as the root user.

1. 1. 1. Install the necessary Perl dependencies.For Debian-based systems:

         apt-get install libwww-perl libcrypt-ssleay-perl

         For RHEL-based systems:

         yum install perl-libwww-perl perl-Crypt-SSLeay

      2. Download pagerduty_nagios.cfg from GitHub:

         wget https://raw.github.com/PagerDuty/pagerduty-nagios-pl/master/pagerduty_nagios.cfg

      3. Open pagerduty_nagios.cfg in a text editor.
      4. Enter the integration key corresponding to your Nagios service into the pager field. The key is a 32-character string that can be found on the service's detail page (step 3 in the PagerDuty section above).If you're using Nagios 4: Slight modifications are needed for the commands specified in pagerduty_nagios.cfg as shown below. Do not make these changes if you are using Nagios 2 or 3.

         define command {
               command_name    notify-service-by-pagerduty
               command_line   /usr/local/bin/pagerduty_nagios.pl enqueue -f pd_nagios_object=service -f CONTACTPAGER="$CONTACTPAGER$" -f NOTIFICATIONTYPE="$NOTIFICATIONTYPE$" -f HOSTNAME="$HOSTNAME$" -f SERVICEDESC="$SERVICEDESC$" -f SERVICESTATE="$SERVICESTATE$" -f HOSTDISPLAYNAME="$HOSTDISPLAYNAME$" -f SERVICEDISPLAYNAME="$SERVICEDISPLAYNAME$" -f SERVICEPROBLEMID="$SERVICEPROBLEMID$"
         }
         define command {
               command_name    notify-host-by-pagerduty
               command_line    /usr/local/bin/pagerduty_nagios.pl enqueue -f pd_nagios_object=host -f CONTACTPAGER="$CONTACTPAGER$" -f NOTIFICATIONTYPE="$NOTIFICATIONTYPE$" -f HOSTNAME="$HOSTNAME$" -f HOSTSTATE="$HOSTSTATE$" -f HOSTDISPLAYNAME="$HOSTDISPLAYNAME$" -f HOSTPROBLEMID="$HOSTPROBLEMID$"
         }

      5. Copy the Nagios configuration file into place:For Debian-based systems this is usually /etc/nagios3/conf.d:

         cp pagerduty_nagios.cfg /etc/nagios3/conf.d

         For RHEL-based systems this is usually /etc/nagios:

         cp pagerduty_nagios.cfg /etc/nagios

      6. Skip this step if you are using a Debian-based distribution. If you are using a RHEL-based distribution, you will need to edit the Nagios config to load the PagerDuty config. To do this, open /etc/nagios/nagios.cfg and add this line to the file:

         cfg_file=/etc/nagios/pagerduty_nagios.cfg

      7. Add the contact "pagerduty" to your Nagios configuration's main contact group. If you're using the default configuration, open /etc/nagios3/conf.d/contacts_nagios2.cfg (for Debian-based systems) or /etc/nagios/localhost.cfg (for RHEL-based systems) and look for the "admins" contact group. Then, simply add the "pagerduty" contact.

         define contactgroup{
              contactgroup_name admins 
              alias Nagios Administrators 
              members root,pagerduty ; Add pagerduty here
         }

      8. Download pagerduty_nagios.pl from GitHub and copy it to /usr/local/bin.

         wget https://raw.github.com/PagerDuty/pagerduty-nagios-pl/master/pagerduty_nagios.pl 
         cp pagerduty_nagios.pl /usr/local/bin

      9. Make sure the file is executable by Nagios:

         chmod 755 /usr/local/bin/pagerduty_nagios.pl

      10. If not enabled already, enable environment variable macros in /etc/nagios3/nagios.cfg (for Debian-based systems) or /etc/nagios/nagios.cfg (for RHEL-based systems):

          enable_environment_macros=1

          This step is required. The integration will not work if you do not have environment macros enabled.
      11. Edit the nagios user's cron table:

          crontab -u nagios -e

      12. Add the following line to the file:

          * * * * * /usr/local/bin/pagerduty_nagios.pl flush

      13. Restart Nagios.For Debian-based systems:

          /etc/init.d/nagios3 restart

          For RHEL-based systems:

          service nagios restart

FAQ

How does this integration work, and how is it different from the newer integration?

This integration uses a Perl script which is compatible with older Linux distributions than the newer agent-based integration, however it lacks the two-way functionality of the new integration. You can not use the two-way script from the agent-based integration with the Perl script as-is; the two-way integration is only supported when using the new integration. For this reason, we recommend using the agent-based integration unless you are using a system which the agent is not compatible with (i.e. CentOS 5). The Perl script works by adding events received from Nagios to a queue at /tmp/pagerduty_nagios. The script tries to flush the queue (send events to PagerDuty) immediately after receiving them from Nagios. If the script is unable to reach PagerDuty for any reason (i.e. due to connectivity issues), events are kept in the queue. The cron job exists to retry flushing the queue if there's anything that hasn't been sent. The script has a few options you can set when executing it, including the ability to change the directory used for the queue or to set a proxy. The options are documented in the script. Note that you must set any options in both of the commands defined in your Nagios configuration as well as the cron job. Tip: If you run in to issues with the integration and accumulate several events in the queue that you do not want to be sent to PagerDuty, simply remove the event files in /tmp/pagerduty_nagios to prevent them from being sent when the script is able to send alerts to PagerDuty successfully.

How do I configure Nagios to work with multiple PagerDuty services?

This is easy to do with the current integration, as a Nagios service in PagerDuty is directly mapped to a "contact" in Nagios. By default, this contact is named pagerduty and defined in the pagerduty_nagios.cfg file. In order to configure multiple services, just duplicate the existing contact definition and rename it (i.e. pagerduty_database, pagerduty_network, etc.). Then copy and paste the corresponding Integration Key from PagerDuty into the pager field. Don't forget to restart your Nagios for the changes to take effect.

What if a Nagios event happens while my network is down?

If a PagerDuty server can't be reached for any reason, events will be stored to an on-disk queue. The cron job will attempt to re-send the events once every minute.

Since Nagios needs my external Internet connection to send failure reports to PagerDuty, how will I receive notification if our site loses external connectivity?

You should configure an external ping check service such as StatusCake or NodePing to monitor your site's external connectivity. Of course, you can use PagerDuty to forward alerts from these services as well.

The integration doesn't seem to be working. What's going on?

First, check syslog for messages containing pagerduty_nagios. For Debian-based systems:

grep pagerduty_nagios /var/log/syslog

For RHEL-based systems:

grep pagerduty_nagios /var/log/messages

More troubleshooting tips can be found in our Nagios Troubleshooting Guide.

What sort of Nagios messages does PagerDuty understand?

PagerDuty can process PROBLEM, ACKNOWLEDGEMENT, and RECOVERY messages. All other messages, including FLAPPINGSTART and FLAPPINGSTOP, are ignored.

How can I customize my Nagios alerts?

We have a guide for Customizing Notifications Sent to PagerDuty from Nagios to help you get started.