Nagios XI Two-Way Integration Guide

Nagios XI expands upon the capabilities of the Nagios Core software to provide you with detailed host and service monitoring for your critical IT systems. The guide below describes how to integrate your Nagios XI installation with PagerDuty using our easy to install agent. Note that you must be logged in as root to complete the installation. Please contact our support team if you have any trouble completing the integration. Note: If you are running Nagios on CentOS 5, you will need to use the Perl-based integration for Nagios XI instead of following this guide.

In PagerDuty

1. Go to the Services menu and select Service Directory.
2. On the Service Directory page:
   • If you are creating a new service for your integration, click +New Service and follow the steps outlined, selecting this integration in step 4.
   • If you are adding your integration to an existing service, click the name of the service you want to add the integration to. Then click the Integrations tab and click Add a new integration.
3. Under Select the integration(s) you use to send alerts to this service search and select this integration.
4. Click the Add Service or Add Integration button to save your new integration. You will be redirected to the Integrations page for your service.
5. Find the integration in the list, copy the Integration Key and keep it in a safe place for later use.

On Your Nagios XI Server

1. Install the PagerDuty Agent. The agent receives events from Nagios XI and sends them to PagerDuty using a queue, provides logging that helps troubleshoot any problems, and automatically retries sending alerts in the event of any connection failure (i.e. if your Nagios XI server temporarily loses connectivity).Note: The Agent does not run on CentOS 5 or lower, as it requires a newer version of Python than the version included with CentOS 5. Please use the Perl-based integration for Nagios XI on older operating systems.
2. Log in to the Nagios XI web interface as nagiosadmin, then go to the Configure menu and select Core Config Manager. Click Commands on the left menu, then click Add New.
3. Set the Command Name, Line and Type as follows, then click Save.
   • Command Name:

     notify-service-by-pagerduty

   • Command Line:

     /usr/share/pdagent-integrations/bin/pd-nagios -n service -k $CONTACTPAGER$ -t "$NOTIFICATIONTYPE$" -f SERVICEDESC="$SERVICEDESC$" -f SERVICESTATE="$SERVICESTATE$" -f HOSTNAME="$HOSTNAME$" -f HOSTDISPLAYNAME="$HOSTDISPLAYNAME$" -f SERVICEDISPLAYNAME="$SERVICEDISPLAYNAME$" -f SERVICEPROBLEMID="$SERVICEPROBLEMID$" -f SERVICEOUTPUT="$SERVICEOUTPUT$"

   • Command Type:

     misc command

   
4. Click Add New to add another command.
5. Set the Command Name, Line and Type as follows, then click Save.
   • Command Name:

     notify-host-by-pagerduty

   • Command Line:

     /usr/share/pdagent-integrations/bin/pd-nagios -n host -k $CONTACTPAGER$ -t "$NOTIFICATIONTYPE$" -f HOSTNAME="$HOSTNAME$" -f HOSTSTATE="$HOSTSTATE$" -f HOSTDISPLAYNAME="$HOSTDISPLAYNAME$" -f HOSTPROBLEMID="$HOSTPROBLEMID$"

   • misc command

   
6. Click Contacts on the left menu, then click Add New.
7. Go to the Alert Settings tab and set the following host options, then click Manage Host Notification Commands:
   • Host Notifications Enabled: on
   • Host Notifications Timeperiod: 24x7
   • Host Notification options: d, r
   
8. Select the notify-host-by-pagerduty command and click Add Selected. Make sure that notify-host-by-pagerduty moves to the Assigned column, then click Close.
9. Back in the Alert Settings tab, set the following service options, then click Manage Service Notification Commands:
   • Service Notifications Enabled: on
   • Service Notifications Timeperiod: 24x7
   • Service Notification options: w, u, c, r
   
10. Select the notify-service-by-pagerduty command and click Add Selected. Make sure that notify-service-by-pagerduty moves to the Assigned column, then click Close.
11. Go to the Misc Settings tab and click Manage Variable Definitions.
12. Set Variable name to pager and Variable value to the Integration Key you copied from PagerDuty earlier and click Insert. Make sure that the variable is added to the right column, then click Close.
13. Go to the Common Settings tab and set a Contact Name (i.e. pagerduty, or if you wish to integrate with multiple PagerDuty services, more descriptive names such as pagerduty_database, pagerduty_network, etc.) then click Manage Contact Groups.
14. Select the admins group, or another group you want the contact to be part of which your hosts and services will also be associated with, and click Add Selected. Make sure that admins (or your alternate group name) moves to the Assigned column, then click Close.
15. Click Save to save your new contact.
16. Click Host Templates on the left menu, then select a template you want to use for PagerDuty notifications. For this guide, we're using generic-host, as this will usually allow you to receive notifications for all hosts as soon as the integration process is complete.
17. Go to the Alert Settings tab and set the following options, then click Manage Contactgroups:
    • Notification period: 24x7
    • Notification options: d, u, r
    • Notification interval: 1 min
    • Notification enabled: on
    
18. Select the admins group (or any alternative group you may have selected in step 15) and click Add Selected. Make sure that the group name moves to the Assigned column, then click Close.
19. Click Save to save your host template.
20. Click Apply Configuration to activate your new commands, contact(s), and host template(s) modifications.
21. Download pagerduty.cgi for the two-way integration:

    wget https://raw.githubusercontent.com/mdcollins05/pd-nag-connector/master/pagerduty.cgi

22. Edit the pagerduty.cgi file so that the command_file variable points to your Nagios command file. The path can be found by running the command grep "^command_file" nagios.cfg in the directory where nagios.cfg resides (/usr/local/nagios/etc if you're using the virtual machine image provided by Nagios Enterprises).If you don't see any information returned, make sure the command_file variable is uncommented (doesn't start with a #).
23. Move pagerduty.cgi to the Nagios Core cgi-bin (not the Nagios XI PHP user interface directory).For the virtual machine image provided by Nagios Enterprises this is /usr/local/nagios/sbin:

    mv pagerduty.cgi /usr/local/nagios/sbin

24. Make pagerduty.cgi executable.For the virtual machine image provided by Nagios Enterprises:

    chmod +x /usr/local/nagios/sbin/pagerduty.cgi

25. Install the required Perl libraries for the script to work.

    yum install perl-CGI perl-JSON perl-libwww-perl

26. In PagerDuty, go to your Nagios XI service Integrations tab and click Add an extension.
27. For the Extension Type select Nagios, then enter in a Name for your webhook, select a Service, paste in the URL, the Username, the Password, and click Save. The URL will look similar to this: ip-or-domain/nagios3/cgi-bin/pagerduty.cgi (no http:// or http:// is required in the URL). Note: Unless you've disabled it, the Nagios web interface requires a username and password. We highly recommend configuring a user that is able to run Nagios commands that is only used for the webhook.If you go to this URL in your browser you should see 400 Requests must be POSTs. If you do not see this, check your web server logs for details on what happened when you tried to call this URL.
28. At this point, you should be all set. To test it out, you'd need to have an issue within Nagios that generates an incident. From there, acknowledging the incident should add a comment stating the incident has been "Acknowledged by PagerDuty."

FAQ

Can I have the two-way integration without having my Nagios install be accessible over the internet?

The two-way integration requires that your Nagios server be accessible over the internet to receive webhook calls from PagerDuty. If you wish to restrict access to your Nagios web server to specific IPs, you can find PagerDuty's webhook IPs in our knowledge base: What are PagerDuty's IPs for whitelisting and firewall purposes? Alternatively, you can use a service like ngrok to create a secure tunnel to your Nagios server with a public URL that can be used for your webhook or the poller script originally developed by Zoosk and now maintained by PagerDuty to reach out instead of having your Nagios server publicly accessible.

How do I configure Nagios XI to work with multiple PagerDuty services?

This is easy to do with the current integration, as a Nagios XI service in PagerDuty is directly mapped to a "contact" in Nagios. In order to configure multiple services, simply create additional contacts in Nagios XI with different names (i.e. pagerduty_database, pagerduty_network, etc.), then copy and paste the corresponding Integration Key from PagerDuty into the pager variable definition field, and make the contact part of a contact group which will receive notifications for your desired services. Don't forget to apply your changes after saving your contact so they take effect.

What if a Nagios event happens while my network is down?

If a PagerDuty server can't be reached for any reason, events will be stored to an on-disk queue. The PagerDuty agent will attempt to re-send the events when connectivity is restored.

Since Nagios needs my external Internet connection to send failure reports to PagerDuty, how will I receive notification if our site loses external connectivity?

You should configure an external ping check service such as StatusCake or NodePing to monitor your site's external connectivity. Of course, you can use PagerDuty to receive alerts from these services as well.

The integration doesn't seem to be working. What's going on?

First, make sure you've installed the PagerDuty Agent, and that there were no errors from your package manager when attempting to install it. Failed installs (i.e. due to an incompatible distribution, such as CentOS 5) are the most common issue with the integration not working. Check that the pagerduty contact is getting the HOST or SERVICE NOTIFICATIONS in syslog. You can grep your syslog to see if the pagerduty contact is being notified. Here's an example:

grep NOTIFICATION /var/log/messages
May 28 18:20:57 ip-10-11-139-249 nagios3: SERVICE NOTIFICATION: pagerduty;localhost;Current Users;CRITICAL;notify-service-by-pagerduty;USERS CRITICAL - 3 users currently logged in

As you can see, the pagerduty contact was notified for this SERVICE NOTIFICATION.  If the pagerduty contact never shows up, that means that the pagerduty contact is not associated with notifications for the host/service in question.  If you're using the default configuration, make sure that the pagerduty contact is a member of the admins contact group. If the pagerduty contact is getting notified, check the agent log at /var/log/pdagent/pdagentd.log. More troubleshooting tips can be found in our Nagios Troubleshooting Guide.

What sort of Nagios messages does PagerDuty understand?

PagerDuty can process PROBLEM, ACKNOWLEDGEMENT, and RECOVERY messages. All other messages, including FLAPPINGSTART and FLAPPINGSTOP, are ignored.

How can I customize my Nagios alerts?

We have a guide for Customizing Notifications Sent to PagerDuty from Nagios to help you get started.