SolarWinds Orion Integration Guide

SolarWinds Orion platform products, such as Network Performance Monitor (NPM) and Server & Application Monitor (SAM), provide network, performance, availability and server monitoring for multi-vendor networks, Linux and Windows Server systems, and detailed application performance monitoring metrics for over 200 applications such as Apache HTTP Server, Microsoft Exchange, and Oracle Database. Both NPM and SAM allow you to trigger and reset alerts based on custom thresholds that can trigger and resolve incidents in PagerDuty. When SolarWinds Orion creates an alert notification for a node, a component or an application, it will send an event to a corresponding service in PagerDuty. This event will trigger a new incident on the associated PagerDuty service, or group as alerts into an existing incident. This integration works by sending SolarWinds Orion alerts formatted as JSON, using our Events API.  When the node, component or application reaches a recovered state in SolarWinds, a resolve event will be sent to the PagerDuty service to resolve the alert, and associated incident on that service

Compatibility

PagerDuty integrations require a Manager base role or higher for account authorization. If you do not have this role, please reach out to an Admin or Account Owner within your organization to configure the integration. This guide and HTTP Request body were written and tested on Windows Server 2012 R2 with SolarWinds Orion platform 2015.1, 2016.1, 2018.4 and 2019.2. Please see our Legacy SolarWinds NPM Integration Guide if you have an older version of NPM. Earlier versions of SAM/APM and SolarWinds products not using the Orion platform can be configured to send alerts to PagerDuty using an email integration instead.

In PagerDuty

There are two ways that SolarWinds Orion can be integrated with PagerDuty: via Global Event Routing or directly through an integration on a PagerDuty Service. If you are integrating SolarWinds Orion with an existing PagerDuty service, please skip to the Integrating with a PagerDuty Service section of this guide.

Integrating With Global Event Routing

Integrating with Global Event Routing enables you to route events to specific services based on the payload of the event from SolarWinds Orion. If you would like to learn more, please visit our article on Global Event Routing.

1. From the Automation menu, select Event Rules and click your Default Global Ruleset.
2. On the Event Rules screen, click on the arrow next to Incoming Event Source to display the Integration key information. Copy your Integration Key and keep it in a safe place for later use. When you have finished setting up the integration in SolarWinds Orion, you will return to this interface to specify how to route events from SolarWinds Orion to services in PagerDuty.

Note:  If your team has already integrated using Global Event Routing, you can verify by checking for the PagerDuty Event Rules Integration Key (found in steps 1 and 2 above) in an existing configuration file. You can then use the existing integration and pipe SolarWinds Orion events into your service by writing new Global Event Rules for your service. You can now proceed to the In SolarWinds Orion section of this guide.  

Integrating With a PagerDuty Service

Integrating with a PagerDuty service directly can be beneficial if you don’t need to route alerts from SolarWinds Orion to different responders based on the event payload. You can still use service-level event rules to perform actions such as suppressing non actionable alerts and adjusting the severity.

1. From the Services menu, select Service Directory.
2. If you are adding your integration to an existing service, click the name of the service you want to add the integration to. Then select the Integrations tab and click the Add a new integration.If you are creating a new service for your integration, please read our documentation in section Configuring Services and Integrations and follow the steps outlined in the Create a New Service section.
3. Select SolarWinds Orion from the Integration Type menu.
4. Click Add Integration button to save your new integration. You will be redirected to the Integrations tab for your service.
5. Find your integration in the list and click ∨ to the right. An Integration Key will be generated on this screen. Keep this key saved in a safe place, as it will be used when you configure the integration with SolarWinds Orion in the next section.

On Your SolarWinds Server

1. Log in to SolarWinds as an admin, then go to the Settings menu and select All Settings (or click Settings in the top right corner of the page if you have an older version of SolarWinds). Click Manage Custom Properties under the Node & Group Management section.
2. Click Add Custom Property. 
3. Select Nodes and click Next. 
4. Set the Property Name to "PDIntegrationKey" and, optionally, set a Description, such as “PagerDuty Integration,” then click Next. 
5. Click Select Nodes and check all the nodes in the Available Nodes column that you want to receive alerts for in PagerDuty, click Add to move the nodes to the Selected Nodes column, then click Select Nodes. 
6. Paste the PagerDuty Integration Key you copied earlier in the field under Assign value for “PDIntegrationKey” to selected Nodes, then click Submit. 
7. Download the sample alerts, saving anywhere you like on your workstation or SolarWinds server.
   • Sample Alerts
8. Extract Sample Alerts.zip on your workstation or SolarWinds server (i.e. on your desktop, Downloads folder, or C:\PagerDuty\Sample Alerts).
9. Navigate back to the Main Settings & Administration page in SolarWinds Orion by going to the Settings menu and selecting All Settings (or click Settings in the top right corner of the page if you have an older version of SolarWinds). Click Manage Alerts under the Alerts & Reports section.
10. Next, click Export/Import to import some of the most popular alerts we’ve created templates for. Click Import Alert and select the alert definitions you would like to import to SolarWinds from the extracted Sample Alerts location.Congratulations! SolarWinds will now be able to trigger and resolve incidents in PagerDuty.

Testing an Alert

Open an alert from SolarWinds panel and navigate to the Trigger Action tab. Simulate the action to send an alert via POST. If successful, a PagerDuty incident will be triggered. If it fails, double check the JSON format and SolarWinds variables that are being used in the HTTP Request Body.

Building Your Own Alerts

You can send additional alerts to PagerDuty by selecting the alert you want to use and going to Trigger Actions, then simply add an action using the Log the Alert to a File option. The file should be logged to C:\PagerDuty\Queue with a .txt file extension, and the content should be JSON that is formatted for our Events API. Check out the alert definitions we provide for examples of how to write your own alerts to be sent to PagerDuty. Special Characters: you must avoid including quotation mark characters in the alert template. These will cause string termination in the output, resulting in invalid JSON syntax, which will result in the events failing to submit to the Events API. If you need to include quotation marks inside of the template, then it is highly recommended that you use curly quotes, i.e. ” and “, rather than straight quotes (").

FAQ

Can I trigger incidents on multiple PagerDuty services?

Yes! Simply go to the details page for the node that you want to trigger incidents on a different PagerDuty service, scroll down to the Custom Properties section, then click Edit Custom Property Values to set a different value for PDIntegrationKey on that node. No need to change your alerts (unless you’ve hard-coded the integration key for your custom alerts), as the templates we provide use a SQL query to pull the integration key from this custom property on each node.  

Why isn't my alert triggering?

First try modifying the alert's Trigger Conditions to something that is presently true to be sure that the problem is not a failure to meet the trigger conditions. If after trying this you see "This alert would be immediately triggered on 0 object(s) in alert scope" in the final Summary step above the Submit button, even though you know that the conditions you have entered under the Trigger Conditions are presently met, a particular bug is likely the culprit. Sometimes SolarWinds alerts can become corrupted and need to be replaced with a new alert.  To resolve the issue, create a new alert from scratch with all trigger conditions, actions, etc. identical to the original alert; at the final Summary step you should then see a message indicating that the new alert will trigger for at least one object. After saving the new alert you can then remove the original, corrupted alert.  

Why do I keep getting an HTTP error when testing/simulating the action?

If you get a "Failed to execute HTTP request" when trying to simulate a trigger or resolve action on an alert, it's usually either because the JSON is invalid or because the SolarWinds variables are not populating properly.

• Try pasting the entire contents of the action's HttpRequestBody field in a JSON validator tool to catch any formatting errors.
• Try testing by replacing any SolarWinds variables (e.g., ${N=SwisEntity;M=FullName}) with dummy data, then testing again with the simulate button.  Once you've managed to successfully create a test alert/incident on PagerDuty by clicking simulate on the action, you can add back variables incrementally; you can do this most easily by clicking the Insert variables button while editing the action.  Any variables that do not have a preview value will not populate properly for the type of resource that the alert in question is set to monitor (e.g., node, interface, etc.).