Riemann Integration Guide

Riemann aggregates events from your servers and applications with a powerful stream processing language. Riemann can notify you of exceptions raised by your code, track the latency distribution of your web application, see the top processes on any host, by memory and CPU, send alerts when a key process fails to check in, know how many users signed up right this second, as well as a variety of other use cases.

In PagerDuty

1. Go to the Configuration menu and select Services.

2. On the Services page:

   • If you are creating a new service for your integration, click Add New Service.

   • If you are adding your integration to an existing service, click the name of the service you want to add the integration to. Then click the Integrations tab and click the New Integration button.

3. Select your app from the Integration Type menu

4. Enter an Integration Name. If you are creating a new service for your integration, in General Settings, enter a Name for your new service. Then, in Incident Settings, specify the Escalation Policy, Notification Urgency, and Incident Behavior for your new service.

5. Click the Add Service or Add Integration button to save your new integration. You will be redirected to the Integrations page for your service.

6. Copy the Integration Key for your new integration.

On Your Riemann Server

Note: This guide only deals with the setup of the actual Riemann monitoring tool with PagerDuty to trigger incidents, it does not review how to set up riemann-dash which provides a web-based UI that contains visualizations of Riemann's activities. This integration requires a Linux-based system which has up-to-date versions of both Java and Ruby installed which will be used to install Riemann.

1. Edit the Riemann configuration file (etc/riemann.config in Riemann's main directory) to set the correct hostname for the host you'd like to be notified about in PagerDuty. In this example, we'll use localhost (i.e. 127.0.0.1):

2. You'll now want to edit the Riemann configuration file to send notifications to PagerDuty. Although you can add the default code seen here, this will cause any notifications that aren't in "ok" state to trigger PagerDuty incidents. This may cause an alert storm, which will subsequently cause you to be rate-limited by PagerDuty's Events API. We recommend you add a modified version to the bottom of your etc/riemann.config file, which will only trigger on conditions you explicitly set:

3. You'll want to test that everything is working by manually triggering an alert in Riemann. This can be accomplished manually via the Ruby client from the Ruby interactive interpreter. Start irb as follows:

4. Trigger an incident in the following format: Note: Although the ruby client will assume your events come from the local hostname unless you pass host: nil, you may want to specify the hostname explicitly just for testing purposes.

5. The event should show up in your Riemann streaming logs. An incident should also be triggered in PagerDuty. When the state changes back to "ok" the incident should also be resolved in PagerDuty.