Scalyr Integration Guide

Scalyr unifies multiple functions into a single tool: log aggregation, search, and analysis; server metrics; dashboards and alerts, external monitoring, and more.  Combined with PagerDuty's on-call scheduling and alerting capabilities, you can easily make sure that your systems are being monitored and that your on-call team is notified when an issue arises. When a Scalyr alert is triggered, Scalyr will report an incident to PagerDuty. If the alert resolves, Scalyr will mark the PagerDuty incident as resolved. You can use PagerDuty for some or all of your Scalyr alerts. You can also choose to have Scalyr send notifications to both PagerDuty and a list of e-mail addresses.

In PagerDuty

1. From the Configuration menu, select Services. 

2. On your Services page: If you are creating a new service for your integration, click +Add New Service. If you are adding your integration to an existing service, click the name of the service you want to add the integration to. Then click the Integrations tab and click the +New Integration button.



3. Select your app from the Integration Type menu and enter an Integration Name. If you are creating a new service for your integration, in General Settings, enter a Name for your new service. Then, in Incident Settings, specify the Escalation Policy, Notification Urgency, and Incident Behavior for your new service.

4. Click the Add Service or Add Integration button to save your new integration. You will be redirected to the Integrations page for your service.

5. Copy the Integration Key for your new integration:

In Scalyr

1. Click the Alerts navigation link, then click Edit Alerts.

2. Edit the Alerts configuration file to specify PagerDuty as the alert recipient. To do this for all Scalyr alerts, create or edit analertAddress field at the top level of the file:

   {
       "alertAddress": "pagerduty:XXXXX",
   
       alerts: [
         ...
       ]
   }

   Replace XXXXX with the Integration Key you generated in PagerDuty (Step 6 above). To send notifications to one or more e-mail addresses in addition to PagerDuty, list them all in alertAddress:

     "alertAddress": "pagerduty:XXXXX, foo@example.com, bar@example.com",

   If you only want to use PagerDuty for certain alerts, you can specify an alertAddress field for those alerts:

   {
       alertAddress: "email@example.com",
   
       alerts: [
         // This alert will be sent to PagerDuty
         {
           trigger: "count:1m(error) > 10",
           "alertAddress": "pagerduty:XXXXX",
       },
   
       // This alert will send notifications to email@example.com
       {
         trigger: "mean:10m($source='tsdb' $serverHost='server1' metric='proc.stat.cpu_rate' type='user') > 50"
       }
     ]
   }

   To link a whole group of alerts to PagerDuty, specify an appropriatealertAddress for the group. See Specifying Alert Recipients.

Verify that Scalyr and PagerDuty are Communicating

You can verify that they are communicating by triggering a test alert. Once the alert clears, the incident will automatically be resolved within PagerDuty.  Please note, it can take approximately five minutes before Scalyr will send out the RESOLVE notification to PagerDuty.

FAQ

How can I setup Scalyr to be tied to multiple PagerDuty services?

You can create any number of Scalyr/Generic API services in PagerDuty, each with its own escalation policy. To link Scalyr to multiple services, simply enter the appropriate Integration Key in each alertAddress field. If you are having trouble completing the installation, please contact support.