SCOM Integration Guide – Email

SCOM is a central component of the Microsoft System Center suite. It is used to monitor the state, health, and performance of everything from servers to individual applications in Microsoft Windows environments. Alerts generated in SCOM can trigger incidents in PagerDuty to alert the appropriate technician via SMS, phone, email or push notification. This Integration was tested with the following versions of software:

• Windows Server 2012 R2
• Microsoft System Center 2012 R2 (Operations Manager)

In PagerDuty

1. From the Services menu, select Service Directory.
2. If you are adding your integration to an existing service, click the name of the service you want to add the integration to. Then select the Integrations tab and click the Add a new integration. If you are creating a new service for your integration, please read our documentation in section Configuring Services and Integrations and follow the steps outlined in the Create a New Service section.
3. Select your integration from the Integration Type menu.
4. Click the Add Service or Add Integration button to save your new integration. You will be redirected to the Integrations page for your service.
5. Find your integration in the list and click ∨ to the right. Copy the Integration Email and keep it in a safe place for later use.

In SCOM

Microsoft System Center Operations Manager can trigger and resolve PagerDuty incidents via email. Below are the steps that are required to setup communication between SCOM and PagerDuty.

1. Open the SCOM 'Operations Manager'.
2. Click on 'Administration' at the bottom left.
3. Expand the 'Notifications' menu and select 'Channels'.
4. Create a new 'Email Notification Channel'.
5. Name it 'PagerDuty Email' , then select 'Next'.
6. Click the 'Add' button.
7. This should bring up an 'Add SMTP Server' window. Now fill in your appropriate 'SMTP Server', 'Port Number' and 'Authentication Method' and click 'OK'. *Warning*: An SMTP Server is not provided by PagerDuty and would need to built up on your own. 
8. Enter in your 'Return Address' (this is the email address that will show up as the 'From' address in your 'PagerDuty" incidents. This can be customized to your liking.) Enter your 'Retry Interval' and click 'Next'.
9. The next screen will display the default 'Subject' and 'Email Message'. This information can be altered to your choosing. Once done, click 'Finish'.
10. Next under the 'Notifications' menu select 'Subscribers', then click 'New'.
11. Enter 'PagerDuty Email' for the name and select 'Next'.
12. In the Schedule menu select 'Always send notifications' and click 'Next'.
13. Next Click 'Add'.
14. Enter the 'Address Name' as 'PagerDuty Email' and click 'Next'.
15. Now use the drop down menu on 'Channel Type' and select 'Email (SMTP)'. Then enter in your 'PagerDuty Integration Email' endpoint that was provided in when you created your PagerDuty Service and click 'Next'.
16. Select 'Always Send Notifications' and select 'Finish'.
17. Now confirm your 'Subscriber Address', then select 'Next'.
18. Next under the 'Notifications' menu select 'Subscriptions', then click 'New'.
19. Under 'Subscription name' enter in 'PagerDuty Email', then click 'Next'.
20. The 'Subscription Criteria' can be completely customized to your liking. These are the Criteria's that the subscription will use to fire off. For this case we are going to leave all options unchecked. You should notice a 'Notify on all alerts' in the description, then click 'Next'.
21. Click the 'Add' option to add a subscriber to the subscription, then click the 'search' button.
22. Start typing 'PagerDuty' in the 'Filter by', then click the 'search' button. Click the 'PagerDuty Email' option under channels and select 'Add', then 'Ok'.
23. Now click 'Next'.
24. Now in the 'Channels' area you will need to add the PagerDuty Channel that was configured earlier. Click the 'Add' button.
25. Start typing 'PagerDuty' in the 'Filter by', then click the 'search' button. Click the 'PagerDuty Email' option under channels and select 'Add', then 'Ok'.
26. Make sure to send notifications without delay and select 'Next'.
27. Confirm your notification subscription settings and select 'Finish'.

In PagerDuty

Per your SCOM instance there are several 'Resolution States' that can alert and send an email out to your SCOM email endpoint.   In this example we are going to ignore the following 'Resolution States' : 'Acknowledged', 'Awaiting Evidence', 'Assigned To Engineering' , and 'Scheduled'. We are going to 'Trigger' an incident based off of the 'New' state and 'Resolve' an incident based off of the 'Resolved' and 'Closed' states. To enable this option you will need to make the following changes to your PagerDuty SCOM Email Service:

1.  In your PagerDuty Dashboard select the 'Configuration' dropdown menu and select 'Services'.
2. Click on your SCOM Email Integration, then click the gear wheel to the right and select 'edit'.
3. In order to ignore the 'Resolution States' we will need to use the drop down menu for 'Email Filters' and select 'Accept email only if it matches ONE OR MORE rules below', then add four rules and use the option in 'The email subject' with the drop down selection of 'does not match the regex' and for each ruled add one of the states that SCOM produces.
4. Now under 'Email Management' select the option 'Create and resolve incidents based on custom rules', then create three rules.
   • The first rule will be a 'trigger' with the options selected for the 'Condition' to 'The email subject contains New' and for the 'Incident Key' field use 'In the email body, match all text between { and }' .
   • The second rule will be a 'resolve' with the options selected for the 'Condition' to 'The email subject contains Resolved' and for the 'Incident Key' field use 'In the email body, match all text between { and }' .
   • The third rule will be a 'resolve' with the options selected for the 'Condition' to 'The email subject contains Closed' and for the 'Incident Key' field use 'In the email body, match all text between { and }'.
   • Then select the catch all rule to list the following: If an email does not match any of the rules above,'discard it'.
5. Then click 'Save changes'.
6. Your Service should now look like this:
7. Now you have auto-resolution setup with email parsing for you SCOM email service. Your SCOM Integration with PagerDuty is complete. Now when a alert populates in SCOM you should receive a notification in PagerDuty.

FAQ

Will SCOM incidents automatically resolve?

Yes. This will need to be configured within your PagerDuty SCOM Email service.

Can I setup SCOM to be tied to multiple PagerDuty services?

Yes, you will need to create multiple SCOM Email Services, then create the additional Channel, Subscribers, and Subscriptions within your SCOM server.

Are incidents de-duplicated?

Yes, incidents are de-duplicated using text between { and } in the email body as the incident key (this is configured in step 4).

Is there a 2-way ack-back integration?

No, acknowledging or resolving an incident in PagerDuty will not be forwarded to SCOM.

Are there logs I can reference in SCOM?

Yes, all alerts are stored and displayed for users within SCOM and the local Event Viewer.

Is there a retry mechanism if the Email call does not succeed?

Yes, you may configure retries in SCOM's Channel Settings.