PagerDuty
PagerDuty
PagerDuty is an enterprise incident resolution service that integrates with ITOps and DevOps monitoring stacks to improve operational reliability and agility. From enriching and aggregating events to correlating them into incidents, PagerDuty streamlines the incident management process by reducing alert noise and resolution times. You can send notifications to PagerDuty using Netdata’s Agent alert notification feature, which supports dozens of endpoints, user roles, and more.
Setup
Prerequisites
- An installation of the PagerDuty Agent on the node running the Netdata Agent
- A PagerDuty Generic API service using either the
Events API v2orEvents API v1 - Add a new service to PagerDuty. Click Use our API directly and select either
Events API v2orEvents API v1. Once you finish creating the service, click on the Integrations tab to find your Integration Key. - Access to the terminal where Netdata Agent is running
Configuration
File
The configuration file name for this integration is health_alarm_notify.conf.
You can edit the configuration file using the edit-config script from the
Netdata config directory.
cd /etc/netdata 2>/dev/null || cd /opt/netdata/etc/netdata
sudo ./edit-config health_alarm_notify.conf
Options
The following options can be defined for this notification
| Name | Description | Default | Required |
|---|---|---|---|
| SEND_PD | Set SEND_PD to YES |
YES | yes |
| DEFAULT_RECIPIENT_PD | Set DEFAULT_RECIPIENT_PD to the PagerDuty service key you want the alert notifications to be sent to. You can define multiple service keys like this: pd_service_key_1 pd_service_key_2. |
yes |
DEFAULT_RECIPIENT_PD
All roles will default to this variable if left unconfigured.
The DEFAULT_RECIPIENT_PD can be edited in the following entries at the bottom of the same file:
role_recipients_pd[sysadmin]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxa"
role_recipients_pd[domainadmin]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxb"
role_recipients_pd[dba]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxc"
role_recipients_pd[webmaster]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxd"
role_recipients_pd[proxyadmin]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxe"
role_recipients_pd[sitemgr]="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxf"
Examples
Basic Configuration
#------------------------------------------------------------------------------
# pagerduty.com notification options
SEND_PD="YES"
DEFAULT_RECIPIENT_PD="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
USE_PD_VERSION="2"
Troubleshooting
Test Notification
You can run the following command by hand, to test alerts configuration:
# become user netdata
sudo su -s /bin/bash netdata
# enable debugging info on the console
export NETDATA_ALARM_NOTIFY_DEBUG=1
# send test alarms to sysadmin
/usr/libexec/netdata/plugins.d/alarm-notify.sh test
# send test alarms to any role
/usr/libexec/netdata/plugins.d/alarm-notify.sh test "ROLE"
Note that this will test all alert mechanisms for the selected role.