├── LICENSE ├── README.md ├── pagerduty-resolve └── pagerduty-trigger /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2012 Cold Brew Labs, Inc. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy 4 | of this software and associated documentation files (the "Software"), to 5 | deal in the Software without restriction, including without limitation the 6 | rights to use, copy, modify, merge, publish, distribute, sublicense, and/or 7 | sell copies of the Software, and to permit persons to whom the Software is 8 | furnished to do so, subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in 11 | all copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 15 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 16 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 17 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING 18 | FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS 19 | IN THE SOFTWARE. -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | **Note:** This project is no longer actively maintained by Pinterest. 2 | 3 | --- 4 | 5 | ## pagerduty-monit 6 | 7 | The scripts in this repository are wrappers for the PagerDuty API which will 8 | open and close PagerDuty incidents from the command line. These scripts are 9 | designed to help you integrate monit with PagerDuty. 10 | 11 | Copyright (c) 2012 Cold Brew Labs, Inc. See LICENSE for details. 12 | 13 | **Usage:** `pagerduty-trigger ` 14 | 15 | **Example:** `pagerduty-trigger nginx` on the server "webserver1.example.com" 16 | will trigger a PagerDuty incident with the subject "nginx failed on 17 | webserver1". 18 | 19 | **Prerequisites:** Requires pagerduty library for Python. You can download this 20 | from http://pypi.python.org/pypi/pagerduty/0.2.1, or run `pip install 21 | pagerduty` to download and install automatically. 22 | 23 | **To configure PagerDuty:** To configure PagerDuty, login as an administrator 24 | and go to the Services tab; then click "Add New Service". You can give the 25 | service any name you'd like, and choose "Generic API system." PagerDuty will 26 | give you an API service key for this service. Be sure to set the 27 | `PAGERDUTY_SERVICE_KEY` variable in both scripts. 28 | 29 | **To configure Monit:** To configure Monit to trigger an event with this 30 | script, save the script in a location like /etc/monit; make sure it's executable; 31 | and use an action like this in your monitrc file: 32 | 33 | exec "/etc/monit/pagerduty-trigger " 34 | 35 | Example monit stanza to trigger an incident if nginx is not present: 36 | 37 | check process nginx with pidfile /var/run/nginx.pid 38 | if does not exist for 3 cycles 39 | then exec "/etc/monit/pagerduty-trigger nginx" 40 | else if passed for 3 cycles 41 | then exec "/etc/monit/pagerduty-resolve nginx" 42 | -------------------------------------------------------------------------------- /pagerduty-resolve: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # pagerduty-resolve is a wrapper around the PagerDuty API to close an incident 4 | # from the command line. 5 | # 6 | # Copyright (c) 2012 Cold Brew Labs, Inc. See LICENSE for details. 7 | 8 | # Uncomment and enter your service key here. 9 | # PAGERDUTY_SERVICE_KEY="1234567890abcdef1234567890abcdef" 10 | 11 | if [ -z "$1" ]; then 12 | echo "Usage: $0 " 13 | exit 1 14 | elif [ -z "$PAGERDUTY_SERVICE_KEY" ]; then 15 | echo "Failed to resolve event: you must set the PAGERDUTY_SERVICE_KEY variable." 16 | exit 1 17 | elif [ ! -x "/usr/local/bin/pagerduty" ]; then 18 | echo "Failed to resolve event: /usr/local/bin/pagerduty does not exist or is not executable." 19 | exit 1 20 | fi 21 | 22 | EVENT=$1 23 | HOST=`hostname -s` 24 | INCIDENT_KEY=`echo "$HOST:$EVENT" | md5sum | cut -f 1 -d ' '` 25 | TMP_FILE="/tmp/pagerduty-$INCIDENT_KEY" 26 | 27 | DESCRIPTION="$EVENT resolved on $HOST" 28 | 29 | rm -f $TMP_FILE 30 | /usr/local/bin/pagerduty -k "$PAGERDUTY_SERVICE_KEY" -i "$INCIDENT_KEY" --description="$DESCRIPTION" resolve 31 | 32 | if [ "$?" -ne "0" ]; then 33 | echo "Failed to resolve incident" 34 | exit 1 35 | fi 36 | 37 | echo "Incident resolved successfully" 38 | exit 0 39 | -------------------------------------------------------------------------------- /pagerduty-trigger: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # 3 | # pagerduty-trigger is a wrapper around the PagerDuty API to open a new 4 | # incident from the command line. This script is designed to help integrate 5 | # monit with PagerDuty. 6 | # 7 | # Copyright (c) 2012 Cold Brew Labs, Inc. See LICENSE for details. 8 | # 9 | # Uncomment and enter your service key here. 10 | # PAGERDUTY_SERVICE_KEY="1234567890abcdef1234567890abcdef" 11 | 12 | if [ -z "$1" ]; then 13 | echo "Usage: $0 " 14 | exit 1 15 | elif [ -z "$PAGERDUTY_SERVICE_KEY" ]; then 16 | echo "Failed to trigger event: you must set the PAGERDUTY_SERVICE_KEY variable." 17 | exit 1 18 | elif [ ! -x "/usr/local/bin/pagerduty" ]; then 19 | echo "Failed to trigger event: /usr/local/bin/pagerduty does not exist or is not executable." 20 | exit 1 21 | fi 22 | 23 | EVENT="$1" 24 | HOST=`hostname -s` 25 | INCIDENT_KEY=`echo "$HOST:$EVENT" | md5sum | cut -f 1 -d ' '` 26 | TMP_FILE="/tmp/pagerduty-$INCIDENT_KEY" 27 | 28 | if [ -f "$TMP_FILE" ]; then 29 | # re-trigger after 4 hrs 1 min (the extra minute is to ensure the incident auto-resolved with PagerDuty) 30 | if [ "$(( $(date +"%s") - $(stat -c "%Y" $TMP_FILE) ))" -lt "14460" ]; then 31 | echo "$TMP_FILE exists, aborting" 32 | exit 0 33 | else 34 | echo "$TMP_FILE exists but is older than 4 hours; re-triggering" 35 | fi 36 | fi 37 | 38 | DESCRIPTION="$EVENT failed on $HOST" 39 | DATE=`date` 40 | 41 | echo "$DATE: $DESCRIPTION" >> $TMP_FILE 42 | /usr/local/bin/pagerduty -k "$PAGERDUTY_SERVICE_KEY" -i "$INCIDENT_KEY" --description="$DESCRIPTION" trigger 43 | 44 | if [ "$?" -ne "0" ]; then 45 | echo "Failed to trigger incident" 46 | exit 1 47 | fi 48 | 49 | echo "Incident triggered successfully" 50 | exit 0 51 | --------------------------------------------------------------------------------