├── .ruby-version ├── Procfile ├── Gemfile ├── images ├── Heroku_Resources.jpg └── Heroku_Scheduler.jpg ├── .gitignore ├── Gemfile.lock ├── LICENSE ├── pagerduty-incident-scraper.rb └── README.md /.ruby-version: -------------------------------------------------------------------------------- 1 | 2.4.2 2 | -------------------------------------------------------------------------------- /Procfile: -------------------------------------------------------------------------------- 1 | poller: ./pagerduty-incident-scraper.rb 2 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | ruby '2.4.2' 2 | 3 | source 'https://rubygems.org' 4 | 5 | gem 'yajl-ruby' 6 | gem 'httparty' 7 | -------------------------------------------------------------------------------- /images/Heroku_Resources.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/newrelic/insights-about-pagerduty/HEAD/images/Heroku_Resources.jpg -------------------------------------------------------------------------------- /images/Heroku_Scheduler.jpg: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/newrelic/insights-about-pagerduty/HEAD/images/Heroku_Scheduler.jpg -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.gem 2 | *.rbc 3 | .bundle 4 | .config 5 | coverage 6 | InstalledFiles 7 | lib/bundler/man 8 | pkg 9 | rdoc 10 | spec/reports 11 | test/tmp 12 | test/version_tmp 13 | tmp 14 | 15 | # YARD artifacts 16 | .yardoc 17 | _yardoc 18 | doc/ 19 | -------------------------------------------------------------------------------- /Gemfile.lock: -------------------------------------------------------------------------------- 1 | GEM 2 | remote: https://rubygems.org/ 3 | specs: 4 | httparty (0.15.6) 5 | multi_xml (>= 0.5.2) 6 | multi_xml (0.6.0) 7 | yajl-ruby (1.3.1) 8 | 9 | PLATFORMS 10 | ruby 11 | 12 | DEPENDENCIES 13 | httparty 14 | yajl-ruby 15 | 16 | RUBY VERSION 17 | ruby 2.4.2p198 18 | 19 | BUNDLED WITH 20 | 1.15.4 21 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2014 New Relic, Inc. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy of 4 | this software and associated documentation files (the "Software"), to deal in 5 | the Software without restriction, including without limitation the rights to 6 | use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies 7 | of the Software, and to permit persons to whom the Software is furnished to do 8 | so, subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in all 11 | copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 15 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 16 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 17 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 18 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 19 | SOFTWARE. 20 | -------------------------------------------------------------------------------- /pagerduty-incident-scraper.rb: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | require 'pp' 4 | require 'yajl' 5 | require 'httparty' 6 | require 'time' 7 | 8 | PAGERDUTY_TOKEN = ENV['PAGERDUTY_TOKEN'] 9 | PAGERDUTY_DOMAIN = ENV['PAGERDUTY_DOMAIN'] 10 | INSIGHTS_INSERT_KEY = ENV['INSIGHTS_INSERT_KEY'] 11 | INSIGHTS_EVENT_URL = ENV['INSIGHTS_EVENT_URL'] 12 | FETCH_INCIDENTS_SINCE = (ENV['FETCH_INCIDENTS_SINCE'] || 10 * 60).to_i 13 | 14 | incidents_this_tw = HTTParty.get( 15 | "https://#{PAGERDUTY_DOMAIN}.pagerduty.com/api/v1/incidents", 16 | :query => {:until => Time.now, :since => (Time.now - FETCH_INCIDENTS_SINCE), 17 | :status => 'resolved', }, 18 | :headers => {"Authorization" => "Token token=#{PAGERDUTY_TOKEN}"}) 19 | 20 | if incidents_this_tw['incidents'].nil? 21 | raise incidents_this_tw.inspect 22 | end 23 | 24 | puts "Fetched #{incidents_this_tw['incidents'].count} incidents in last #{FETCH_INCIDENTS_SINCE} seconds" 25 | 26 | events = [] 27 | incidents_this_tw['incidents'].each do |incident| 28 | created_on = Time.parse(incident['created_on']) 29 | closed_on = Time.parse(incident['last_status_change_on']) 30 | open_duration = closed_on - created_on 31 | 32 | incidents_log = HTTParty.get( 33 | "https://#{PAGERDUTY_DOMAIN}.pagerduty.com/api/v1/incidents/#{incident['id']}/log_entries", 34 | :query => {:offset => 0, :limit => 100}, 35 | :headers => {"Authorization" => "Token token=#{PAGERDUTY_TOKEN}"}) 36 | 37 | first_assignment = 38 | incidents_log['log_entries'].detect {|entry| entry['type'] == 'assign'} 39 | 40 | first_assigned_to_user = first_assignment.nil? ? 41 | nil : first_assignment['assigned_user']['email'] 42 | 43 | events << { 44 | 'eventType' => 'PagerdutyIncident', 45 | 'eventVersion' => 2, 46 | 'incident_number' => incident['incident_number'].to_i, 47 | 'incident_url' => incident['html_url'], 48 | 'incident_key' => incident['incident_key'], 49 | 'created_on' => created_on.iso8601, 50 | 'last_status_change_on' => closed_on.iso8601, 51 | 'timestamp' => closed_on.to_i, 52 | 'open_duration' => open_duration.ceil, 53 | 'created_on_hour' => created_on.strftime('%H').to_i, 54 | 'last_status_change_on_hour' => closed_on.strftime('%H').to_i, 55 | 'service_name' => incident['service']['name'], 56 | 'service_id' => incident['service']['id'], 57 | 'escalation_policy_name' => incident['escalation_policy']['name'], 58 | 'escalation_policy_id' => incident['escalation_policy']['id'], 59 | 'trigger_type' => incident['trigger_type'], 60 | 'number_of_escalations' => incident['number_of_escalations'].to_i, 61 | 'resolved_by_user' => incident['resolved_by_user'].nil? ? 62 | nil : incident['resolved_by_user']['email'], 63 | 'first_assigned_to_user' => first_assigned_to_user, 64 | } 65 | end 66 | 67 | puts "Reporting #{events.count} events to Insights" 68 | 69 | response = HTTParty.post(INSIGHTS_EVENT_URL, 70 | :body => Yajl::Encoder.encode(events), 71 | :headers => {'Content-Type' => 'application/json', 72 | 'X-Insert-Key' => INSIGHTS_INSERT_KEY}) 73 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | [![Archived header](https://github.com/newrelic/open-source-office/raw/master/examples/categories/images/Archived.png)](https://github.com/newrelic/open-source-office/blob/master/examples/categories/index.md#archived) 2 | 3 | Insights about Pagerduty 4 | ======================== 5 | 6 | ## Purpose 7 | How is your oncall shift going? How many incidents did you field this week, and for what services? Is your alert noise getting worse or better? 8 | 9 | Find answers to these and more by using New Relic Insights to query your PagerDuty incident history! This is a small Ruby script, suitable for running in Heroku, that queries PagerDuty every 10 minutes and populates Insights with information about resolved PagerDuy incidents. 10 | 11 | ## Usage 12 | ### Software required: 13 | * Ruby 2.1.1 14 | * Bundler 15 | 16 | ### Required environment variables: 17 | * `PAGERDUTY_TOKEN` - your [PagerDuty API key](https://support.pagerduty.com/entries/23761081-Generating-an-API-Key). 18 | * `PAGERDUTY_DOMAIN` - your PagerDuty subdomain. 19 | * `INSIGHTS_INSERT_KEY` - your New Relic Insights [insert key](http://docs.newrelic.com/docs/insights/inserting-events#register). 20 | * `INSIGHTS_EVENT_URL` - The Endpoint to send Insights events to. Can be found in the examples in your Data Management -> API Keys page. Will look like: `https://insights-collector.newrelic.com/v1/accounts//events`. 21 | 22 | ### Optional environment variables: 23 | * `FETCH_INCIDENTS_SINCE` - The number of seconds to look in the past for pagerduty incidents. Defaults to 10 minutes. Set this to your run interval for the script. You may set this to be wider when running manually to backfill events, but note that Insights only allows backdated events up to 24 hours old. [NB: The code makes no attempt to prevent duplicate entries in Insights, so be careful when experimenting.] 24 | 25 | ### Command line: 26 | ```bash 27 | bundle install 28 | bundle exec ./pagerduty-incident-scraper.rb 29 | ``` 30 | 31 | ### Heroku: 32 | Set the required ENV inside your heroku app and configure the Heroku Scheduler add-on to the script every 10 minutes. Each run is very short, so it should fit easily into free dyno hours. 33 | 34 | When configured, it will look something like this: 35 | 36 | ![](images/Heroku_Resources.jpg) 37 | 38 | Then configure the Heroku Scheduler Add-on: 39 | 40 | ![](images/Heroku_Scheduler.jpg) 41 | 42 | 43 | ## Example NRQL Queries 44 | 45 | Once you've loaded some of your data, what kind of things can you find out? 46 | 47 | ### Who fielded the most alerts this week? 48 | ```sql 49 | SELECT count(*) from PagerdutyIncident FACET first_assigned_to_user SINCE 1 week ago TIMESERIES 50 | ``` 51 | ![alert recipient chart](http://i.imgur.com/aMjuk7Q.png) 52 | 53 | ### What hours of the day are incidents created? 54 | ```sql 55 | SELECT histogram(created_on_hour, 24, 24) from PagerdutyIncident WHERE eventVersion >= 2 SINCE 1 week ago 56 | ``` 57 | ![alert hour histogram](http://i.imgur.com/hzgEWoZ.png) 58 | 59 | #### Wait, what's eventVersion? 60 | Because Insights doesn't currently allow modification of stored events, if the schema of the events sent in changes, it can be hard to make queries that depend on the schema looking a certain way. This script sends an `eventVersion` integer attribute that you should increment if you change the schema of the events you send. For this query, `created_on_hour` was changed from a string to an integer in eventVersion 2, so it's scoped to those. 61 | 62 | ### How long are our incidents open? 63 | ```sql 64 | SELECT histogram(open_duration, 600) from PagerdutyIncident SINCE 1 week ago 65 | ``` 66 | ![incident open time histogram](http://i.imgur.com/i2m9LBt.png) 67 | 68 | ### How many incidents are being routed to my team? 69 | ```sql 70 | SELECT count(*) from PagerdutyIncident WHERE escalation_policy_name='Site Services' SINCE 1 week ago TIMESERIES 71 | ``` 72 | ![Escalation policy alert count chart](http://i.imgur.com/drTSyAG.png) 73 | 74 | ## Contributing 75 | 76 | You are welcome to send pull requests to us - however, by doing so you agree that you are granting New Relic a non-exclusive, non-revokable, no-cost license to use the code, algorithms, patents, and ideas in that code in our products if we so choose. You also agree the code is provided as-is and you provide no warranties as to its fitness or correctness for any purpose. 77 | --------------------------------------------------------------------------------