├── .gitignore ├── project.clj ├── README.md └── src └── cognician └── dogstatsd.clj /.gitignore: -------------------------------------------------------------------------------- 1 | target 2 | *.sublime-* 3 | .nrepl-history 4 | .nrepl-port 5 | pom.xml 6 | pom.xml.asc 7 | -------------------------------------------------------------------------------- /project.clj: -------------------------------------------------------------------------------- 1 | (defproject cognician/dogstatsd-clj "0.1.2" 2 | :description "Clojure client for DogStatsD, Datadog’s StatsD agent" 3 | :license {:name "Eclipse Public License" 4 | :url "http://www.eclipse.org/legal/epl-v10.html"} 5 | :url "https://github.com/Cognician/dogstatsd-clj" 6 | :dependencies [[org.clojure/clojure "1.7.0"]]) 7 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Clojure client for DogStatsD, Datadog’s StatsD agent 2 | 3 | For general information about DataDog, DogStatsD, how they're useful 4 | and why all this is useful - read the _Rationale, Context, additional 5 | documentation_ section below. 6 | 7 | ## Setting things up 8 | 9 | Add to project.clj: 10 | 11 | ```clj 12 | [cognician/dogstatsd-clj "0.1.1"] 13 | ``` 14 | 15 | Require it: 16 | 17 | ```clj 18 | (require '[cognician.dogstatsd :as dogstatsd]) 19 | ``` 20 | 21 | 22 | ## Configuring 23 | 24 | To configure, provide URL of DogStatsD: 25 | 26 | ```clj 27 | (dogstatsd/configure! "localhost:8125") 28 | ``` 29 | 30 | Optionally, you can provide set of global tags to be appended to every metric: 31 | 32 | ```clj 33 | (dogstatsd/configure! "localhost:8125" { :tags {:env "production", :project "Secret"} }) 34 | ``` 35 | 36 | 37 | ## Reporting 38 | 39 | After that, you can start reporting metrics: 40 | 41 | Total value/rate: 42 | 43 | ```clj 44 | (dogstatsd/increment! "chat.request.count" 1) 45 | ``` 46 | 47 | In-the-moment value: 48 | 49 | ```clj 50 | (dogstatsd/gauge! "chat.ws.connections" 17) 51 | ``` 52 | 53 | Values distribution (mean, avg, max, percentiles): 54 | 55 | ```clj 56 | (dogstatsd/histogram! "chat.request.time" 188.17) 57 | ``` 58 | 59 | To measure function execution time, use `d/measure!`: 60 | 61 | ```clj 62 | (dogstatsd/measure! "thread.sleep.time" {} 63 | (Thread/sleep 1000)) 64 | ``` 65 | 66 | Counting unique values: 67 | 68 | ```clj 69 | (dogstatsd/set! "chat.user.email" "nikita@mailforspam.com") 70 | ``` 71 | 72 | 73 | ## Tags and throttling 74 | 75 | Additional options can be specified as third argument to report functions: 76 | 77 | ```clj 78 | { :tags => [String+] | { Keyword -> Any | Nil } 79 | :sample-rate => Double[0..1] } 80 | ``` 81 | 82 | Tags can be specified as map: 83 | 84 | ```clj 85 | {:tags { :env "production", :chat nil }} ;; => |#env:production,chat 86 | ``` 87 | 88 | or as a vector: 89 | 90 | ```clj 91 | {:tags [ "env:production", "chat" ]} ;; => |#env:production,chat 92 | ``` 93 | 94 | 95 | ## Events: 96 | 97 | ```clj 98 | (dogstatsd/event! "title" "text" opts) 99 | ``` 100 | 101 | where opts could contain any subset of: 102 | 103 | ```clj 104 | { :tags => [String+] | { Keyword -> Any | Nil } 105 | :date-happened => java.util.Date 106 | :hostname => String 107 | :aggregation-key => String 108 | :priority => :normal | :low 109 | :source-type=name => String 110 | :alert-type => :error | :warning | :info | :success } 111 | ``` 112 | 113 | 114 | ## Example 115 | 116 | ```clj 117 | (require '[cognician/dogstatsd :as dogstatsd]) 118 | 119 | (dogstatsd/configure! "localhost:8125" {:tags {:env "production"}}) 120 | 121 | (dogstatsd/increment! "request.count" 1 {:tags ["endpoint:messages__list"] 122 | :sample-rate 0.5}) 123 | ``` 124 | 125 | ## Rationale, context, additional documentation ## 126 | 127 | ### Rationale and Context ### 128 | DataDog, being a monitoring service, has the ability, through their 129 | DogStatsD implementation, to collect and show important information 130 | like when things are happening, and how long those things take. 131 | 132 | An example is here: 133 | [Cog Validation 134 | Time](https://app.datadoghq.com/dash/211555/production-monolith?screenId=211555&screenName=production-monolith&from_ts=1544104800000&is_auto=false&live=true&page=0&to_ts=1544191200000&fullscreen_widget=399429687&tile_size=m) 135 | 136 | (It should show how long a validation function takes, which, over 137 | time, we hope to correlate with core dumps or slow service events) 138 | 139 | Because the data is pulled into DataDog, the graph widgets can be 140 | pulled into dashboards, so synchronisation and correlation can take 141 | place. 142 | 143 | ### Local testing ### 144 | 145 | Since DogStatsD is DataDog's service, you'll want to tighten the loop 146 | on feedback and prevent contamination of production data with 147 | dev/testing info. 148 | 149 | An excellent package is 150 | [https://github.com/jonmorehouse/dogstatsd-local](https://github.com/jonmorehouse/dogstatsd-local) 151 | 152 | It allows you to create a StatsD listener on localhost - and spits 153 | results out in the terminal when you make calls. The process is pretty 154 | straightforward: 155 | - Install go `brew install go` worked nicely enough 156 | - Clone the repository: `git clone https://github.com/jonmorehouse/dogstatsd-local.git` 157 | - cd into the repository and enter `go build` 158 | - then, run `./dogstatsd-local -port=8126` 159 | 160 | dogstatsd-local is now listening on port 8126. 161 | 162 | You'll need to then tell whatever is using this library at 163 | configure-time to send requests to localhost:8126. 164 | 165 | To make this work for Manage, the following was 166 | added to the ~/.zshrc file: 167 | 168 | `export COGNICIAN_STATSD_URI="localhost:8126"` 169 | 170 | ... then, when manage runs, it uses the configuration library, which 171 | ultimately reads this value from the system environment. 172 | 173 | Now, when manage is run in dev mode and instrumented code is hit, the results are 174 | available immediately in the terminal :) 175 | 176 | ### Conventions ### 177 | Of course, you can do whatever you want, but it's much more convenient 178 | for everyone if you include it as "dogstatsd" - so searching across 179 | codebases is easier ;) 180 | 181 | ## CHANGES 182 | 183 | *0.1.2* 184 | 185 | - Remove reflection warnings 186 | 187 | *0.1.1* 188 | 189 | - Metric reporting methods now catch all errors, print them to stderr and continue 190 | 191 | *0.1.0* 192 | 193 | - Initial release 194 | 195 | ## License 196 | 197 | Copyright © 2015 Cognician Software (Pty) Ltd 198 | 199 | Distributed under the Eclipse Public License, the same as Clojure. 200 | -------------------------------------------------------------------------------- /src/cognician/dogstatsd.clj: -------------------------------------------------------------------------------- 1 | (ns ^{:doc 2 | " (configure! \"localhost:8125\") 3 | 4 | Total value/rate: 5 | 6 | (increment! \"chat.request.count\" 1) 7 | 8 | In-the-moment value: 9 | 10 | (gauge! \"chat.ws.connections\" 17) 11 | 12 | Values distribution (mean, avg, max, percentiles): 13 | 14 | (histogram! \"chat.request.time\" 188.17) 15 | 16 | Counting unique values: 17 | 18 | (set! \"chat.user.email\" \"nikita@mailforspam.com\") 19 | 20 | Supported opts (third argument): 21 | 22 | { :tags => [String+] | { Keyword -> Any | Nil } 23 | :sample-rate => Double[0..1] } 24 | 25 | E.g. (increment! \"chat.request.count\" 1 26 | { :tags { :env \"production\", :chat nil } ;; => |#env:production,chat 27 | :tags [ \"env:production\" \"chat\" ] ;; => |#env:production,chat 28 | :sample-rate 0.5 } ;; Throttling 50%"} 29 | cognician.dogstatsd 30 | (:require 31 | [clojure.string :as str]) 32 | (:import 33 | [java.net InetSocketAddress DatagramSocket DatagramPacket])) 34 | 35 | 36 | (defonce *state (atom nil)) 37 | 38 | 39 | (defn configure! 40 | "Just pass StatsD server URI: 41 | 42 | (configure! \"localhost:8125\") 43 | (configure! \":8125\") 44 | (configure! \"localhost\") 45 | 46 | Pass system-wide tags to opts: 47 | 48 | (configure! \"localhost:8125\" {:tags {:env \"production\"}})" 49 | ([uri] (configure! uri {})) 50 | ([uri opts] 51 | (when-let [[_ host port] (and uri (re-matches #"([^:]*)(?:\:(\d+))?" uri))] 52 | (let [host (if (str/blank? host) "localhost" host) 53 | port (if (str/blank? port) 8125 port) 54 | port (if (string? port) (Integer/parseInt port) port) 55 | socket ^java.net.SocketAddress (DatagramSocket.) 56 | addr ^java.net.InetSocketAddress (InetSocketAddress. ^String host ^Long port)] 57 | (reset! *state (merge (select-keys opts [:tags]) 58 | {:socket socket 59 | :addr addr})))))) 60 | 61 | 62 | (defn- send! [^String payload] 63 | ;; (println "[ metrics ]" payload) 64 | (if-let [{:keys [socket addr]} @*state] 65 | (let [bytes (.getBytes payload "UTF-8")] 66 | (try 67 | (.send ^DatagramSocket socket 68 | (DatagramPacket. bytes (alength bytes) ^InetSocketAddress addr)) 69 | (catch Exception e 70 | (.printStackTrace e)))))) 71 | 72 | 73 | (defn- format-tags [& tag-colls] 74 | (->> tag-colls 75 | (mapcat (fn [tags] 76 | (cond->> tags 77 | (map? tags) (map (fn [[k v]] 78 | (if (nil? v) 79 | (name k) 80 | (str (name k) ":" v))))))) 81 | (str/join ","))) 82 | 83 | 84 | (defn- format-metric [metric type value tags sample-rate] 85 | (assert (re-matches #"[a-zA-Z][a-zA-Z0-9_.]*" metric) (str "Invalid metric name: " metric)) 86 | (assert (< (count metric) 200) (str "Metric name too long: " metric)) 87 | (str metric 88 | ":" value 89 | "|" type 90 | (when-not (== 1 sample-rate) 91 | (str "|@" sample-rate)) 92 | (let [global-tags (:tags @*state)] 93 | (when (or (not-empty tags) 94 | (not-empty global-tags)) 95 | (str "|#" (format-tags global-tags tags)))))) 96 | 97 | 98 | (defn- report-fn [type] 99 | (fn report! 100 | ([name value] (report! name value {})) 101 | ([name value opts] 102 | (let [tags (:tags opts []) 103 | sample-rate (:sample-rate opts 1)] 104 | (when (or (== sample-rate 1) 105 | (< (rand) sample-rate)) 106 | (send! (format-metric name type value tags sample-rate))))))) 107 | 108 | 109 | (def increment! (report-fn "c")) 110 | 111 | 112 | (def gauge! (report-fn "g")) 113 | 114 | 115 | (def histogram! (report-fn "h")) 116 | 117 | 118 | (defmacro measure! [metric opts & body] 119 | `(let [t0# (System/currentTimeMillis) 120 | res# (do ~@body)] 121 | (histogram! ~metric (- (System/currentTimeMillis) t0#) ~opts) 122 | res#)) 123 | 124 | 125 | (def set! (report-fn "s")) 126 | 127 | 128 | (defn- escape-event-string [s] 129 | (str/replace s "\n" "\\n")) 130 | 131 | 132 | (defn- format-event [title text opts] 133 | (let [title' (escape-event-string title) 134 | text' (escape-event-string text) 135 | {:keys [tags ^java.util.Date date-happened hostname aggregation-key 136 | priority source-type-name alert-type]} opts] 137 | (str "_e{" (count title') "," (count text') "}:" title' "|" text' 138 | (when date-happened 139 | (assert (instance? java.util.Date date-happened)) 140 | (str "|d:" (-> date-happened .getTime (/ 1000) long))) 141 | (when hostname 142 | (str "|h:" hostname)) 143 | (when aggregation-key 144 | (str "|k:" aggregation-key)) 145 | (when priority 146 | (assert (#{:normal :low} priority)) 147 | (str "|p:" (name priority))) 148 | (when source-type-name 149 | (str "|s:" source-type-name)) 150 | (when alert-type 151 | (assert (#{:error :warning :info :success} alert-type)) 152 | (str "|t:" (name alert-type))) 153 | (let [global-tags (:tags @*state)] 154 | (when (or (not-empty tags) 155 | (not-empty global-tags)) 156 | (str "|#" (format-tags global-tags tags))))))) 157 | 158 | 159 | (defn event! 160 | "title => String 161 | text => String 162 | opts => { :tags => [String+] | { Keyword -> Any | Nil } 163 | :date-happened => #inst 164 | :hostname => String 165 | :aggregation-key => String 166 | :priority => :normal | :low 167 | :source-type=name => String 168 | :alert-type => :error | :warning | :info | :success }" 169 | [title text opts] 170 | (let [payload (format-event title text opts)] 171 | (assert (< (count payload) (* 8 1024)) (str "Payload too big: " title text payload)) 172 | (send! payload))) 173 | --------------------------------------------------------------------------------