├── .gitignore ├── LICENSE ├── README.md ├── assets ├── gelf_message.png └── graylog_dashboard.png ├── graylog ├── contentpacks │ └── grok-patterns.json └── graylog.conf ├── pom.xml ├── sigar-libs.zip └── src └── main ├── java └── com │ └── objectpartners │ └── plummer │ └── graylog │ ├── Application.java │ ├── aspects │ ├── ErrorAspect.java │ └── QuoteAspect.java │ ├── config │ ├── SchedulingConfiguration.java │ ├── SwaggerConfiguration.java │ └── WebMvcConfiguration.java │ ├── controller │ └── QuotesController.java │ ├── data │ └── MongoInstance.java │ ├── domain │ ├── Exchange.java │ └── QuoteResource.java │ ├── graylog │ ├── ConcreteDashboardList.java │ ├── GelfMessage.java │ ├── GraylogInstance.java │ └── GraylogRestInterface.java │ ├── jobs │ └── ScheduledJobs.java │ └── service │ └── QuoteService.java └── resources ├── application.yml └── logback.xml /.gitignore: -------------------------------------------------------------------------------- 1 | /.gradle 2 | /.idea 3 | *.iml 4 | /build 5 | /data 6 | /sigar-libs 7 | /target 8 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2016 Mike Plummer 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # graylog-springboot 2 | Example of using Graylog with Spring Boot 3 | 4 | Check out my [blog post](https://objectpartners.com/2016/05/04/graylog-with-spring-boot-open-source-log-and-event-analysis/) describing the goals and process (and compromises) behind this code. 5 | 6 | ## Infrastructure 7 | Graylog uses MongoDB and ElasticSearch behind the scenes. To make this example as simple as possible I've configured the SpringBoot application to fire up embedded versions of both of these. I highly recommend not using this setup in a Production environment. 8 | 9 | ## URLs 10 | Graylog: http://localhost:9000 11 | 12 | Graylog Rest API: http://localhost:12900/api-browser 13 | 14 | App Rest API: http://localhost:8080/swagger-ui.html 15 | 16 | ## Instructions 17 | 1. Make sure you have Java and Maven installed 18 | 2. Run `mvn spring-boot:run` from project root 19 | 3. The first time you run this a LOT of stuff will download. This is a one-time occurrence. 20 | - Maven dependencies 21 | - MongoDB binaries to launch an embedded Mongo instance 22 | 4. You'll see a lot of logging - this is the Spring application initializing itself and booting up an ElasticSearch cluster, MongoDB database, and kicking off the Graylog instance 23 | 5. After about 30 seconds (depending on your machine's horsepower) you should start seeing logging like `Generated quote - VKL@59.0` 24 | 6. Open `http://localhost:9000` to open the Graylog GUI, login with username/password `admin/admin` 25 | 7. Explore! The application sets up some default GUI elements - start with the 'Stock Market' dashboard from the top menu. 26 | 8. When you're done just end the Maven process with `Ctrl-C` 27 | 28 | ## Cleanup 29 | ### MongoDB 30 | - To delete persisted Mongo data you can delete the 'data/mongodb/data' directory - the next time you launch the application a fresh database will get set up 31 | - Mongo executables get downloaded into the project directory - they can be deleted by removing the 'data/mongodb' directory. This will delete the persisted data as well. If you do this and try to re-launch the application it will re-download the executables. 32 | 33 | ### ElasticSearch 34 | If you delete the MongoDB data you'll need to fix the ElasticSearch indices or else they'll be out of sync. This can be done from the Graylog GUI or you can just delete `data/graylog/nodes` to rebuild them on next launch. 35 | 36 | ## Configuration 37 | If you're interested in tweaking the application's config you can look in three main places: 38 | 39 | 1. The code! Lots of setup and defaults are being set in the code since this is just an example and not really intended for extension/reuse. 40 | 2. application.yml: Spring Boot configs are set here (including ElasticSearch which is launched by Spring Boot) 41 | 3. graylog.conf: Config file for Graylog - lots of values in here you can mess with. Take a look at the Graylog docs for info on what they all do. 42 | 43 | ## Versions 44 | You'll need versions of Java and Maven available. I've tested with the specified versions. 45 | 46 | Java: 1.8.0_131 47 | 48 | Maven: 3.5.0 49 | 50 | Application has been verified on OSX 10.13.0. Your mileage may vary on other operating systems. 51 | 52 | ## License 53 | This code is provided under the terms of the MIT license: basically you're free to do whatever you want with it, but no guarantees are made to its validity, stability, or safety. All works referenced by or utilized by this project are the property of their respective copyright holders and retain licensing that may be more restrictive. 54 | -------------------------------------------------------------------------------- /assets/gelf_message.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mike-plummer/graylog-springboot/28d7cc71f3413b356cacf44b71eadfd9a3d44714/assets/gelf_message.png -------------------------------------------------------------------------------- /assets/graylog_dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mike-plummer/graylog-springboot/28d7cc71f3413b356cacf44b71eadfd9a3d44714/assets/graylog_dashboard.png -------------------------------------------------------------------------------- /graylog/contentpacks/grok-patterns.json: -------------------------------------------------------------------------------- 1 | { 2 | "name": "Core Grok Patterns", 3 | "description": "Core grok patterns", 4 | "category": "Grok", 5 | "grok_patterns": [ 6 | { 7 | "name": "USERNAME", 8 | "pattern": "[a-zA-Z0-9._-]+" 9 | }, 10 | { 11 | "name": "USER", 12 | "pattern": "%{USERNAME}" 13 | }, 14 | { 15 | "name": "EMAILLOCALPART", 16 | "pattern": "[a-zA-Z][a-zA-Z0-9_.+-=:]+" 17 | }, 18 | { 19 | "name": "EMAILADDRESS", 20 | "pattern": "%{EMAILLOCALPART}@%{HOSTNAME}" 21 | }, 22 | { 23 | "name": "HTTPDUSER", 24 | "pattern": "%{EMAILADDRESS}|%{USER}" 25 | }, 26 | { 27 | "name": "INT", 28 | "pattern": "(?:[+-]?(?:[0-9]+))" 29 | }, 30 | { 31 | "name": "BASE10NUM", 32 | "pattern": "(?[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))" 33 | }, 34 | { 35 | "name": "NUMBER", 36 | "pattern": "(?:%{BASE10NUM})" 37 | }, 38 | { 39 | "name": "BASE16NUM", 40 | "pattern": "(?(?\"(?>\\\\.|[^\\\\\"]+)+\"|\"\"|(?>'(?>\\\\.|[^\\\\']+)+')|''|(?>`(?>\\\\.|[^\\\\`]+)+`)|``))" 77 | }, 78 | { 79 | "name": "UUID", 80 | "pattern": "[A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12}" 81 | }, 82 | { 83 | "name": "MAC", 84 | "pattern": "(?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC})" 85 | }, 86 | { 87 | "name": "CISCOMAC", 88 | "pattern": "(?:(?:[A-Fa-f0-9]{4}\\.){2}[A-Fa-f0-9]{4})" 89 | }, 90 | { 91 | "name": "WINDOWSMAC", 92 | "pattern": "(?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2})" 93 | }, 94 | { 95 | "name": "COMMONMAC", 96 | "pattern": "(?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2})" 97 | }, 98 | { 99 | "name": "IPV6", 100 | "pattern": "((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)(\\.(25[0-5]|2[0-4]\\d|1\\d\\d|[1-9]?\\d)){3}))|:)))(%.+)?" 101 | }, 102 | { 103 | "name": "IPV4", 104 | "pattern": "(?[A-Za-z]+:|\\\\)(?:\\\\[^\\\\?*]*)+" 137 | }, 138 | { 139 | "name": "URIPROTO", 140 | "pattern": "[A-Za-z]+(\\+[A-Za-z+]+)?" 141 | }, 142 | { 143 | "name": "URIHOST", 144 | "pattern": "%{IPORHOST}(?::%{POSINT:port})?" 145 | }, 146 | { 147 | "name": "URIPATH", 148 | "pattern": "(?:/[A-Za-z0-9$.+!*'(){},~:;=@#%_\\-]*)+" 149 | }, 150 | { 151 | "name": "URIPARAM", 152 | "pattern": "\\?[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\\-\\[\\]<>]*" 153 | }, 154 | { 155 | "name": "URIPATHPARAM", 156 | "pattern": "%{URIPATH}(?:%{URIPARAM})?" 157 | }, 158 | { 159 | "name": "URI", 160 | "pattern": "%{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})?" 161 | }, 162 | { 163 | "name": "MONTH", 164 | "pattern": "\\b(?:Jan(?:uary|uar)?|Feb(?:ruary|ruar)?|M(?:a|ä)?r(?:ch|z)?|Apr(?:il)?|Ma(?:y|i)?|Jun(?:e|i)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|O(?:c|k)?t(?:ober)?|Nov(?:ember)?|De(?:c|z)(?:ember)?)\\b" 165 | }, 166 | { 167 | "name": "MONTHNUM", 168 | "pattern": "(?:0?[1-9]|1[0-2])" 169 | }, 170 | { 171 | "name": "MONTHNUM2", 172 | "pattern": "(?:0[1-9]|1[0-2])" 173 | }, 174 | { 175 | "name": "MONTHDAY", 176 | "pattern": "(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])" 177 | }, 178 | { 179 | "name": "DAY", 180 | "pattern": "(?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?)" 181 | }, 182 | { 183 | "name": "YEAR", 184 | "pattern": "(?>\\d\\d){1,2}" 185 | }, 186 | { 187 | "name": "HOUR", 188 | "pattern": "(?:2[0123]|[01]?[0-9])" 189 | }, 190 | { 191 | "name": "MINUTE", 192 | "pattern": "(?:[0-5][0-9])" 193 | }, 194 | { 195 | "name": "SECOND", 196 | "pattern": "(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)" 197 | }, 198 | { 199 | "name": "TIME", 200 | "pattern": "(?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9])" 201 | }, 202 | { 203 | "name": "DATE_US", 204 | "pattern": "%{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR}" 205 | }, 206 | { 207 | "name": "DATE_EU", 208 | "pattern": "%{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR}" 209 | }, 210 | { 211 | "name": "ISO8601_TIMEZONE", 212 | "pattern": "(?:Z|[+-]%{HOUR}(?::?%{MINUTE}))" 213 | }, 214 | { 215 | "name": "ISO8601_SECOND", 216 | "pattern": "(?:%{SECOND}|60)" 217 | }, 218 | { 219 | "name": "TIMESTAMP_ISO8601", 220 | "pattern": "%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?" 221 | }, 222 | { 223 | "name": "DATE", 224 | "pattern": "%{DATE_US}|%{DATE_EU}" 225 | }, 226 | { 227 | "name": "DATESTAMP", 228 | "pattern": "%{DATE}[- ]%{TIME}" 229 | }, 230 | { 231 | "name": "TZ", 232 | "pattern": "(?:[PMCE][SD]T|UTC)" 233 | }, 234 | { 235 | "name": "DATESTAMP_RFC822", 236 | "pattern": "%{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ}" 237 | }, 238 | { 239 | "name": "DATESTAMP_RFC2822", 240 | "pattern": "%{DAY}, %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} %{ISO8601_TIMEZONE}" 241 | }, 242 | { 243 | "name": "DATESTAMP_OTHER", 244 | "pattern": "%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR}" 245 | }, 246 | { 247 | "name": "DATESTAMP_EVENTLOG", 248 | "pattern": "%{YEAR}%{MONTHNUM2}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND}" 249 | }, 250 | { 251 | "name": "HTTPDERROR_DATE", 252 | "pattern": "%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}" 253 | }, 254 | { 255 | "name": "SYSLOGTIMESTAMP", 256 | "pattern": "%{MONTH} +%{MONTHDAY} %{TIME}" 257 | }, 258 | { 259 | "name": "PROG", 260 | "pattern": "[\\x21-\\x5a\\x5c\\x5e-\\x7e]+" 261 | }, 262 | { 263 | "name": "SYSLOGPROG", 264 | "pattern": "%{PROG:program}(?:\\[%{POSINT:pid}\\])?" 265 | }, 266 | { 267 | "name": "SYSLOGHOST", 268 | "pattern": "%{IPORHOST}" 269 | }, 270 | { 271 | "name": "SYSLOGFACILITY", 272 | "pattern": "<%{NONNEGINT:facility}.%{NONNEGINT:priority}>" 273 | }, 274 | { 275 | "name": "HTTPDATE", 276 | "pattern": "%{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT}" 277 | }, 278 | { 279 | "name": "QS", 280 | "pattern": "%{QUOTEDSTRING}" 281 | }, 282 | { 283 | "name": "SYSLOGBASE", 284 | "pattern": "%{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}:" 285 | }, 286 | { 287 | "name": "COMMONAPACHELOG", 288 | "pattern": "%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \\[%{HTTPDATE:timestamp}\\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-)" 289 | }, 290 | { 291 | "name": "COMBINEDAPACHELOG", 292 | "pattern": "%{COMMONAPACHELOG} %{QS:referrer} %{QS:agent}" 293 | }, 294 | { 295 | "name": "HTTPD20_ERRORLOG", 296 | "pattern": "\\[%{HTTPDERROR_DATE:timestamp}\\] \\[%{LOGLEVEL:loglevel}\\] (?:\\[client %{IPORHOST:clientip}\\] ){0,1}%{GREEDYDATA:errormsg}" 297 | }, 298 | { 299 | "name": "HTTPD24_ERRORLOG", 300 | "pattern": "\\[%{HTTPDERROR_DATE:timestamp}\\] \\[%{WORD:module}:%{LOGLEVEL:loglevel}\\] \\[pid %{POSINT:pid}:tid %{NUMBER:tid}\\]( \\(%{POSINT:proxy_errorcode}\\)%{DATA:proxy_errormessage}:)?( \\[client %{IPORHOST:client}:%{POSINT:clientport}\\])? %{DATA:errorcode}: %{GREEDYDATA:message}" 301 | }, 302 | { 303 | "name": "HTTPD_ERRORLOG", 304 | "pattern": "%{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}" 305 | }, 306 | { 307 | "name": "LOGLEVEL", 308 | "pattern": "([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)" 309 | } 310 | ] 311 | } 312 | -------------------------------------------------------------------------------- /graylog/graylog.conf: -------------------------------------------------------------------------------- 1 | is_master = true 2 | node_id_file = data/node-id 3 | password_secret = replacethiswithyourownsecret! 4 | root_password_sha2 = 8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918 5 | plugin_dir = data/plugin 6 | rest_listen_uri = http://0.0.0.0:12900/ 7 | rest_transport_uri = http://localhost:12900/ 8 | web_listen_uri = http://0.0.0.0:9000/ 9 | web_enable_cors = true 10 | rotation_strategy = count 11 | elasticsearch_max_docs_per_index = 20000000 12 | elasticsearch_max_number_of_indices = 20 13 | retention_strategy = delete 14 | elasticsearch_shards = 4 15 | elasticsearch_replicas = 0 16 | elasticsearch_index_prefix = graylog 17 | allow_leading_wildcard_searches = true 18 | allow_highlighting = true 19 | elasticsearch_cluster_name = graylog 20 | # elasticsearch_transport_tcp_port = 9350 21 | elasticsearch_http_enabled = false 22 | # elasticsearch_discovery_zen_ping_multicast_enabled = true 23 | elasticsearch_discovery_zen_ping_unicast_hosts = localhost:9300 24 | elasticsearch_network_host = 0.0.0.0 25 | elasticsearch_analyzer = standard 26 | output_batch_size = 500 27 | output_flush_interval = 1 28 | output_fault_count_threshold = 5 29 | output_fault_penalty_seconds = 30 30 | processbuffer_processors = 5 31 | outputbuffer_processors = 3 32 | processor_wait_strategy = blocking 33 | ring_size = 65536 34 | inputbuffer_ring_size = 65536 35 | inputbuffer_processors = 2 36 | inputbuffer_wait_strategy = blocking 37 | message_journal_enabled = true 38 | message_journal_dir = data/journal 39 | lb_recognition_period_seconds = 3 40 | mongodb_uri = mongodb://localhost/graylog 41 | mongodb_max_connections = 100 42 | mongodb_threads_allowed_to_block_multiplier = 5 43 | content_packs_loader_enabled = true 44 | content_packs_dir = graylog/contentpacks 45 | content_packs_auto_load = grok-patterns.json -------------------------------------------------------------------------------- /pom.xml: -------------------------------------------------------------------------------- 1 | 2 | 4 | 4.0.0 5 | 6 | com.objectpartners.plummer 7 | graylog-springboot 8 | 0.0.1 9 | jar 10 | 11 | graylog-springboot 12 | Example of using GrayLog with Spring Boot 13 | https://github.com/mike-plummer/graylog-springboot 14 | 15 | 16 | 3.6 17 | 2.0.0 18 | 2.3.1 19 | 21.0 20 | 1 21 | 1.8 22 | 1.1.11 23 | 1.16.18 24 | 1.5.7.RELEASE 25 | 2.7.0 26 | 1.2 27 | 28 | 29 | 30 | org.springframework.boot 31 | spring-boot-starter-parent 32 | 1.5.7.RELEASE 33 | 34 | 35 | 36 | 37 | 38 | org.apache.maven.plugins 39 | maven-dependency-plugin 40 | 2.10 41 | 42 | 43 | org.springframework.boot 44 | spring-boot-maven-plugin 45 | 1.5.7.RELEASE 46 | 47 | 48 | 49 | repackage 50 | 51 | 52 | 53 | 54 | 55 | org.codehaus.mojo 56 | truezip-maven-plugin 57 | ${truezip.version} 58 | 59 | 60 | copy-package 61 | 62 | copy 63 | 64 | compile 65 | 66 | true 67 | 68 | sigar-libs.zip 69 | sigar-libs 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79 | 80 | org.springframework.boot 81 | spring-boot-starter-aop 82 | 83 | 84 | org.springframework.boot 85 | spring-boot-starter-data-elasticsearch 86 | 87 | 88 | org.springframework.boot 89 | spring-boot-starter-web 90 | 91 | 92 | io.springfox 93 | springfox-swagger-ui 94 | ${springfox.version} 95 | 96 | 97 | io.springfox 98 | springfox-swagger2 99 | ${springfox.version} 100 | 101 | 102 | javax.inject 103 | javax.inject 104 | ${javax-inject.version} 105 | 106 | 107 | org.projectlombok 108 | lombok 109 | ${lombok.version} 110 | 111 | 112 | de.flapdoodle.embed 113 | de.flapdoodle.embed.mongo 114 | ${embedded-mongo.version} 115 | 116 | 117 | org.apache.commons 118 | commons-lang3 119 | ${commons-lang3.version} 120 | 121 | 122 | org.graylog2 123 | graylog2-server 124 | ${graylog.version} 125 | 126 | 127 | 128 | org.apache.logging.log4j 129 | log4j-slf4j-impl 130 | 131 | 132 | org.apache.logging.log4j 133 | log4j-slf4j-impl 134 | 135 | 136 | org.slf4j 137 | log4j-over-slf4j 138 | 139 | 140 | 141 | 142 | com.fasterxml.jackson.datatype 143 | jackson-datatype-joda 144 | 145 | 146 | com.github.pukkaone 147 | logback-gelf 148 | ${logback-gelf.version} 149 | runtime 150 | 151 | 152 | org.springframework.boot 153 | spring-boot-starter-test 154 | test 155 | 156 | 157 | com.google.guava 158 | guava 159 | ${guava.version} 160 | 161 | 162 | 163 | -------------------------------------------------------------------------------- /sigar-libs.zip: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mike-plummer/graylog-springboot/28d7cc71f3413b356cacf44b71eadfd9a3d44714/sigar-libs.zip -------------------------------------------------------------------------------- /src/main/java/com/objectpartners/plummer/graylog/Application.java: -------------------------------------------------------------------------------- 1 | package com.objectpartners.plummer.graylog; 2 | 3 | import com.objectpartners.plummer.graylog.config.*; 4 | import org.slf4j.Logger; 5 | import org.slf4j.LoggerFactory; 6 | import org.springframework.boot.SpringApplication; 7 | import org.springframework.boot.autoconfigure.EnableAutoConfiguration; 8 | import org.springframework.boot.autoconfigure.SpringBootApplication; 9 | import org.springframework.boot.autoconfigure.elasticsearch.jest.JestAutoConfiguration; 10 | import org.springframework.boot.autoconfigure.mongo.embedded.EmbeddedMongoAutoConfiguration; 11 | import org.springframework.context.ConfigurableApplicationContext; 12 | import org.springframework.context.annotation.ComponentScan; 13 | import org.springframework.context.annotation.Import; 14 | 15 | @SpringBootApplication 16 | @EnableAutoConfiguration(exclude = { 17 | EmbeddedMongoAutoConfiguration.class, 18 | JestAutoConfiguration.class 19 | }) 20 | @Import({ 21 | SchedulingConfiguration.class, 22 | SwaggerConfiguration.class, 23 | WebMvcConfiguration.class 24 | }) 25 | @ComponentScan("com.objectpartners.plummer.graylog") 26 | public class Application { 27 | private static final Logger LOGGER = LoggerFactory.getLogger(Application.class); 28 | 29 | public static void main(String[] args) throws Exception { 30 | ConfigurableApplicationContext application = SpringApplication.run(Application.class, args); 31 | LOGGER.info("Application started, registering shutdown hook..."); 32 | application.registerShutdownHook(); 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /src/main/java/com/objectpartners/plummer/graylog/aspects/ErrorAspect.java: -------------------------------------------------------------------------------- 1 | package com.objectpartners.plummer.graylog.aspects; 2 | 3 | import com.objectpartners.plummer.graylog.graylog.GelfMessage; 4 | import com.objectpartners.plummer.graylog.graylog.GraylogRestInterface; 5 | import org.aspectj.lang.annotation.AfterThrowing; 6 | import org.aspectj.lang.annotation.Aspect; 7 | 8 | import javax.inject.Inject; 9 | import javax.inject.Named; 10 | import java.io.IOException; 11 | import java.io.PrintWriter; 12 | import java.io.StringWriter; 13 | 14 | @Aspect 15 | @Named 16 | public class ErrorAspect { 17 | 18 | @Inject 19 | protected GraylogRestInterface graylog; 20 | 21 | @AfterThrowing(pointcut = "execution(* com.objectpartners.plummer.graylog.jobs..*(..))", throwing = "e") 22 | public void errorThrown(Throwable e) throws IOException { 23 | GelfMessage message = new GelfMessage(); 24 | message.setShortMessage("Error"); 25 | message.setFullMessage(e.getMessage()); 26 | 27 | try (StringWriter stringWriter = new StringWriter(); 28 | PrintWriter printWriter = new PrintWriter(stringWriter)) { 29 | e.printStackTrace(printWriter); 30 | message.getAdditionalProperties().put("stacktrace", stringWriter.toString()); 31 | } 32 | 33 | graylog.logEvent(message); 34 | } 35 | } 36 | -------------------------------------------------------------------------------- /src/main/java/com/objectpartners/plummer/graylog/aspects/QuoteAspect.java: -------------------------------------------------------------------------------- 1 | package com.objectpartners.plummer.graylog.aspects; 2 | 3 | import com.fasterxml.jackson.databind.ObjectMapper; 4 | import com.google.common.base.Stopwatch; 5 | import com.objectpartners.plummer.graylog.domain.QuoteResource; 6 | import com.objectpartners.plummer.graylog.graylog.GelfMessage; 7 | import com.objectpartners.plummer.graylog.graylog.GraylogRestInterface; 8 | import org.aspectj.lang.ProceedingJoinPoint; 9 | import org.aspectj.lang.annotation.Around; 10 | import org.aspectj.lang.annotation.Aspect; 11 | import org.slf4j.Logger; 12 | import org.slf4j.LoggerFactory; 13 | 14 | import javax.inject.Inject; 15 | import javax.inject.Named; 16 | import java.util.concurrent.TimeUnit; 17 | 18 | @Aspect 19 | @Named 20 | public class QuoteAspect { 21 | 22 | private static final Logger LOGGER = LoggerFactory.getLogger(QuoteAspect.class); 23 | 24 | @Inject 25 | protected ObjectMapper mapper; 26 | 27 | @Inject 28 | protected GraylogRestInterface graylog; 29 | 30 | @Around("execution(* com.objectpartners.plummer.graylog.service.QuoteService.*(..))") 31 | public QuoteResource errorThrown(ProceedingJoinPoint joinPoint) throws Throwable { 32 | Stopwatch timer = Stopwatch.createStarted(); 33 | 34 | QuoteResource quote = (QuoteResource) joinPoint.proceed(); 35 | 36 | // Log message via Graylog UDP Logback Appender 37 | LOGGER.info("Generated quote - {}@{}", quote.getSymbol(), quote.getPrice()); 38 | 39 | // Log message via Graylog HTTP Input 40 | GelfMessage message = new GelfMessage(); 41 | message.setShortMessage(String.format("Quote %s@%2.2f", quote.getSymbol(), quote.getPrice())); 42 | message.setFullMessage(mapper.writeValueAsString(quote)); 43 | message.getAdditionalProperties().put("elapsed_time", timer.stop().elapsed(TimeUnit.MICROSECONDS)); 44 | graylog.logEvent(message); 45 | 46 | return quote; 47 | } 48 | } 49 | -------------------------------------------------------------------------------- /src/main/java/com/objectpartners/plummer/graylog/config/SchedulingConfiguration.java: -------------------------------------------------------------------------------- 1 | package com.objectpartners.plummer.graylog.config; 2 | 3 | import org.springframework.context.annotation.Bean; 4 | import org.springframework.context.annotation.Configuration; 5 | import org.springframework.scheduling.annotation.EnableScheduling; 6 | import org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler; 7 | 8 | @Configuration 9 | @EnableScheduling 10 | public class SchedulingConfiguration { 11 | 12 | @Bean 13 | public ThreadPoolTaskScheduler taskScheduler() { 14 | return new ThreadPoolTaskScheduler(); 15 | } 16 | } 17 | -------------------------------------------------------------------------------- /src/main/java/com/objectpartners/plummer/graylog/config/SwaggerConfiguration.java: -------------------------------------------------------------------------------- 1 | package com.objectpartners.plummer.graylog.config; 2 | 3 | import com.google.common.collect.Sets; 4 | import org.springframework.beans.factory.annotation.Value; 5 | import org.springframework.context.annotation.Bean; 6 | import org.springframework.context.annotation.Configuration; 7 | import org.springframework.http.MediaType; 8 | import springfox.documentation.builders.ApiInfoBuilder; 9 | import springfox.documentation.service.ApiInfo; 10 | import springfox.documentation.service.Contact; 11 | import springfox.documentation.spi.DocumentationType; 12 | import springfox.documentation.spring.web.plugins.Docket; 13 | import springfox.documentation.swagger2.annotations.EnableSwagger2; 14 | 15 | import java.util.Set; 16 | 17 | import static springfox.documentation.builders.RequestHandlerSelectors.basePackage; 18 | 19 | @EnableSwagger2 20 | @Configuration 21 | public class SwaggerConfiguration { 22 | 23 | @Value("${info.build.version}") 24 | private String buildVersion; 25 | 26 | @Value("${info.build.description}") 27 | private String projectDescription; 28 | 29 | @Value("${info.build.name}") 30 | private String projectName; 31 | 32 | @Bean 33 | public Docket stockmarketApi() { 34 | return new Docket(DocumentationType.SWAGGER_2) 35 | .consumes(apiContentTypes()) 36 | .produces(apiContentTypes()) 37 | .groupName("stockmarket") 38 | .apiInfo(apiInfo()) 39 | .select() 40 | .apis(basePackage("com.objectpartners.plummer.graylog")) 41 | .build(); 42 | } 43 | 44 | private ApiInfo apiInfo() { 45 | return new ApiInfoBuilder() 46 | .title(projectName) 47 | .description(projectDescription) 48 | .contact(new Contact("Mike Plummer", "https://mike-plummer.github.io", "mike.plummer@objectpartners.com")) 49 | .license("MIT") 50 | .version(buildVersion) 51 | .build(); 52 | } 53 | 54 | private Set apiContentTypes() { 55 | return Sets.newHashSet(MediaType.APPLICATION_JSON_VALUE); 56 | } 57 | } 58 | -------------------------------------------------------------------------------- /src/main/java/com/objectpartners/plummer/graylog/config/WebMvcConfiguration.java: -------------------------------------------------------------------------------- 1 | package com.objectpartners.plummer.graylog.config; 2 | 3 | import com.fasterxml.jackson.datatype.joda.JodaModule; 4 | import com.fasterxml.jackson.datatype.joda.cfg.JacksonJodaDateFormat; 5 | import com.fasterxml.jackson.datatype.joda.ser.DateTimeSerializer; 6 | import org.joda.time.DateTime; 7 | import org.springframework.context.annotation.Bean; 8 | import org.springframework.context.annotation.Configuration; 9 | import org.springframework.format.annotation.DateTimeFormat; 10 | import org.springframework.format.datetime.joda.DateTimeFormatterFactory; 11 | import org.springframework.http.MediaType; 12 | import org.springframework.web.servlet.config.annotation.ContentNegotiationConfigurer; 13 | import org.springframework.web.servlet.config.annotation.EnableWebMvc; 14 | import org.springframework.web.servlet.config.annotation.ResourceHandlerRegistry; 15 | import org.springframework.web.servlet.config.annotation.WebMvcConfigurerAdapter; 16 | 17 | @EnableWebMvc 18 | @Configuration 19 | public class WebMvcConfiguration extends WebMvcConfigurerAdapter { 20 | @Override 21 | public void addResourceHandlers(ResourceHandlerRegistry registry) { 22 | registry.addResourceHandler("swagger-ui.html") 23 | .addResourceLocations("classpath:/META-INF/resources/"); 24 | 25 | registry.addResourceHandler("/webjars/**") 26 | .addResourceLocations("classpath:/META-INF/resources/webjars/"); 27 | } 28 | 29 | @Override 30 | public void configureContentNegotiation(ContentNegotiationConfigurer configurer) { 31 | configurer.defaultContentType(MediaType.APPLICATION_JSON); 32 | } 33 | 34 | @Bean 35 | public JodaModule jacksonJodaModule() { 36 | JodaModule module = new JodaModule(); 37 | DateTimeFormatterFactory formatterFactory = new DateTimeFormatterFactory(); 38 | formatterFactory.setIso(DateTimeFormat.ISO.DATE); 39 | module.addSerializer(DateTime.class, new DateTimeSerializer( 40 | new JacksonJodaDateFormat(formatterFactory.createDateTimeFormatter().withZoneUTC()))); 41 | return module; 42 | } 43 | } 44 | -------------------------------------------------------------------------------- /src/main/java/com/objectpartners/plummer/graylog/controller/QuotesController.java: -------------------------------------------------------------------------------- 1 | package com.objectpartners.plummer.graylog.controller; 2 | 3 | import com.objectpartners.plummer.graylog.domain.QuoteResource; 4 | import com.objectpartners.plummer.graylog.service.QuoteService; 5 | import io.swagger.annotations.Api; 6 | import io.swagger.annotations.ApiOperation; 7 | import io.swagger.annotations.ApiResponse; 8 | import io.swagger.annotations.ApiResponses; 9 | import org.springframework.web.bind.annotation.PathVariable; 10 | import org.springframework.web.bind.annotation.RequestMapping; 11 | import org.springframework.web.bind.annotation.RequestMethod; 12 | import org.springframework.web.bind.annotation.RestController; 13 | 14 | import javax.inject.Inject; 15 | 16 | import static com.objectpartners.plummer.graylog.controller.QuotesController.RESOURCE_ROOT_URL; 17 | 18 | @RestController 19 | @RequestMapping(RESOURCE_ROOT_URL) 20 | @Api(value = "stocks", tags = "Stocks") 21 | public class QuotesController { 22 | 23 | public static final String RESOURCE_ROOT_URL = "/quotes"; 24 | 25 | @Inject 26 | protected QuoteService quoteService; 27 | 28 | @RequestMapping(method = RequestMethod.GET, value = "/{symbol}") 29 | @ApiOperation(value = "Get Quote") 30 | @ApiResponses(value = { 31 | @ApiResponse(code = 200, message = "Success"), 32 | @ApiResponse(code = 401, message = "Unauthorized"), 33 | @ApiResponse(code = 403, message = "Forbidden"), 34 | @ApiResponse(code = 404, message = "Not Found"), 35 | @ApiResponse(code = 500, message = "Failure") 36 | }) 37 | public QuoteResource getQuote(@PathVariable("symbol") String symbol) { 38 | return quoteService.quote(symbol); 39 | } 40 | } -------------------------------------------------------------------------------- /src/main/java/com/objectpartners/plummer/graylog/data/MongoInstance.java: -------------------------------------------------------------------------------- 1 | package com.objectpartners.plummer.graylog.data; 2 | 3 | import de.flapdoodle.embed.mongo.Command; 4 | import de.flapdoodle.embed.mongo.MongodExecutable; 5 | import de.flapdoodle.embed.mongo.MongodProcess; 6 | import de.flapdoodle.embed.mongo.MongodStarter; 7 | import de.flapdoodle.embed.mongo.config.*; 8 | import de.flapdoodle.embed.mongo.distribution.Version; 9 | import de.flapdoodle.embed.process.config.IRuntimeConfig; 10 | import de.flapdoodle.embed.process.config.store.IDownloadConfig; 11 | import de.flapdoodle.embed.process.io.directories.FixedPath; 12 | import de.flapdoodle.embed.process.io.directories.IDirectory; 13 | import de.flapdoodle.embed.process.runtime.Network; 14 | import org.springframework.beans.factory.DisposableBean; 15 | import org.springframework.boot.autoconfigure.mongo.MongoProperties; 16 | import org.springframework.core.Ordered; 17 | import org.springframework.core.annotation.Order; 18 | 19 | import javax.inject.Named; 20 | import javax.inject.Singleton; 21 | import java.io.IOException; 22 | 23 | @Named 24 | @Singleton 25 | @Order(Ordered.HIGHEST_PRECEDENCE) 26 | public class MongoInstance implements DisposableBean { 27 | 28 | private static final IDirectory ARTIFACT_STORE_PATH = new FixedPath("./data/mongodb"); 29 | private static final IDirectory EXTRACTED_STORE_PATH = new FixedPath("./data/mongodb/extracted"); 30 | 31 | private MongodExecutable mongo; 32 | private MongodProcess process; 33 | 34 | public MongoInstance() throws IOException, InterruptedException { 35 | 36 | // Download Mongo artifacts into the project directory to ease cleanup 37 | IDownloadConfig downloadConfig = new DownloadConfigBuilder() 38 | .defaultsForCommand(Command.MongoD) 39 | .artifactStorePath(ARTIFACT_STORE_PATH) 40 | .build(); 41 | 42 | // Extract Mongo artifacts into the project directory to ease cleanup 43 | IRuntimeConfig runtimeConfig = new RuntimeConfigBuilder() 44 | .defaults(Command.MongoD) 45 | .artifactStore(new ExtractedArtifactStoreBuilder() 46 | .defaults(Command.MongoD) 47 | .download(downloadConfig) 48 | .extractDir(EXTRACTED_STORE_PATH) 49 | ) 50 | .build(); 51 | 52 | // Store Mongo data into the project directory to ease cleanup 53 | Storage replication = new Storage("./data/mongodb/data", null, 0); 54 | 55 | MongodStarter starter = MongodStarter.getInstance(runtimeConfig); 56 | 57 | IMongodConfig mongodConfig = new MongodConfigBuilder() 58 | .version(Version.Main.PRODUCTION) 59 | .cmdOptions(new MongoCmdOptionsBuilder() 60 | .useNoJournal(false) 61 | .useSmallFiles(true) 62 | .build()) 63 | .net(new Net(MongoProperties.DEFAULT_PORT, Network.localhostIsIPv6())) 64 | .replication(replication) 65 | .build(); 66 | 67 | mongo = starter.prepare(mongodConfig); 68 | process = mongo.start(); 69 | } 70 | 71 | @Override 72 | public void destroy() throws Exception { 73 | if (process != null) { 74 | process.stop(); 75 | } 76 | if (mongo != null) { 77 | mongo.stop(); 78 | } 79 | } 80 | } 81 | -------------------------------------------------------------------------------- /src/main/java/com/objectpartners/plummer/graylog/domain/Exchange.java: -------------------------------------------------------------------------------- 1 | package com.objectpartners.plummer.graylog.domain; 2 | 3 | public enum Exchange { 4 | NYSE, 5 | NASDAQ 6 | } 7 | -------------------------------------------------------------------------------- /src/main/java/com/objectpartners/plummer/graylog/domain/QuoteResource.java: -------------------------------------------------------------------------------- 1 | package com.objectpartners.plummer.graylog.domain; 2 | 3 | import lombok.AllArgsConstructor; 4 | import lombok.Data; 5 | import lombok.EqualsAndHashCode; 6 | import lombok.NoArgsConstructor; 7 | 8 | @Data 9 | @NoArgsConstructor 10 | @AllArgsConstructor 11 | @EqualsAndHashCode(callSuper = false) 12 | public class QuoteResource { 13 | private String symbol; 14 | private Double price; 15 | private Exchange exchange; 16 | } 17 | -------------------------------------------------------------------------------- /src/main/java/com/objectpartners/plummer/graylog/graylog/ConcreteDashboardList.java: -------------------------------------------------------------------------------- 1 | package com.objectpartners.plummer.graylog.graylog; 2 | 3 | import lombok.AllArgsConstructor; 4 | import lombok.EqualsAndHashCode; 5 | import lombok.NoArgsConstructor; 6 | import org.graylog2.rest.models.dashboards.responses.DashboardList; 7 | 8 | import java.util.ArrayList; 9 | import java.util.List; 10 | import java.util.Map; 11 | 12 | /** 13 | * Graylog rest classes don't supply a DashboardList implementation that can be easily used with Jackson deserialization. 14 | * Create one here to make it easier to pull and deserialize JSON from Graylog rest services. 15 | */ 16 | @NoArgsConstructor 17 | @AllArgsConstructor 18 | @EqualsAndHashCode(callSuper = true) 19 | public class ConcreteDashboardList extends DashboardList { 20 | private int total; 21 | private List> dashboards; 22 | 23 | @Override 24 | public int total() { 25 | return total; 26 | } 27 | 28 | @Override 29 | public List> dashboards() { 30 | if (dashboards == null) { 31 | dashboards = new ArrayList<>(0); 32 | } 33 | return dashboards; 34 | } 35 | } 36 | -------------------------------------------------------------------------------- /src/main/java/com/objectpartners/plummer/graylog/graylog/GelfMessage.java: -------------------------------------------------------------------------------- 1 | package com.objectpartners.plummer.graylog.graylog; 2 | 3 | import com.fasterxml.jackson.annotation.JsonAnyGetter; 4 | import com.fasterxml.jackson.annotation.JsonIgnore; 5 | import com.fasterxml.jackson.annotation.JsonProperty; 6 | import lombok.Data; 7 | 8 | import javax.validation.constraints.Min; 9 | import javax.validation.constraints.NotNull; 10 | import java.net.InetAddress; 11 | import java.net.UnknownHostException; 12 | import java.util.Collections; 13 | import java.util.HashMap; 14 | import java.util.Map; 15 | import java.util.concurrent.TimeUnit; 16 | import java.util.stream.Collectors; 17 | 18 | /** 19 | * Class that will serialize into JSON compliant the GELF format. This version is a bit easier to use than the built-in 20 | * versions in Graylog. 21 | */ 22 | @Data 23 | public class GelfMessage { 24 | 25 | private final String version = "1.1"; 26 | 27 | @NotNull 28 | private String host; 29 | 30 | @NotNull 31 | @JsonProperty("short_message") 32 | private String shortMessage; 33 | 34 | @JsonProperty("full_message") 35 | private String fullMessage; 36 | 37 | private Double timestamp; 38 | 39 | @Min(1) 40 | private Integer level = 1; 41 | 42 | @JsonIgnore 43 | private Map additionalProperties; 44 | 45 | public GelfMessage() { 46 | try { 47 | setHost(InetAddress.getLocalHost().getHostName()); 48 | } catch (UnknownHostException e) { 49 | setHost("localhost"); 50 | } 51 | setTimestamp(System.currentTimeMillis()); 52 | } 53 | 54 | public Map getAdditionalProperties() { 55 | if (additionalProperties == null) { 56 | additionalProperties = new HashMap<>(0); 57 | } 58 | return additionalProperties; 59 | } 60 | 61 | @JsonAnyGetter 62 | public Map getGelfAdditionalProperties() { 63 | if (additionalProperties == null) { 64 | return Collections.emptyMap(); 65 | } 66 | return additionalProperties.entrySet().stream() 67 | .collect(Collectors.toMap(entry -> { 68 | if (entry.getKey().startsWith("_")) { 69 | return entry.getKey(); 70 | } 71 | return "_" + entry.getKey(); 72 | }, Map.Entry::getValue)); 73 | } 74 | 75 | public void setTimestamp(Long millisSinceEpoch) { 76 | Long seconds = TimeUnit.MILLISECONDS.toSeconds(millisSinceEpoch); 77 | Double millis = (millisSinceEpoch - seconds * 1000) / 1000.0; 78 | timestamp = seconds + millis; 79 | } 80 | } 81 | -------------------------------------------------------------------------------- /src/main/java/com/objectpartners/plummer/graylog/graylog/GraylogInstance.java: -------------------------------------------------------------------------------- 1 | package com.objectpartners.plummer.graylog.graylog; 2 | 3 | import com.github.rvesse.airline.Cli; 4 | import com.github.rvesse.airline.builder.CliBuilder; 5 | import com.objectpartners.plummer.graylog.data.MongoInstance; 6 | import org.graylog2.bootstrap.CliCommand; 7 | import org.graylog2.bootstrap.CliCommandsProvider; 8 | import org.graylog2.bootstrap.commands.CliCommandHelp; 9 | import org.graylog2.bootstrap.commands.ShowVersion; 10 | import org.graylog2.inputs.gelf.http.GELFHttpInput; 11 | import org.graylog2.inputs.gelf.udp.GELFUDPInput; 12 | import org.graylog2.plugin.inputs.MessageInput; 13 | import org.graylog2.rest.models.dashboards.requests.AddWidgetRequest; 14 | import org.graylog2.rest.models.dashboards.requests.CreateDashboardRequest; 15 | import org.graylog2.rest.models.system.inputs.extractors.requests.CreateExtractorRequest; 16 | import org.graylog2.rest.models.system.inputs.requests.InputCreateRequest; 17 | import org.slf4j.Logger; 18 | import org.slf4j.LoggerFactory; 19 | import org.springframework.beans.factory.annotation.Value; 20 | import org.springframework.core.DecoratingClassLoader; 21 | import org.springframework.scheduling.TaskScheduler; 22 | import org.springframework.util.ReflectionUtils; 23 | 24 | import javax.annotation.PostConstruct; 25 | import javax.inject.Inject; 26 | import javax.inject.Named; 27 | import javax.inject.Singleton; 28 | import java.io.IOException; 29 | import java.lang.reflect.Field; 30 | import java.net.URL; 31 | import java.util.*; 32 | 33 | @Named 34 | @Singleton 35 | public class GraylogInstance { 36 | 37 | private static final Logger LOGGER = LoggerFactory.getLogger(GraylogInstance.class); 38 | 39 | // Hint to Spring that we need to run this config AFTER Mongo is started 40 | @Inject 41 | private MongoInstance mongo; 42 | 43 | @Inject 44 | private GraylogRestInterface graylogRestInterface; 45 | 46 | @Value("${graylog.configFile}") 47 | private String graylogConfigFile; 48 | 49 | @Inject 50 | protected TaskScheduler scheduler; 51 | 52 | @PostConstruct 53 | public void init() throws IOException, InterruptedException { 54 | System.setProperty("java.library.path", "./sigar-libs"); 55 | final CliBuilder builder = Cli.builder("graylog") 56 | .withDefaultCommand(CliCommandHelp.class) 57 | .withCommands(CliCommandHelp.class, ShowVersion.class); 58 | 59 | ServiceLoader commandsProviders = ServiceLoader.load(CliCommandsProvider.class); 60 | 61 | for (Object commandsProvider : commandsProviders) { 62 | CliCommandsProvider command = (CliCommandsProvider) commandsProvider; 63 | command.addTopLevelCommandsOrGroups(builder); 64 | } 65 | 66 | final Cli cli = builder.build(); 67 | final Runnable command = cli.parse("server", "-f", graylogConfigFile, "-np"); 68 | final Thread graylogThread = new Thread(command); 69 | 70 | /* 71 | This is a nasty workaround to get Graylog's Swagger instance working when embedded within SpringBoot. 72 | DocumentationBrowserResource uses the SystemClassLoader (which it 100% should not be doing) which means 73 | it fails to load resources when run in an environment like SpringBoot 74 | */ 75 | try { 76 | Field sclField = ReflectionUtils.findField(ClassLoader.class, "scl"); 77 | ReflectionUtils.makeAccessible(sclField); 78 | ReflectionUtils.setField(sclField, null, new DecoratingClassLoader() { 79 | @Override 80 | public URL getResource(String name) { 81 | if (name.startsWith("swagger/")) { 82 | return Thread.currentThread().getContextClassLoader().getResource(name); 83 | } 84 | return super.getResource(name); 85 | } 86 | }); 87 | } catch(Exception e) { 88 | LOGGER.error("Failed to replace SystemClassLoader, this means Graylog's Swagger won't work.", e); 89 | } 90 | 91 | graylogThread.start(); 92 | 93 | LOGGER.info("Graylog started"); 94 | 95 | configureGraylog(); 96 | } 97 | 98 | private void configureGraylog() { 99 | try { 100 | graylogRestInterface.inputExists("test"); 101 | LOGGER.info("Graylog rest interface ready to go!"); 102 | 103 | setupUdpInput(); 104 | setupHttpInput(); 105 | setupDashboard(); 106 | } catch (Exception e) { 107 | LOGGER.warn("Graylog rest interface not ready yet, rescheduling..."); 108 | delay(this::configureGraylog, 1); 109 | } 110 | } 111 | 112 | private void delay(Runnable runnable, int delaySeconds) { 113 | Calendar cal = Calendar.getInstance(); 114 | cal.add(Calendar.SECOND, delaySeconds); 115 | scheduler.schedule(runnable, cal.getTime()); 116 | } 117 | 118 | private void setupUdpInput() { 119 | String inputName = "SpringBoot GELF UDP"; 120 | if (graylogRestInterface.inputExists(inputName)) { 121 | LOGGER.info("Graylog UDP input already exists, skipping..."); 122 | return; 123 | } 124 | setupInput(inputName, GELFUDPInput.class, 12201); 125 | } 126 | 127 | private void setupHttpInput() { 128 | String inputName = "SpringBoot GELF HTTP"; 129 | if (graylogRestInterface.inputExists(inputName)) { 130 | LOGGER.info("Graylog HTTP input already exists, skipping..."); 131 | return; 132 | } 133 | String inputId = setupInput(inputName, GELFHttpInput.class, 12202); 134 | 135 | Map extractorConfig = new HashMap<>(); 136 | extractorConfig.put("key_separator", "_"); 137 | extractorConfig.put("list_separator", ", "); 138 | extractorConfig.put("kv_separator", "="); 139 | CreateExtractorRequest extractorRequest = CreateExtractorRequest.create( 140 | "Quote JSON", 141 | "copy", 142 | "full_message", 143 | "", 144 | "json", 145 | extractorConfig, 146 | Collections.emptyMap(), 147 | "none", 148 | "", 149 | 0 150 | ); 151 | graylogRestInterface.createExtractor(inputId, extractorRequest); 152 | LOGGER.info("Graylog JSON extractor created."); 153 | } 154 | 155 | private String setupInput(String name, Class inputType, int port) { 156 | Map properties = new HashMap<>(); 157 | properties.put("use_null_delimiter", true); 158 | properties.put("bind_address", "0.0.0.0"); 159 | properties.put("port", port); 160 | 161 | InputCreateRequest request = InputCreateRequest.create( 162 | name, 163 | inputType.getName(), 164 | true, 165 | properties, 166 | null); 167 | 168 | String inputId = graylogRestInterface.createInput(request); 169 | LOGGER.info(String.format("%s input created.", name)); 170 | return inputId; 171 | } 172 | 173 | private void setupDashboard() { 174 | String dashboardName = "Stock Market"; 175 | if (graylogRestInterface.dashboardExists(dashboardName)) { 176 | LOGGER.info("Graylog dashboard already exists, skipping..."); 177 | return; 178 | } 179 | CreateDashboardRequest dashboardRequest = CreateDashboardRequest.create( 180 | dashboardName, 181 | "Measurements from Stock Market example application" 182 | ); 183 | 184 | String dashboardId = graylogRestInterface.createDashboard(dashboardRequest); 185 | LOGGER.info("Graylog dashboard created."); 186 | 187 | Map timerangeMap = new HashMap() {{ 188 | put("type", "relative"); 189 | put("range", 300); 190 | }}; 191 | 192 | AddWidgetRequest widgetRequest = AddWidgetRequest.create( 193 | "Stock Quotes - Symbols", 194 | "QUICKVALUES", 195 | 10, 196 | new HashMap(){{ 197 | put("timerange", timerangeMap); 198 | put("field", "symbol"); 199 | put("show_pie_chart", true); 200 | put("query", ""); 201 | put("show_data_table", true); 202 | }} 203 | ); 204 | graylogRestInterface.createWidget(dashboardId, widgetRequest); 205 | LOGGER.info("Symbol widget created."); 206 | 207 | widgetRequest = AddWidgetRequest.create( 208 | "Generated Quote Count", 209 | "SEARCH_RESULT_COUNT", 210 | 10, 211 | new HashMap(){{ 212 | put("timerange", timerangeMap); 213 | put("lower_is_better", false); 214 | put("trend", false); 215 | put("query", "_exists_:symbol"); 216 | }} 217 | ); 218 | graylogRestInterface.createWidget(dashboardId, widgetRequest); 219 | LOGGER.info("Quote count widget created."); 220 | 221 | widgetRequest = AddWidgetRequest.create( 222 | "Quote Generation Time (us)", 223 | "FIELD_CHART", 224 | 10, 225 | new HashMap(){{ 226 | put("timerange", new HashMap() {{ 227 | put("type", "relative"); 228 | put("range", 300); 229 | }}); 230 | put("valuetype", "mean"); 231 | put("renderer", "line"); 232 | put("interpolation", "linear"); 233 | put("field", "elapsed_time"); 234 | put("interval", "minute"); 235 | put("rangeType", "relative"); 236 | put("relative", 300); 237 | put("query", "_exists_:elapsed_time"); 238 | }} 239 | ); 240 | graylogRestInterface.createWidget(dashboardId, widgetRequest); 241 | LOGGER.info("Quote generation time widget created."); 242 | 243 | widgetRequest = AddWidgetRequest.create( 244 | "Error Count", 245 | "SEARCH_RESULT_COUNT", 246 | 10, 247 | new HashMap(){{ 248 | put("timerange", new HashMap() {{ 249 | put("type", "relative"); 250 | put("range", 300); 251 | }}); 252 | put("lower_is_better", false); 253 | put("trend", false); 254 | put("query", "_exists_:stacktrace"); 255 | }} 256 | ); 257 | graylogRestInterface.createWidget(dashboardId, widgetRequest); 258 | LOGGER.info("Error count widget created."); 259 | } 260 | } 261 | -------------------------------------------------------------------------------- /src/main/java/com/objectpartners/plummer/graylog/graylog/GraylogRestInterface.java: -------------------------------------------------------------------------------- 1 | package com.objectpartners.plummer.graylog.graylog; 2 | 3 | import com.google.common.collect.Lists; 4 | import org.apache.tomcat.util.codec.binary.Base64; 5 | import org.graylog2.rest.models.dashboards.requests.AddWidgetRequest; 6 | import org.graylog2.rest.models.dashboards.requests.CreateDashboardRequest; 7 | import org.graylog2.rest.models.dashboards.responses.CreateDashboardResponse; 8 | import org.graylog2.rest.models.dashboards.responses.DashboardList; 9 | import org.graylog2.rest.models.system.inputs.extractors.requests.CreateExtractorRequest; 10 | import org.graylog2.rest.models.system.inputs.requests.InputCreateRequest; 11 | import org.graylog2.rest.models.system.inputs.responses.InputCreated; 12 | import org.graylog2.rest.models.system.inputs.responses.InputsList; 13 | import org.springframework.beans.factory.annotation.Value; 14 | import org.springframework.http.*; 15 | import org.springframework.web.client.RestTemplate; 16 | import org.springframework.web.util.UriComponentsBuilder; 17 | 18 | import javax.inject.Named; 19 | import java.nio.charset.StandardCharsets; 20 | 21 | @Named 22 | public class GraylogRestInterface { 23 | 24 | @Value("${graylog.username}") 25 | private String graylogUsername; 26 | 27 | @Value("${graylog.password}") 28 | private String graylogPassword; 29 | 30 | private final UriComponentsBuilder uriBuilder = UriComponentsBuilder.newInstance() 31 | .scheme("http").host("localhost").port(12900); 32 | 33 | private final RestTemplate restTemplate = new RestTemplate(); 34 | 35 | public String createInput(InputCreateRequest request) { 36 | HttpEntity entity = new HttpEntity<>(request, buildHeaders()); 37 | ResponseEntity response = restTemplate.postForEntity(uriBuilder.cloneBuilder().path("system/inputs").toUriString(), entity, InputCreated.class); 38 | return response.getBody().id(); 39 | } 40 | 41 | public boolean inputExists(String inputName) { 42 | HttpEntity entity = new HttpEntity<>(null, buildHeaders()); 43 | ResponseEntity response = restTemplate.exchange(uriBuilder.cloneBuilder().path("system/inputs").toUriString(), HttpMethod.GET, entity, InputsList.class); 44 | InputsList inputs = response.getBody(); 45 | return inputs.inputs().stream().anyMatch(input -> inputName.equals(input.title())); 46 | } 47 | 48 | public void logEvent(GelfMessage message) { 49 | HttpEntity entity = new HttpEntity<>(message, buildHeaders()); 50 | restTemplate.postForEntity(uriBuilder.cloneBuilder().port(12202).path("gelf").toUriString(), entity, null); 51 | } 52 | 53 | public void createExtractor(String inputId, CreateExtractorRequest extractorRequest) { 54 | HttpEntity entity = new HttpEntity<>(extractorRequest, buildHeaders()); 55 | restTemplate.postForEntity(uriBuilder.cloneBuilder().path("system/inputs/"+inputId+"/extractors").toUriString(), entity, null); 56 | } 57 | 58 | public String createDashboard(CreateDashboardRequest dashboardRequest) { 59 | HttpEntity entity = new HttpEntity<>(dashboardRequest, buildHeaders()); 60 | ResponseEntity response = restTemplate.postForEntity(uriBuilder.cloneBuilder().path("dashboards").toUriString(), entity, CreateDashboardResponse.class); 61 | return response.getBody().dashboardId(); 62 | } 63 | 64 | public boolean dashboardExists(String dashboardName) { 65 | HttpEntity entity = new HttpEntity<>(null, buildHeaders()); 66 | ResponseEntity response = restTemplate.exchange(uriBuilder.cloneBuilder().path("dashboards").toUriString(), HttpMethod.GET, entity, ConcreteDashboardList.class); 67 | DashboardList dashboards = response.getBody(); 68 | return dashboards.dashboards().stream().anyMatch(dashboard -> dashboardName.equals(dashboard.get("title"))); 69 | } 70 | 71 | public void createWidget(String dashboardId, AddWidgetRequest widgetRequest) { 72 | HttpEntity entity = new HttpEntity<>(widgetRequest, buildHeaders()); 73 | restTemplate.postForEntity(uriBuilder.cloneBuilder().path("dashboards/"+dashboardId+"/widgets").toUriString(), entity, null); 74 | } 75 | 76 | /** 77 | * Graylog rest services require authentication - construct an HTTP BASIC Auth header and attach to all outgoing requests 78 | * Also tell service that JSON is outgoing and is requested for incoming data 79 | * @return headers 80 | */ 81 | private HttpHeaders buildHeaders() { 82 | String auth = graylogUsername + ":" + graylogPassword; 83 | byte[] encodedAuth = Base64.encodeBase64(auth.getBytes(StandardCharsets.US_ASCII) ); 84 | final String authHeader = "Basic " + new String( encodedAuth ); 85 | 86 | HttpHeaders headers = new HttpHeaders(); 87 | headers.add(HttpHeaders.AUTHORIZATION, authHeader ); 88 | headers.setContentType(MediaType.APPLICATION_JSON); 89 | headers.setAccept(Lists.newArrayList(MediaType.APPLICATION_JSON)); 90 | return headers; 91 | } 92 | } 93 | -------------------------------------------------------------------------------- /src/main/java/com/objectpartners/plummer/graylog/jobs/ScheduledJobs.java: -------------------------------------------------------------------------------- 1 | package com.objectpartners.plummer.graylog.jobs; 2 | 3 | import com.fasterxml.jackson.databind.ObjectMapper; 4 | import com.objectpartners.plummer.graylog.graylog.GraylogRestInterface; 5 | import com.objectpartners.plummer.graylog.service.QuoteService; 6 | import org.apache.commons.lang3.RandomStringUtils; 7 | import org.springframework.scheduling.annotation.Scheduled; 8 | 9 | import javax.inject.Inject; 10 | import javax.inject.Named; 11 | 12 | @Named 13 | public class ScheduledJobs { 14 | 15 | @Inject 16 | protected ObjectMapper mapper; 17 | 18 | @Inject 19 | protected GraylogRestInterface graylog; 20 | 21 | @Inject 22 | protected QuoteService quoteService; 23 | 24 | @Scheduled(initialDelay = 15000L, fixedDelay = 500L) 25 | public void generateQuote() throws Exception { 26 | String symbol = RandomStringUtils.randomAlphabetic(3).toUpperCase(); 27 | quoteService.quote(symbol); 28 | } 29 | 30 | @Scheduled(initialDelay = 15000L, fixedDelay = 1000L) 31 | public void generateException() throws Exception { 32 | throw new Exception("Error during quote generation"); 33 | } 34 | } 35 | -------------------------------------------------------------------------------- /src/main/java/com/objectpartners/plummer/graylog/service/QuoteService.java: -------------------------------------------------------------------------------- 1 | package com.objectpartners.plummer.graylog.service; 2 | 3 | import com.objectpartners.plummer.graylog.domain.Exchange; 4 | import com.objectpartners.plummer.graylog.domain.QuoteResource; 5 | import org.apache.commons.lang3.RandomUtils; 6 | 7 | import javax.inject.Named; 8 | import java.math.BigDecimal; 9 | import java.math.RoundingMode; 10 | 11 | @Named 12 | public class QuoteService { 13 | 14 | public QuoteResource quote(String symbol) { 15 | return new QuoteResource(symbol, 16 | BigDecimal.valueOf(RandomUtils.nextDouble(0, 100)).setScale(2, RoundingMode.HALF_UP).doubleValue(), 17 | Exchange.values()[RandomUtils.nextInt(0, Exchange.values().length)]); 18 | } 19 | } 20 | -------------------------------------------------------------------------------- /src/main/resources/application.yml: -------------------------------------------------------------------------------- 1 | info: 2 | build: 3 | description: "@project.description@" 4 | name: "@project.name@" 5 | version: "@project.version@" 6 | 7 | graylog: 8 | username: "admin" 9 | password: "admin" 10 | configFile: "./graylog/graylog.conf" 11 | 12 | spring: 13 | data: 14 | elasticsearch: 15 | clusterName: "graylog" 16 | properties: 17 | node: 18 | local: false 19 | data: true 20 | http: 21 | enabled: true -------------------------------------------------------------------------------- /src/main/resources/logback.xml: -------------------------------------------------------------------------------- 1 | 2 | 3 | 4 | 5 | 6 | localhost 7 | localhost 8 | true 9 | false 10 | true 11 | false 12 | false 13 | false 14 | gelf-java 15 | 16 | 17 | 18 | 19 | %d %level %logger{0}:%line : %message%n 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | --------------------------------------------------------------------------------