├── LICENSE ├── README.md ├── alertmanager-server ├── README.md ├── alertmanager.json ├── config │ └── prometheus-config.yml └── docker-compose.yaml ├── concat-container ├── Dockerfile ├── README.md └── concat.sh ├── images ├── alertmanager-to-mailhog.png ├── import-redis-dashboard.png ├── redis-dashboard.png └── scope-dashboard.png ├── mailhog-server ├── README.md ├── alertmanager.yml └── docker-compose.yaml ├── memcached-server ├── docker-compose.yaml └── memcached.json ├── node-exporter ├── docker-compose.yaml └── node-exporter.json ├── postgresql-server ├── docker-compose.yaml └── postgresql-server.json ├── prometheus-server ├── config │ ├── base_prometheus.yml │ ├── datasource.yaml │ └── targets │ │ ├── grafana.json │ │ └── prometheus.json └── docker-compose.yaml ├── pushgateway ├── docker-compose.yaml └── pushgateway.json ├── redis-server ├── README.md ├── docker-compose.yaml ├── redis.json └── redis.rules └── scope ├── README.md └── docker-compose.yaml /LICENSE: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2018 Dean Wilson 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # docker-compose-prometheus 2 | 3 | A set of Docker Compose configs to run a local Prometheus test environment 4 | 5 | ### Table of contents 6 | 7 | * [Introduction](/README.md#introduction) 8 | * [Getting started](/README.md#getting-started) 9 | - [Existing services](/README.md#existing-services) 10 | * [Architecture and layout](/README.md#architecture-and-layout) 11 | - [Networking](/README.md#networking) 12 | * [Extending](/README.md#extending) 13 | 14 | ## Introduction 15 | 16 | This repository contains the configuration required to create local 17 | [Prometheus](https://prometheus.io/) and [Grafana](https://grafana.com/) 18 | containers and link them together so you can experiment with metric 19 | collection and graphing. 20 | 21 | Although it makes it possible to bootstrap a very simple, 2 container 22 | infrastructure the most useful additions are the configs in the other 23 | directories. Each of these will add a service and exporter that link 24 | in to the base prometheus and allow you to experiment and learn 25 | how to monitor and graph other services. A good place to start is 26 | with the the Prometheus and Redis combination documented below. 27 | 28 | ## Getting started 29 | 30 | The only things you need to run these examples are `docker-compose` and a copy 31 | of this repo. Everything else happens inside the docker containers. 32 | 33 | A note about the hideous command lines. In order to make this a modular experiment 34 | I've extracted the separate sections of config in to different directories. 35 | While this allows you to spin up a test site with any combination of services 36 | and exporters it does mean you'll need to add a `-f $foo/docker-compose.yaml` 37 | argument for each service you want to include in the test. I avoid the pain by 38 | setting an alias: 39 | 40 | alias dc='docker-compose -f prometheus-server/docker-compose.yaml -f redis-server/docker-compose.yaml' 41 | 42 | And then use commands like `dc up -d` and `dc logs`. In the README examples I'll 43 | use the full commands for clarity but you won't have to. 44 | 45 | ### Creating Prometheus 46 | 47 | The first part of the infrastructure you should build, and the one depended on by 48 | all the example service configurations in other directories, is 49 | [prometheus-server](./prometheus-server/docker-compose.yaml). This will create 50 | both a prometheus and grafana container. At the moment we'll have to manually 51 | link these together. 52 | 53 | From the root of this repo run the command to create the docker containers. 54 | 55 | docker-compose -f prometheus-server/docker-compose.yaml up -d 56 | 57 | On the first run this might take a little while as it fetches all the 58 | containers. Once it returns you can confirm the containers are running: 59 | 60 | docker-compose -f prometheus-server/docker-compose.yaml ps 61 | 62 | ``` 63 | > docker-compose -f prometheus-server/docker-compose.yaml ps 64 | Name Command State Ports 65 | ----------------------------------------------------------------------------------------- 66 | prometheusserver_grafana_1 /run.sh Up 0.0.0.0:3000->3000/tcp 67 | prometheusserver_prometheus_1 /bin/prometheus --config.f ... Up 0.0.0.0:9090->9090/tcp 68 | 69 | ``` 70 | 71 | and view their output: 72 | 73 | docker-compose -f prometheus-server/docker-compose.yaml logs 74 | 75 | When you're finished you can remove the containers, but don't do that yet. 76 | 77 | docker-compose -f prometheus-server/docker-compose.yaml down 78 | 79 | Once the containers have been created you can view the 80 | [Prometheus dashboard](http://127.0.0.1:9090/graph) and the 81 | [Grafana Dashboard](http://127.0.0.1:3000/) (login with admin / secret). 82 | 83 | Now Grafana 5 has been released we can move to configuring the Prometheus data 84 | source with a [datasource.yaml](/prometheus-server/config/datasource.yaml) file 85 | and remove the manual steps. You should still click through to 86 | the [import dashboard page](http://127.0.0.1:3000/datasources/edit/1/dashboards) 87 | screen and import each of those available. You can then view the 88 | [Prometheus graphs](http://127.0.0.1:3000/dashboard/db/prometheus-stats). 89 | 90 | Congratulations! You now have a prometheus and grafana test instance and you can 91 | experiment with making your own scrape backed graphs. You'll soon want to expand 92 | into data from other services, and an ideal place to start is with 93 | [Redis](./redis-server/README.md). 94 | 95 | ### Existing Services 96 | 97 | This repo currently contains example configurations for the following 98 | services and their respective exporters: 99 | 100 | * [Memcached](/memcached-server) 101 | * [Node exporter](/node-exporter) - just an exporter running against your local 102 | host 103 | * [PostgreSQL](/postgresql-server) 104 | * [Prometheus and Grafana](/prometheus-server) 105 | * [Redis Server](/redis-server) 106 | 107 | ### Networking 108 | 109 | All the containers are created inside a single docker network and reference each 110 | other by the magic of their service names. They can also be reached from the 111 | host on `127.0.0.1`. This allows easier access to the prometheus and grafana 112 | dashboards and means you can easily add sample data to the graphs by running 113 | command such as `redis-cli` in a loop or pointing a load tester at them. 114 | 115 | ## Architecture and layout 116 | 117 | One of the key goals in this experiment is to keep it as modular as possible 118 | and allow you to create container networks of whichever combination you need. 119 | Does your application use PostgreSQL and redis? Add a new `docker- 120 | compose.yaml` for your application itself and just include `redis-server/docker- 121 | compose.yaml` and `postgresql-server/docker-compose.yaml` on the 122 | command line to create those backing services and collect metrics on them all. 123 | 124 | To implement this we have a subdirectory for each different thing we 125 | want to collect metrics for. This contains the prometheus target 126 | configuration file, mostly in `${subdirectory_name}.json` and a `docker- 127 | compose.yaml` file that defines how to run the service inside a 128 | container. Critically, the compose file contains 129 | an additional `prometheus` service definition. 130 | 131 | ``` 132 | prometheus: 133 | volumes: 134 | - ${PWD}/redis-server/redis.json:/etc/prometheus/targets/redis.json 135 | ``` 136 | 137 | Docker compose has a wonderful feature that ensures additional values for a 138 | service, even one defined in a separate docker-compose file, are 139 | merged to create a configuration that contains all encountered keys. In 140 | the case of this repo it means we can define the basic prometheus checks 141 | in the base `docker-compose.yaml` file and add the additional checks as we 142 | include the services they target. 143 | 144 | ## Extending 145 | 146 | Adding a new data source or your own application should be a simple process. 147 | 148 | * create a new subdirectory 149 | * add a `docker-compose.yaml` file that can run your container 150 | * ensure the `prometheus:` volume line points to your own service 151 | * add the prometheus target config in the service specific `json` file 152 | * run `docker-compose up` with all the services you want specified using 153 | multiple `-f $foo/docker-compose.yaml` arguments 154 | * add a README detailing your work 155 | 156 | ## Debugging 157 | 158 | To extract the complete base prometheus config file so you can check 159 | its contents after it is concatenated. 160 | 161 | docker cp prometheus-server_config-concat_1:/fragments/complete/prometheus.yml /tmp/prometheus.yml 162 | 163 | ### Author ### 164 | 165 | [Dean Wilson](http://www.unixdaemon.net) 166 | -------------------------------------------------------------------------------- /alertmanager-server/README.md: -------------------------------------------------------------------------------- 1 | # AlertManager Server 2 | 3 | Run a local alertmanager server and configure the prometheus container 4 | to send alerts to it. 5 | 6 | ## Useful commands 7 | 8 | View running config 9 | 10 | docker exec -ti promdocker_alert-manager_1 amtool --alertmanager.url http://127.0.0.1:9093 config 11 | 12 | View active alerts 13 | 14 | docker exec -ti promdocker_alert-manager_1 amtool --alertmanager.url http://127.0.0.1:9093 alert 15 | 16 | Alertname Starts At Summary 17 | RedisDown 2018-03-04 22:23:25 UTC Redis Availability alert. 18 | 19 | -------------------------------------------------------------------------------- /alertmanager-server/alertmanager.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "targets": [ "alert-manager:9093" ], 4 | "labels": { 5 | "job": "alertmanager" 6 | } 7 | } 8 | ] 9 | -------------------------------------------------------------------------------- /alertmanager-server/config/prometheus-config.yml: -------------------------------------------------------------------------------- 1 | 2 | rule_files: 3 | - "/etc/prometheus/*.rules" 4 | 5 | alerting: 6 | alertmanagers: 7 | - static_configs: 8 | - targets: 9 | - alert-manager:9093 10 | 11 | -------------------------------------------------------------------------------- /alertmanager-server/docker-compose.yaml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | 3 | services: 4 | 5 | alert-manager: 6 | image: prom/alertmanager:v0.20.0 7 | ports: 8 | - 9093:9093 9 | networks: 10 | - public 11 | command: 12 | - '--log.level=debug' 13 | 14 | 15 | config-concat: 16 | volumes: 17 | - ${PWD}/alertmanager-server/config/prometheus-config.yml:/fragments/alertmanager-server.yml 18 | 19 | 20 | prometheus: 21 | volumes: 22 | - ${PWD}/alertmanager-server/alertmanager.json:/etc/prometheus/targets/alertmanager.json 23 | -------------------------------------------------------------------------------- /concat-container/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM busybox:1.28 2 | 3 | LABEL maintainer="dean.wilson@gmail.com" 4 | 5 | COPY concat.sh /concat.sh 6 | 7 | ENTRYPOINT ["/concat.sh"] 8 | -------------------------------------------------------------------------------- /concat-container/README.md: -------------------------------------------------------------------------------- 1 | # promconf-concat docker container 2 | 3 | This directory contains all the files needed to build the prometheus 4 | config file concatentor used by the 5 | [docker-compose-prometheus](https://github.com/deanwilson/docker-compose-prometheus) 6 | project to enable using different fragments of configuration. 7 | 8 | ## Introduction 9 | 10 | Prometheus only allows you to specify a single configuration file 11 | which must contain all the config, without any ability to use any kind 12 | of include functionality. In my docker-compose-prometheus experiments 13 | I want to be able to specify a number of discrete components, some of which 14 | cannot be configured via `file_sd_configs` so instead I've built this container 15 | image which will build a single config from a number of separate config file 16 | fragments. 17 | 18 | ## How it works 19 | 20 | The `promconf-concat` container mounts a docker volume under 21 | `/fragments/` and runs a while loop that essentially combines all the file 22 | fragments found in that directory and then writes them into a single, combined 23 | prometheus config file. The actual implementation is written like this: 24 | 25 | cat /base_prometheus.yml /fragments/*.yml > /prometheus.yml 26 | 27 | As this is using standard shell globbing to select the files you can add 28 | ordering with careful selection of file names. This combined file is then 29 | checked against the current `prometheus.yml` file and if they differ the 30 | new one overwrites the old. Prometheus then detects this change and reloads its 31 | config. 32 | 33 | A concrete example of this is the 34 | [alertmanager-server/docker-compose.yaml](/alertmanager-server/docker-compose.yaml) 35 | config, which adds alertmanager configuration under the `alerting:` key. 36 | 37 | To add another fragment to `promconf-concat` you should include config like this 38 | in your `docker-compose.yaml` file. 39 | 40 | ``` 41 | config-concat: 42 | volumes: 43 | - ${PWD}/alertmanager-server/config/prometheus-config.yml:/fragments/alertmanager-server.yml 44 | ``` 45 | 46 | This will then be combined with the other fragments into one config file, written to a location 47 | under the volume and presented to the prometheus container. The prometheus 48 | container also mounts this volume and then runs with the given config. 49 | 50 | ## Container build command 51 | 52 | To build the container you'll need docker installed: 53 | 54 | docker build -t promconf-concat:0.1.0 . 55 | 56 | ## Push the container to dockerhub 57 | 58 | I'm testing this on Fedora 26 which doesn't default to docker.io 59 | as the remote registery so I'm being very specific in my commandlines. 60 | 61 | docker login docker.io 62 | 63 | Tag the container. Use your own name here 64 | 65 | docker tag promconf-concat docker.io/deanwilson/promconf-concat:0.1.0 66 | 67 | And then push to dockerhub 68 | 69 | docker push docker.io/deanwilson/promconf-concat:0.1.0 70 | -------------------------------------------------------------------------------- /concat-container/concat.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | # store the combined config file here 4 | mkdir -p /fragments/complete 5 | # the final, combined config file 6 | completed_config=/fragments/complete/prometheus.yml 7 | 8 | # set a default sha for /fragments/complete/prometheus.yml 9 | # as it won't exist on first run 10 | old_sha=hello 11 | 12 | # loop around and concatenate any fragment files from the docker volume 13 | while true 14 | do 15 | # combine all the yml fragments to the end of /base_prometheus.yml 16 | cat /base_prometheus.yml /fragments/*.yml > /prometheus.yml 2>/dev/null 17 | 18 | new_sha=$(sha256sum /prometheus.yml | cut -d ' ' -f1) 19 | 20 | if [ -e $completed_config ];then 21 | old_sha=$(sha256sum $completed_config | cut -d ' ' -f1) 22 | fi 23 | 24 | # only replace the config if it's changed. prometheus restarts on change 25 | if [ "$new_sha" != "$old_sha" ];then 26 | mv /prometheus.yml $completed_config 27 | fi 28 | 29 | sleep 60 30 | done 31 | -------------------------------------------------------------------------------- /images/alertmanager-to-mailhog.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/deanwilson/docker-compose-prometheus/c38967ab563fac1b2ee5c8149ab39e9799d833f1/images/alertmanager-to-mailhog.png -------------------------------------------------------------------------------- /images/import-redis-dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/deanwilson/docker-compose-prometheus/c38967ab563fac1b2ee5c8149ab39e9799d833f1/images/import-redis-dashboard.png -------------------------------------------------------------------------------- /images/redis-dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/deanwilson/docker-compose-prometheus/c38967ab563fac1b2ee5c8149ab39e9799d833f1/images/redis-dashboard.png -------------------------------------------------------------------------------- /images/scope-dashboard.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/deanwilson/docker-compose-prometheus/c38967ab563fac1b2ee5c8149ab39e9799d833f1/images/scope-dashboard.png -------------------------------------------------------------------------------- /mailhog-server/README.md: -------------------------------------------------------------------------------- 1 | # MailHog Server for AlertManager alerts 2 | 3 | [MailHog](https://github.com/mailhog/MailHog) is a fake SMTP server 4 | that receives emails and then displays them via its internal web server. 5 | In this repository we use it as an AlertManager target so we can visually 6 | see what the triggered alerts look like. 7 | 8 | If you include MailHog via `docker-compose` it will add a basic AlertManager 9 | config that matches every alert and then sends them to MailHog for processing. 10 | 11 | ![MailHog UI with alert](/images/alertmanager-to-mailhog.png?raw=true "MailHog displaying AlertManager alert") 12 | -------------------------------------------------------------------------------- /mailhog-server/alertmanager.yml: -------------------------------------------------------------------------------- 1 | global: 2 | # The smarthost and SMTP sender used for mail notifications. 3 | smtp_smarthost: 'mailhog:1025' 4 | smtp_from: 'alertmanager@example.org' 5 | smtp_require_tls: false 6 | 7 | # The root route on which each incoming alert enters. 8 | route: 9 | # The root route must not have any matchers as it is the entry point for 10 | # all alerts. It needs to have a receiver configured so alerts that do not 11 | # match any of the sub-routes are sent to someone. 12 | receiver: 'unmatched-default-root-route' 13 | 14 | # When a new group of alerts is created by an incoming alert, wait at 15 | # least 'group_wait' to send the initial notification. 16 | # This way ensures that you get multiple alerts for the same group that start 17 | # firing shortly after another are batched together on the first 18 | # notification. 19 | group_wait: 30s 20 | 21 | receivers: 22 | - name: 'unmatched-default-root-route' 23 | email_configs: 24 | - to: 'devopsteam@example.org' 25 | send_resolved: true 26 | -------------------------------------------------------------------------------- /mailhog-server/docker-compose.yaml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | 3 | services: 4 | 5 | mailhog: 6 | image: mailhog/mailhog:v1.0.0 7 | ports: 8 | - 1025:1025 # SMTP 9 | - 8025:8025 # HTTP 10 | networks: 11 | - public 12 | 13 | alert-manager: 14 | volumes: 15 | - ${PWD}/mailhog-server/alertmanager.yml:/alertmanager.yml 16 | command: 17 | - '--config.file=/alertmanager.yml' 18 | 19 | -------------------------------------------------------------------------------- /memcached-server/docker-compose.yaml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | 3 | services: 4 | memcached-server: 5 | image: library/memcached:1.6.5 6 | ports: 7 | - 11211:11211 8 | networks: 9 | - public 10 | 11 | memcached-exporter: 12 | image: prom/memcached-exporter:v0.6.0 13 | ports: 14 | - 9150:9150 15 | networks: 16 | - public 17 | command: 18 | - '--memcached.address=memcached-server:11211' 19 | 20 | prometheus: 21 | volumes: 22 | - ${PWD}/memcached-server/memcached.json:/etc/prometheus/targets/memcached.json 23 | 24 | -------------------------------------------------------------------------------- /memcached-server/memcached.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "targets": [ "memcached-exporter:9150" ], 4 | "labels": { 5 | "job": "memcached_exporter" 6 | } 7 | } 8 | ] 9 | -------------------------------------------------------------------------------- /node-exporter/docker-compose.yaml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | 3 | services: 4 | node-exporter: 5 | image: prom/node-exporter:v0.18.1 6 | volumes: 7 | - /proc:/host/proc:ro 8 | - /sys:/host/sys:ro 9 | - /:/rootfs:ro 10 | command: 11 | - '--path.procfs=/host/proc' 12 | - '--path.sysfs=/host/sys' 13 | - --collector.filesystem.ignored-mount-points 14 | - "^/(sys|proc|dev|host|etc|rootfs/var/lib/docker/containers|rootfs/var/lib/docker/overlay2|rootfs/run/docker/netns|rootfs/var/lib/docker/aufs)($$|/)" 15 | ports: 16 | - 9100:9100 17 | networks: 18 | - public 19 | 20 | prometheus: 21 | volumes: 22 | - ${PWD}/node-exporter/node-exporter.json:/etc/prometheus/targets/node-exporter.json 23 | -------------------------------------------------------------------------------- /node-exporter/node-exporter.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "targets": [ "node-exporter:9100" ], 4 | "labels": { 5 | "job": "node-exporter" 6 | } 7 | } 8 | ] 9 | -------------------------------------------------------------------------------- /postgresql-server/docker-compose.yaml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | 3 | services: 4 | postgresql-server: 5 | image: library/postgres:12.2 6 | ports: 7 | - 5432:5432 8 | networks: 9 | - public 10 | environment: 11 | - POSTGRES_PASSWORD=secretpassword 12 | 13 | postgresql-exporter: 14 | image: wrouesnel/postgres_exporter:v0.8.0 15 | ports: 16 | - 9187:9187 17 | networks: 18 | - public 19 | environment: 20 | # very insecure client connection 21 | - DATA_SOURCE_NAME=postgresql://postgres:secretpassword@postgresql-server:5432/?sslmode=disable 22 | 23 | prometheus: 24 | volumes: 25 | - ${PWD}/postgresql-server/postgresql-server.json:/etc/prometheus/targets/postgresql-server.json 26 | -------------------------------------------------------------------------------- /postgresql-server/postgresql-server.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "targets": [ "postgresql-exporter:9187" ], 4 | "labels": { 5 | "job": "postgresql-exporter" 6 | } 7 | } 8 | ] 9 | -------------------------------------------------------------------------------- /prometheus-server/config/base_prometheus.yml: -------------------------------------------------------------------------------- 1 | # from base_prometheus.yml 2 | global: 3 | scrape_interval: 15s 4 | 5 | # Load a directory of JSON configs 6 | scrape_configs: 7 | - job_name: 'stub' # This is a default value, it is mandatory. 8 | file_sd_configs: 9 | - files: 10 | - /etc/prometheus/targets/*.json 11 | -------------------------------------------------------------------------------- /prometheus-server/config/datasource.yaml: -------------------------------------------------------------------------------- 1 | # config file version 2 | apiVersion: 1 3 | 4 | # list of datasources that should be deleted from the database 5 | deleteDatasources: 6 | - name: Graphite 7 | orgId: 1 8 | 9 | datasources: 10 | - name: Prometheus 11 | type: prometheus 12 | access: proxy 13 | isDefault: true 14 | url: http://prometheus:9090 15 | # don't use this in prod 16 | editable: true 17 | -------------------------------------------------------------------------------- /prometheus-server/config/targets/grafana.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "targets": [ "grafana:3000" ], 4 | "labels": { 5 | "job": "grafana" 6 | } 7 | } 8 | ] 9 | -------------------------------------------------------------------------------- /prometheus-server/config/targets/prometheus.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "targets": [ "localhost:9090" ], 4 | "labels": { 5 | "job": "prometheus" 6 | } 7 | } 8 | ] 9 | -------------------------------------------------------------------------------- /prometheus-server/docker-compose.yaml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | 3 | networks: 4 | public: {} 5 | 6 | volumes: 7 | grafana_lib: {} 8 | prometheus_data: {} 9 | prometheus_config: 10 | driver_opts: 11 | type: tmpfs 12 | device: tmpfs 13 | 14 | services: 15 | prometheus: 16 | image: prom/prometheus:v2.19.0 17 | ports: 18 | - 9090:9090 19 | networks: 20 | - public 21 | volumes: 22 | - prometheus_data:/prometheus 23 | - prometheus_config:/fragments/ 24 | - ${PWD}/prometheus-server/config/targets/prometheus.json:/etc/prometheus/targets/prometheus.json 25 | - ${PWD}/prometheus-server/config/targets/grafana.json:/etc/prometheus/targets/grafana.json 26 | command: 27 | - '--config.file=/fragments/complete/prometheus.yml' 28 | - '--storage.tsdb.path=/prometheus' 29 | 30 | grafana: 31 | image: grafana/grafana:7.0.3 32 | ports: 33 | - "3000:3000" 34 | networks: 35 | - public 36 | volumes: 37 | - grafana_lib:/var/lib/grafana 38 | - ${PWD}/prometheus-server/config/datasource.yaml:/etc/grafana/provisioning/datasources/datasource.yaml 39 | environment: 40 | - GF_SECURITY_ADMIN_PASSWORD=secret 41 | 42 | config-concat: 43 | image: deanwilson/promconf-concat:0.1.0 44 | volumes: 45 | - prometheus_config:/fragments/ 46 | - ${PWD}/prometheus-server/config/base_prometheus.yml:/base_prometheus.yml 47 | -------------------------------------------------------------------------------- /pushgateway/docker-compose.yaml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | 3 | services: 4 | pushgateway: 5 | image: prom/pushgateway:v1.2.0 6 | ports: 7 | - 9091:9091 8 | networks: 9 | - public 10 | 11 | prometheus: 12 | volumes: 13 | - ${PWD}/pushgateway/pushgateway.json:/etc/prometheus/targets/pushgateway.json 14 | -------------------------------------------------------------------------------- /pushgateway/pushgateway.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "targets": [ "pushgateway:9091" ], 4 | "labels": { 5 | "job": "pushgateway" 6 | } 7 | } 8 | ] 9 | -------------------------------------------------------------------------------- /redis-server/README.md: -------------------------------------------------------------------------------- 1 | ## Redis monitoring and metrics 2 | 3 | In this document we will add Redis, and the Redis exporter that presents 4 | information to prometheus, to our test stack and then import a Grafana 5 | dashboard to show us the real time results. 6 | 7 | ### Warning 8 | 9 | Before going any further in this document please ensure you've read 10 | the [main README](/README.md) which explains the requirements 11 | and steps you'll need to do before adding redis will succeed. 12 | 13 | ### Running redis 14 | 15 | In this directory is a [redis docker-compose](/redis-server/docker-compose.yaml) 16 | config file. This will run a redis server and a prometheus exporter that will 17 | connect to it. The [redis target config](redis-server/redis.json) is then 18 | added to the prometheus configs, which is enough to enable prometheus to start 19 | scraping and collecting the metrics. 20 | 21 | To spin the containers up run 22 | 23 | docker-compose -f prometheus-server/docker-compose.yaml -f redis-server/docker-compose.yaml up -d 24 | 25 | You can confirm that the containers are running with 26 | 27 | docker-compose -f prometheus-server/docker-compose.yaml -f redis-server/docker-compose.yaml ps 28 | 29 | and view any output: 30 | 31 | docker-compose -f prometheus-server/docker-compose.yaml -f redis-server/docker-compose.yaml logs 32 | 33 | Now everything is running you can view the exporter, and its status, 34 | on the [Prometheus Targets](http://127.0.0.1:9090/targets) page. You 35 | can confirm data is being loaded by querying the 36 | [redis_instance_info metric](http://127.0.0.1:9090/graph?g0.range_input=1h&g0.expr=redis_instance_info&g0.tab=1) 37 | 38 | Now we have all the pieces configured and running we'll add a third party 39 | [Grafana dashboard](https://grafana.com/dashboards?dataSource=prometheus) 40 | to our instance to help us visualise the results. This part of the 41 | process is currently very manual but hopefully this will become more automated 42 | over time. 43 | 44 | * Click on the "Home" button on the top left of the Grafana dashboard 45 | * Click "Import Dashboard" (on the right of the dashboard list) 46 | * Add the dashboard ID, [763](https://grafana.com/dashboards/763) in this case, 47 | and click else where in the dialog box. 48 | 49 | You will then be presented with the "Import Dashboard" dialog box 50 | 51 | ![Import dashboard](/images/import-redis-dashboard.png?raw=true "Import redis dashboard") 52 | 53 | Select your prometheus data source, which we created in the main README earlier 54 | and named "Prometheus". This will import and enable the dashboard. You can now 55 | click over to the 56 | [Prometheus Redis Dashboard](http://127.0.0.1:3000/dashboard/db/prometheus-redis) 57 | and view its details. 58 | 59 | ![Redis dashboard](/images/redis-dashboard.png?raw=true "Sample redis dashboard") 60 | 61 | ### Access the redis container 62 | 63 | The redis container is assigned its default port on the host loopback 64 | interface so you can use tools like `redis-cli` to connect directly to it. 65 | This allows you to execute commands you can then view on the dashboards or use 66 | load testing tools to generate additional traffic. 67 | 68 | $ redis-cli ping 69 | 70 | By default we require password auth to successfully connect. The default password 71 | can be found in the [redis docker-compose file](/redis-server/docker-compose.yaml) 72 | in the `command` section. 73 | -------------------------------------------------------------------------------- /redis-server/docker-compose.yaml: -------------------------------------------------------------------------------- 1 | version: "3" 2 | 3 | services: 4 | redis-server: 5 | image: library/redis:6.0.1 6 | command: "--requirepass slightly-locked" 7 | ports: 8 | - 6379:6379 9 | networks: 10 | - public 11 | 12 | redis-exporter: 13 | image: oliver006/redis_exporter:v1.6.0-alpine 14 | ports: 15 | - 9121:9121 16 | networks: 17 | - public 18 | command: 19 | - '--redis.addr=redis://redis-server:6379' 20 | - '--redis.password=slightly-locked' 21 | 22 | prometheus: 23 | volumes: 24 | - ${PWD}/redis-server/redis.json:/etc/prometheus/targets/redis.json 25 | - ${PWD}/redis-server/redis.rules:/etc/prometheus/redis.rules 26 | 27 | -------------------------------------------------------------------------------- /redis-server/redis.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "targets": [ "redis-exporter:9121" ], 4 | "labels": { 5 | "job": "redis_exporter" 6 | } 7 | } 8 | ] 9 | -------------------------------------------------------------------------------- /redis-server/redis.rules: -------------------------------------------------------------------------------- 1 | groups: 2 | - name: redis-available 3 | rules: 4 | - alert: RedisDown 5 | expr: redis_up == 0 6 | for: 1m 7 | labels: 8 | severity: "high" 9 | annotations: 10 | summary: Redis Availability alert. 11 | description: "Checks if the redis exporter shows the redis server as available" 12 | -------------------------------------------------------------------------------- /scope/README.md: -------------------------------------------------------------------------------- 1 | ## Weave Scope 2 | 3 | "_[Weave Scope is a visualisation and monitoring tool for Docker and Kubernetes.](https://www.weave.works/oss/scope/)_" 4 | 5 | ![Weave Scope Dashboard](/images/scope-dashboard.png?raw=true "Weave Scope Dashboard") 6 | 7 | As I've added more services and exporters to the 8 | [Docker Compose Prometheus project](https://github.com/deanwilson/docker-compose-prometheus) 9 | the number of possible different arrangements and configurations has 10 | exploded and I decided it was time to find a more visual way to describe 11 | exactly what was running. Weave Scopes ability to automatically generate 12 | a map of the services and show some connectivity details between the containers 13 | made it the perfect tool for an initial experiment. 14 | 15 | To run Scope add an additional `docker-compose.yaml` to your command line: 16 | 17 | ``` 18 | docker-compose \ 19 | -f prometheus-server/docker-compose.yaml \ 20 | -f alertmanager-server/docker-compose.yaml \ 21 | -f node-exporter/docker-compose.yaml \ 22 | -f mailhog-server/docker-compose.yaml \ 23 | -f pushgateway/docker-compose.yaml 24 | -f redis-server/docker-compose.yaml 25 | -f scope/docker-compose.yaml \ 26 | up -d 27 | ``` 28 | 29 | and then view the [scope dashboard](http://localhost:4040/) on your local machine. 30 | -------------------------------------------------------------------------------- /scope/docker-compose.yaml: -------------------------------------------------------------------------------- 1 | version: '3' 2 | 3 | services: 4 | scope: 5 | image: weaveworks/scope:1.13.1 6 | network_mode: "host" 7 | pid: "host" 8 | privileged: true 9 | labels: 10 | - "works.weave.role=system" 11 | volumes: 12 | - "/var/run/docker.sock:/var/run/docker.sock:rw" 13 | command: 14 | - "--probe.docker=true" 15 | --------------------------------------------------------------------------------