├── .gitignore ├── README.md ├── docker ├── alertmanager │ └── config.yml ├── caddy │ └── Caddyfile ├── docker-compose.yml ├── elasticsearch │ └── .keep ├── filebeat │ ├── config │ │ └── filebeat.yml │ └── data │ │ └── .keep ├── grafana │ ├── dashboards │ │ ├── docker_containers.json │ │ ├── docker_host.json │ │ ├── monitor_services.json │ │ └── nginx_container.json │ ├── datasources │ │ └── Prometheus.json │ └── setup.sh ├── kibana │ └── config.yml ├── logstash │ ├── config │ │ └── logstash.yml │ └── pipeline │ │ └── logstash.conf └── prometheus │ ├── alert.rules │ └── prometheus.yml ├── docs ├── architecture_diagram.drawio └── architecture_diagram.png └── k8s └── .keep /.gitignore: -------------------------------------------------------------------------------- 1 | docker/elasticsearch/nodes/* 2 | docker/filebeat/data/meta.json 3 | docker/filebeat/data/registry/* 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # monitoring-stack 2 | 3 | Monitoring stack containing ELK + Grafana + Prometheus (with multiple exporters) and authentication through a reverse proxy (Caddy) 4 | 5 | ## Overview 6 | 7 | This repository is inteended to assist implementing a set of tools for Log Consolidation and monitoring. 8 | 9 | This monitoring stack is composed by the following tools: 10 | 11 | - [Elastic stack (formerly known as ELK):](https://www.elastic.co/webinars/introduction-elk-stack) Consists of Elasticsearch, Logstash, Kibana and Beats. Although they've all been built to work exceptionally well together, each one is a separate project that is driven by the open-source vendor Elastic -- which itself began as an enterprise search platform vendor. 12 | 13 | - [Elasticsearch:](https://www.elastic.co/products/elasticsearch) is a search engine based on Lucene. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in Java and is released as open source under the terms of the Apache License. 14 | 15 | - [Logstash:](https://www.elastic.co/products/logstash) Is an open source tool for collecting, parsing, and storing logs for future use. Kibana 3 is a web interface that can be used to search and view the logs that Logstash has indexed. Both of these tools are based on Elasticsearch. Elasticsearch, Logstash, and Kibana, when used together is known as an ELK stack. 16 | 17 | - [Kibana:](https://www.elastic.co/products/kibana) Is an open source data visualization plugin for Elasticsearch. It provides visualization capabilities on top of the content indexed on an Elasticsearch cluster. Users can create bar, line and scatter plots, or pie charts and maps on top of large volumes of data. 18 | 19 | - [Beats](https://www.elastic.co/products/beats): Is a lightweight agent from Elastic, which is great for gathering data. They sit on your servers and centralize data in Elasticsearch. Idea is that we use it to increase the the processing muscle, Beats can also ship to Logstash for transformation and parsing. 20 | 21 | - [Prometheus](https://github.com/prometheus): Prometheus, a Cloud Native Computing Foundation project, is a systems and service monitoring system. It collects metrics from configured targets at given intervals, evaluates rule expressions, displays the results, and can trigger alerts if some condition is observed to be true. 22 | 23 | - [Node_Exporter:](https://github.com/prometheus/node_exporter) Prometheus exporter for hardware and OS metrics exposed by \*NIX kernels, written in Go with pluggable metric collectors. 24 | 25 | - [Elasticsearch_Exporter:](https://github.com/justwatchcom/elasticsearch_exporter) Prometheus exporter for various metrics about ElasticSearch, written in Go. 26 | 27 | - [CAdvisor:](https://github.com/google/cadvisor) (Container Advisor) provides container users an understanding of the resource usage and performance characteristics of their running containers. It is a running daemon that collects, aggregates, processes, and exports information about running containers. Specifically, for each container it keeps resource isolation parameters, historical resource usage, histograms of complete historical resource usage and network statistics. This data is exported by container and machine-wide. 28 | 29 | - [Grafana:](https://github.com/grafana/grafana) is an open source metric analytics & visualization suite. It is most commonly used for visualizing time series data for infrastructure and application analytics but many use it in other domains including industrial sensors, home automation, weather, and process control. 30 | 31 | - [Caddy:](https://github.com/stefanprodan/caddy-builder) Caddy is the HTTP/2 web server with automatic HTTPS, including additional plugins to make it act in a similar way as an Ingress/Proxy. 32 | 33 | ## Requirements 34 | 35 | - [Docker](https://docs.docker.com/engine/installation/) 36 | - [Docker Compose](https://docs.docker.com/compose/install/) 37 | 38 | ## Architecture 39 | 40 | The following is a visual representation of how the different tools will be integrated together: 41 | 42 | ![diagram](docs/architecture_diagram.png) 43 | 44 | ## Deployment steps on Docker Swarm Cluster 45 | 46 | **1.** After installing `docker` and `docker-compose`, as described in the requirements session, initialize the swarm, as shown right below: 47 | 48 | ```shell 49 | docker swarm init 50 | ``` 51 | 52 | A similar output should be displayed if the command was successfull: 53 | 54 | ```shell 55 | Swarm initialized: current node (a8pml3unconooa7t7qnsw7knp) is now a manager. 56 | 57 | To add a worker to this swarm, run the following command: 58 | 59 | docker swarm join --token SWMTKN-1-5jk292hj9b12by0byz4xfnntq07nd58ozrqxxyjx91kx03oqhw-8pl8cm17pj1lam8lq10gmuih0 192.168.122.60:2377 60 | 61 | To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions. 62 | ``` 63 | 64 | **2.** Clone this repository to the target machine where the monitoring stack will be deployed: 65 | 66 | ```shell 67 | git clone git@github.com:fsilveir/monitoring-stack.git 68 | ``` 69 | 70 | **3.** From the repository base directory, go into the `docker` directory and execute the following command on your target machine: 71 | 72 | ```shell 73 | cd monitoring-stack/docker 74 | docker stack deploy monitoring --compose-file docker-compose.yml 75 | ``` 76 | 77 | A similar output should be displayed if the applications were successfully deployed: 78 | 79 | ```shell 80 | docker stack deploy monitoring --compose-file docker-compose.yml 81 | Creating network monitoring_net 82 | Creating service monitoring_pushgateway 83 | Creating service monitoring_prometheus 84 | Creating service monitoring_kibana 85 | Creating service monitoring_elasticsearch_exporter 86 | Creating service monitoring_caddy 87 | Creating service monitoring_cadvisor 88 | Creating service monitoring_elasticsearch 89 | Creating service monitoring_logstash 90 | Creating service monitoring_nodeexporter 91 | Creating service monitoring_grafana 92 | Creating service monitoring_filebeat 93 | Creating service monitoring_alertmanager 94 | ``` 95 | 96 | You can also confirm if all the required services were successfully started by executing `docker ps`, as shown below: 97 | 98 | ```shell 99 | $ docker ps 100 | CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 101 | 186f2cb28c7e prom/alertmanager:v0.19.0 "/bin/alertmanager -…" 7 seconds ago Up 4 seconds 9093/tcp monitoring_alertmanager.1.ktll54qa0ttk9ljr1bpba6f04 102 | cedda4dd3185 docker.elastic.co/beats/filebeat:7.3.2 "/usr/local/bin/dock…" 11 seconds ago Up 8 seconds monitoring_filebeat.1.apc6mygodsypsdvyb84zfcgpa 103 | 3910e960856a grafana/grafana:6.3.5 "/setup.sh" 18 seconds ago Up 14 seconds 3000/tcp monitoring_grafana.1.lslqeve2o95djzx24ji3ltqjy 104 | 755c577b9f9a prom/node-exporter:v0.18.1 "/bin/node_exporter …" 22 seconds ago Up 18 seconds 9100/tcp monitoring_nodeexporter.1.tet69f5xrdantjc5omuqfyy6s 105 | 5fdb167dcae0 docker.elastic.co/logstash/logstash:7.3.2 "/usr/local/bin/dock…" 26 seconds ago Up 22 seconds 5044/tcp, 9600/tcp monitoring_logstash.1.ou571n1now2c9go6o21gkpbt2 106 | 945f02693a24 docker.elastic.co/elasticsearch/elasticsearch:7.3.2 "/usr/local/bin/dock…" 33 seconds ago Up 31 seconds 9200/tcp, 9300/tcp monitoring_elasticsearch.1.fa1a2dn9wt6027qfzqpfavsn5 107 | 8be996662db6 google/cadvisor:v0.33.0 "/usr/bin/cadvisor -…" 40 seconds ago Up 38 seconds (healthy) 8080/tcp monitoring_cadvisor.1.65i1noof89ca1wzcq4dynet11 108 | 3d7b098d445d stefanprodan/caddy:latest "/sbin/tini -- caddy…" 46 seconds ago Up 42 seconds monitoring_caddy.1.yje4slt3myl0w1btlofyf69hg 109 | d05e14053300 justwatch/elasticsearch_exporter:1.1.0 "/bin/elasticsearch_…" 51 seconds ago Up 49 seconds 9114/tcp monitoring_elasticsearch_exporter.1.ph0aea6j5dxtmuwg2fy105jh9 110 | ba7a199a8a86 docker.elastic.co/kibana/kibana:7.3.2 "/usr/local/bin/dumb…" 57 seconds ago Up 55 seconds 5601/tcp monitoring_kibana.1.sd0kawrjy12btkrr1yge97m0t 111 | 41773e80e777 prom/prometheus:v2.12.0 "/bin/prometheus --c…" About a minute ago Up About a minute 9090/tcp monitoring_prometheus.1.gpmeg1bgp3dywsmbp23pndil3 112 | d79840c54b78 prom/pushgateway:v0.9.1 "/bin/pushgateway" About a minute ago Up About a minute 9091/tcp monitoring_pushgateway.1.7rzking41qjisqcglh52qeo04 113 | ``` 114 | 115 | ## Accessing the Tools 116 | 117 | **Caddy** will work as a reverse proxy and provide basic authentication for accessing the different UI's. The default password is defined during the deployment, default values can be found inside the `docker-compose` fle -- we strongly recommend changing the default passwords before deploying the stack. 118 | 119 | To access the different UI's click at the following URL's: 120 | 121 | | Tool | URL | 122 | | ------------ | --------------------- | 123 | | Grafana | http://localhost:3000 | 124 | | Prometheus | http://localhost:9090 | 125 | | Alertmanager | http://localhost:9093 | 126 | | Pushgateway | http://localhost:9091 | 127 | | Kibana | http://localhost:5601 | 128 | 129 | ## Testing Data Injection to Logstash 130 | 131 | After confirming that the monitoring stack is running, you can inject some log entries for testing purposes. The shipped Logstash configuration allows you to send content via TCP, as shown in the example below: 132 | 133 | ```shell 134 | $ nc localhost 5000 < /path/to/logfile.log 135 | ``` 136 | 137 | You should be able to confirm through Kibana that the entry you've injected is shown as a new document. 138 | 139 | ## Common Issues 140 | 141 | ### Logstash / Filebeat index pattern not displayed in Kibana 142 | 143 | By default, the index pattern should be automatically detected the first time you access the Kibana dashboard, however if that does not happen, you can manually create the index patterns with the following `curl` command from within the Kibana container. 144 | 145 | #### Manually creating Logstash index pattern on Kibana 146 | 147 | ```shell 148 | curl -v -XPOST -D- 'http://kibana:5601/api/saved_objects/index-pattern' \ 149 | -H 'Content-Type: application/json' \ 150 | -H 'kbn-version: 7.3.2' \ 151 | -d '{"attributes":{"title":"logstash-*","timeFieldName":"@timestamp"}}' 152 | ``` 153 | 154 | #### Manually creating Filebeat index pattern on Kibana 155 | 156 | ```shell 157 | curl -v -XPOST -D- 'http://kibana:5601/api/saved_objects/index-pattern' \ 158 | -H 'Content-Type: application/json' \ 159 | -H 'kbn-version: 7.3.2' \ 160 | -d '{"attributes":{"title":"filebeat-*","timeFieldName":"@timestamp"}}' 161 | ``` 162 | 163 | **4.** To fully stop the monitoring stack, execute the following command on your target machine: 164 | 165 | ```shell 166 | docker stack rm monitoring 167 | ``` 168 | 169 | A similar output should be displayed if all the applications was successfully removed: 170 | 171 | ```shell 172 | Removing service monitoring_alertmanager 173 | Removing service monitoring_caddy 174 | Removing service monitoring_cadvisor 175 | Removing service monitoring_elasticsearch 176 | Removing service monitoring_elasticsearch_exporter 177 | Removing service monitoring_filebeat 178 | Removing service monitoring_grafana 179 | Removing service monitoring_kibana 180 | Removing service monitoring_logstash 181 | Removing service monitoring_nodeexporter 182 | Removing service monitoring_prometheus 183 | Removing service monitoring_pushgateway 184 | Removing network monitoring_net 185 | ``` 186 | 187 | You can confirm if all the required services were successfully removed by executing `docker ps`, a similar output should be displayed: 188 | 189 | ```shell 190 | docker ps 191 | CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 192 | ``` 193 | 194 | ## Get support 195 | 196 | Create an [issue](https://github.com/fsilveir/docker-monitoring-stack/issues) if you want to report a problem or ask for a new functionality any feedback is highly appreciated! 197 | -------------------------------------------------------------------------------- /docker/alertmanager/config.yml: -------------------------------------------------------------------------------- 1 | route: 2 | receiver: 'slack' 3 | 4 | receivers: 5 | - name: 'slack' 6 | slack_configs: 7 | - send_resolved: true 8 | text: "{{ .CommonAnnotations.description }}" 9 | username: 'Prometheus' 10 | channel: '#' 11 | api_url: 'https://hooks.slack.com/services/' 12 | -------------------------------------------------------------------------------- /docker/caddy/Caddyfile: -------------------------------------------------------------------------------- 1 | :9090 { 2 | basicauth / {$ADMIN_USER} {$ADMIN_PASSWORD} 3 | proxy / prometheus:9090 { 4 | transparent 5 | } 6 | 7 | errors stderr 8 | tls off 9 | } 10 | 11 | # :9093 { 12 | # basicauth / {$ADMIN_USER} {$ADMIN_PASSWORD} 13 | # proxy / alertmanager:9093 { 14 | # transparent 15 | # } 16 | 17 | # errors stderr 18 | # tls off 19 | # } 20 | 21 | :9091 { 22 | basicauth / {$ADMIN_USER} {$ADMIN_PASSWORD} 23 | proxy / pushgateway:9091 { 24 | transparent 25 | } 26 | 27 | errors stderr 28 | tls off 29 | } 30 | 31 | :5601 { 32 | basicauth / {$ADMIN_USER} {$ADMIN_PASSWORD} 33 | proxy / kibana:5601 { 34 | transparent 35 | } 36 | 37 | errors stderr 38 | tls off 39 | } 40 | 41 | 42 | :3000 { 43 | proxy / grafana:3000 { 44 | transparent 45 | websocket 46 | } 47 | 48 | errors stderr 49 | tls off 50 | } -------------------------------------------------------------------------------- /docker/docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3.7' 2 | 3 | volumes: 4 | elasticsearch: {} 5 | filebeat: {} 6 | prometheus: {} 7 | grafana: {} 8 | 9 | networks: 10 | net: 11 | driver: overlay 12 | 13 | services: 14 | # ----------------------------------------------------------------------------- 15 | elasticsearch: 16 | # ----------------------------------------------------------------------------- 17 | image: docker.elastic.co/elasticsearch/elasticsearch:7.3.2 18 | hostname: elasticsearch 19 | environment: 20 | - node.name=elasticsearch 21 | - discovery.type=single-node 22 | - cluster.name=monitoring 23 | - bootstrap.memory_lock=true 24 | - "ES_JAVA_OPTS=-Xms512m -Xmx512m" 25 | 26 | volumes: 27 | - ./elasticsearch:/usr/share/elasticsearch/data 28 | # ports: 29 | # - 9200:9200/tcp 30 | # - 9300:3200/tcp 31 | networks: 32 | - net 33 | # ----------------------------------------------------------------------------- 34 | elasticsearch_exporter: 35 | # ----------------------------------------------------------------------------- 36 | hostname: elasticsearch_exporter 37 | image: justwatch/elasticsearch_exporter:1.1.0 38 | depends_on: 39 | - elasticsearch 40 | command: 41 | - '--es.uri=http://elasticsearch:9200' 42 | # ports: 43 | # - 9114:9114/tcp 44 | networks: 45 | - net 46 | 47 | # ----------------------------------------------------------------------------- 48 | logstash: 49 | # ----------------------------------------------------------------------------- 50 | hostname: logstash 51 | image: docker.elastic.co/logstash/logstash:7.3.2 52 | depends_on: 53 | - elasticsearch 54 | volumes: 55 | - type: bind 56 | source: ./logstash/config/logstash.yml 57 | target: /usr/share/logstash/config/logstash.yml 58 | read_only: true 59 | - type: bind 60 | source: ./logstash/pipeline 61 | target: /usr/share/logstash/pipeline 62 | read_only: true 63 | ports: 64 | - "5000:5000" 65 | - "9600:9600" 66 | environment: 67 | LS_JAVA_OPTS: "-Xmx256m -Xms256m" 68 | networks: 69 | - net 70 | 71 | # ----------------------------------------------------------------------------- 72 | filebeat: 73 | # ----------------------------------------------------------------------------- 74 | image: docker.elastic.co/beats/filebeat:7.3.2 75 | hostname: filebeat 76 | user: root 77 | volumes: 78 | - ./filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml 79 | - ./filebeat/data:/usr/share/filebeat/data 80 | - /var/run/docker.sock:/var/run/docker.sock 81 | # This is needed for filebeat to load container log path as specified in filebeat.yml 82 | - /var/lib/docker/containers/:/var/lib/docker/containers/:ro 83 | 84 | # # This is needed for filebeat to load jenkins build log path as specified in filebeat.yml 85 | # - /var/lib/docker/volumes/jenkins_home/_data/jobs/:/var/lib/docker/volumes/jenkins_home/_data/jobs/:ro 86 | 87 | # This is needed for filebeat to load logs for system and auth modules 88 | - /var/log/:/var/log/:ro 89 | # This is needed for filebeat to load logs for auditd module 90 | # - /var/log/audit/:/var/log/audit/:ro 91 | environment: 92 | - ELASTICSEARCH_HOST="elasticsearch" 93 | - KIBANA_HOST="kibana" 94 | # - ELASTICSEARCH_USERNAME=${ELASTICSEARCH_USERNAME:-elastic} 95 | # - ELASTICSEARCH_PASSWORD=${ELASTICSEARCH_PASSWORD:-changeme} 96 | # disable strict permission checks 97 | command: ["--strict.perms=false"] 98 | networks: 99 | - net 100 | 101 | # ----------------------------------------------------------------------------- 102 | kibana: 103 | # ----------------------------------------------------------------------------- 104 | image: docker.elastic.co/kibana/kibana:7.3.2 105 | hostname: kibana 106 | depends_on: 107 | - elasticsearch 108 | environment: 109 | elasticsearch.url: "http://elasticsearch:9200" 110 | networks: 111 | - net 112 | 113 | # ----------------------------------------------------------------------------- 114 | prometheus: 115 | # ----------------------------------------------------------------------------- 116 | image: prom/prometheus:v2.12.0 117 | hostname: prometheus 118 | volumes: 119 | - ./prometheus/:/etc/prometheus/ 120 | - prometheus:/prometheus 121 | command: 122 | - '--config.file=/etc/prometheus/prometheus.yml' 123 | - '--storage.tsdb.path=/prometheus' 124 | - '--web.console.libraries=/etc/prometheus/console_libraries' 125 | - '--web.console.templates=/etc/prometheus/consoles' 126 | - '--storage.tsdb.retention.time=200h' 127 | - '--web.enable-lifecycle' 128 | networks: 129 | - net 130 | 131 | # ----------------------------------------------------------------------------- 132 | alertmanager: 133 | # ----------------------------------------------------------------------------- 134 | image: prom/alertmanager:v0.19.0 135 | hostname: alertmanager 136 | depends_on: 137 | - prometheus 138 | volumes: 139 | - ./alertmanager/:/etc/alertmanager/ 140 | command: 141 | - '--config.file=/etc/alertmanager/config.yml' 142 | - '--storage.path=/alertmanager' 143 | networks: 144 | - net 145 | 146 | # ----------------------------------------------------------------------------- 147 | nodeexporter: 148 | # ----------------------------------------------------------------------------- 149 | image: prom/node-exporter:v0.18.1 150 | hostname: nodeexporter 151 | user: root 152 | volumes: 153 | - /proc:/host/proc:ro 154 | - /sys:/host/sys:ro 155 | - /:/rootfs:ro 156 | command: 157 | - '--path.procfs=/host/proc' 158 | - '--path.rootfs=/rootfs' 159 | - '--path.sysfs=/host/sys' 160 | - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)' 161 | networks: 162 | - net 163 | # ----------------------------------------------------------------------------- 164 | cadvisor: 165 | # ----------------------------------------------------------------------------- 166 | image: google/cadvisor:v0.33.0 167 | hostname: cadvisor 168 | volumes: 169 | - /:/rootfs:ro 170 | - /var/run:/var/run:rw 171 | - /sys:/sys:ro 172 | - /var/lib/docker/:/var/lib/docker:ro 173 | - /cgroup:/cgroup:ro 174 | networks: 175 | - net 176 | 177 | # ----------------------------------------------------------------------------- 178 | grafana: 179 | # ----------------------------------------------------------------------------- 180 | image: grafana/grafana:6.3.5 181 | hostname: grafana 182 | volumes: 183 | - grafana:/var/lib/grafana 184 | - ./grafana/datasources:/etc/grafana/datasources 185 | - ./grafana/dashboards:/etc/grafana/dashboards 186 | - ./grafana/setup.sh:/setup.sh 187 | entrypoint: /setup.sh 188 | environment: 189 | - GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin} 190 | - GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin} 191 | - GF_USERS_ALLOW_SIGN_UP=false 192 | networks: 193 | - net 194 | 195 | # ----------------------------------------------------------------------------- 196 | pushgateway: 197 | # ----------------------------------------------------------------------------- 198 | image: prom/pushgateway:v0.9.1 199 | hostname: pushgateway 200 | networks: 201 | - net 202 | 203 | # ----------------------------------------------------------------------------- 204 | caddy: 205 | # ----------------------------------------------------------------------------- 206 | image: stefanprodan/caddy 207 | hostname: caddy 208 | ports: 209 | - "3000:3000" 210 | - "9090:9090" 211 | - "9093:9093" 212 | - "9091:9091" 213 | - "5601:5601" 214 | volumes: 215 | - ./caddy/:/etc/caddy/ 216 | environment: 217 | - ADMIN_USER=${ADMIN_USER:-admin} 218 | - ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin} 219 | networks: 220 | - net 221 | 222 | # ----------------------------------------------------------------------------- 223 | # END OF FILE 224 | # ----------------------------------------------------------------------------- -------------------------------------------------------------------------------- /docker/elasticsearch/.keep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/silveiralexf/monitoring-stack/d0d37baed30ddc017e4b94888d5b13880171085a/docker/elasticsearch/.keep -------------------------------------------------------------------------------- /docker/filebeat/config/filebeat.yml: -------------------------------------------------------------------------------- 1 | # https://github.com/elastic/beats/blob/master/filebeat/filebeat.reference.yml 2 | 3 | filebeat.modules: 4 | - module: system 5 | syslog: 6 | enabled: true 7 | auth: 8 | enabled: true 9 | - module: auditd 10 | log: 11 | # Does not look like Auditd is supported in Alpine linux: https://github.com/linuxkit/linuxkit/issues/52 12 | enabled: false 13 | 14 | filebeat.inputs: 15 | - type: docker 16 | enabled: true 17 | containers: 18 | stream: all # can be all, stdout or stderr 19 | ids: 20 | - '*' 21 | # exclude_lines: ["^\\s+[\\-`('.|_]"] # drop asciiart lines 22 | # multiline.pattern: "^\t|^[[:space:]]+(at|...)|^Caused by:" 23 | # multiline.match: after 24 | 25 | #========================== Filebeat autodiscover ============================== 26 | # See this URL on how to run Apache2 Filebeat module: # https://www.elastic.co/guide/en/beats/filebeat/current/running-on-docker.html 27 | filebeat.autodiscover: 28 | providers: 29 | - type: docker 30 | # https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html 31 | # This URL alos contains instructions on multi-line logs 32 | hints.enabled: true 33 | 34 | #================================ Processors =================================== 35 | processors: 36 | - add_cloud_metadata: ~ 37 | - add_docker_metadata: ~ 38 | - add_locale: 39 | format: offset 40 | - add_host_metadata: 41 | netinfo.enabled: true 42 | 43 | #========================== Elasticsearch output =============================== 44 | output.elasticsearch: 45 | hosts: ["elasticsearch:9200"] 46 | # username: ${ELASTICSEARCH_USERNAME} 47 | # password: ${ELASTICSEARCH_PASSWORD} 48 | 49 | #============================== Dashboards ===================================== 50 | setup.dashboards: 51 | enabled: true 52 | 53 | #============================== Kibana ========================================= 54 | setup.kibana: 55 | host: "kibana:5601" 56 | # username: ${ELASTICSEARCH_USERNAME} 57 | # password: ${ELASTICSEARCH_PASSWORD} 58 | 59 | #============================== Xpack Monitoring =============================== 60 | xpack.monitoring: 61 | enabled: false 62 | elasticsearch: -------------------------------------------------------------------------------- /docker/filebeat/data/.keep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/silveiralexf/monitoring-stack/d0d37baed30ddc017e4b94888d5b13880171085a/docker/filebeat/data/.keep -------------------------------------------------------------------------------- /docker/grafana/dashboards/docker_containers.json: -------------------------------------------------------------------------------- 1 | {"dashboard": { 2 | "id": null, 3 | "title": "Docker Containers", 4 | "description": "Containers metrics", 5 | "tags": [ 6 | "docker" 7 | ], 8 | "style": "dark", 9 | "timezone": "browser", 10 | "editable": true, 11 | "hideControls": false, 12 | "sharedCrosshair": true, 13 | "rows": [ 14 | { 15 | "collapse": false, 16 | "editable": true, 17 | "height": "150px", 18 | "panels": [ 19 | { 20 | "cacheTimeout": null, 21 | "colorBackground": false, 22 | "colorValue": false, 23 | "colors": [ 24 | "rgba(50, 172, 45, 0.97)", 25 | "rgba(237, 129, 40, 0.89)", 26 | "rgba(245, 54, 54, 0.9)" 27 | ], 28 | "datasource": "Prometheus", 29 | "decimals": 2, 30 | "editable": true, 31 | "error": false, 32 | "format": "percent", 33 | "gauge": { 34 | "maxValue": 100, 35 | "minValue": 0, 36 | "show": true, 37 | "thresholdLabels": false, 38 | "thresholdMarkers": true 39 | }, 40 | "id": 4, 41 | "interval": null, 42 | "isNew": true, 43 | "links": [], 44 | "mappingType": 1, 45 | "mappingTypes": [ 46 | { 47 | "name": "value to text", 48 | "value": 1 49 | }, 50 | { 51 | "name": "range to text", 52 | "value": 2 53 | } 54 | ], 55 | "maxDataPoints": 100, 56 | "nullPointMode": "connected", 57 | "nullText": null, 58 | "postfix": "", 59 | "postfixFontSize": "50%", 60 | "prefix": "", 61 | "prefixFontSize": "50%", 62 | "rangeMaps": [ 63 | { 64 | "from": "null", 65 | "text": "N/A", 66 | "to": "null" 67 | } 68 | ], 69 | "span": 2, 70 | "sparkline": { 71 | "fillColor": "rgba(31, 118, 189, 0.18)", 72 | "full": false, 73 | "lineColor": "rgb(31, 120, 193)", 74 | "show": false 75 | }, 76 | "targets": [ 77 | { 78 | "expr": "sum(rate(container_cpu_user_seconds_total{image!=\"\"}[1m])) / count(node_cpu_seconds_total{mode=\"user\"}) * 100", 79 | "interval": "10s", 80 | "intervalFactor": 1, 81 | "legendFormat": "", 82 | "refId": "A", 83 | "step": 10 84 | } 85 | ], 86 | "thresholds": "65, 90", 87 | "title": "CPU Load", 88 | "transparent": false, 89 | "type": "singlestat", 90 | "valueFontSize": "80%", 91 | "valueMaps": [ 92 | { 93 | "op": "=", 94 | "text": "N/A", 95 | "value": "null" 96 | } 97 | ], 98 | "valueName": "avg", 99 | "timeFrom": "10s", 100 | "hideTimeOverride": true 101 | }, 102 | { 103 | "cacheTimeout": null, 104 | "colorBackground": false, 105 | "colorValue": false, 106 | "colors": [ 107 | "rgba(245, 54, 54, 0.9)", 108 | "rgba(237, 129, 40, 0.89)", 109 | "rgba(50, 172, 45, 0.97)" 110 | ], 111 | "datasource": "Prometheus", 112 | "editable": true, 113 | "error": false, 114 | "format": "none", 115 | "gauge": { 116 | "maxValue": 100, 117 | "minValue": 0, 118 | "show": false, 119 | "thresholdLabels": false, 120 | "thresholdMarkers": true 121 | }, 122 | "id": 7, 123 | "interval": null, 124 | "isNew": true, 125 | "links": [], 126 | "mappingType": 1, 127 | "mappingTypes": [ 128 | { 129 | "name": "value to text", 130 | "value": 1 131 | }, 132 | { 133 | "name": "range to text", 134 | "value": 2 135 | } 136 | ], 137 | "maxDataPoints": 100, 138 | "nullPointMode": "connected", 139 | "nullText": null, 140 | "postfix": "", 141 | "postfixFontSize": "50%", 142 | "prefix": "", 143 | "prefixFontSize": "50%", 144 | "rangeMaps": [ 145 | { 146 | "from": "null", 147 | "text": "N/A", 148 | "to": "null" 149 | } 150 | ], 151 | "span": 2, 152 | "sparkline": { 153 | "fillColor": "rgba(31, 118, 189, 0.18)", 154 | "full": false, 155 | "lineColor": "rgb(31, 120, 193)", 156 | "show": false 157 | }, 158 | "targets": [ 159 | { 160 | "expr": "machine_cpu_cores", 161 | "interval": "", 162 | "intervalFactor": 2, 163 | "legendFormat": "", 164 | "metric": "machine_cpu_cores", 165 | "refId": "A", 166 | "step": 20 167 | } 168 | ], 169 | "thresholds": "", 170 | "title": "CPU Cores", 171 | "type": "singlestat", 172 | "valueFontSize": "80%", 173 | "valueMaps": [ 174 | { 175 | "op": "=", 176 | "text": "N/A", 177 | "value": "null" 178 | } 179 | ], 180 | "valueName": "avg" 181 | }, 182 | { 183 | "cacheTimeout": null, 184 | "colorBackground": false, 185 | "colorValue": false, 186 | "colors": [ 187 | "rgba(50, 172, 45, 0.97)", 188 | "rgba(237, 129, 40, 0.89)", 189 | "rgba(245, 54, 54, 0.9)" 190 | ], 191 | "datasource": "Prometheus", 192 | "editable": true, 193 | "error": false, 194 | "format": "percent", 195 | "gauge": { 196 | "maxValue": 100, 197 | "minValue": 0, 198 | "show": true, 199 | "thresholdLabels": false, 200 | "thresholdMarkers": true 201 | }, 202 | "id": 5, 203 | "interval": null, 204 | "isNew": true, 205 | "links": [], 206 | "mappingType": 1, 207 | "mappingTypes": [ 208 | { 209 | "name": "value to text", 210 | "value": 1 211 | }, 212 | { 213 | "name": "range to text", 214 | "value": 2 215 | } 216 | ], 217 | "maxDataPoints": 100, 218 | "nullPointMode": "connected", 219 | "nullText": null, 220 | "postfix": "", 221 | "postfixFontSize": "50%", 222 | "prefix": "", 223 | "prefixFontSize": "50%", 224 | "rangeMaps": [ 225 | { 226 | "from": "null", 227 | "text": "N/A", 228 | "to": "null" 229 | } 230 | ], 231 | "span": 2, 232 | "sparkline": { 233 | "fillColor": "rgba(31, 118, 189, 0.18)", 234 | "full": false, 235 | "lineColor": "rgb(31, 120, 193)", 236 | "show": false 237 | }, 238 | "targets": [ 239 | { 240 | "expr": "(sum(node_memory_MemTotal_bytes) - sum(node_memory_MemFree_bytes+node_memory_Buffers_bytes+node_memory_Cached_bytes) ) / sum(node_memory_MemTotal_bytes) * 100", 241 | "interval": "10s", 242 | "intervalFactor": 2, 243 | "legendFormat": "", 244 | "refId": "A", 245 | "step": 20 246 | } 247 | ], 248 | "thresholds": "65, 90", 249 | "title": "Memory Load", 250 | "transparent": false, 251 | "type": "singlestat", 252 | "valueFontSize": "80%", 253 | "valueMaps": [ 254 | { 255 | "op": "=", 256 | "text": "N/A", 257 | "value": "null" 258 | } 259 | ], 260 | "valueName": "avg", 261 | "timeFrom": "10s", 262 | "hideTimeOverride": true 263 | }, 264 | { 265 | "cacheTimeout": null, 266 | "colorBackground": false, 267 | "colorValue": false, 268 | "colors": [ 269 | "rgba(245, 54, 54, 0.9)", 270 | "rgba(237, 129, 40, 0.89)", 271 | "rgba(50, 172, 45, 0.97)" 272 | ], 273 | "datasource": "Prometheus", 274 | "decimals": 2, 275 | "editable": true, 276 | "error": false, 277 | "format": "bytes", 278 | "gauge": { 279 | "maxValue": 100, 280 | "minValue": 0, 281 | "show": false, 282 | "thresholdLabels": false, 283 | "thresholdMarkers": true 284 | }, 285 | "id": 2, 286 | "interval": null, 287 | "isNew": true, 288 | "links": [], 289 | "mappingType": 1, 290 | "mappingTypes": [ 291 | { 292 | "name": "value to text", 293 | "value": 1 294 | }, 295 | { 296 | "name": "range to text", 297 | "value": 2 298 | } 299 | ], 300 | "maxDataPoints": 100, 301 | "nullPointMode": "connected", 302 | "nullText": null, 303 | "postfix": "", 304 | "postfixFontSize": "50%", 305 | "prefix": "", 306 | "prefixFontSize": "50%", 307 | "rangeMaps": [ 308 | { 309 | "from": "null", 310 | "text": "N/A", 311 | "to": "null" 312 | } 313 | ], 314 | "span": 2, 315 | "sparkline": { 316 | "fillColor": "rgba(31, 118, 189, 0.18)", 317 | "full": false, 318 | "lineColor": "rgb(31, 120, 193)", 319 | "show": false 320 | }, 321 | "targets": [ 322 | { 323 | "expr": "sum(container_memory_usage_bytes{image!=\"\"})", 324 | "interval": "10s", 325 | "intervalFactor": 2, 326 | "legendFormat": "", 327 | "refId": "A", 328 | "step": 20 329 | } 330 | ], 331 | "thresholds": "", 332 | "timeFrom": "10s", 333 | "title": "Used Memory", 334 | "transparent": false, 335 | "type": "singlestat", 336 | "valueFontSize": "80%", 337 | "valueMaps": [ 338 | { 339 | "op": "=", 340 | "text": "N/A", 341 | "value": "null" 342 | } 343 | ], 344 | "valueName": "avg", 345 | "hideTimeOverride": true 346 | }, 347 | { 348 | "cacheTimeout": null, 349 | "colorBackground": false, 350 | "colorValue": false, 351 | "colors": [ 352 | "rgba(50, 172, 45, 0.97)", 353 | "rgba(237, 129, 40, 0.89)", 354 | "rgba(245, 54, 54, 0.9)" 355 | ], 356 | "datasource": "Prometheus", 357 | "decimals": null, 358 | "editable": true, 359 | "error": false, 360 | "format": "percent", 361 | "gauge": { 362 | "maxValue": 100, 363 | "minValue": 0, 364 | "show": true, 365 | "thresholdLabels": false, 366 | "thresholdMarkers": true 367 | }, 368 | "id": 6, 369 | "interval": null, 370 | "isNew": true, 371 | "links": [], 372 | "mappingType": 1, 373 | "mappingTypes": [ 374 | { 375 | "name": "value to text", 376 | "value": 1 377 | }, 378 | { 379 | "name": "range to text", 380 | "value": 2 381 | } 382 | ], 383 | "maxDataPoints": 100, 384 | "nullPointMode": "connected", 385 | "nullText": null, 386 | "postfix": "", 387 | "postfixFontSize": "50%", 388 | "prefix": "", 389 | "prefixFontSize": "50%", 390 | "rangeMaps": [ 391 | { 392 | "from": "null", 393 | "text": "N/A", 394 | "to": "null" 395 | } 396 | ], 397 | "span": 2, 398 | "sparkline": { 399 | "fillColor": "rgba(31, 118, 189, 0.18)", 400 | "full": false, 401 | "lineColor": "rgb(31, 120, 193)", 402 | "show": false 403 | }, 404 | "targets": [ 405 | { 406 | "expr": "(node_filesystem_size_bytes{fstype=\"aufs\"} - node_filesystem_free_bytes{fstype=\"aufs\"}) / node_filesystem_size_bytes{fstype=\"aufs\"} * 100", 407 | "interval": "30s", 408 | "intervalFactor": 1, 409 | "legendFormat": "", 410 | "refId": "A", 411 | "step": 30 412 | } 413 | ], 414 | "thresholds": "65, 90", 415 | "title": "Storage Load", 416 | "transparent": false, 417 | "type": "singlestat", 418 | "valueFontSize": "80%", 419 | "valueMaps": [ 420 | { 421 | "op": "=", 422 | "text": "N/A", 423 | "value": "null" 424 | } 425 | ], 426 | "valueName": "avg", 427 | "timeFrom": "10s", 428 | "hideTimeOverride": true 429 | }, 430 | { 431 | "cacheTimeout": null, 432 | "colorBackground": false, 433 | "colorValue": false, 434 | "colors": [ 435 | "rgba(245, 54, 54, 0.9)", 436 | "rgba(237, 129, 40, 0.89)", 437 | "rgba(50, 172, 45, 0.97)" 438 | ], 439 | "datasource": "Prometheus", 440 | "decimals": 2, 441 | "editable": true, 442 | "error": false, 443 | "format": "bytes", 444 | "gauge": { 445 | "maxValue": 100, 446 | "minValue": 0, 447 | "show": false, 448 | "thresholdLabels": false, 449 | "thresholdMarkers": true 450 | }, 451 | "id": 3, 452 | "interval": null, 453 | "isNew": true, 454 | "links": [], 455 | "mappingType": 1, 456 | "mappingTypes": [ 457 | { 458 | "name": "value to text", 459 | "value": 1 460 | }, 461 | { 462 | "name": "range to text", 463 | "value": 2 464 | } 465 | ], 466 | "maxDataPoints": 100, 467 | "nullPointMode": "connected", 468 | "nullText": null, 469 | "postfix": "", 470 | "postfixFontSize": "50%", 471 | "prefix": "", 472 | "prefixFontSize": "50%", 473 | "rangeMaps": [ 474 | { 475 | "from": "null", 476 | "text": "N/A", 477 | "to": "null" 478 | } 479 | ], 480 | "span": 2, 481 | "sparkline": { 482 | "fillColor": "rgba(31, 118, 189, 0.18)", 483 | "full": false, 484 | "lineColor": "rgb(31, 120, 193)", 485 | "show": false 486 | }, 487 | "targets": [ 488 | { 489 | "expr": "sum(container_fs_usage_bytes)", 490 | "interval": "30s", 491 | "intervalFactor": 2, 492 | "refId": "A", 493 | "step": 60 494 | } 495 | ], 496 | "thresholds": "", 497 | "title": "Used Storage", 498 | "transparent": false, 499 | "type": "singlestat", 500 | "valueFontSize": "80%", 501 | "valueMaps": [ 502 | { 503 | "op": "=", 504 | "text": "N/A", 505 | "value": "null" 506 | } 507 | ], 508 | "valueName": "avg", 509 | "timeFrom": "10s", 510 | "hideTimeOverride": true 511 | } 512 | ], 513 | "title": "Overview" 514 | }, 515 | { 516 | "collapse": false, 517 | "editable": true, 518 | "height": "150px", 519 | "panels": [ 520 | { 521 | "aliasColors": {}, 522 | "bars": true, 523 | "datasource": "Prometheus", 524 | "decimals": 0, 525 | "editable": true, 526 | "error": false, 527 | "fill": 1, 528 | "grid": { 529 | "threshold1": null, 530 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 531 | "threshold2": null, 532 | "threshold2Color": "rgba(234, 112, 112, 0.22)", 533 | "thresholdLine": false 534 | }, 535 | "id": 9, 536 | "isNew": true, 537 | "legend": { 538 | "avg": false, 539 | "current": false, 540 | "max": false, 541 | "min": false, 542 | "show": false, 543 | "total": false, 544 | "values": false 545 | }, 546 | "lines": false, 547 | "linewidth": 2, 548 | "links": [], 549 | "nullPointMode": "connected", 550 | "percentage": false, 551 | "pointradius": 5, 552 | "points": false, 553 | "renderer": "flot", 554 | "seriesOverrides": [], 555 | "span": 4, 556 | "stack": false, 557 | "steppedLine": false, 558 | "targets": [ 559 | { 560 | "expr": "scalar(count(container_memory_usage_bytes{image!=\"\"}) > 0)", 561 | "interval": "", 562 | "intervalFactor": 2, 563 | "legendFormat": "containers", 564 | "refId": "A", 565 | "step": 2 566 | } 567 | ], 568 | "timeFrom": null, 569 | "timeShift": null, 570 | "title": "Running Containers", 571 | "tooltip": { 572 | "msResolution": true, 573 | "shared": true, 574 | "sort": 0, 575 | "value_type": "cumulative" 576 | }, 577 | "type": "graph", 578 | "xaxis": { 579 | "show": true 580 | }, 581 | "yaxes": [ 582 | { 583 | "format": "none", 584 | "label": "", 585 | "logBase": 1, 586 | "max": null, 587 | "min": 0, 588 | "show": true 589 | }, 590 | { 591 | "format": "short", 592 | "label": null, 593 | "logBase": 1, 594 | "max": null, 595 | "min": null, 596 | "show": false 597 | } 598 | ] 599 | }, 600 | { 601 | "aliasColors": {}, 602 | "bars": true, 603 | "datasource": "Prometheus", 604 | "decimals": 2, 605 | "editable": true, 606 | "error": false, 607 | "fill": 1, 608 | "grid": { 609 | "threshold1": null, 610 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 611 | "threshold2": null, 612 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 613 | }, 614 | "id": 10, 615 | "isNew": true, 616 | "legend": { 617 | "avg": false, 618 | "current": false, 619 | "max": false, 620 | "min": false, 621 | "show": false, 622 | "total": false, 623 | "values": false 624 | }, 625 | "lines": false, 626 | "linewidth": 2, 627 | "links": [], 628 | "nullPointMode": "connected", 629 | "percentage": false, 630 | "pointradius": 5, 631 | "points": false, 632 | "renderer": "flot", 633 | "seriesOverrides": [ 634 | { 635 | "alias": "load 1m", 636 | "color": "#BF1B00" 637 | } 638 | ], 639 | "span": 4, 640 | "stack": false, 641 | "steppedLine": false, 642 | "targets": [ 643 | { 644 | "expr": "node_load1", 645 | "interval": "", 646 | "intervalFactor": 2, 647 | "legendFormat": "load 1m", 648 | "metric": "node_load1", 649 | "refId": "A", 650 | "step": 2 651 | } 652 | ], 653 | "timeFrom": null, 654 | "timeShift": null, 655 | "title": "System Load", 656 | "tooltip": { 657 | "msResolution": true, 658 | "shared": true, 659 | "sort": 0, 660 | "value_type": "cumulative" 661 | }, 662 | "type": "graph", 663 | "xaxis": { 664 | "show": true 665 | }, 666 | "yaxes": [ 667 | { 668 | "format": "short", 669 | "label": null, 670 | "logBase": 1, 671 | "max": null, 672 | "min": 0, 673 | "show": true 674 | }, 675 | { 676 | "format": "short", 677 | "label": null, 678 | "logBase": 1, 679 | "max": null, 680 | "min": null, 681 | "show": false 682 | } 683 | ] 684 | }, 685 | { 686 | "aliasColors": {}, 687 | "bars": false, 688 | "datasource": "Prometheus", 689 | "editable": true, 690 | "error": false, 691 | "fill": 1, 692 | "grid": { 693 | "threshold1": null, 694 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 695 | "threshold2": null, 696 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 697 | }, 698 | "id": 15, 699 | "isNew": true, 700 | "legend": { 701 | "alignAsTable": true, 702 | "avg": true, 703 | "current": false, 704 | "max": true, 705 | "min": true, 706 | "rightSide": true, 707 | "show": false, 708 | "total": false, 709 | "values": true 710 | }, 711 | "lines": true, 712 | "linewidth": 2, 713 | "links": [], 714 | "nullPointMode": "connected", 715 | "percentage": false, 716 | "pointradius": 5, 717 | "points": false, 718 | "renderer": "flot", 719 | "seriesOverrides": [ 720 | { 721 | "alias": "read", 722 | "yaxis": 1 723 | }, 724 | { 725 | "alias": "written", 726 | "yaxis": 1 727 | }, 728 | { 729 | "alias": "io time", 730 | "yaxis": 2 731 | } 732 | ], 733 | "span": 4, 734 | "stack": false, 735 | "steppedLine": false, 736 | "targets": [ 737 | { 738 | "expr": "sum(irate(node_disk_read_bytes_total[5m]))", 739 | "interval": "2s", 740 | "intervalFactor": 4, 741 | "legendFormat": "read", 742 | "metric": "", 743 | "refId": "A", 744 | "step": 8 745 | }, 746 | { 747 | "expr": "sum(irate(node_disk_written_bytes_total[5m]))", 748 | "interval": "2s", 749 | "intervalFactor": 4, 750 | "legendFormat": "written", 751 | "metric": "", 752 | "refId": "B", 753 | "step": 8 754 | }, 755 | { 756 | "expr": "sum(irate(node_disk_io_time_seconds_total[5m]))", 757 | "interval": "2s", 758 | "intervalFactor": 4, 759 | "legendFormat": "io time", 760 | "metric": "", 761 | "refId": "C", 762 | "step": 8 763 | } 764 | ], 765 | "timeFrom": null, 766 | "timeShift": null, 767 | "title": "I/O Usage", 768 | "tooltip": { 769 | "msResolution": true, 770 | "shared": true, 771 | "sort": 0, 772 | "value_type": "cumulative" 773 | }, 774 | "type": "graph", 775 | "xaxis": { 776 | "show": true 777 | }, 778 | "yaxes": [ 779 | { 780 | "format": "bytes", 781 | "label": null, 782 | "logBase": 1, 783 | "max": null, 784 | "min": null, 785 | "show": true 786 | }, 787 | { 788 | "format": "ms", 789 | "label": null, 790 | "logBase": 1, 791 | "max": null, 792 | "min": null, 793 | "show": true 794 | } 795 | ] 796 | } 797 | ], 798 | "title": "Host stats" 799 | }, 800 | { 801 | "collapse": false, 802 | "editable": true, 803 | "height": "250px", 804 | "panels": [ 805 | { 806 | "aliasColors": {}, 807 | "bars": false, 808 | "datasource": "Prometheus", 809 | "decimals": 2, 810 | "editable": true, 811 | "error": false, 812 | "fill": 1, 813 | "grid": { 814 | "threshold1": null, 815 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 816 | "threshold2": null, 817 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 818 | }, 819 | "id": 8, 820 | "isNew": true, 821 | "legend": { 822 | "alignAsTable": true, 823 | "avg": true, 824 | "current": false, 825 | "max": true, 826 | "min": true, 827 | "rightSide": true, 828 | "show": true, 829 | "total": false, 830 | "values": true 831 | }, 832 | "lines": true, 833 | "linewidth": 2, 834 | "links": [], 835 | "nullPointMode": "connected", 836 | "percentage": false, 837 | "pointradius": 5, 838 | "points": false, 839 | "renderer": "flot", 840 | "seriesOverrides": [], 841 | "span": 12, 842 | "stack": false, 843 | "steppedLine": false, 844 | "targets": [ 845 | { 846 | "expr": "sum by (name) (rate(container_cpu_usage_seconds_total{image!=\"\",container_label_org_label_schema_group=\"\"}[1m])) / scalar(count(node_cpu_seconds_total{mode=\"user\"})) * 100", 847 | "intervalFactor": 10, 848 | "legendFormat": "{{ name }}", 849 | "metric": "container_cpu_user_seconds_total", 850 | "refId": "A", 851 | "step": 10 852 | } 853 | ], 854 | "timeFrom": null, 855 | "timeShift": null, 856 | "title": "Container CPU Usage", 857 | "tooltip": { 858 | "msResolution": true, 859 | "shared": true, 860 | "sort": 2, 861 | "value_type": "cumulative" 862 | }, 863 | "type": "graph", 864 | "xaxis": { 865 | "show": true 866 | }, 867 | "yaxes": [ 868 | { 869 | "format": "percent", 870 | "label": null, 871 | "logBase": 1, 872 | "max": null, 873 | "min": 0, 874 | "show": true 875 | }, 876 | { 877 | "format": "short", 878 | "label": null, 879 | "logBase": 1, 880 | "max": null, 881 | "min": null, 882 | "show": false 883 | } 884 | ] 885 | } 886 | ], 887 | "title": "CPU" 888 | }, 889 | { 890 | "collapse": false, 891 | "editable": true, 892 | "height": "250px", 893 | "panels": [ 894 | { 895 | "aliasColors": {}, 896 | "bars": false, 897 | "datasource": "Prometheus", 898 | "decimals": 2, 899 | "editable": true, 900 | "error": false, 901 | "fill": 1, 902 | "grid": { 903 | "threshold1": null, 904 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 905 | "threshold2": null, 906 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 907 | }, 908 | "id": 11, 909 | "isNew": true, 910 | "legend": { 911 | "alignAsTable": true, 912 | "avg": true, 913 | "current": false, 914 | "max": true, 915 | "min": true, 916 | "rightSide": true, 917 | "show": true, 918 | "total": false, 919 | "values": true 920 | }, 921 | "lines": true, 922 | "linewidth": 2, 923 | "links": [], 924 | "nullPointMode": "connected", 925 | "percentage": false, 926 | "pointradius": 5, 927 | "points": false, 928 | "renderer": "flot", 929 | "seriesOverrides": [], 930 | "span": 12, 931 | "stack": false, 932 | "steppedLine": false, 933 | "targets": [ 934 | { 935 | "expr": "sum by (name)(container_memory_usage_bytes{image!=\"\",container_label_org_label_schema_group=\"\"})", 936 | "intervalFactor": 1, 937 | "legendFormat": "{{ name }}", 938 | "metric": "container_memory_usage", 939 | "refId": "A", 940 | "step": 1 941 | } 942 | ], 943 | "timeFrom": null, 944 | "timeShift": null, 945 | "title": "Container Memory Usage", 946 | "tooltip": { 947 | "msResolution": true, 948 | "shared": true, 949 | "sort": 0, 950 | "value_type": "cumulative" 951 | }, 952 | "type": "graph", 953 | "xaxis": { 954 | "show": true 955 | }, 956 | "yaxes": [ 957 | { 958 | "format": "bytes", 959 | "label": null, 960 | "logBase": 1, 961 | "max": null, 962 | "min": 0, 963 | "show": true 964 | }, 965 | { 966 | "format": "short", 967 | "label": null, 968 | "logBase": 1, 969 | "max": null, 970 | "min": null, 971 | "show": false 972 | } 973 | ] 974 | }, 975 | { 976 | "aliasColors": {}, 977 | "bars": false, 978 | "datasource": "Prometheus", 979 | "decimals": 2, 980 | "editable": true, 981 | "error": false, 982 | "fill": 1, 983 | "grid": { 984 | "threshold1": null, 985 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 986 | "threshold2": null, 987 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 988 | }, 989 | "id": 12, 990 | "isNew": true, 991 | "legend": { 992 | "alignAsTable": true, 993 | "avg": true, 994 | "current": false, 995 | "max": true, 996 | "min": true, 997 | "rightSide": true, 998 | "show": true, 999 | "total": false, 1000 | "values": true 1001 | }, 1002 | "lines": true, 1003 | "linewidth": 2, 1004 | "links": [], 1005 | "nullPointMode": "connected", 1006 | "percentage": false, 1007 | "pointradius": 5, 1008 | "points": false, 1009 | "renderer": "flot", 1010 | "seriesOverrides": [], 1011 | "span": 12, 1012 | "stack": false, 1013 | "steppedLine": false, 1014 | "targets": [ 1015 | { 1016 | "expr": "sum by (name) (container_memory_cache{image!=\"\",container_label_org_label_schema_group=\"\"})", 1017 | "intervalFactor": 2, 1018 | "legendFormat": "{{name}}", 1019 | "metric": "container_memory_cache", 1020 | "refId": "A", 1021 | "step": 2 1022 | } 1023 | ], 1024 | "timeFrom": null, 1025 | "timeShift": null, 1026 | "title": "Container Cached Memory Usage", 1027 | "tooltip": { 1028 | "msResolution": true, 1029 | "shared": true, 1030 | "sort": 0, 1031 | "value_type": "cumulative" 1032 | }, 1033 | "type": "graph", 1034 | "xaxis": { 1035 | "show": true 1036 | }, 1037 | "yaxes": [ 1038 | { 1039 | "format": "bytes", 1040 | "label": null, 1041 | "logBase": 1, 1042 | "max": null, 1043 | "min": 0, 1044 | "show": true 1045 | }, 1046 | { 1047 | "format": "short", 1048 | "label": null, 1049 | "logBase": 1, 1050 | "max": null, 1051 | "min": null, 1052 | "show": false 1053 | } 1054 | ] 1055 | } 1056 | ], 1057 | "title": "Memory" 1058 | }, 1059 | { 1060 | "collapse": false, 1061 | "editable": true, 1062 | "height": "250px", 1063 | "panels": [ 1064 | { 1065 | "aliasColors": {}, 1066 | "bars": false, 1067 | "datasource": "Prometheus", 1068 | "decimals": 2, 1069 | "editable": true, 1070 | "error": false, 1071 | "fill": 1, 1072 | "grid": { 1073 | "threshold1": null, 1074 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 1075 | "threshold2": null, 1076 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 1077 | }, 1078 | "id": 13, 1079 | "isNew": true, 1080 | "legend": { 1081 | "alignAsTable": true, 1082 | "avg": true, 1083 | "current": false, 1084 | "max": true, 1085 | "min": true, 1086 | "rightSide": true, 1087 | "show": true, 1088 | "total": false, 1089 | "values": true 1090 | }, 1091 | "lines": true, 1092 | "linewidth": 2, 1093 | "links": [], 1094 | "nullPointMode": "connected", 1095 | "percentage": false, 1096 | "pointradius": 5, 1097 | "points": false, 1098 | "renderer": "flot", 1099 | "seriesOverrides": [], 1100 | "span": 12, 1101 | "stack": false, 1102 | "steppedLine": false, 1103 | "targets": [ 1104 | { 1105 | "expr": "sum by (name) (rate(container_network_receive_bytes_total{image!=\"\",container_label_org_label_schema_group=\"\"}[1m]))", 1106 | "intervalFactor": 10, 1107 | "legendFormat": "{{ name }}", 1108 | "metric": "container_network_receive_bytes_total", 1109 | "refId": "A", 1110 | "step": 10 1111 | } 1112 | ], 1113 | "timeFrom": null, 1114 | "timeShift": null, 1115 | "title": "Container Network Input", 1116 | "tooltip": { 1117 | "msResolution": true, 1118 | "shared": true, 1119 | "sort": 2, 1120 | "value_type": "cumulative" 1121 | }, 1122 | "type": "graph", 1123 | "xaxis": { 1124 | "show": true 1125 | }, 1126 | "yaxes": [ 1127 | { 1128 | "format": "bytes", 1129 | "label": null, 1130 | "logBase": 1, 1131 | "max": null, 1132 | "min": 0, 1133 | "show": true 1134 | }, 1135 | { 1136 | "format": "short", 1137 | "label": null, 1138 | "logBase": 1, 1139 | "max": null, 1140 | "min": null, 1141 | "show": false 1142 | } 1143 | ] 1144 | }, 1145 | { 1146 | "aliasColors": {}, 1147 | "bars": false, 1148 | "datasource": "Prometheus", 1149 | "decimals": 2, 1150 | "editable": true, 1151 | "error": false, 1152 | "fill": 1, 1153 | "grid": { 1154 | "threshold1": null, 1155 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 1156 | "threshold2": null, 1157 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 1158 | }, 1159 | "id": 14, 1160 | "isNew": true, 1161 | "legend": { 1162 | "alignAsTable": true, 1163 | "avg": true, 1164 | "current": false, 1165 | "max": true, 1166 | "min": true, 1167 | "rightSide": true, 1168 | "show": true, 1169 | "total": false, 1170 | "values": true 1171 | }, 1172 | "lines": true, 1173 | "linewidth": 2, 1174 | "links": [], 1175 | "nullPointMode": "connected", 1176 | "percentage": false, 1177 | "pointradius": 5, 1178 | "points": false, 1179 | "renderer": "flot", 1180 | "seriesOverrides": [], 1181 | "span": 12, 1182 | "stack": false, 1183 | "steppedLine": false, 1184 | "targets": [ 1185 | { 1186 | "expr": "sum by (name) (rate(container_network_transmit_bytes_total{image!=\"\",container_label_org_label_schema_group=\"\"}[1m]))", 1187 | "intervalFactor": 10, 1188 | "legendFormat": "{{ name }}", 1189 | "metric": "container_network_transmit_bytes_total", 1190 | "refId": "A", 1191 | "step": 10 1192 | } 1193 | ], 1194 | "timeFrom": null, 1195 | "timeShift": null, 1196 | "title": "Container Network Output", 1197 | "tooltip": { 1198 | "msResolution": true, 1199 | "shared": true, 1200 | "sort": 2, 1201 | "value_type": "cumulative" 1202 | }, 1203 | "type": "graph", 1204 | "xaxis": { 1205 | "show": true 1206 | }, 1207 | "yaxes": [ 1208 | { 1209 | "format": "bytes", 1210 | "label": null, 1211 | "logBase": 1, 1212 | "max": null, 1213 | "min": 0, 1214 | "show": true 1215 | }, 1216 | { 1217 | "format": "short", 1218 | "label": null, 1219 | "logBase": 1, 1220 | "max": null, 1221 | "min": null, 1222 | "show": false 1223 | } 1224 | ] 1225 | } 1226 | ], 1227 | "title": "Network" 1228 | } 1229 | ], 1230 | "time": { 1231 | "from": "now-15m", 1232 | "to": "now" 1233 | }, 1234 | "timepicker": { 1235 | "refresh_intervals": [ 1236 | "5s", 1237 | "10s", 1238 | "30s", 1239 | "1m", 1240 | "5m", 1241 | "15m", 1242 | "30m", 1243 | "1h", 1244 | "2h", 1245 | "1d" 1246 | ], 1247 | "time_options": [ 1248 | "5m", 1249 | "15m", 1250 | "1h", 1251 | "6h", 1252 | "12h", 1253 | "24h", 1254 | "2d", 1255 | "7d", 1256 | "30d" 1257 | ] 1258 | }, 1259 | "templating": { 1260 | "list": [] 1261 | }, 1262 | "annotations": { 1263 | "list": [] 1264 | }, 1265 | "refresh": "10s", 1266 | "schemaVersion": 12, 1267 | "version": 8, 1268 | "links": [], 1269 | "gnetId": null 1270 | }} 1271 | -------------------------------------------------------------------------------- /docker/grafana/dashboards/docker_host.json: -------------------------------------------------------------------------------- 1 | {"dashboard": { 2 | "id": null, 3 | "title": "Docker Host", 4 | "description": "Docker host metrics", 5 | "tags": [ 6 | "system" 7 | ], 8 | "style": "dark", 9 | "timezone": "browser", 10 | "editable": true, 11 | "hideControls": false, 12 | "sharedCrosshair": true, 13 | "rows": [ 14 | { 15 | "collapse": false, 16 | "editable": true, 17 | "height": "100px", 18 | "panels": [ 19 | { 20 | "cacheTimeout": null, 21 | "colorBackground": false, 22 | "colorValue": false, 23 | "colors": [ 24 | "rgba(245, 54, 54, 0.9)", 25 | "rgba(237, 129, 40, 0.89)", 26 | "rgba(50, 172, 45, 0.97)" 27 | ], 28 | "datasource": "Prometheus", 29 | "decimals": 1, 30 | "editable": true, 31 | "error": false, 32 | "format": "s", 33 | "gauge": { 34 | "maxValue": 100, 35 | "minValue": 0, 36 | "show": false, 37 | "thresholdLabels": false, 38 | "thresholdMarkers": true 39 | }, 40 | "id": 1, 41 | "interval": null, 42 | "isNew": true, 43 | "links": [], 44 | "mappingType": 1, 45 | "mappingTypes": [ 46 | { 47 | "name": "value to text", 48 | "value": 1 49 | }, 50 | { 51 | "name": "range to text", 52 | "value": 2 53 | } 54 | ], 55 | "maxDataPoints": 100, 56 | "nullPointMode": "connected", 57 | "nullText": null, 58 | "postfix": "s", 59 | "postfixFontSize": "80%", 60 | "prefix": "", 61 | "prefixFontSize": "50%", 62 | "rangeMaps": [ 63 | { 64 | "from": "null", 65 | "text": "N/A", 66 | "to": "null" 67 | } 68 | ], 69 | "span": 2, 70 | "sparkline": { 71 | "fillColor": "rgba(31, 118, 189, 0.18)", 72 | "full": false, 73 | "lineColor": "rgb(31, 120, 193)", 74 | "show": false 75 | }, 76 | "targets": [ 77 | { 78 | "expr": "node_time_seconds - node_boot_time_seconds", 79 | "interval": "30s", 80 | "intervalFactor": 1, 81 | "refId": "A", 82 | "step": 30 83 | } 84 | ], 85 | "thresholds": "", 86 | "title": "Uptime", 87 | "type": "singlestat", 88 | "valueFontSize": "80%", 89 | "valueMaps": [ 90 | { 91 | "op": "=", 92 | "text": "N/A", 93 | "value": "null" 94 | } 95 | ], 96 | "valueName": "avg", 97 | "timeFrom": "10s", 98 | "hideTimeOverride": true 99 | }, 100 | { 101 | "cacheTimeout": null, 102 | "colorBackground": false, 103 | "colorValue": false, 104 | "colors": [ 105 | "rgba(245, 54, 54, 0.9)", 106 | "rgba(237, 129, 40, 0.89)", 107 | "rgba(50, 172, 45, 0.97)" 108 | ], 109 | "datasource": "Prometheus", 110 | "editable": true, 111 | "error": false, 112 | "format": "percent", 113 | "gauge": { 114 | "maxValue": 100, 115 | "minValue": 0, 116 | "show": false, 117 | "thresholdLabels": false, 118 | "thresholdMarkers": true 119 | }, 120 | "id": 13, 121 | "interval": null, 122 | "isNew": true, 123 | "links": [], 124 | "mappingType": 1, 125 | "mappingTypes": [ 126 | { 127 | "name": "value to text", 128 | "value": 1 129 | }, 130 | { 131 | "name": "range to text", 132 | "value": 2 133 | } 134 | ], 135 | "maxDataPoints": 100, 136 | "nullPointMode": "connected", 137 | "nullText": null, 138 | "postfix": "", 139 | "postfixFontSize": "50%", 140 | "prefix": "", 141 | "prefixFontSize": "50%", 142 | "rangeMaps": [ 143 | { 144 | "from": "null", 145 | "text": "N/A", 146 | "to": "null" 147 | } 148 | ], 149 | "span": 2, 150 | "sparkline": { 151 | "fillColor": "rgba(31, 118, 189, 0.18)", 152 | "full": false, 153 | "lineColor": "rgb(31, 120, 193)", 154 | "show": false 155 | }, 156 | "targets": [ 157 | { 158 | "expr": "sum(rate(node_cpu_seconds_total{mode=\"idle\"}[1m])) * 100 / scalar(count(node_cpu_seconds_total{mode=\"user\"}))", 159 | "interval": "10s", 160 | "intervalFactor": 2, 161 | "legendFormat": "", 162 | "refId": "A", 163 | "step": 20 164 | } 165 | ], 166 | "thresholds": "", 167 | "title": "CPU Idle", 168 | "type": "singlestat", 169 | "valueFontSize": "80%", 170 | "valueMaps": [ 171 | { 172 | "op": "=", 173 | "text": "N/A", 174 | "value": "null" 175 | } 176 | ], 177 | "valueName": "avg", 178 | "timeFrom": "10s", 179 | "hideTimeOverride": true 180 | }, 181 | { 182 | "cacheTimeout": null, 183 | "colorBackground": false, 184 | "colorValue": false, 185 | "colors": [ 186 | "rgba(245, 54, 54, 0.9)", 187 | "rgba(237, 129, 40, 0.89)", 188 | "rgba(50, 172, 45, 0.97)" 189 | ], 190 | "datasource": "Prometheus", 191 | "editable": true, 192 | "error": false, 193 | "format": "none", 194 | "gauge": { 195 | "maxValue": 100, 196 | "minValue": 0, 197 | "show": false, 198 | "thresholdLabels": false, 199 | "thresholdMarkers": true 200 | }, 201 | "id": 12, 202 | "interval": null, 203 | "isNew": true, 204 | "links": [], 205 | "mappingType": 1, 206 | "mappingTypes": [ 207 | { 208 | "name": "value to text", 209 | "value": 1 210 | }, 211 | { 212 | "name": "range to text", 213 | "value": 2 214 | } 215 | ], 216 | "maxDataPoints": 100, 217 | "nullPointMode": "connected", 218 | "nullText": null, 219 | "postfix": "", 220 | "postfixFontSize": "50%", 221 | "prefix": "", 222 | "prefixFontSize": "50%", 223 | "rangeMaps": [ 224 | { 225 | "from": "null", 226 | "text": "N/A", 227 | "to": "null" 228 | } 229 | ], 230 | "span": 2, 231 | "sparkline": { 232 | "fillColor": "rgba(31, 118, 189, 0.18)", 233 | "full": false, 234 | "lineColor": "rgb(31, 120, 193)", 235 | "show": false 236 | }, 237 | "targets": [ 238 | { 239 | "expr": "machine_cpu_cores", 240 | "intervalFactor": 2, 241 | "metric": "machine_cpu_cores", 242 | "refId": "A", 243 | "step": 2 244 | } 245 | ], 246 | "thresholds": "", 247 | "title": "CPU Cores", 248 | "type": "singlestat", 249 | "valueFontSize": "80%", 250 | "valueMaps": [ 251 | { 252 | "op": "=", 253 | "text": "N/A", 254 | "value": "null" 255 | } 256 | ], 257 | "valueName": "avg", 258 | "timeFrom": "10s", 259 | "hideTimeOverride": true 260 | }, 261 | { 262 | "cacheTimeout": null, 263 | "colorBackground": false, 264 | "colorValue": false, 265 | "colors": [ 266 | "rgba(245, 54, 54, 0.9)", 267 | "rgba(237, 129, 40, 0.89)", 268 | "rgba(50, 172, 45, 0.97)" 269 | ], 270 | "datasource": "Prometheus", 271 | "editable": true, 272 | "error": false, 273 | "format": "bytes", 274 | "gauge": { 275 | "maxValue": 100, 276 | "minValue": 0, 277 | "show": false, 278 | "thresholdLabels": false, 279 | "thresholdMarkers": true 280 | }, 281 | "id": 2, 282 | "interval": null, 283 | "isNew": true, 284 | "links": [], 285 | "mappingType": 1, 286 | "mappingTypes": [ 287 | { 288 | "name": "value to text", 289 | "value": 1 290 | }, 291 | { 292 | "name": "range to text", 293 | "value": 2 294 | } 295 | ], 296 | "maxDataPoints": 100, 297 | "nullPointMode": "connected", 298 | "nullText": null, 299 | "postfix": "", 300 | "postfixFontSize": "50%", 301 | "prefix": "", 302 | "prefixFontSize": "50%", 303 | "rangeMaps": [ 304 | { 305 | "from": "null", 306 | "text": "N/A", 307 | "to": "null" 308 | } 309 | ], 310 | "span": 2, 311 | "sparkline": { 312 | "fillColor": "rgba(31, 118, 189, 0.18)", 313 | "full": false, 314 | "lineColor": "rgb(31, 120, 193)", 315 | "show": false 316 | }, 317 | "targets": [ 318 | { 319 | "expr": "node_memory_MemAvailable_bytes", 320 | "interval": "30s", 321 | "intervalFactor": 2, 322 | "legendFormat": "", 323 | "refId": "A", 324 | "step": 60 325 | } 326 | ], 327 | "thresholds": "", 328 | "title": "Available Memory", 329 | "type": "singlestat", 330 | "valueFontSize": "80%", 331 | "valueMaps": [ 332 | { 333 | "op": "=", 334 | "text": "N/A", 335 | "value": "null" 336 | } 337 | ], 338 | "valueName": "avg", 339 | "timeFrom": "10s", 340 | "hideTimeOverride": true 341 | }, 342 | { 343 | "cacheTimeout": null, 344 | "colorBackground": false, 345 | "colorValue": false, 346 | "colors": [ 347 | "rgba(245, 54, 54, 0.9)", 348 | "rgba(237, 129, 40, 0.89)", 349 | "rgba(50, 172, 45, 0.97)" 350 | ], 351 | "datasource": "Prometheus", 352 | "editable": true, 353 | "error": false, 354 | "format": "bytes", 355 | "gauge": { 356 | "maxValue": 100, 357 | "minValue": 0, 358 | "show": false, 359 | "thresholdLabels": false, 360 | "thresholdMarkers": true 361 | }, 362 | "id": 3, 363 | "interval": null, 364 | "isNew": true, 365 | "links": [], 366 | "mappingType": 1, 367 | "mappingTypes": [ 368 | { 369 | "name": "value to text", 370 | "value": 1 371 | }, 372 | { 373 | "name": "range to text", 374 | "value": 2 375 | } 376 | ], 377 | "maxDataPoints": 100, 378 | "nullPointMode": "connected", 379 | "nullText": null, 380 | "postfix": "", 381 | "postfixFontSize": "50%", 382 | "prefix": "", 383 | "prefixFontSize": "50%", 384 | "rangeMaps": [ 385 | { 386 | "from": "null", 387 | "text": "N/A", 388 | "to": "null" 389 | } 390 | ], 391 | "span": 2, 392 | "sparkline": { 393 | "fillColor": "rgba(31, 118, 189, 0.18)", 394 | "full": false, 395 | "lineColor": "rgb(31, 120, 193)", 396 | "show": false 397 | }, 398 | "targets": [ 399 | { 400 | "expr": "node_memory_SwapFree_bytes", 401 | "interval": "30s", 402 | "intervalFactor": 2, 403 | "refId": "A", 404 | "step": 60 405 | } 406 | ], 407 | "thresholds": "", 408 | "title": "Free Swap", 409 | "type": "singlestat", 410 | "valueFontSize": "80%", 411 | "valueMaps": [ 412 | { 413 | "op": "=", 414 | "text": "N/A", 415 | "value": "null" 416 | } 417 | ], 418 | "valueName": "avg", 419 | "timeFrom": "10s", 420 | "hideTimeOverride": true 421 | }, 422 | { 423 | "cacheTimeout": null, 424 | "colorBackground": false, 425 | "colorValue": false, 426 | "colors": [ 427 | "rgba(245, 54, 54, 0.9)", 428 | "rgba(237, 129, 40, 0.89)", 429 | "rgba(50, 172, 45, 0.97)" 430 | ], 431 | "datasource": "Prometheus", 432 | "editable": true, 433 | "error": false, 434 | "format": "bytes", 435 | "gauge": { 436 | "maxValue": 100, 437 | "minValue": 0, 438 | "show": false, 439 | "thresholdLabels": false, 440 | "thresholdMarkers": true 441 | }, 442 | "id": 4, 443 | "interval": null, 444 | "isNew": true, 445 | "links": [], 446 | "mappingType": 1, 447 | "mappingTypes": [ 448 | { 449 | "name": "value to text", 450 | "value": 1 451 | }, 452 | { 453 | "name": "range to text", 454 | "value": 2 455 | } 456 | ], 457 | "maxDataPoints": 100, 458 | "nullPointMode": "connected", 459 | "nullText": null, 460 | "postfix": "", 461 | "postfixFontSize": "50%", 462 | "prefix": "", 463 | "prefixFontSize": "50%", 464 | "rangeMaps": [ 465 | { 466 | "from": "null", 467 | "text": "N/A", 468 | "to": "null" 469 | } 470 | ], 471 | "span": 2, 472 | "sparkline": { 473 | "fillColor": "rgba(31, 118, 189, 0.18)", 474 | "full": false, 475 | "lineColor": "rgb(31, 120, 193)", 476 | "show": false 477 | }, 478 | "targets": [ 479 | { 480 | "expr": "sum(node_filesystem_free_bytes{fstype=\"aufs\"})", 481 | "interval": "30s", 482 | "intervalFactor": 1, 483 | "legendFormat": "", 484 | "refId": "A", 485 | "step": 30 486 | } 487 | ], 488 | "thresholds": "", 489 | "title": "Free Storage", 490 | "type": "singlestat", 491 | "valueFontSize": "80%", 492 | "valueMaps": [ 493 | { 494 | "op": "=", 495 | "text": "N/A", 496 | "value": "null" 497 | } 498 | ], 499 | "valueName": "avg", 500 | "timeFrom": "10s", 501 | "hideTimeOverride": true 502 | } 503 | ], 504 | "title": "Available resources" 505 | }, 506 | { 507 | "collapse": false, 508 | "editable": true, 509 | "height": "150px", 510 | "panels": [ 511 | { 512 | "aliasColors": {}, 513 | "bars": true, 514 | "datasource": "Prometheus", 515 | "decimals": 2, 516 | "editable": true, 517 | "error": false, 518 | "fill": 1, 519 | "grid": { 520 | "threshold1": null, 521 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 522 | "threshold2": null, 523 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 524 | }, 525 | "id": 9, 526 | "isNew": true, 527 | "legend": { 528 | "avg": false, 529 | "current": false, 530 | "max": false, 531 | "min": false, 532 | "show": false, 533 | "total": false, 534 | "values": false 535 | }, 536 | "lines": false, 537 | "linewidth": 2, 538 | "links": [], 539 | "nullPointMode": "connected", 540 | "percentage": false, 541 | "pointradius": 5, 542 | "points": false, 543 | "renderer": "flot", 544 | "seriesOverrides": [ 545 | { 546 | "alias": "load 1m", 547 | "color": "#1F78C1" 548 | } 549 | ], 550 | "span": 4, 551 | "stack": false, 552 | "steppedLine": false, 553 | "targets": [ 554 | { 555 | "expr": "node_load1", 556 | "interval": "10s", 557 | "intervalFactor": 1, 558 | "legendFormat": "load 1m", 559 | "refId": "A", 560 | "step": 10 561 | } 562 | ], 563 | "timeFrom": null, 564 | "timeShift": null, 565 | "title": "Load Average 1m", 566 | "tooltip": { 567 | "msResolution": true, 568 | "shared": true, 569 | "sort": 0, 570 | "value_type": "cumulative" 571 | }, 572 | "type": "graph", 573 | "xaxis": { 574 | "show": true 575 | }, 576 | "yaxes": [ 577 | { 578 | "format": "short", 579 | "label": null, 580 | "logBase": 1, 581 | "max": null, 582 | "min": 0, 583 | "show": true 584 | }, 585 | { 586 | "format": "short", 587 | "label": null, 588 | "logBase": 1, 589 | "max": null, 590 | "min": null, 591 | "show": false 592 | } 593 | ] 594 | }, 595 | { 596 | "aliasColors": {}, 597 | "bars": true, 598 | "datasource": "Prometheus", 599 | "editable": true, 600 | "error": false, 601 | "fill": 1, 602 | "grid": { 603 | "threshold1": null, 604 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 605 | "threshold2": null, 606 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 607 | }, 608 | "id": 10, 609 | "isNew": true, 610 | "legend": { 611 | "avg": false, 612 | "current": false, 613 | "max": false, 614 | "min": false, 615 | "show": false, 616 | "total": false, 617 | "values": false 618 | }, 619 | "lines": false, 620 | "linewidth": 2, 621 | "links": [], 622 | "nullPointMode": "connected", 623 | "percentage": false, 624 | "pointradius": 5, 625 | "points": false, 626 | "renderer": "flot", 627 | "seriesOverrides": [ 628 | { 629 | "alias": "blocked by I/O", 630 | "color": "#58140C" 631 | } 632 | ], 633 | "span": 4, 634 | "stack": true, 635 | "steppedLine": false, 636 | "targets": [ 637 | { 638 | "expr": "node_procs_running", 639 | "interval": "10s", 640 | "intervalFactor": 1, 641 | "legendFormat": "running", 642 | "metric": "node_procs_running", 643 | "refId": "A", 644 | "step": 10 645 | }, 646 | { 647 | "expr": "node_procs_blocked", 648 | "interval": "10s", 649 | "intervalFactor": 1, 650 | "legendFormat": "blocked by I/O", 651 | "metric": "node_procs_blocked", 652 | "refId": "B", 653 | "step": 10 654 | } 655 | ], 656 | "timeFrom": null, 657 | "timeShift": null, 658 | "title": "Processes", 659 | "tooltip": { 660 | "msResolution": true, 661 | "shared": true, 662 | "sort": 2, 663 | "value_type": "individual" 664 | }, 665 | "type": "graph", 666 | "xaxis": { 667 | "show": true 668 | }, 669 | "yaxes": [ 670 | { 671 | "format": "short", 672 | "label": null, 673 | "logBase": 1, 674 | "max": null, 675 | "min": 0, 676 | "show": true 677 | }, 678 | { 679 | "format": "short", 680 | "label": null, 681 | "logBase": 1, 682 | "max": null, 683 | "min": null, 684 | "show": false 685 | } 686 | ] 687 | }, 688 | { 689 | "aliasColors": {}, 690 | "bars": true, 691 | "datasource": "Prometheus", 692 | "editable": true, 693 | "error": false, 694 | "fill": 1, 695 | "grid": { 696 | "threshold1": null, 697 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 698 | "threshold2": null, 699 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 700 | }, 701 | "id": 11, 702 | "isNew": true, 703 | "legend": { 704 | "avg": false, 705 | "current": false, 706 | "max": false, 707 | "min": false, 708 | "show": false, 709 | "total": false, 710 | "values": false 711 | }, 712 | "lines": false, 713 | "linewidth": 2, 714 | "links": [], 715 | "nullPointMode": "connected", 716 | "percentage": false, 717 | "pointradius": 5, 718 | "points": false, 719 | "renderer": "flot", 720 | "seriesOverrides": [ 721 | { 722 | "alias": "interrupts", 723 | "color": "#806EB7" 724 | } 725 | ], 726 | "span": 4, 727 | "stack": false, 728 | "steppedLine": false, 729 | "targets": [ 730 | { 731 | "expr": " irate(node_intr_total[5m])", 732 | "interval": "10s", 733 | "intervalFactor": 1, 734 | "legendFormat": "interrupts", 735 | "metric": "node_intr_total", 736 | "refId": "A", 737 | "step": 10 738 | } 739 | ], 740 | "timeFrom": null, 741 | "timeShift": null, 742 | "title": "Interrupts", 743 | "tooltip": { 744 | "msResolution": true, 745 | "shared": true, 746 | "sort": 0, 747 | "value_type": "cumulative" 748 | }, 749 | "type": "graph", 750 | "xaxis": { 751 | "show": true 752 | }, 753 | "yaxes": [ 754 | { 755 | "format": "short", 756 | "label": null, 757 | "logBase": 1, 758 | "max": null, 759 | "min": null, 760 | "show": true 761 | }, 762 | { 763 | "format": "short", 764 | "label": null, 765 | "logBase": 1, 766 | "max": null, 767 | "min": null, 768 | "show": false 769 | } 770 | ] 771 | } 772 | ], 773 | "title": "Load" 774 | }, 775 | { 776 | "collapse": false, 777 | "editable": true, 778 | "height": "250px", 779 | "panels": [ 780 | { 781 | "aliasColors": {}, 782 | "bars": false, 783 | "datasource": "Prometheus", 784 | "decimals": 2, 785 | "editable": true, 786 | "error": false, 787 | "fill": 4, 788 | "grid": { 789 | "threshold1": null, 790 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 791 | "threshold2": null, 792 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 793 | }, 794 | "id": 5, 795 | "isNew": true, 796 | "legend": { 797 | "alignAsTable": true, 798 | "avg": true, 799 | "current": false, 800 | "max": true, 801 | "min": true, 802 | "rightSide": true, 803 | "show": true, 804 | "total": false, 805 | "values": true 806 | }, 807 | "lines": true, 808 | "linewidth": 2, 809 | "links": [], 810 | "nullPointMode": "connected", 811 | "percentage": false, 812 | "pointradius": 5, 813 | "points": false, 814 | "renderer": "flot", 815 | "seriesOverrides": [], 816 | "span": 12, 817 | "stack": true, 818 | "steppedLine": false, 819 | "targets": [ 820 | { 821 | "expr": "sum(rate(node_cpu_seconds_total[1m])) by (mode) * 100 / scalar(count(node_cpu_seconds_total{mode=\"user\"}))", 822 | "intervalFactor": 10, 823 | "legendFormat": "{{ mode }}", 824 | "metric": "node_cpu_seconds_total", 825 | "refId": "A", 826 | "step": 10 827 | } 828 | ], 829 | "timeFrom": null, 830 | "timeShift": null, 831 | "title": "CPU Usage", 832 | "tooltip": { 833 | "msResolution": true, 834 | "shared": true, 835 | "sort": 2, 836 | "value_type": "individual" 837 | }, 838 | "type": "graph", 839 | "xaxis": { 840 | "show": true 841 | }, 842 | "yaxes": [ 843 | { 844 | "format": "percent", 845 | "label": null, 846 | "logBase": 1, 847 | "max": 100, 848 | "min": 0, 849 | "show": true 850 | }, 851 | { 852 | "format": "short", 853 | "label": null, 854 | "logBase": 1, 855 | "max": null, 856 | "min": 0, 857 | "show": true 858 | } 859 | ] 860 | } 861 | ], 862 | "title": "CPU" 863 | }, 864 | { 865 | "collapse": false, 866 | "editable": true, 867 | "height": "250px", 868 | "panels": [ 869 | { 870 | "aliasColors": {}, 871 | "bars": false, 872 | "datasource": "Prometheus", 873 | "decimals": 2, 874 | "editable": true, 875 | "error": false, 876 | "fill": 4, 877 | "grid": { 878 | "threshold1": null, 879 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 880 | "threshold2": null, 881 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 882 | }, 883 | "id": 6, 884 | "isNew": true, 885 | "legend": { 886 | "alignAsTable": true, 887 | "avg": true, 888 | "current": false, 889 | "max": true, 890 | "min": true, 891 | "rightSide": true, 892 | "show": true, 893 | "total": false, 894 | "values": true 895 | }, 896 | "lines": true, 897 | "linewidth": 2, 898 | "links": [], 899 | "nullPointMode": "null", 900 | "percentage": false, 901 | "pointradius": 5, 902 | "points": false, 903 | "renderer": "flot", 904 | "seriesOverrides": [ 905 | { 906 | "alias": "Used", 907 | "color": "#BF1B00" 908 | }, 909 | { 910 | "alias": "Free", 911 | "color": "#7EB26D" 912 | }, 913 | { 914 | "alias": "Buffers", 915 | "color": "#6ED0E0" 916 | }, 917 | { 918 | "alias": "Cached", 919 | "color": "#EF843C" 920 | } 921 | ], 922 | "span": 12, 923 | "stack": true, 924 | "steppedLine": false, 925 | "targets": [ 926 | { 927 | "expr": "node_memory_MemTotal_bytes - (node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes)", 928 | "intervalFactor": 1, 929 | "legendFormat": "Used", 930 | "refId": "A", 931 | "step": 1 932 | }, 933 | { 934 | "expr": "node_memory_MemFree_bytes", 935 | "intervalFactor": 1, 936 | "legendFormat": "Free", 937 | "refId": "B", 938 | "step": 1 939 | }, 940 | { 941 | "expr": "node_memory_Buffers_bytes", 942 | "intervalFactor": 1, 943 | "legendFormat": "Buffers", 944 | "refId": "C", 945 | "step": 1 946 | }, 947 | { 948 | "expr": "node_memory_Cached_bytes", 949 | "intervalFactor": 1, 950 | "legendFormat": "Cached", 951 | "refId": "D", 952 | "step": 1 953 | } 954 | ], 955 | "timeFrom": null, 956 | "timeShift": null, 957 | "title": "Memory Usage", 958 | "tooltip": { 959 | "msResolution": true, 960 | "shared": true, 961 | "sort": 2, 962 | "value_type": "individual" 963 | }, 964 | "type": "graph", 965 | "xaxis": { 966 | "show": true 967 | }, 968 | "yaxes": [ 969 | { 970 | "format": "bytes", 971 | "label": null, 972 | "logBase": 1, 973 | "max": null, 974 | "min": null, 975 | "show": true 976 | }, 977 | { 978 | "format": "short", 979 | "label": null, 980 | "logBase": 1, 981 | "max": null, 982 | "min": null, 983 | "show": true 984 | } 985 | ] 986 | } 987 | ], 988 | "title": "Memory" 989 | }, 990 | { 991 | "collapse": false, 992 | "editable": true, 993 | "height": "250px", 994 | "panels": [ 995 | { 996 | "aliasColors": {}, 997 | "bars": false, 998 | "datasource": "Prometheus", 999 | "decimals": 2, 1000 | "editable": true, 1001 | "error": false, 1002 | "fill": 1, 1003 | "grid": { 1004 | "threshold1": null, 1005 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 1006 | "threshold2": null, 1007 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 1008 | }, 1009 | "id": 7, 1010 | "isNew": true, 1011 | "legend": { 1012 | "alignAsTable": true, 1013 | "avg": true, 1014 | "current": false, 1015 | "max": true, 1016 | "min": true, 1017 | "rightSide": true, 1018 | "show": true, 1019 | "total": false, 1020 | "values": true 1021 | }, 1022 | "lines": true, 1023 | "linewidth": 2, 1024 | "links": [], 1025 | "nullPointMode": "connected", 1026 | "percentage": false, 1027 | "pointradius": 5, 1028 | "points": false, 1029 | "renderer": "flot", 1030 | "seriesOverrides": [ 1031 | { 1032 | "alias": "read", 1033 | "yaxis": 1 1034 | }, 1035 | { 1036 | "alias": "written", 1037 | "yaxis": 1 1038 | }, 1039 | { 1040 | "alias": "io time", 1041 | "yaxis": 2 1042 | } 1043 | ], 1044 | "span": 12, 1045 | "stack": false, 1046 | "steppedLine": false, 1047 | "targets": [ 1048 | { 1049 | "expr": "sum(irate(node_disk_read_bytes_total[1m]))", 1050 | "interval": "", 1051 | "intervalFactor": 1, 1052 | "legendFormat": "read", 1053 | "metric": "node_disk_read_bytes_total", 1054 | "refId": "A", 1055 | "step": 1 1056 | }, 1057 | { 1058 | "expr": "sum(irate(node_disk_written_bytes_total[1m]))", 1059 | "intervalFactor": 1, 1060 | "legendFormat": "written", 1061 | "metric": "node_disk_written_bytes_total", 1062 | "refId": "B", 1063 | "step": 1 1064 | }, 1065 | { 1066 | "expr": "sum(irate(node_disk_io_time_seconds_total[1m]))", 1067 | "intervalFactor": 1, 1068 | "legendFormat": "io time", 1069 | "metric": "node_disk_io_time_seconds_total", 1070 | "refId": "C", 1071 | "step": 1 1072 | } 1073 | ], 1074 | "timeFrom": null, 1075 | "timeShift": null, 1076 | "title": "I/O Usage", 1077 | "tooltip": { 1078 | "msResolution": true, 1079 | "shared": true, 1080 | "sort": 0, 1081 | "value_type": "cumulative" 1082 | }, 1083 | "type": "graph", 1084 | "xaxis": { 1085 | "show": true 1086 | }, 1087 | "yaxes": [ 1088 | { 1089 | "format": "Bps", 1090 | "label": null, 1091 | "logBase": 1, 1092 | "max": null, 1093 | "min": 0, 1094 | "show": true 1095 | }, 1096 | { 1097 | "format": "ms", 1098 | "label": null, 1099 | "logBase": 1, 1100 | "max": null, 1101 | "min": null, 1102 | "show": true 1103 | } 1104 | ] 1105 | } 1106 | ], 1107 | "title": "I/O" 1108 | }, 1109 | { 1110 | "collapse": false, 1111 | "editable": true, 1112 | "height": "250px", 1113 | "panels": [ 1114 | { 1115 | "aliasColors": {}, 1116 | "bars": false, 1117 | "datasource": "Prometheus", 1118 | "decimals": 2, 1119 | "editable": true, 1120 | "error": false, 1121 | "fill": 4, 1122 | "grid": { 1123 | "threshold1": null, 1124 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 1125 | "threshold2": null, 1126 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 1127 | }, 1128 | "id": 8, 1129 | "isNew": true, 1130 | "legend": { 1131 | "alignAsTable": true, 1132 | "avg": true, 1133 | "current": false, 1134 | "max": true, 1135 | "min": true, 1136 | "rightSide": true, 1137 | "show": true, 1138 | "total": false, 1139 | "values": true 1140 | }, 1141 | "lines": true, 1142 | "linewidth": 2, 1143 | "links": [], 1144 | "nullPointMode": "connected", 1145 | "percentage": false, 1146 | "pointradius": 5, 1147 | "points": false, 1148 | "renderer": "flot", 1149 | "seriesOverrides": [], 1150 | "span": 12, 1151 | "stack": true, 1152 | "steppedLine": false, 1153 | "targets": [ 1154 | { 1155 | "expr": "irate(node_network_receive_bytes_total{device!=\"lo\"}[1m])", 1156 | "intervalFactor": 1, 1157 | "legendFormat": "In: {{ device }}", 1158 | "metric": "node_network_receive_bytes_total", 1159 | "refId": "A", 1160 | "step": 1 1161 | }, 1162 | { 1163 | "expr": "irate(node_network_transmit_bytes_total{device!=\"lo\"}[1m])", 1164 | "intervalFactor": 1, 1165 | "legendFormat": "Out: {{ device }}", 1166 | "metric": "node_network_transmit_bytes_total", 1167 | "refId": "B", 1168 | "step": 1 1169 | } 1170 | ], 1171 | "timeFrom": null, 1172 | "timeShift": null, 1173 | "title": "Network Usage", 1174 | "tooltip": { 1175 | "msResolution": true, 1176 | "shared": true, 1177 | "sort": 2, 1178 | "value_type": "individual" 1179 | }, 1180 | "type": "graph", 1181 | "xaxis": { 1182 | "show": true 1183 | }, 1184 | "yaxes": [ 1185 | { 1186 | "format": "Bps", 1187 | "label": null, 1188 | "logBase": 1, 1189 | "max": null, 1190 | "min": 0, 1191 | "show": true 1192 | }, 1193 | { 1194 | "format": "short", 1195 | "label": null, 1196 | "logBase": 1, 1197 | "max": null, 1198 | "min": null, 1199 | "show": false 1200 | } 1201 | ] 1202 | } 1203 | ], 1204 | "title": "Network" 1205 | }, 1206 | { 1207 | "collapse": false, 1208 | "editable": true, 1209 | "height": "250px", 1210 | "panels": [ 1211 | { 1212 | "aliasColors": {}, 1213 | "bars": false, 1214 | "datasource": "Prometheus", 1215 | "decimals": 2, 1216 | "editable": true, 1217 | "error": false, 1218 | "fill": 4, 1219 | "grid": { 1220 | "threshold1": null, 1221 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 1222 | "threshold2": null, 1223 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 1224 | }, 1225 | "id": 14, 1226 | "isNew": true, 1227 | "legend": { 1228 | "alignAsTable": true, 1229 | "avg": true, 1230 | "current": false, 1231 | "max": true, 1232 | "min": true, 1233 | "rightSide": false, 1234 | "show": true, 1235 | "total": false, 1236 | "values": true 1237 | }, 1238 | "lines": true, 1239 | "linewidth": 2, 1240 | "links": [], 1241 | "nullPointMode": "connected", 1242 | "percentage": false, 1243 | "pointradius": 5, 1244 | "points": false, 1245 | "renderer": "flot", 1246 | "seriesOverrides": [ 1247 | { 1248 | "alias": "Used", 1249 | "color": "#890F02" 1250 | }, 1251 | { 1252 | "alias": "Free", 1253 | "color": "#7EB26D" 1254 | } 1255 | ], 1256 | "span": 6, 1257 | "stack": true, 1258 | "steppedLine": false, 1259 | "targets": [ 1260 | { 1261 | "expr": "node_memory_SwapTotal_bytes - node_memory_SwapFree_bytes", 1262 | "interval": "10s", 1263 | "intervalFactor": 1, 1264 | "legendFormat": "Used", 1265 | "refId": "A", 1266 | "step": 10 1267 | }, 1268 | { 1269 | "expr": "node_memory_SwapFree_bytes", 1270 | "interval": "10s", 1271 | "intervalFactor": 1, 1272 | "legendFormat": "Free", 1273 | "refId": "B", 1274 | "step": 10 1275 | } 1276 | ], 1277 | "timeFrom": null, 1278 | "timeShift": null, 1279 | "title": "Swap Usage", 1280 | "tooltip": { 1281 | "msResolution": true, 1282 | "shared": true, 1283 | "sort": 2, 1284 | "value_type": "individual" 1285 | }, 1286 | "type": "graph", 1287 | "xaxis": { 1288 | "show": true 1289 | }, 1290 | "yaxes": [ 1291 | { 1292 | "format": "bytes", 1293 | "label": null, 1294 | "logBase": 1, 1295 | "max": null, 1296 | "min": 0, 1297 | "show": true 1298 | }, 1299 | { 1300 | "format": "short", 1301 | "label": null, 1302 | "logBase": 1, 1303 | "max": null, 1304 | "min": null, 1305 | "show": false 1306 | } 1307 | ] 1308 | }, 1309 | { 1310 | "aliasColors": {}, 1311 | "bars": false, 1312 | "datasource": "Prometheus", 1313 | "decimals": 2, 1314 | "editable": true, 1315 | "error": false, 1316 | "fill": 1, 1317 | "grid": { 1318 | "threshold1": null, 1319 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 1320 | "threshold2": null, 1321 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 1322 | }, 1323 | "id": 15, 1324 | "isNew": true, 1325 | "legend": { 1326 | "alignAsTable": true, 1327 | "avg": true, 1328 | "current": false, 1329 | "max": true, 1330 | "min": true, 1331 | "show": true, 1332 | "total": false, 1333 | "values": true 1334 | }, 1335 | "lines": true, 1336 | "linewidth": 2, 1337 | "links": [], 1338 | "nullPointMode": "connected", 1339 | "percentage": false, 1340 | "pointradius": 5, 1341 | "points": false, 1342 | "renderer": "flot", 1343 | "seriesOverrides": [], 1344 | "span": 6, 1345 | "stack": false, 1346 | "steppedLine": false, 1347 | "targets": [ 1348 | { 1349 | "expr": "rate(node_vmstat_pswpin[1m]) * 4096 or irate(node_vmstat_pswpin[5m]) * 4096", 1350 | "interval": "10s", 1351 | "intervalFactor": 1, 1352 | "legendFormat": "In", 1353 | "refId": "A", 1354 | "step": 10 1355 | }, 1356 | { 1357 | "expr": "rate(node_vmstat_pswpout[1m]) * 4096 or irate(node_vmstat_pswpout[5m]) * 4096", 1358 | "interval": "10s", 1359 | "intervalFactor": 1, 1360 | "legendFormat": "Out", 1361 | "refId": "B", 1362 | "step": 10 1363 | } 1364 | ], 1365 | "timeFrom": null, 1366 | "timeShift": null, 1367 | "title": "Swap I/O", 1368 | "tooltip": { 1369 | "msResolution": true, 1370 | "shared": true, 1371 | "sort": 0, 1372 | "value_type": "cumulative" 1373 | }, 1374 | "type": "graph", 1375 | "xaxis": { 1376 | "show": true 1377 | }, 1378 | "yaxes": [ 1379 | { 1380 | "format": "Bps", 1381 | "label": null, 1382 | "logBase": 1, 1383 | "max": null, 1384 | "min": 0, 1385 | "show": true 1386 | }, 1387 | { 1388 | "format": "short", 1389 | "label": null, 1390 | "logBase": 1, 1391 | "max": null, 1392 | "min": null, 1393 | "show": false 1394 | } 1395 | ] 1396 | } 1397 | ], 1398 | "title": "New row" 1399 | } 1400 | ], 1401 | "time": { 1402 | "from": "now-15m", 1403 | "to": "now" 1404 | }, 1405 | "timepicker": { 1406 | "refresh_intervals": [ 1407 | "5s", 1408 | "10s", 1409 | "30s", 1410 | "1m", 1411 | "5m", 1412 | "15m", 1413 | "30m", 1414 | "1h", 1415 | "2h", 1416 | "1d" 1417 | ], 1418 | "time_options": [ 1419 | "5m", 1420 | "15m", 1421 | "1h", 1422 | "6h", 1423 | "12h", 1424 | "24h", 1425 | "2d", 1426 | "7d", 1427 | "30d" 1428 | ] 1429 | }, 1430 | "templating": { 1431 | "list": [] 1432 | }, 1433 | "annotations": { 1434 | "list": [] 1435 | }, 1436 | "refresh": "10s", 1437 | "schemaVersion": 12, 1438 | "version": 2, 1439 | "links": [], 1440 | "gnetId": null 1441 | }} 1442 | -------------------------------------------------------------------------------- /docker/grafana/dashboards/monitor_services.json: -------------------------------------------------------------------------------- 1 | {"dashboard": { 2 | "id": null, 3 | "title": "Monitor Services", 4 | "tags": [ 5 | "prometheus" 6 | ], 7 | "style": "dark", 8 | "timezone": "browser", 9 | "editable": true, 10 | "hideControls": false, 11 | "sharedCrosshair": true, 12 | "panels": [ 13 | { 14 | "cacheTimeout": null, 15 | "colorBackground": false, 16 | "colorValue": false, 17 | "colors": [ 18 | "rgba(245, 54, 54, 0.9)", 19 | "rgba(237, 129, 40, 0.89)", 20 | "rgba(50, 172, 45, 0.97)" 21 | ], 22 | "datasource": "Prometheus", 23 | "decimals": 1, 24 | "editable": true, 25 | "error": false, 26 | "format": "s", 27 | "gauge": { 28 | "maxValue": 100, 29 | "minValue": 0, 30 | "show": false, 31 | "thresholdLabels": false, 32 | "thresholdMarkers": true 33 | }, 34 | "gridPos": { 35 | "h": 3, 36 | "w": 6, 37 | "x": 0, 38 | "y": 0 39 | }, 40 | "hideTimeOverride": true, 41 | "id": 1, 42 | "interval": null, 43 | "links": [], 44 | "mappingType": 1, 45 | "mappingTypes": [ 46 | { 47 | "name": "value to text", 48 | "value": 1 49 | }, 50 | { 51 | "name": "range to text", 52 | "value": 2 53 | } 54 | ], 55 | "maxDataPoints": 100, 56 | "nullPointMode": "connected", 57 | "nullText": null, 58 | "postfix": "s", 59 | "postfixFontSize": "80%", 60 | "prefix": "", 61 | "prefixFontSize": "50%", 62 | "rangeMaps": [ 63 | { 64 | "from": "null", 65 | "text": "N/A", 66 | "to": "null" 67 | } 68 | ], 69 | "sparkline": { 70 | "fillColor": "rgba(31, 118, 189, 0.18)", 71 | "full": false, 72 | "lineColor": "rgb(31, 120, 193)", 73 | "show": false 74 | }, 75 | "tableColumn": "", 76 | "targets": [ 77 | { 78 | "expr": "(time() - process_start_time_seconds{instance=\"localhost:9090\",job=\"prometheus\"})", 79 | "interval": "10s", 80 | "intervalFactor": 1, 81 | "legendFormat": "", 82 | "refId": "A", 83 | "step": 10 84 | } 85 | ], 86 | "thresholds": "", 87 | "timeFrom": "10s", 88 | "timeShift": null, 89 | "title": "Prometheus Uptime", 90 | "type": "singlestat", 91 | "valueFontSize": "80%", 92 | "valueMaps": [ 93 | { 94 | "op": "=", 95 | "text": "N/A", 96 | "value": "null" 97 | } 98 | ], 99 | "valueName": "avg" 100 | }, 101 | { 102 | "cacheTimeout": null, 103 | "colorBackground": false, 104 | "colorValue": false, 105 | "colors": [ 106 | "rgba(245, 54, 54, 0.9)", 107 | "rgba(237, 129, 40, 0.89)", 108 | "rgba(50, 172, 45, 0.97)" 109 | ], 110 | "datasource": "Prometheus", 111 | "decimals": 2, 112 | "editable": true, 113 | "error": false, 114 | "format": "bytes", 115 | "gauge": { 116 | "maxValue": 100, 117 | "minValue": 0, 118 | "show": false, 119 | "thresholdLabels": false, 120 | "thresholdMarkers": true 121 | }, 122 | "gridPos": { 123 | "h": 3, 124 | "w": 6, 125 | "x": 6, 126 | "y": 0 127 | }, 128 | "hideTimeOverride": true, 129 | "id": 5, 130 | "interval": null, 131 | "links": [], 132 | "mappingType": 1, 133 | "mappingTypes": [ 134 | { 135 | "name": "value to text", 136 | "value": 1 137 | }, 138 | { 139 | "name": "range to text", 140 | "value": 2 141 | } 142 | ], 143 | "maxDataPoints": 100, 144 | "nullPointMode": "connected", 145 | "nullText": null, 146 | "postfix": "", 147 | "postfixFontSize": "50%", 148 | "prefix": "", 149 | "prefixFontSize": "50%", 150 | "rangeMaps": [ 151 | { 152 | "from": "null", 153 | "text": "N/A", 154 | "to": "null" 155 | } 156 | ], 157 | "sparkline": { 158 | "fillColor": "rgba(31, 118, 189, 0.18)", 159 | "full": false, 160 | "lineColor": "rgb(31, 120, 193)", 161 | "show": false 162 | }, 163 | "tableColumn": "", 164 | "targets": [ 165 | { 166 | "expr": "sum(container_memory_usage_bytes{container_label_org_label_schema_group=\"monitoring\"})", 167 | "format": "time_series", 168 | "interval": "10s", 169 | "intervalFactor": 1, 170 | "legendFormat": "", 171 | "refId": "A", 172 | "step": 10 173 | } 174 | ], 175 | "thresholds": "", 176 | "timeFrom": "10s", 177 | "timeShift": null, 178 | "title": "Memory Usage", 179 | "type": "singlestat", 180 | "valueFontSize": "80%", 181 | "valueMaps": [ 182 | { 183 | "op": "=", 184 | "text": "N/A", 185 | "value": "null" 186 | } 187 | ], 188 | "valueName": "avg" 189 | }, 190 | { 191 | "cacheTimeout": null, 192 | "colorBackground": false, 193 | "colorValue": false, 194 | "colors": [ 195 | "rgba(245, 54, 54, 0.9)", 196 | "rgba(237, 129, 40, 0.89)", 197 | "rgba(50, 172, 45, 0.97)" 198 | ], 199 | "datasource": "Prometheus", 200 | "editable": true, 201 | "error": false, 202 | "format": "none", 203 | "gauge": { 204 | "maxValue": 100, 205 | "minValue": 0, 206 | "show": false, 207 | "thresholdLabels": false, 208 | "thresholdMarkers": true 209 | }, 210 | "gridPos": { 211 | "h": 3, 212 | "w": 6, 213 | "x": 12, 214 | "y": 0 215 | }, 216 | "hideTimeOverride": true, 217 | "id": 3, 218 | "interval": null, 219 | "links": [], 220 | "mappingType": 1, 221 | "mappingTypes": [ 222 | { 223 | "name": "value to text", 224 | "value": 1 225 | }, 226 | { 227 | "name": "range to text", 228 | "value": 2 229 | } 230 | ], 231 | "maxDataPoints": 100, 232 | "nullPointMode": "connected", 233 | "nullText": null, 234 | "postfix": "", 235 | "postfixFontSize": "50%", 236 | "prefix": "", 237 | "prefixFontSize": "50%", 238 | "rangeMaps": [ 239 | { 240 | "from": "null", 241 | "text": "N/A", 242 | "to": "null" 243 | } 244 | ], 245 | "sparkline": { 246 | "fillColor": "rgba(31, 118, 189, 0.18)", 247 | "full": false, 248 | "lineColor": "rgb(31, 120, 193)", 249 | "show": false 250 | }, 251 | "tableColumn": "", 252 | "targets": [ 253 | { 254 | "expr": "prometheus_tsdb_head_chunks", 255 | "interval": "10s", 256 | "intervalFactor": 1, 257 | "refId": "A", 258 | "step": 10 259 | } 260 | ], 261 | "thresholds": "", 262 | "timeFrom": "10s", 263 | "timeShift": null, 264 | "title": "In-Memory Chunks", 265 | "type": "singlestat", 266 | "valueFontSize": "80%", 267 | "valueMaps": [ 268 | { 269 | "op": "=", 270 | "text": "N/A", 271 | "value": "null" 272 | } 273 | ], 274 | "valueName": "avg" 275 | }, 276 | { 277 | "cacheTimeout": null, 278 | "colorBackground": false, 279 | "colorValue": false, 280 | "colors": [ 281 | "rgba(245, 54, 54, 0.9)", 282 | "rgba(237, 129, 40, 0.89)", 283 | "rgba(50, 172, 45, 0.97)" 284 | ], 285 | "datasource": "Prometheus", 286 | "editable": true, 287 | "error": false, 288 | "format": "none", 289 | "gauge": { 290 | "maxValue": 100, 291 | "minValue": 0, 292 | "show": false, 293 | "thresholdLabels": false, 294 | "thresholdMarkers": true 295 | }, 296 | "gridPos": { 297 | "h": 3, 298 | "w": 6, 299 | "x": 18, 300 | "y": 0 301 | }, 302 | "hideTimeOverride": true, 303 | "id": 2, 304 | "interval": null, 305 | "links": [], 306 | "mappingType": 1, 307 | "mappingTypes": [ 308 | { 309 | "name": "value to text", 310 | "value": 1 311 | }, 312 | { 313 | "name": "range to text", 314 | "value": 2 315 | } 316 | ], 317 | "maxDataPoints": 100, 318 | "nullPointMode": "connected", 319 | "nullText": null, 320 | "postfix": "", 321 | "postfixFontSize": "50%", 322 | "prefix": "", 323 | "prefixFontSize": "50%", 324 | "rangeMaps": [ 325 | { 326 | "from": "null", 327 | "text": "N/A", 328 | "to": "null" 329 | } 330 | ], 331 | "sparkline": { 332 | "fillColor": "rgba(31, 118, 189, 0.18)", 333 | "full": false, 334 | "lineColor": "rgb(31, 120, 193)", 335 | "show": false 336 | }, 337 | "tableColumn": "", 338 | "targets": [ 339 | { 340 | "expr": "prometheus_tsdb_head_series", 341 | "interval": "10s", 342 | "intervalFactor": 1, 343 | "refId": "A", 344 | "step": 10 345 | } 346 | ], 347 | "thresholds": "", 348 | "timeFrom": "10s", 349 | "timeShift": null, 350 | "title": "In-Memory Series", 351 | "type": "singlestat", 352 | "valueFontSize": "80%", 353 | "valueMaps": [ 354 | { 355 | "op": "=", 356 | "text": "N/A", 357 | "value": "null" 358 | } 359 | ], 360 | "valueName": "avg" 361 | }, 362 | { 363 | "aliasColors": {}, 364 | "bars": false, 365 | "dashLength": 10, 366 | "dashes": false, 367 | "datasource": "Prometheus", 368 | "decimals": 2, 369 | "editable": true, 370 | "error": false, 371 | "fill": 1, 372 | "grid": {}, 373 | "gridPos": { 374 | "h": 7, 375 | "w": 24, 376 | "x": 0, 377 | "y": 3 378 | }, 379 | "id": 6, 380 | "legend": { 381 | "alignAsTable": true, 382 | "avg": true, 383 | "current": false, 384 | "max": true, 385 | "min": true, 386 | "rightSide": true, 387 | "show": true, 388 | "total": false, 389 | "values": true 390 | }, 391 | "lines": true, 392 | "linewidth": 2, 393 | "links": [], 394 | "nullPointMode": "connected", 395 | "percentage": false, 396 | "pointradius": 5, 397 | "points": false, 398 | "renderer": "flot", 399 | "seriesOverrides": [], 400 | "spaceLength": 10, 401 | "stack": false, 402 | "steppedLine": false, 403 | "targets": [ 404 | { 405 | "expr": "sum by (name) (rate(container_cpu_user_seconds_total{container_label_org_label_schema_group=\"monitoring\"}[1m]) * 100 / scalar(count(node_cpu{mode=\"user\"})))", 406 | "format": "time_series", 407 | "hide": true, 408 | "intervalFactor": 10, 409 | "legendFormat": "{{ name }}", 410 | "refId": "A", 411 | "step": 10 412 | }, 413 | { 414 | "expr": "sum by (name) (rate(container_cpu_user_seconds_total{container_label_org_label_schema_group=\"monitoring\"}[1m]) * 100 / scalar(count(node_cpu_seconds_total{mode=\"user\"})))", 415 | "format": "time_series", 416 | "intervalFactor": 10, 417 | "legendFormat": "{{ name }}", 418 | "refId": "B" 419 | } 420 | ], 421 | "thresholds": [], 422 | "timeFrom": null, 423 | "timeShift": null, 424 | "title": "Container CPU Usage", 425 | "tooltip": { 426 | "msResolution": true, 427 | "shared": true, 428 | "sort": 2, 429 | "value_type": "cumulative" 430 | }, 431 | "type": "graph", 432 | "xaxis": { 433 | "buckets": null, 434 | "mode": "time", 435 | "name": null, 436 | "show": true, 437 | "values": [] 438 | }, 439 | "yaxes": [ 440 | { 441 | "format": "percent", 442 | "label": null, 443 | "logBase": 1, 444 | "max": null, 445 | "min": null, 446 | "show": true 447 | }, 448 | { 449 | "format": "short", 450 | "label": null, 451 | "logBase": 1, 452 | "max": null, 453 | "min": null, 454 | "show": true 455 | } 456 | ], 457 | "yaxis": { 458 | "align": false, 459 | "alignLevel": null 460 | } 461 | }, 462 | { 463 | "aliasColors": {}, 464 | "bars": false, 465 | "dashLength": 10, 466 | "dashes": false, 467 | "datasource": "Prometheus", 468 | "decimals": 2, 469 | "editable": true, 470 | "error": false, 471 | "fill": 1, 472 | "grid": {}, 473 | "gridPos": { 474 | "h": 7, 475 | "w": 24, 476 | "x": 0, 477 | "y": 10 478 | }, 479 | "id": 7, 480 | "legend": { 481 | "alignAsTable": true, 482 | "avg": true, 483 | "current": false, 484 | "max": true, 485 | "min": true, 486 | "rightSide": true, 487 | "show": true, 488 | "total": false, 489 | "values": true 490 | }, 491 | "lines": true, 492 | "linewidth": 2, 493 | "links": [], 494 | "nullPointMode": "connected", 495 | "percentage": false, 496 | "pointradius": 5, 497 | "points": false, 498 | "renderer": "flot", 499 | "seriesOverrides": [], 500 | "spaceLength": 10, 501 | "stack": false, 502 | "steppedLine": false, 503 | "targets": [ 504 | { 505 | "expr": "sum by (name) (container_memory_usage_bytes{container_label_org_label_schema_group=\"monitoring\"})", 506 | "format": "time_series", 507 | "interval": "", 508 | "intervalFactor": 10, 509 | "legendFormat": "{{ name }}", 510 | "metric": "container_memory_usage_bytes", 511 | "refId": "A", 512 | "step": 10 513 | } 514 | ], 515 | "thresholds": [], 516 | "timeFrom": null, 517 | "timeShift": null, 518 | "title": "Container Memory Usage", 519 | "tooltip": { 520 | "msResolution": true, 521 | "shared": true, 522 | "sort": 2, 523 | "value_type": "cumulative" 524 | }, 525 | "type": "graph", 526 | "xaxis": { 527 | "buckets": null, 528 | "mode": "time", 529 | "name": null, 530 | "show": true, 531 | "values": [] 532 | }, 533 | "yaxes": [ 534 | { 535 | "format": "bytes", 536 | "label": null, 537 | "logBase": 1, 538 | "max": null, 539 | "min": null, 540 | "show": true 541 | }, 542 | { 543 | "format": "short", 544 | "label": null, 545 | "logBase": 1, 546 | "max": null, 547 | "min": null, 548 | "show": false 549 | } 550 | ], 551 | "yaxis": { 552 | "align": false, 553 | "alignLevel": null 554 | } 555 | }, 556 | { 557 | "aliasColors": {}, 558 | "bars": true, 559 | "dashLength": 10, 560 | "dashes": false, 561 | "datasource": "Prometheus", 562 | "decimals": 0, 563 | "editable": true, 564 | "error": false, 565 | "fill": 1, 566 | "grid": {}, 567 | "gridPos": { 568 | "h": 7, 569 | "w": 24, 570 | "x": 0, 571 | "y": 17 572 | }, 573 | "id": 73, 574 | "legend": { 575 | "alignAsTable": true, 576 | "avg": false, 577 | "current": true, 578 | "max": false, 579 | "min": false, 580 | "rightSide": true, 581 | "show": true, 582 | "total": false, 583 | "values": true 584 | }, 585 | "lines": false, 586 | "linewidth": 2, 587 | "links": [], 588 | "nullPointMode": "connected", 589 | "percentage": false, 590 | "pointradius": 5, 591 | "points": true, 592 | "renderer": "flot", 593 | "seriesOverrides": [], 594 | "spaceLength": 10, 595 | "stack": false, 596 | "steppedLine": false, 597 | "targets": [ 598 | { 599 | "expr": "sum(ALERTS{alertstate=\"firing\"}) by (alertname)", 600 | "format": "time_series", 601 | "interval": "30s", 602 | "intervalFactor": 1, 603 | "legendFormat": "{{ alertname }}", 604 | "metric": "container_memory_usage_bytes", 605 | "refId": "A", 606 | "step": 10 607 | } 608 | ], 609 | "thresholds": [], 610 | "timeFrom": null, 611 | "timeShift": null, 612 | "title": "Alerts", 613 | "tooltip": { 614 | "msResolution": true, 615 | "shared": false, 616 | "sort": 0, 617 | "value_type": "cumulative" 618 | }, 619 | "type": "graph", 620 | "xaxis": { 621 | "buckets": null, 622 | "mode": "time", 623 | "name": null, 624 | "show": true, 625 | "values": [] 626 | }, 627 | "yaxes": [ 628 | { 629 | "format": "short", 630 | "label": null, 631 | "logBase": 1, 632 | "max": null, 633 | "min": "0", 634 | "show": true 635 | }, 636 | { 637 | "format": "short", 638 | "label": null, 639 | "logBase": 1, 640 | "max": null, 641 | "min": null, 642 | "show": false 643 | } 644 | ], 645 | "yaxis": { 646 | "align": false, 647 | "alignLevel": null 648 | } 649 | }, 650 | { 651 | "collapsed": false, 652 | "gridPos": { 653 | "h": 1, 654 | "w": 24, 655 | "x": 0, 656 | "y": 24 657 | }, 658 | "id": 22, 659 | "panels": [], 660 | "repeat": null, 661 | "title": "Prometheus Metrics", 662 | "type": "row" 663 | }, 664 | { 665 | "aliasColors": { 666 | "Max": "#e24d42", 667 | "Open": "#508642" 668 | }, 669 | "bars": false, 670 | "dashLength": 10, 671 | "dashes": false, 672 | "datasource": "Prometheus", 673 | "fill": 1, 674 | "gridPos": { 675 | "h": 7, 676 | "w": 8, 677 | "x": 0, 678 | "y": 25 679 | }, 680 | "id": 18, 681 | "legend": { 682 | "alignAsTable": true, 683 | "avg": true, 684 | "current": true, 685 | "max": true, 686 | "min": true, 687 | "show": true, 688 | "total": false, 689 | "values": true 690 | }, 691 | "lines": true, 692 | "linewidth": 2, 693 | "links": [], 694 | "nullPointMode": "null", 695 | "percentage": false, 696 | "pointradius": 5, 697 | "points": false, 698 | "renderer": "flot", 699 | "seriesOverrides": [], 700 | "spaceLength": 10, 701 | "stack": false, 702 | "steppedLine": false, 703 | "targets": [ 704 | { 705 | "expr": "process_max_fds{job=\"prometheus\"}", 706 | "format": "time_series", 707 | "intervalFactor": 1, 708 | "legendFormat": "Max", 709 | "refId": "A" 710 | }, 711 | { 712 | "expr": "process_open_fds{job=\"prometheus\"}", 713 | "format": "time_series", 714 | "intervalFactor": 1, 715 | "legendFormat": "Open", 716 | "refId": "B" 717 | } 718 | ], 719 | "thresholds": [], 720 | "timeFrom": null, 721 | "timeShift": null, 722 | "title": "File Descriptors", 723 | "tooltip": { 724 | "shared": true, 725 | "sort": 0, 726 | "value_type": "individual" 727 | }, 728 | "type": "graph", 729 | "xaxis": { 730 | "buckets": null, 731 | "mode": "time", 732 | "name": null, 733 | "show": true, 734 | "values": [] 735 | }, 736 | "yaxes": [ 737 | { 738 | "format": "short", 739 | "label": null, 740 | "logBase": 1, 741 | "max": null, 742 | "min": null, 743 | "show": true 744 | }, 745 | { 746 | "format": "short", 747 | "label": null, 748 | "logBase": 1, 749 | "max": null, 750 | "min": null, 751 | "show": false 752 | } 753 | ], 754 | "yaxis": { 755 | "align": false, 756 | "alignLevel": null 757 | } 758 | }, 759 | { 760 | "aliasColors": { 761 | "Allocated bytes": "#7EB26D", 762 | "Allocated bytes - 1m max": "#BF1B00", 763 | "Allocated bytes - 1m min": "#BF1B00", 764 | "Allocated bytes - 5m max": "#BF1B00", 765 | "Allocated bytes - 5m min": "#BF1B00", 766 | "Chunks": "#1F78C1", 767 | "Chunks to persist": "#508642", 768 | "Max chunks": "#052B51", 769 | "Max to persist": "#3F6833", 770 | "RSS": "#447EBC" 771 | }, 772 | "bars": false, 773 | "dashLength": 10, 774 | "dashes": false, 775 | "datasource": "Prometheus", 776 | "decimals": null, 777 | "editable": true, 778 | "error": false, 779 | "fill": 1, 780 | "gridPos": { 781 | "h": 7, 782 | "w": 8, 783 | "x": 8, 784 | "y": 25 785 | }, 786 | "id": 58, 787 | "legend": { 788 | "alignAsTable": true, 789 | "avg": true, 790 | "current": true, 791 | "max": true, 792 | "min": true, 793 | "show": true, 794 | "total": false, 795 | "values": true 796 | }, 797 | "lines": true, 798 | "linewidth": 2, 799 | "links": [], 800 | "nullPointMode": "null", 801 | "percentage": false, 802 | "pointradius": 5, 803 | "points": false, 804 | "renderer": "flot", 805 | "seriesOverrides": [ 806 | { 807 | "alias": "/-/", 808 | "fill": 0 809 | } 810 | ], 811 | "spaceLength": 10, 812 | "stack": false, 813 | "steppedLine": false, 814 | "targets": [ 815 | { 816 | "expr": "process_resident_memory_bytes{job=\"prometheus\"}", 817 | "format": "time_series", 818 | "intervalFactor": 2, 819 | "legendFormat": "RSS", 820 | "metric": "process_resident_memory_bytes", 821 | "refId": "B", 822 | "step": 10 823 | }, 824 | { 825 | "expr": "prometheus_local_storage_target_heap_size_bytes{job=\"prometheus\"}", 826 | "format": "time_series", 827 | "intervalFactor": 2, 828 | "legendFormat": "Target heap size", 829 | "metric": "go_memstats_alloc_bytes", 830 | "refId": "D", 831 | "step": 10 832 | }, 833 | { 834 | "expr": "go_memstats_next_gc_bytes{job=\"prometheus\"}", 835 | "format": "time_series", 836 | "intervalFactor": 2, 837 | "legendFormat": "Next GC", 838 | "metric": "go_memstats_next_gc_bytes", 839 | "refId": "C", 840 | "step": 10 841 | }, 842 | { 843 | "expr": "go_memstats_alloc_bytes{job=\"prometheus\"}", 844 | "format": "time_series", 845 | "intervalFactor": 2, 846 | "legendFormat": "Allocated", 847 | "metric": "go_memstats_alloc_bytes", 848 | "refId": "A", 849 | "step": 10 850 | } 851 | ], 852 | "thresholds": [], 853 | "timeFrom": null, 854 | "timeShift": null, 855 | "title": "Memory", 856 | "tooltip": { 857 | "msResolution": false, 858 | "shared": true, 859 | "sort": 0, 860 | "value_type": "individual" 861 | }, 862 | "type": "graph", 863 | "xaxis": { 864 | "buckets": null, 865 | "mode": "time", 866 | "name": null, 867 | "show": true, 868 | "values": [] 869 | }, 870 | "yaxes": [ 871 | { 872 | "format": "bytes", 873 | "label": null, 874 | "logBase": 1, 875 | "max": null, 876 | "min": "0", 877 | "show": true 878 | }, 879 | { 880 | "format": "short", 881 | "label": null, 882 | "logBase": 1, 883 | "max": null, 884 | "min": null, 885 | "show": false 886 | } 887 | ], 888 | "yaxis": { 889 | "align": false, 890 | "alignLevel": null 891 | } 892 | }, 893 | { 894 | "aliasColors": { 895 | "Allocated bytes": "#F9BA8F", 896 | "Chunks": "#1F78C1", 897 | "Chunks to persist": "#508642", 898 | "Max chunks": "#052B51", 899 | "Max to persist": "#3F6833", 900 | "RSS": "#890F02" 901 | }, 902 | "bars": false, 903 | "dashLength": 10, 904 | "dashes": false, 905 | "datasource": "Prometheus", 906 | "editable": true, 907 | "error": false, 908 | "fill": 1, 909 | "gridPos": { 910 | "h": 7, 911 | "w": 8, 912 | "x": 16, 913 | "y": 25 914 | }, 915 | "id": 60, 916 | "legend": { 917 | "alignAsTable": true, 918 | "avg": true, 919 | "current": true, 920 | "max": true, 921 | "min": true, 922 | "show": true, 923 | "total": false, 924 | "values": true 925 | }, 926 | "lines": true, 927 | "linewidth": 2, 928 | "links": [], 929 | "nullPointMode": "null", 930 | "percentage": false, 931 | "pointradius": 5, 932 | "points": false, 933 | "renderer": "flot", 934 | "seriesOverrides": [], 935 | "spaceLength": 10, 936 | "stack": false, 937 | "steppedLine": false, 938 | "targets": [ 939 | { 940 | "expr": "rate(go_memstats_alloc_bytes_total{job=\"prometheus\"}[1m])", 941 | "format": "time_series", 942 | "intervalFactor": 2, 943 | "legendFormat": "Allocated", 944 | "metric": "go_memstats_alloc_bytes", 945 | "refId": "A", 946 | "step": 10 947 | } 948 | ], 949 | "thresholds": [], 950 | "timeFrom": null, 951 | "timeShift": null, 952 | "title": "Allocations", 953 | "tooltip": { 954 | "msResolution": false, 955 | "shared": true, 956 | "sort": 0, 957 | "value_type": "individual" 958 | }, 959 | "type": "graph", 960 | "xaxis": { 961 | "buckets": null, 962 | "mode": "time", 963 | "name": null, 964 | "show": true, 965 | "values": [] 966 | }, 967 | "yaxes": [ 968 | { 969 | "format": "Bps", 970 | "label": null, 971 | "logBase": 1, 972 | "max": null, 973 | "min": "0", 974 | "show": true 975 | }, 976 | { 977 | "format": "short", 978 | "label": null, 979 | "logBase": 1, 980 | "max": null, 981 | "min": null, 982 | "show": false 983 | } 984 | ], 985 | "yaxis": { 986 | "align": false, 987 | "alignLevel": null 988 | } 989 | }, 990 | { 991 | "aliasColors": { 992 | "Chunks": "#1F78C1", 993 | "Chunks to persist": "#508642", 994 | "Max chunks": "#052B51", 995 | "Max to persist": "#3F6833", 996 | "Time series": "#70dbed" 997 | }, 998 | "bars": false, 999 | "dashLength": 10, 1000 | "dashes": false, 1001 | "datasource": "Prometheus", 1002 | "editable": true, 1003 | "error": false, 1004 | "fill": 1, 1005 | "gridPos": { 1006 | "h": 7, 1007 | "w": 8, 1008 | "x": 0, 1009 | "y": 32 1010 | }, 1011 | "id": 20, 1012 | "legend": { 1013 | "alignAsTable": true, 1014 | "avg": true, 1015 | "current": true, 1016 | "max": true, 1017 | "min": true, 1018 | "show": true, 1019 | "total": false, 1020 | "values": true 1021 | }, 1022 | "lines": true, 1023 | "linewidth": 2, 1024 | "links": [], 1025 | "nullPointMode": "null", 1026 | "percentage": false, 1027 | "pointradius": 5, 1028 | "points": false, 1029 | "renderer": "flot", 1030 | "seriesOverrides": [], 1031 | "spaceLength": 10, 1032 | "stack": false, 1033 | "steppedLine": false, 1034 | "targets": [ 1035 | { 1036 | "expr": "prometheus_tsdb_head_series", 1037 | "format": "time_series", 1038 | "intervalFactor": 2, 1039 | "legendFormat": "Time series", 1040 | "metric": "prometheus_local_storage_memory_series", 1041 | "refId": "A", 1042 | "step": 10 1043 | } 1044 | ], 1045 | "thresholds": [], 1046 | "timeFrom": null, 1047 | "timeShift": null, 1048 | "title": "Head Time series", 1049 | "tooltip": { 1050 | "msResolution": false, 1051 | "shared": true, 1052 | "sort": 0, 1053 | "value_type": "individual" 1054 | }, 1055 | "type": "graph", 1056 | "xaxis": { 1057 | "buckets": null, 1058 | "mode": "time", 1059 | "name": null, 1060 | "show": true, 1061 | "values": [] 1062 | }, 1063 | "yaxes": [ 1064 | { 1065 | "format": "short", 1066 | "label": null, 1067 | "logBase": 1, 1068 | "max": null, 1069 | "min": "0", 1070 | "show": true 1071 | }, 1072 | { 1073 | "format": "short", 1074 | "label": null, 1075 | "logBase": 1, 1076 | "max": null, 1077 | "min": null, 1078 | "show": false 1079 | } 1080 | ], 1081 | "yaxis": { 1082 | "align": false, 1083 | "alignLevel": null 1084 | } 1085 | }, 1086 | { 1087 | "aliasColors": { 1088 | "Chunks": "#1F78C1", 1089 | "Chunks to persist": "#508642", 1090 | "Max chunks": "#052B51", 1091 | "Max to persist": "#3F6833" 1092 | }, 1093 | "bars": false, 1094 | "dashLength": 10, 1095 | "dashes": false, 1096 | "datasource": "Prometheus", 1097 | "editable": true, 1098 | "error": false, 1099 | "fill": 1, 1100 | "gridPos": { 1101 | "h": 7, 1102 | "w": 8, 1103 | "x": 8, 1104 | "y": 32 1105 | }, 1106 | "id": 24, 1107 | "legend": { 1108 | "alignAsTable": true, 1109 | "avg": true, 1110 | "current": true, 1111 | "max": true, 1112 | "min": true, 1113 | "show": true, 1114 | "total": false, 1115 | "values": true 1116 | }, 1117 | "lines": true, 1118 | "linewidth": 2, 1119 | "links": [], 1120 | "nullPointMode": "null", 1121 | "percentage": false, 1122 | "pointradius": 5, 1123 | "points": false, 1124 | "renderer": "flot", 1125 | "seriesOverrides": [], 1126 | "spaceLength": 10, 1127 | "stack": false, 1128 | "steppedLine": false, 1129 | "targets": [ 1130 | { 1131 | "expr": "prometheus_tsdb_head_active_appenders", 1132 | "format": "time_series", 1133 | "intervalFactor": 2, 1134 | "legendFormat": "Head Appenders", 1135 | "metric": "prometheus_local_storage_memory_series", 1136 | "refId": "A", 1137 | "step": 10 1138 | } 1139 | ], 1140 | "thresholds": [], 1141 | "timeFrom": null, 1142 | "timeShift": null, 1143 | "title": "Head Active Appenders", 1144 | "tooltip": { 1145 | "msResolution": false, 1146 | "shared": true, 1147 | "sort": 0, 1148 | "value_type": "individual" 1149 | }, 1150 | "type": "graph", 1151 | "xaxis": { 1152 | "buckets": null, 1153 | "mode": "time", 1154 | "name": null, 1155 | "show": true, 1156 | "values": [] 1157 | }, 1158 | "yaxes": [ 1159 | { 1160 | "format": "short", 1161 | "label": null, 1162 | "logBase": 1, 1163 | "max": null, 1164 | "min": "0", 1165 | "show": true 1166 | }, 1167 | { 1168 | "format": "short", 1169 | "label": null, 1170 | "logBase": 1, 1171 | "max": null, 1172 | "min": null, 1173 | "show": false 1174 | } 1175 | ], 1176 | "yaxis": { 1177 | "align": false, 1178 | "alignLevel": null 1179 | } 1180 | }, 1181 | { 1182 | "aliasColors": { 1183 | "samples/s": "#e5a8e2" 1184 | }, 1185 | "bars": false, 1186 | "dashLength": 10, 1187 | "dashes": false, 1188 | "datasource": "Prometheus", 1189 | "editable": true, 1190 | "error": false, 1191 | "fill": 1, 1192 | "gridPos": { 1193 | "h": 7, 1194 | "w": 8, 1195 | "x": 16, 1196 | "y": 32 1197 | }, 1198 | "id": 26, 1199 | "legend": { 1200 | "alignAsTable": true, 1201 | "avg": true, 1202 | "current": true, 1203 | "max": true, 1204 | "min": true, 1205 | "show": true, 1206 | "total": false, 1207 | "values": true 1208 | }, 1209 | "lines": true, 1210 | "linewidth": 2, 1211 | "links": [], 1212 | "nullPointMode": "connected", 1213 | "percentage": false, 1214 | "pointradius": 5, 1215 | "points": false, 1216 | "renderer": "flot", 1217 | "seriesOverrides": [], 1218 | "spaceLength": 10, 1219 | "stack": false, 1220 | "steppedLine": false, 1221 | "targets": [ 1222 | { 1223 | "expr": "rate(prometheus_tsdb_head_samples_appended_total[1m])", 1224 | "format": "time_series", 1225 | "intervalFactor": 2, 1226 | "legendFormat": "Samples", 1227 | "metric": "prometheus_local_storage_ingested_samples_total", 1228 | "refId": "A", 1229 | "step": 10 1230 | } 1231 | ], 1232 | "thresholds": [], 1233 | "timeFrom": null, 1234 | "timeShift": null, 1235 | "title": "Samples Appended", 1236 | "tooltip": { 1237 | "msResolution": false, 1238 | "shared": true, 1239 | "sort": 0, 1240 | "value_type": "individual" 1241 | }, 1242 | "type": "graph", 1243 | "xaxis": { 1244 | "buckets": null, 1245 | "mode": "time", 1246 | "name": null, 1247 | "show": true, 1248 | "values": [] 1249 | }, 1250 | "yaxes": [ 1251 | { 1252 | "format": "Bps", 1253 | "label": "", 1254 | "logBase": 1, 1255 | "max": null, 1256 | "min": "0", 1257 | "show": true 1258 | }, 1259 | { 1260 | "format": "short", 1261 | "label": null, 1262 | "logBase": 1, 1263 | "max": null, 1264 | "min": null, 1265 | "show": false 1266 | } 1267 | ], 1268 | "yaxis": { 1269 | "align": false, 1270 | "alignLevel": null 1271 | } 1272 | }, 1273 | { 1274 | "aliasColors": { 1275 | "Chunks": "#1F78C1", 1276 | "Chunks to persist": "#508642", 1277 | "Max chunks": "#052B51", 1278 | "Max to persist": "#3F6833", 1279 | "To persist": "#9AC48A" 1280 | }, 1281 | "bars": false, 1282 | "dashLength": 10, 1283 | "dashes": false, 1284 | "datasource": "Prometheus", 1285 | "editable": true, 1286 | "error": false, 1287 | "fill": 1, 1288 | "gridPos": { 1289 | "h": 7, 1290 | "w": 8, 1291 | "x": 0, 1292 | "y": 39 1293 | }, 1294 | "id": 28, 1295 | "legend": { 1296 | "alignAsTable": true, 1297 | "avg": true, 1298 | "current": true, 1299 | "max": true, 1300 | "min": true, 1301 | "show": true, 1302 | "total": false, 1303 | "values": true 1304 | }, 1305 | "lines": true, 1306 | "linewidth": 2, 1307 | "links": [], 1308 | "nullPointMode": "null", 1309 | "percentage": false, 1310 | "pointradius": 5, 1311 | "points": false, 1312 | "renderer": "flot", 1313 | "seriesOverrides": [ 1314 | { 1315 | "alias": "/Max.*/", 1316 | "fill": 0 1317 | } 1318 | ], 1319 | "spaceLength": 10, 1320 | "stack": false, 1321 | "steppedLine": false, 1322 | "targets": [ 1323 | { 1324 | "expr": "prometheus_tsdb_head_chunks", 1325 | "format": "time_series", 1326 | "intervalFactor": 2, 1327 | "legendFormat": "Chunks", 1328 | "metric": "prometheus_local_storage_memory_chunks", 1329 | "refId": "A", 1330 | "step": 10 1331 | } 1332 | ], 1333 | "thresholds": [], 1334 | "timeFrom": null, 1335 | "timeShift": null, 1336 | "title": "Head Chunks", 1337 | "tooltip": { 1338 | "msResolution": false, 1339 | "shared": true, 1340 | "sort": 0, 1341 | "value_type": "individual" 1342 | }, 1343 | "type": "graph", 1344 | "xaxis": { 1345 | "buckets": null, 1346 | "mode": "time", 1347 | "name": null, 1348 | "show": true, 1349 | "values": [] 1350 | }, 1351 | "yaxes": [ 1352 | { 1353 | "format": "short", 1354 | "label": null, 1355 | "logBase": 1, 1356 | "max": null, 1357 | "min": "0", 1358 | "show": true 1359 | }, 1360 | { 1361 | "format": "short", 1362 | "label": null, 1363 | "logBase": 1, 1364 | "max": null, 1365 | "min": null, 1366 | "show": false 1367 | } 1368 | ], 1369 | "yaxis": { 1370 | "align": false, 1371 | "alignLevel": null 1372 | } 1373 | }, 1374 | { 1375 | "aliasColors": { 1376 | "Chunks": "#1F78C1", 1377 | "Chunks to persist": "#508642", 1378 | "Max chunks": "#052B51", 1379 | "Max to persist": "#3F6833" 1380 | }, 1381 | "bars": false, 1382 | "dashLength": 10, 1383 | "dashes": false, 1384 | "datasource": "Prometheus", 1385 | "editable": true, 1386 | "error": false, 1387 | "fill": 1, 1388 | "gridPos": { 1389 | "h": 7, 1390 | "w": 8, 1391 | "x": 8, 1392 | "y": 39 1393 | }, 1394 | "id": 30, 1395 | "legend": { 1396 | "alignAsTable": true, 1397 | "avg": true, 1398 | "current": true, 1399 | "max": true, 1400 | "min": true, 1401 | "show": true, 1402 | "total": false, 1403 | "values": true 1404 | }, 1405 | "lines": true, 1406 | "linewidth": 2, 1407 | "links": [], 1408 | "nullPointMode": "null", 1409 | "percentage": false, 1410 | "pointradius": 5, 1411 | "points": false, 1412 | "renderer": "flot", 1413 | "seriesOverrides": [], 1414 | "spaceLength": 10, 1415 | "stack": false, 1416 | "steppedLine": false, 1417 | "targets": [ 1418 | { 1419 | "expr": "rate(prometheus_tsdb_head_chunks_created_total[1m])", 1420 | "format": "time_series", 1421 | "intervalFactor": 2, 1422 | "legendFormat": "Created", 1423 | "metric": "prometheus_local_storage_chunk_ops_total", 1424 | "refId": "A", 1425 | "step": 10 1426 | } 1427 | ], 1428 | "thresholds": [], 1429 | "timeFrom": null, 1430 | "timeShift": null, 1431 | "title": "Head Chunks Created", 1432 | "tooltip": { 1433 | "msResolution": false, 1434 | "shared": true, 1435 | "sort": 0, 1436 | "value_type": "individual" 1437 | }, 1438 | "type": "graph", 1439 | "xaxis": { 1440 | "buckets": null, 1441 | "mode": "time", 1442 | "name": null, 1443 | "show": true, 1444 | "values": [] 1445 | }, 1446 | "yaxes": [ 1447 | { 1448 | "format": "ops", 1449 | "label": null, 1450 | "logBase": 1, 1451 | "max": null, 1452 | "min": "0", 1453 | "show": true 1454 | }, 1455 | { 1456 | "format": "short", 1457 | "label": null, 1458 | "logBase": 1, 1459 | "max": null, 1460 | "min": null, 1461 | "show": false 1462 | } 1463 | ], 1464 | "yaxis": { 1465 | "align": false, 1466 | "alignLevel": null 1467 | } 1468 | }, 1469 | { 1470 | "aliasColors": { 1471 | "Chunks": "#1F78C1", 1472 | "Chunks to persist": "#508642", 1473 | "Max chunks": "#052B51", 1474 | "Max to persist": "#3F6833", 1475 | "Removed": "#e5ac0e" 1476 | }, 1477 | "bars": false, 1478 | "dashLength": 10, 1479 | "dashes": false, 1480 | "datasource": "Prometheus", 1481 | "editable": true, 1482 | "error": false, 1483 | "fill": 1, 1484 | "gridPos": { 1485 | "h": 7, 1486 | "w": 8, 1487 | "x": 16, 1488 | "y": 39 1489 | }, 1490 | "id": 32, 1491 | "legend": { 1492 | "alignAsTable": true, 1493 | "avg": true, 1494 | "current": true, 1495 | "max": true, 1496 | "min": true, 1497 | "show": true, 1498 | "total": false, 1499 | "values": true 1500 | }, 1501 | "lines": true, 1502 | "linewidth": 1, 1503 | "links": [], 1504 | "nullPointMode": "null", 1505 | "percentage": false, 1506 | "pointradius": 5, 1507 | "points": false, 1508 | "renderer": "flot", 1509 | "seriesOverrides": [], 1510 | "spaceLength": 10, 1511 | "stack": false, 1512 | "steppedLine": false, 1513 | "targets": [ 1514 | { 1515 | "expr": "rate(prometheus_tsdb_head_chunks_removed_total[1m])", 1516 | "format": "time_series", 1517 | "intervalFactor": 2, 1518 | "legendFormat": "Removed", 1519 | "metric": "prometheus_local_storage_chunk_ops_total", 1520 | "refId": "B", 1521 | "step": 10 1522 | } 1523 | ], 1524 | "thresholds": [], 1525 | "timeFrom": null, 1526 | "timeShift": null, 1527 | "title": "Head Chunks Removed", 1528 | "tooltip": { 1529 | "msResolution": false, 1530 | "shared": true, 1531 | "sort": 0, 1532 | "value_type": "individual" 1533 | }, 1534 | "type": "graph", 1535 | "xaxis": { 1536 | "buckets": null, 1537 | "mode": "time", 1538 | "name": null, 1539 | "show": true, 1540 | "values": [] 1541 | }, 1542 | "yaxes": [ 1543 | { 1544 | "format": "ops", 1545 | "label": null, 1546 | "logBase": 1, 1547 | "max": null, 1548 | "min": "0", 1549 | "show": true 1550 | }, 1551 | { 1552 | "format": "short", 1553 | "label": null, 1554 | "logBase": 1, 1555 | "max": null, 1556 | "min": null, 1557 | "show": false 1558 | } 1559 | ], 1560 | "yaxis": { 1561 | "align": false, 1562 | "alignLevel": null 1563 | } 1564 | }, 1565 | { 1566 | "aliasColors": { 1567 | "Chunks": "#1F78C1", 1568 | "Chunks to persist": "#508642", 1569 | "Max": "#447ebc", 1570 | "Max chunks": "#052B51", 1571 | "Max to persist": "#3F6833", 1572 | "Min": "#447ebc", 1573 | "Now": "#7eb26d" 1574 | }, 1575 | "bars": false, 1576 | "dashLength": 10, 1577 | "dashes": false, 1578 | "datasource": "Prometheus", 1579 | "editable": true, 1580 | "error": false, 1581 | "fill": 1, 1582 | "gridPos": { 1583 | "h": 7, 1584 | "w": 8, 1585 | "x": 0, 1586 | "y": 46 1587 | }, 1588 | "id": 34, 1589 | "legend": { 1590 | "alignAsTable": false, 1591 | "avg": false, 1592 | "current": false, 1593 | "max": false, 1594 | "min": false, 1595 | "show": true, 1596 | "total": false, 1597 | "values": false 1598 | }, 1599 | "lines": true, 1600 | "linewidth": 2, 1601 | "links": [], 1602 | "nullPointMode": "null", 1603 | "percentage": false, 1604 | "pointradius": 5, 1605 | "points": false, 1606 | "renderer": "flot", 1607 | "seriesOverrides": [ 1608 | { 1609 | "alias": "Max", 1610 | "fillBelowTo": "Min", 1611 | "lines": false 1612 | } 1613 | ], 1614 | "spaceLength": 10, 1615 | "stack": false, 1616 | "steppedLine": false, 1617 | "targets": [ 1618 | { 1619 | "expr": "prometheus_tsdb_head_min_time", 1620 | "format": "time_series", 1621 | "intervalFactor": 2, 1622 | "legendFormat": "Min", 1623 | "metric": "prometheus_local_storage_series_chunks_persisted_count", 1624 | "refId": "A", 1625 | "step": 10 1626 | }, 1627 | { 1628 | "expr": "time() * 1000", 1629 | "format": "time_series", 1630 | "hide": false, 1631 | "intervalFactor": 2, 1632 | "legendFormat": "Now", 1633 | "refId": "C" 1634 | }, 1635 | { 1636 | "expr": "prometheus_tsdb_head_max_time", 1637 | "format": "time_series", 1638 | "intervalFactor": 2, 1639 | "legendFormat": "Max", 1640 | "metric": "prometheus_local_storage_series_chunks_persisted_count", 1641 | "refId": "B", 1642 | "step": 10 1643 | } 1644 | ], 1645 | "thresholds": [], 1646 | "timeFrom": null, 1647 | "timeShift": null, 1648 | "title": "Head Time Range", 1649 | "tooltip": { 1650 | "msResolution": false, 1651 | "shared": true, 1652 | "sort": 0, 1653 | "value_type": "individual" 1654 | }, 1655 | "type": "graph", 1656 | "xaxis": { 1657 | "buckets": null, 1658 | "mode": "time", 1659 | "name": null, 1660 | "show": true, 1661 | "values": [] 1662 | }, 1663 | "yaxes": [ 1664 | { 1665 | "decimals": null, 1666 | "format": "dateTimeAsIso", 1667 | "label": null, 1668 | "logBase": 1, 1669 | "max": null, 1670 | "min": null, 1671 | "show": true 1672 | }, 1673 | { 1674 | "format": "short", 1675 | "label": null, 1676 | "logBase": 1, 1677 | "max": null, 1678 | "min": null, 1679 | "show": false 1680 | } 1681 | ], 1682 | "yaxis": { 1683 | "align": false, 1684 | "alignLevel": null 1685 | } 1686 | }, 1687 | { 1688 | "aliasColors": { 1689 | "Chunks": "#1F78C1", 1690 | "Chunks to persist": "#508642", 1691 | "Max chunks": "#052B51", 1692 | "Max to persist": "#3F6833" 1693 | }, 1694 | "bars": false, 1695 | "dashLength": 10, 1696 | "dashes": false, 1697 | "datasource": "Prometheus", 1698 | "editable": true, 1699 | "error": false, 1700 | "fill": 1, 1701 | "gridPos": { 1702 | "h": 7, 1703 | "w": 8, 1704 | "x": 8, 1705 | "y": 46 1706 | }, 1707 | "id": 36, 1708 | "legend": { 1709 | "alignAsTable": true, 1710 | "avg": true, 1711 | "current": true, 1712 | "max": true, 1713 | "min": true, 1714 | "show": true, 1715 | "total": false, 1716 | "values": true 1717 | }, 1718 | "lines": true, 1719 | "linewidth": 2, 1720 | "links": [], 1721 | "nullPointMode": "null", 1722 | "percentage": false, 1723 | "pointradius": 5, 1724 | "points": false, 1725 | "renderer": "flot", 1726 | "seriesOverrides": [], 1727 | "spaceLength": 10, 1728 | "stack": false, 1729 | "steppedLine": false, 1730 | "targets": [ 1731 | { 1732 | "expr": "rate(prometheus_tsdb_head_gc_duration_seconds_sum[1m])", 1733 | "format": "time_series", 1734 | "intervalFactor": 2, 1735 | "legendFormat": "GC Time", 1736 | "metric": "prometheus_local_storage_series_chunks_persisted_count", 1737 | "refId": "A", 1738 | "step": 10 1739 | } 1740 | ], 1741 | "thresholds": [], 1742 | "timeFrom": null, 1743 | "timeShift": null, 1744 | "title": "Head GC Time/s", 1745 | "tooltip": { 1746 | "msResolution": false, 1747 | "shared": true, 1748 | "sort": 0, 1749 | "value_type": "individual" 1750 | }, 1751 | "type": "graph", 1752 | "xaxis": { 1753 | "buckets": null, 1754 | "mode": "time", 1755 | "name": null, 1756 | "show": true, 1757 | "values": [] 1758 | }, 1759 | "yaxes": [ 1760 | { 1761 | "format": "s", 1762 | "label": null, 1763 | "logBase": 1, 1764 | "max": null, 1765 | "min": "0", 1766 | "show": true 1767 | }, 1768 | { 1769 | "format": "short", 1770 | "label": null, 1771 | "logBase": 1, 1772 | "max": null, 1773 | "min": null, 1774 | "show": false 1775 | } 1776 | ], 1777 | "yaxis": { 1778 | "align": false, 1779 | "alignLevel": null 1780 | } 1781 | }, 1782 | { 1783 | "aliasColors": { 1784 | "Chunks": "#1F78C1", 1785 | "Chunks to persist": "#508642", 1786 | "Max chunks": "#052B51", 1787 | "Max to persist": "#3F6833" 1788 | }, 1789 | "bars": false, 1790 | "dashLength": 10, 1791 | "dashes": false, 1792 | "datasource": "Prometheus", 1793 | "editable": true, 1794 | "error": false, 1795 | "fill": 1, 1796 | "gridPos": { 1797 | "h": 7, 1798 | "w": 8, 1799 | "x": 16, 1800 | "y": 46 1801 | }, 1802 | "id": 38, 1803 | "legend": { 1804 | "alignAsTable": true, 1805 | "avg": true, 1806 | "current": true, 1807 | "max": true, 1808 | "min": true, 1809 | "show": true, 1810 | "total": false, 1811 | "values": true 1812 | }, 1813 | "lines": true, 1814 | "linewidth": 2, 1815 | "links": [], 1816 | "nullPointMode": "null", 1817 | "percentage": false, 1818 | "pointradius": 5, 1819 | "points": false, 1820 | "renderer": "flot", 1821 | "seriesOverrides": [ 1822 | { 1823 | "alias": "Queue length", 1824 | "yaxis": 2 1825 | } 1826 | ], 1827 | "spaceLength": 10, 1828 | "stack": false, 1829 | "steppedLine": false, 1830 | "targets": [ 1831 | { 1832 | "expr": "prometheus_tsdb_blocks_loaded", 1833 | "format": "time_series", 1834 | "intervalFactor": 2, 1835 | "legendFormat": "Blocks Loaded", 1836 | "metric": "prometheus_local_storage_indexing_batch_sizes_sum", 1837 | "refId": "A", 1838 | "step": 10 1839 | } 1840 | ], 1841 | "thresholds": [], 1842 | "timeFrom": null, 1843 | "timeShift": null, 1844 | "title": "Blocks Loaded", 1845 | "tooltip": { 1846 | "msResolution": false, 1847 | "shared": true, 1848 | "sort": 0, 1849 | "value_type": "individual" 1850 | }, 1851 | "type": "graph", 1852 | "xaxis": { 1853 | "buckets": null, 1854 | "mode": "time", 1855 | "name": null, 1856 | "show": true, 1857 | "values": [] 1858 | }, 1859 | "yaxes": [ 1860 | { 1861 | "format": "short", 1862 | "label": null, 1863 | "logBase": 1, 1864 | "max": null, 1865 | "min": "0", 1866 | "show": true 1867 | }, 1868 | { 1869 | "format": "short", 1870 | "label": null, 1871 | "logBase": 1, 1872 | "max": null, 1873 | "min": "0", 1874 | "show": false 1875 | } 1876 | ], 1877 | "yaxis": { 1878 | "align": false, 1879 | "alignLevel": null 1880 | } 1881 | }, 1882 | { 1883 | "aliasColors": { 1884 | "Chunks": "#1F78C1", 1885 | "Chunks to persist": "#508642", 1886 | "Failed Compactions": "#bf1b00", 1887 | "Failed Reloads": "#bf1b00", 1888 | "Max chunks": "#052B51", 1889 | "Max to persist": "#3F6833" 1890 | }, 1891 | "bars": false, 1892 | "dashLength": 10, 1893 | "dashes": false, 1894 | "datasource": "Prometheus", 1895 | "editable": true, 1896 | "error": false, 1897 | "fill": 1, 1898 | "gridPos": { 1899 | "h": 7, 1900 | "w": 8, 1901 | "x": 0, 1902 | "y": 53 1903 | }, 1904 | "id": 40, 1905 | "legend": { 1906 | "alignAsTable": false, 1907 | "avg": false, 1908 | "current": false, 1909 | "max": false, 1910 | "min": false, 1911 | "show": true, 1912 | "total": false, 1913 | "values": false 1914 | }, 1915 | "lines": true, 1916 | "linewidth": 2, 1917 | "links": [], 1918 | "nullPointMode": "null", 1919 | "percentage": false, 1920 | "pointradius": 5, 1921 | "points": false, 1922 | "renderer": "flot", 1923 | "seriesOverrides": [], 1924 | "spaceLength": 10, 1925 | "stack": false, 1926 | "steppedLine": false, 1927 | "targets": [ 1928 | { 1929 | "expr": "rate(prometheus_tsdb_reloads_total[10m])", 1930 | "format": "time_series", 1931 | "interval": "", 1932 | "intervalFactor": 2, 1933 | "legendFormat": "Reloads", 1934 | "metric": "prometheus_local_storage_series_chunks_persisted_count", 1935 | "refId": "A", 1936 | "step": 10 1937 | } 1938 | ], 1939 | "thresholds": [], 1940 | "timeFrom": null, 1941 | "timeShift": null, 1942 | "title": "TSDB Reloads", 1943 | "tooltip": { 1944 | "msResolution": false, 1945 | "shared": true, 1946 | "sort": 0, 1947 | "value_type": "individual" 1948 | }, 1949 | "type": "graph", 1950 | "xaxis": { 1951 | "buckets": null, 1952 | "mode": "time", 1953 | "name": null, 1954 | "show": true, 1955 | "values": [] 1956 | }, 1957 | "yaxes": [ 1958 | { 1959 | "format": "ops", 1960 | "label": null, 1961 | "logBase": 1, 1962 | "max": null, 1963 | "min": "0", 1964 | "show": true 1965 | }, 1966 | { 1967 | "format": "short", 1968 | "label": null, 1969 | "logBase": 1, 1970 | "max": null, 1971 | "min": null, 1972 | "show": false 1973 | } 1974 | ], 1975 | "yaxis": { 1976 | "align": false, 1977 | "alignLevel": null 1978 | } 1979 | }, 1980 | { 1981 | "aliasColors": { 1982 | "Chunks": "#1F78C1", 1983 | "Chunks to persist": "#508642", 1984 | "Failed Compactions": "#bf1b00", 1985 | "Max chunks": "#052B51", 1986 | "Max to persist": "#3F6833", 1987 | "{instance=\"demo.robustperception.io:9090\",job=\"prometheus\"}": "#bf1b00" 1988 | }, 1989 | "bars": false, 1990 | "dashLength": 10, 1991 | "dashes": false, 1992 | "datasource": "Prometheus", 1993 | "editable": true, 1994 | "error": false, 1995 | "fill": 1, 1996 | "gridPos": { 1997 | "h": 7, 1998 | "w": 8, 1999 | "x": 8, 2000 | "y": 53 2001 | }, 2002 | "id": 44, 2003 | "legend": { 2004 | "alignAsTable": true, 2005 | "avg": true, 2006 | "current": true, 2007 | "max": true, 2008 | "min": true, 2009 | "show": true, 2010 | "total": false, 2011 | "values": true 2012 | }, 2013 | "lines": true, 2014 | "linewidth": 2, 2015 | "links": [], 2016 | "nullPointMode": "null", 2017 | "percentage": false, 2018 | "pointradius": 5, 2019 | "points": false, 2020 | "renderer": "flot", 2021 | "seriesOverrides": [], 2022 | "spaceLength": 10, 2023 | "stack": false, 2024 | "steppedLine": false, 2025 | "targets": [ 2026 | { 2027 | "expr": "rate(prometheus_tsdb_wal_corruptions_total[10m])", 2028 | "format": "time_series", 2029 | "interval": "", 2030 | "intervalFactor": 2, 2031 | "legendFormat": "WAL Corruptions", 2032 | "metric": "prometheus_local_storage_series_chunks_persisted_count", 2033 | "refId": "A", 2034 | "step": 10 2035 | }, 2036 | { 2037 | "expr": "rate(prometheus_tsdb_reloads_failures_total[10m])", 2038 | "format": "time_series", 2039 | "interval": "", 2040 | "intervalFactor": 2, 2041 | "legendFormat": "Reload Failures", 2042 | "metric": "prometheus_local_storage_series_chunks_persisted_count", 2043 | "refId": "B", 2044 | "step": 10 2045 | }, 2046 | { 2047 | "expr": "rate(prometheus_tsdb_head_series_not_found[10m])", 2048 | "format": "time_series", 2049 | "interval": "", 2050 | "intervalFactor": 2, 2051 | "legendFormat": "Head Series Not Found", 2052 | "metric": "prometheus_local_storage_series_chunks_persisted_count", 2053 | "refId": "C", 2054 | "step": 10 2055 | }, 2056 | { 2057 | "expr": "rate(prometheus_tsdb_compactions_failed_total[10m])", 2058 | "format": "time_series", 2059 | "interval": "", 2060 | "intervalFactor": 2, 2061 | "legendFormat": "Compaction Failures", 2062 | "metric": "prometheus_local_storage_series_chunks_persisted_count", 2063 | "refId": "D", 2064 | "step": 10 2065 | }, 2066 | { 2067 | "expr": "rate(prometheus_tsdb_retention_cutoffs_failures_total[10m])", 2068 | "format": "time_series", 2069 | "interval": "", 2070 | "intervalFactor": 2, 2071 | "legendFormat": "Retention Cutoff Failures", 2072 | "metric": "prometheus_local_storage_series_chunks_persisted_count", 2073 | "refId": "E", 2074 | "step": 10 2075 | } 2076 | ], 2077 | "thresholds": [], 2078 | "timeFrom": null, 2079 | "timeShift": null, 2080 | "title": "TSDB Problems", 2081 | "tooltip": { 2082 | "msResolution": false, 2083 | "shared": true, 2084 | "sort": 0, 2085 | "value_type": "individual" 2086 | }, 2087 | "type": "graph", 2088 | "xaxis": { 2089 | "buckets": null, 2090 | "mode": "time", 2091 | "name": null, 2092 | "show": true, 2093 | "values": [] 2094 | }, 2095 | "yaxes": [ 2096 | { 2097 | "format": "none", 2098 | "label": null, 2099 | "logBase": 1, 2100 | "max": null, 2101 | "min": "0", 2102 | "show": true 2103 | }, 2104 | { 2105 | "format": "short", 2106 | "label": null, 2107 | "logBase": 1, 2108 | "max": null, 2109 | "min": null, 2110 | "show": false 2111 | } 2112 | ], 2113 | "yaxis": { 2114 | "align": false, 2115 | "alignLevel": null 2116 | } 2117 | }, 2118 | { 2119 | "aliasColors": { 2120 | "Chunks": "#1F78C1", 2121 | "Chunks to persist": "#508642", 2122 | "Failed Compactions": "#bf1b00", 2123 | "Max chunks": "#052B51", 2124 | "Max to persist": "#3F6833" 2125 | }, 2126 | "bars": false, 2127 | "dashLength": 10, 2128 | "dashes": false, 2129 | "datasource": "Prometheus", 2130 | "editable": true, 2131 | "error": false, 2132 | "fill": 1, 2133 | "gridPos": { 2134 | "h": 7, 2135 | "w": 8, 2136 | "x": 16, 2137 | "y": 53 2138 | }, 2139 | "id": 42, 2140 | "legend": { 2141 | "alignAsTable": true, 2142 | "avg": true, 2143 | "current": true, 2144 | "max": true, 2145 | "min": true, 2146 | "show": true, 2147 | "total": false, 2148 | "values": true 2149 | }, 2150 | "lines": true, 2151 | "linewidth": 2, 2152 | "links": [], 2153 | "nullPointMode": "null", 2154 | "percentage": false, 2155 | "pointradius": 5, 2156 | "points": false, 2157 | "renderer": "flot", 2158 | "seriesOverrides": [], 2159 | "spaceLength": 10, 2160 | "stack": false, 2161 | "steppedLine": false, 2162 | "targets": [ 2163 | { 2164 | "expr": "rate(prometheus_tsdb_wal_fsync_duration_seconds_sum[1m]) / rate(prometheus_tsdb_wal_fsync_duration_seconds_count[1m])", 2165 | "format": "time_series", 2166 | "interval": "", 2167 | "intervalFactor": 2, 2168 | "legendFormat": "Fsync Latency", 2169 | "metric": "prometheus_local_storage_series_chunks_persisted_count", 2170 | "refId": "A", 2171 | "step": 10 2172 | }, 2173 | { 2174 | "expr": "rate(prometheus_tsdb_wal_truncate_duration_seconds_sum[1m]) / rate(prometheus_tsdb_wal_truncate_duration_seconds_count[1m])", 2175 | "format": "time_series", 2176 | "interval": "", 2177 | "intervalFactor": 2, 2178 | "legendFormat": "Truncate Latency", 2179 | "metric": "prometheus_local_storage_series_chunks_persisted_count", 2180 | "refId": "B", 2181 | "step": 10 2182 | } 2183 | ], 2184 | "thresholds": [], 2185 | "timeFrom": null, 2186 | "timeShift": null, 2187 | "title": "WAL Latencies", 2188 | "tooltip": { 2189 | "msResolution": false, 2190 | "shared": true, 2191 | "sort": 0, 2192 | "value_type": "individual" 2193 | }, 2194 | "type": "graph", 2195 | "xaxis": { 2196 | "buckets": null, 2197 | "mode": "time", 2198 | "name": null, 2199 | "show": true, 2200 | "values": [] 2201 | }, 2202 | "yaxes": [ 2203 | { 2204 | "format": "s", 2205 | "label": null, 2206 | "logBase": 1, 2207 | "max": null, 2208 | "min": "0", 2209 | "show": true 2210 | }, 2211 | { 2212 | "format": "short", 2213 | "label": null, 2214 | "logBase": 1, 2215 | "max": null, 2216 | "min": null, 2217 | "show": false 2218 | } 2219 | ], 2220 | "yaxis": { 2221 | "align": false, 2222 | "alignLevel": null 2223 | } 2224 | }, 2225 | { 2226 | "aliasColors": { 2227 | "Chunks": "#1F78C1", 2228 | "Chunks to persist": "#508642", 2229 | "Failed Compactions": "#bf1b00", 2230 | "Max chunks": "#052B51", 2231 | "Max to persist": "#3F6833" 2232 | }, 2233 | "bars": false, 2234 | "dashLength": 10, 2235 | "dashes": false, 2236 | "datasource": "Prometheus", 2237 | "editable": true, 2238 | "error": false, 2239 | "fill": 1, 2240 | "gridPos": { 2241 | "h": 7, 2242 | "w": 8, 2243 | "x": 0, 2244 | "y": 60 2245 | }, 2246 | "id": 46, 2247 | "legend": { 2248 | "alignAsTable": true, 2249 | "avg": true, 2250 | "current": true, 2251 | "max": true, 2252 | "min": true, 2253 | "show": true, 2254 | "total": false, 2255 | "values": true 2256 | }, 2257 | "lines": true, 2258 | "linewidth": 2, 2259 | "links": [], 2260 | "nullPointMode": "null", 2261 | "percentage": false, 2262 | "pointradius": 5, 2263 | "points": false, 2264 | "renderer": "flot", 2265 | "seriesOverrides": [], 2266 | "spaceLength": 10, 2267 | "stack": false, 2268 | "steppedLine": false, 2269 | "targets": [ 2270 | { 2271 | "expr": "rate(prometheus_tsdb_compactions_total[10m])", 2272 | "format": "time_series", 2273 | "interval": "", 2274 | "intervalFactor": 2, 2275 | "legendFormat": "Compactions", 2276 | "metric": "prometheus_local_storage_series_chunks_persisted_count", 2277 | "refId": "A", 2278 | "step": 10 2279 | } 2280 | ], 2281 | "thresholds": [], 2282 | "timeFrom": null, 2283 | "timeShift": null, 2284 | "title": "Compactions", 2285 | "tooltip": { 2286 | "msResolution": false, 2287 | "shared": true, 2288 | "sort": 0, 2289 | "value_type": "individual" 2290 | }, 2291 | "type": "graph", 2292 | "xaxis": { 2293 | "buckets": null, 2294 | "mode": "time", 2295 | "name": null, 2296 | "show": true, 2297 | "values": [] 2298 | }, 2299 | "yaxes": [ 2300 | { 2301 | "format": "none", 2302 | "label": null, 2303 | "logBase": 1, 2304 | "max": null, 2305 | "min": "0", 2306 | "show": true 2307 | }, 2308 | { 2309 | "format": "short", 2310 | "label": null, 2311 | "logBase": 1, 2312 | "max": null, 2313 | "min": null, 2314 | "show": false 2315 | } 2316 | ], 2317 | "yaxis": { 2318 | "align": false, 2319 | "alignLevel": null 2320 | } 2321 | }, 2322 | { 2323 | "aliasColors": { 2324 | "Chunks": "#1F78C1", 2325 | "Chunks to persist": "#508642", 2326 | "Max chunks": "#052B51", 2327 | "Max to persist": "#3F6833" 2328 | }, 2329 | "bars": false, 2330 | "dashLength": 10, 2331 | "dashes": false, 2332 | "datasource": "Prometheus", 2333 | "editable": true, 2334 | "error": false, 2335 | "fill": 1, 2336 | "gridPos": { 2337 | "h": 7, 2338 | "w": 8, 2339 | "x": 8, 2340 | "y": 60 2341 | }, 2342 | "id": 48, 2343 | "legend": { 2344 | "alignAsTable": true, 2345 | "avg": true, 2346 | "current": true, 2347 | "max": true, 2348 | "min": true, 2349 | "show": true, 2350 | "total": false, 2351 | "values": true 2352 | }, 2353 | "lines": true, 2354 | "linewidth": 2, 2355 | "links": [], 2356 | "nullPointMode": "null", 2357 | "percentage": false, 2358 | "pointradius": 5, 2359 | "points": false, 2360 | "renderer": "flot", 2361 | "seriesOverrides": [], 2362 | "spaceLength": 10, 2363 | "stack": false, 2364 | "steppedLine": false, 2365 | "targets": [ 2366 | { 2367 | "expr": "rate(prometheus_tsdb_compaction_duration_seconds_sum[10m])", 2368 | "format": "time_series", 2369 | "intervalFactor": 2, 2370 | "legendFormat": "Compaction Time", 2371 | "metric": "prometheus_local_storage_series_chunks_persisted_count", 2372 | "refId": "A", 2373 | "step": 10 2374 | } 2375 | ], 2376 | "thresholds": [], 2377 | "timeFrom": null, 2378 | "timeShift": null, 2379 | "title": "Compaction Time", 2380 | "tooltip": { 2381 | "msResolution": false, 2382 | "shared": true, 2383 | "sort": 0, 2384 | "value_type": "individual" 2385 | }, 2386 | "type": "graph", 2387 | "xaxis": { 2388 | "buckets": null, 2389 | "mode": "time", 2390 | "name": null, 2391 | "show": true, 2392 | "values": [] 2393 | }, 2394 | "yaxes": [ 2395 | { 2396 | "format": "s", 2397 | "label": null, 2398 | "logBase": 1, 2399 | "max": null, 2400 | "min": "0", 2401 | "show": true 2402 | }, 2403 | { 2404 | "format": "short", 2405 | "label": null, 2406 | "logBase": 1, 2407 | "max": null, 2408 | "min": null, 2409 | "show": false 2410 | } 2411 | ], 2412 | "yaxis": { 2413 | "align": false, 2414 | "alignLevel": null 2415 | } 2416 | }, 2417 | { 2418 | "aliasColors": { 2419 | "Allocated bytes": "#F9BA8F", 2420 | "Chunks": "#1F78C1", 2421 | "Chunks to persist": "#508642", 2422 | "Max chunks": "#052B51", 2423 | "Max to persist": "#3F6833", 2424 | "RSS": "#890F02" 2425 | }, 2426 | "bars": false, 2427 | "dashLength": 10, 2428 | "dashes": false, 2429 | "datasource": "Prometheus", 2430 | "editable": true, 2431 | "error": false, 2432 | "fill": 1, 2433 | "gridPos": { 2434 | "h": 7, 2435 | "w": 8, 2436 | "x": 16, 2437 | "y": 60 2438 | }, 2439 | "id": 50, 2440 | "legend": { 2441 | "alignAsTable": true, 2442 | "avg": true, 2443 | "current": true, 2444 | "max": true, 2445 | "min": true, 2446 | "show": true, 2447 | "total": false, 2448 | "values": true 2449 | }, 2450 | "lines": true, 2451 | "linewidth": 2, 2452 | "links": [], 2453 | "nullPointMode": "null", 2454 | "percentage": false, 2455 | "pointradius": 5, 2456 | "points": false, 2457 | "renderer": "flot", 2458 | "seriesOverrides": [], 2459 | "spaceLength": 10, 2460 | "stack": false, 2461 | "steppedLine": false, 2462 | "targets": [ 2463 | { 2464 | "expr": "rate(prometheus_tsdb_retention_cutoffs_total[10m])", 2465 | "format": "time_series", 2466 | "intervalFactor": 2, 2467 | "legendFormat": "Retention Cutoffs", 2468 | "metric": "last", 2469 | "refId": "A", 2470 | "step": 10 2471 | } 2472 | ], 2473 | "thresholds": [], 2474 | "timeFrom": null, 2475 | "timeShift": null, 2476 | "title": "Retention Cutoffs", 2477 | "tooltip": { 2478 | "msResolution": false, 2479 | "shared": true, 2480 | "sort": 0, 2481 | "value_type": "individual" 2482 | }, 2483 | "type": "graph", 2484 | "xaxis": { 2485 | "buckets": null, 2486 | "mode": "time", 2487 | "name": null, 2488 | "show": true, 2489 | "values": [] 2490 | }, 2491 | "yaxes": [ 2492 | { 2493 | "format": "none", 2494 | "label": null, 2495 | "logBase": 1, 2496 | "max": null, 2497 | "min": "0", 2498 | "show": true 2499 | }, 2500 | { 2501 | "format": "short", 2502 | "label": null, 2503 | "logBase": 1, 2504 | "max": null, 2505 | "min": null, 2506 | "show": false 2507 | } 2508 | ], 2509 | "yaxis": { 2510 | "align": false, 2511 | "alignLevel": null 2512 | } 2513 | }, 2514 | { 2515 | "aliasColors": { 2516 | "Chunks": "#1F78C1", 2517 | "Chunks to persist": "#508642", 2518 | "Max chunks": "#052B51", 2519 | "Max to persist": "#3F6833" 2520 | }, 2521 | "bars": false, 2522 | "dashLength": 10, 2523 | "dashes": false, 2524 | "datasource": "Prometheus", 2525 | "editable": true, 2526 | "error": false, 2527 | "fill": 1, 2528 | "gridPos": { 2529 | "h": 7, 2530 | "w": 8, 2531 | "x": 0, 2532 | "y": 67 2533 | }, 2534 | "id": 56, 2535 | "legend": { 2536 | "alignAsTable": true, 2537 | "avg": true, 2538 | "current": true, 2539 | "max": true, 2540 | "min": true, 2541 | "show": true, 2542 | "total": false, 2543 | "values": true 2544 | }, 2545 | "lines": true, 2546 | "linewidth": 2, 2547 | "links": [], 2548 | "nullPointMode": "null", 2549 | "percentage": false, 2550 | "pointradius": 5, 2551 | "points": false, 2552 | "renderer": "flot", 2553 | "seriesOverrides": [], 2554 | "spaceLength": 10, 2555 | "stack": false, 2556 | "steppedLine": false, 2557 | "targets": [ 2558 | { 2559 | "expr": "rate(prometheus_tsdb_compaction_chunk_samples_sum[10m]) / rate(prometheus_tsdb_compaction_chunk_samples_count[10m])", 2560 | "format": "time_series", 2561 | "intervalFactor": 2, 2562 | "legendFormat": "Chunk Samples", 2563 | "metric": "prometheus_local_storage_series_chunks_persisted_count", 2564 | "refId": "A", 2565 | "step": 10 2566 | } 2567 | ], 2568 | "thresholds": [], 2569 | "timeFrom": null, 2570 | "timeShift": null, 2571 | "title": "First Compaction, Avg Chunk Samples", 2572 | "tooltip": { 2573 | "msResolution": false, 2574 | "shared": true, 2575 | "sort": 0, 2576 | "value_type": "individual" 2577 | }, 2578 | "type": "graph", 2579 | "xaxis": { 2580 | "buckets": null, 2581 | "mode": "time", 2582 | "name": null, 2583 | "show": true, 2584 | "values": [] 2585 | }, 2586 | "yaxes": [ 2587 | { 2588 | "format": "none", 2589 | "label": null, 2590 | "logBase": 1, 2591 | "max": null, 2592 | "min": null, 2593 | "show": true 2594 | }, 2595 | { 2596 | "format": "short", 2597 | "label": null, 2598 | "logBase": 1, 2599 | "max": null, 2600 | "min": null, 2601 | "show": false 2602 | } 2603 | ], 2604 | "yaxis": { 2605 | "align": false, 2606 | "alignLevel": null 2607 | } 2608 | }, 2609 | { 2610 | "aliasColors": {}, 2611 | "bars": false, 2612 | "dashLength": 10, 2613 | "dashes": false, 2614 | "datasource": "Prometheus", 2615 | "editable": true, 2616 | "error": false, 2617 | "fill": 1, 2618 | "grid": {}, 2619 | "gridPos": { 2620 | "h": 7, 2621 | "w": 8, 2622 | "x": 8, 2623 | "y": 67 2624 | }, 2625 | "id": 10, 2626 | "legend": { 2627 | "alignAsTable": false, 2628 | "avg": false, 2629 | "current": false, 2630 | "max": false, 2631 | "min": false, 2632 | "show": true, 2633 | "total": false, 2634 | "values": false 2635 | }, 2636 | "lines": true, 2637 | "linewidth": 2, 2638 | "links": [], 2639 | "nullPointMode": "connected", 2640 | "percentage": false, 2641 | "pointradius": 5, 2642 | "points": false, 2643 | "renderer": "flot", 2644 | "seriesOverrides": [], 2645 | "spaceLength": 10, 2646 | "stack": false, 2647 | "steppedLine": false, 2648 | "targets": [ 2649 | { 2650 | "expr": "rate(prometheus_target_interval_length_seconds_count[5m])", 2651 | "format": "time_series", 2652 | "intervalFactor": 2, 2653 | "legendFormat": "{{ interval }}", 2654 | "refId": "A", 2655 | "step": 2 2656 | } 2657 | ], 2658 | "thresholds": [], 2659 | "timeFrom": null, 2660 | "timeShift": null, 2661 | "title": "Target Scrapes", 2662 | "tooltip": { 2663 | "msResolution": true, 2664 | "shared": true, 2665 | "sort": 0, 2666 | "value_type": "cumulative" 2667 | }, 2668 | "type": "graph", 2669 | "xaxis": { 2670 | "buckets": null, 2671 | "mode": "time", 2672 | "name": null, 2673 | "show": true, 2674 | "values": [] 2675 | }, 2676 | "yaxes": [ 2677 | { 2678 | "format": "short", 2679 | "label": null, 2680 | "logBase": 1, 2681 | "max": null, 2682 | "min": null, 2683 | "show": true 2684 | }, 2685 | { 2686 | "format": "short", 2687 | "label": null, 2688 | "logBase": 1, 2689 | "max": null, 2690 | "min": null, 2691 | "show": false 2692 | } 2693 | ], 2694 | "yaxis": { 2695 | "align": false, 2696 | "alignLevel": null 2697 | } 2698 | }, 2699 | { 2700 | "aliasColors": {}, 2701 | "bars": false, 2702 | "dashLength": 10, 2703 | "dashes": false, 2704 | "datasource": "Prometheus", 2705 | "editable": true, 2706 | "error": false, 2707 | "fill": 1, 2708 | "grid": {}, 2709 | "gridPos": { 2710 | "h": 7, 2711 | "w": 8, 2712 | "x": 16, 2713 | "y": 67 2714 | }, 2715 | "id": 11, 2716 | "legend": { 2717 | "alignAsTable": false, 2718 | "avg": false, 2719 | "current": false, 2720 | "max": false, 2721 | "min": false, 2722 | "show": true, 2723 | "total": false, 2724 | "values": false 2725 | }, 2726 | "lines": true, 2727 | "linewidth": 2, 2728 | "links": [], 2729 | "nullPointMode": "connected", 2730 | "percentage": false, 2731 | "pointradius": 5, 2732 | "points": false, 2733 | "renderer": "flot", 2734 | "seriesOverrides": [], 2735 | "spaceLength": 10, 2736 | "stack": false, 2737 | "steppedLine": false, 2738 | "targets": [ 2739 | { 2740 | "expr": "prometheus_target_interval_length_seconds{quantile!=\"0.01\", quantile!=\"0.05\"}", 2741 | "format": "time_series", 2742 | "intervalFactor": 2, 2743 | "legendFormat": "{{quantile}} ({{interval}})", 2744 | "refId": "A", 2745 | "step": 2 2746 | } 2747 | ], 2748 | "thresholds": [], 2749 | "timeFrom": null, 2750 | "timeShift": null, 2751 | "title": "Scrape Duration", 2752 | "tooltip": { 2753 | "msResolution": true, 2754 | "shared": true, 2755 | "sort": 0, 2756 | "value_type": "cumulative" 2757 | }, 2758 | "type": "graph", 2759 | "xaxis": { 2760 | "buckets": null, 2761 | "mode": "time", 2762 | "name": null, 2763 | "show": true, 2764 | "values": [] 2765 | }, 2766 | "yaxes": [ 2767 | { 2768 | "format": "short", 2769 | "label": null, 2770 | "logBase": 1, 2771 | "max": null, 2772 | "min": null, 2773 | "show": true 2774 | }, 2775 | { 2776 | "format": "short", 2777 | "label": null, 2778 | "logBase": 1, 2779 | "max": null, 2780 | "min": null, 2781 | "show": false 2782 | } 2783 | ], 2784 | "yaxis": { 2785 | "align": false, 2786 | "alignLevel": null 2787 | } 2788 | }, 2789 | { 2790 | "aliasColors": { 2791 | "Chunks": "#1F78C1", 2792 | "Chunks to persist": "#508642", 2793 | "Max chunks": "#052B51", 2794 | "Max to persist": "#3F6833" 2795 | }, 2796 | "bars": false, 2797 | "dashLength": 10, 2798 | "dashes": false, 2799 | "datasource": "Prometheus", 2800 | "description": "", 2801 | "editable": true, 2802 | "error": false, 2803 | "fill": 1, 2804 | "gridPos": { 2805 | "h": 7, 2806 | "w": 8, 2807 | "x": 0, 2808 | "y": 74 2809 | }, 2810 | "id": 62, 2811 | "legend": { 2812 | "alignAsTable": false, 2813 | "avg": false, 2814 | "current": false, 2815 | "max": false, 2816 | "min": false, 2817 | "show": true, 2818 | "total": false, 2819 | "values": false 2820 | }, 2821 | "lines": true, 2822 | "linewidth": 2, 2823 | "links": [], 2824 | "nullPointMode": "null", 2825 | "percentage": false, 2826 | "pointradius": 5, 2827 | "points": false, 2828 | "renderer": "flot", 2829 | "seriesOverrides": [], 2830 | "spaceLength": 10, 2831 | "stack": false, 2832 | "steppedLine": false, 2833 | "targets": [ 2834 | { 2835 | "expr": "rate(prometheus_http_request_duration_seconds_count[1m])", 2836 | "format": "time_series", 2837 | "intervalFactor": 2, 2838 | "legendFormat": "{{handler}}", 2839 | "metric": "prometheus_local_storage_memory_chunkdescs", 2840 | "refId": "A", 2841 | "step": 10 2842 | } 2843 | ], 2844 | "thresholds": [], 2845 | "timeFrom": null, 2846 | "timeShift": null, 2847 | "title": "HTTP requests", 2848 | "tooltip": { 2849 | "msResolution": false, 2850 | "shared": true, 2851 | "sort": 0, 2852 | "value_type": "individual" 2853 | }, 2854 | "type": "graph", 2855 | "xaxis": { 2856 | "buckets": null, 2857 | "mode": "time", 2858 | "name": null, 2859 | "show": true, 2860 | "values": [] 2861 | }, 2862 | "yaxes": [ 2863 | { 2864 | "format": "reqps", 2865 | "label": null, 2866 | "logBase": 1, 2867 | "max": null, 2868 | "min": "0", 2869 | "show": true 2870 | }, 2871 | { 2872 | "format": "short", 2873 | "label": null, 2874 | "logBase": 1, 2875 | "max": null, 2876 | "min": null, 2877 | "show": false 2878 | } 2879 | ], 2880 | "yaxis": { 2881 | "align": false, 2882 | "alignLevel": null 2883 | } 2884 | }, 2885 | { 2886 | "aliasColors": { 2887 | "Chunks": "#1F78C1", 2888 | "Chunks to persist": "#508642", 2889 | "Max chunks": "#052B51", 2890 | "Max to persist": "#3F6833" 2891 | }, 2892 | "bars": false, 2893 | "dashLength": 10, 2894 | "dashes": false, 2895 | "datasource": "Prometheus", 2896 | "description": "", 2897 | "editable": true, 2898 | "error": false, 2899 | "fill": 1, 2900 | "gridPos": { 2901 | "h": 7, 2902 | "w": 8, 2903 | "x": 8, 2904 | "y": 74 2905 | }, 2906 | "id": 64, 2907 | "legend": { 2908 | "alignAsTable": false, 2909 | "avg": false, 2910 | "current": false, 2911 | "max": false, 2912 | "min": false, 2913 | "show": true, 2914 | "total": false, 2915 | "values": false 2916 | }, 2917 | "lines": true, 2918 | "linewidth": 2, 2919 | "links": [], 2920 | "nullPointMode": "null", 2921 | "percentage": false, 2922 | "pointradius": 5, 2923 | "points": false, 2924 | "renderer": "flot", 2925 | "seriesOverrides": [], 2926 | "spaceLength": 10, 2927 | "stack": false, 2928 | "steppedLine": false, 2929 | "targets": [ 2930 | { 2931 | "expr": "rate(prometheus_http_request_duration_seconds_sum[1m]) / rate(prometheus_http_request_duration_seconds_count[1m])", 2932 | "format": "time_series", 2933 | "intervalFactor": 2, 2934 | "legendFormat": "{{handler}}", 2935 | "metric": "prometheus_local_storage_memory_chunkdescs", 2936 | "refId": "A", 2937 | "step": 10 2938 | } 2939 | ], 2940 | "thresholds": [], 2941 | "timeFrom": null, 2942 | "timeShift": null, 2943 | "title": "HTTP request latency", 2944 | "tooltip": { 2945 | "msResolution": false, 2946 | "shared": true, 2947 | "sort": 0, 2948 | "value_type": "individual" 2949 | }, 2950 | "type": "graph", 2951 | "xaxis": { 2952 | "buckets": null, 2953 | "mode": "time", 2954 | "name": null, 2955 | "show": true, 2956 | "values": [] 2957 | }, 2958 | "yaxes": [ 2959 | { 2960 | "format": "s", 2961 | "label": null, 2962 | "logBase": 1, 2963 | "max": null, 2964 | "min": "0", 2965 | "show": true 2966 | }, 2967 | { 2968 | "format": "short", 2969 | "label": null, 2970 | "logBase": 1, 2971 | "max": null, 2972 | "min": null, 2973 | "show": false 2974 | } 2975 | ], 2976 | "yaxis": { 2977 | "align": false, 2978 | "alignLevel": null 2979 | } 2980 | }, 2981 | { 2982 | "aliasColors": { 2983 | "Chunks": "#1F78C1", 2984 | "Chunks to persist": "#508642", 2985 | "Max chunks": "#052B51", 2986 | "Max to persist": "#3F6833" 2987 | }, 2988 | "bars": false, 2989 | "dashLength": 10, 2990 | "dashes": false, 2991 | "datasource": "Prometheus", 2992 | "description": "", 2993 | "editable": true, 2994 | "error": false, 2995 | "fill": 1, 2996 | "gridPos": { 2997 | "h": 7, 2998 | "w": 8, 2999 | "x": 16, 3000 | "y": 74 3001 | }, 3002 | "id": 66, 3003 | "legend": { 3004 | "avg": false, 3005 | "current": false, 3006 | "max": false, 3007 | "min": false, 3008 | "show": true, 3009 | "total": false, 3010 | "values": false 3011 | }, 3012 | "lines": true, 3013 | "linewidth": 2, 3014 | "links": [], 3015 | "nullPointMode": "null", 3016 | "percentage": false, 3017 | "pointradius": 5, 3018 | "points": false, 3019 | "renderer": "flot", 3020 | "seriesOverrides": [], 3021 | "spaceLength": 10, 3022 | "stack": true, 3023 | "steppedLine": false, 3024 | "targets": [ 3025 | { 3026 | "expr": "rate(prometheus_http_request_duration_seconds_sum[1m])", 3027 | "format": "time_series", 3028 | "intervalFactor": 2, 3029 | "legendFormat": "{{handler}}", 3030 | "metric": "prometheus_local_storage_memory_chunkdescs", 3031 | "refId": "A", 3032 | "step": 10 3033 | } 3034 | ], 3035 | "thresholds": [], 3036 | "timeFrom": null, 3037 | "timeShift": null, 3038 | "title": "Time spent in HTTP requests", 3039 | "tooltip": { 3040 | "msResolution": false, 3041 | "shared": true, 3042 | "sort": 0, 3043 | "value_type": "individual" 3044 | }, 3045 | "type": "graph", 3046 | "xaxis": { 3047 | "buckets": null, 3048 | "mode": "time", 3049 | "name": null, 3050 | "show": true, 3051 | "values": [] 3052 | }, 3053 | "yaxes": [ 3054 | { 3055 | "format": "s", 3056 | "label": null, 3057 | "logBase": 1, 3058 | "max": null, 3059 | "min": "0", 3060 | "show": true 3061 | }, 3062 | { 3063 | "format": "short", 3064 | "label": null, 3065 | "logBase": 1, 3066 | "max": null, 3067 | "min": null, 3068 | "show": false 3069 | } 3070 | ], 3071 | "yaxis": { 3072 | "align": false, 3073 | "alignLevel": null 3074 | } 3075 | }, 3076 | { 3077 | "aliasColors": { 3078 | "Chunks": "#1F78C1", 3079 | "Chunks to persist": "#508642", 3080 | "Max chunks": "#052B51", 3081 | "Max to persist": "#3F6833" 3082 | }, 3083 | "bars": false, 3084 | "dashLength": 10, 3085 | "dashes": false, 3086 | "datasource": "Prometheus", 3087 | "description": "Time spent in each mode, per second", 3088 | "editable": true, 3089 | "error": false, 3090 | "fill": 1, 3091 | "gridPos": { 3092 | "h": 7, 3093 | "w": 8, 3094 | "x": 0, 3095 | "y": 81 3096 | }, 3097 | "id": 68, 3098 | "legend": { 3099 | "alignAsTable": true, 3100 | "avg": true, 3101 | "current": true, 3102 | "max": true, 3103 | "min": true, 3104 | "show": true, 3105 | "total": false, 3106 | "values": true 3107 | }, 3108 | "lines": true, 3109 | "linewidth": 1, 3110 | "links": [], 3111 | "nullPointMode": "null", 3112 | "percentage": false, 3113 | "pointradius": 5, 3114 | "points": false, 3115 | "renderer": "flot", 3116 | "seriesOverrides": [], 3117 | "spaceLength": 10, 3118 | "stack": true, 3119 | "steppedLine": false, 3120 | "targets": [ 3121 | { 3122 | "expr": "rate(prometheus_engine_query_duration_seconds_sum[1m])", 3123 | "format": "time_series", 3124 | "intervalFactor": 2, 3125 | "legendFormat": "{{slice}}", 3126 | "metric": "prometheus_local_storage_memory_chunkdescs", 3127 | "refId": "A", 3128 | "step": 10 3129 | } 3130 | ], 3131 | "thresholds": [], 3132 | "timeFrom": null, 3133 | "timeShift": null, 3134 | "title": "Query engine timings", 3135 | "tooltip": { 3136 | "msResolution": false, 3137 | "shared": true, 3138 | "sort": 0, 3139 | "value_type": "individual" 3140 | }, 3141 | "type": "graph", 3142 | "xaxis": { 3143 | "buckets": null, 3144 | "mode": "time", 3145 | "name": null, 3146 | "show": true, 3147 | "values": [] 3148 | }, 3149 | "yaxes": [ 3150 | { 3151 | "format": "s", 3152 | "label": null, 3153 | "logBase": 1, 3154 | "max": null, 3155 | "min": "0", 3156 | "show": true 3157 | }, 3158 | { 3159 | "format": "short", 3160 | "label": null, 3161 | "logBase": 1, 3162 | "max": null, 3163 | "min": null, 3164 | "show": false 3165 | } 3166 | ], 3167 | "yaxis": { 3168 | "align": false, 3169 | "alignLevel": null 3170 | } 3171 | }, 3172 | { 3173 | "aliasColors": { 3174 | "Chunks": "#1F78C1", 3175 | "Chunks to persist": "#508642", 3176 | "Max chunks": "#052B51", 3177 | "Max to persist": "#3F6833" 3178 | }, 3179 | "bars": false, 3180 | "dashLength": 10, 3181 | "dashes": false, 3182 | "datasource": "Prometheus", 3183 | "editable": true, 3184 | "error": false, 3185 | "fill": 1, 3186 | "gridPos": { 3187 | "h": 7, 3188 | "w": 8, 3189 | "x": 8, 3190 | "y": 81 3191 | }, 3192 | "id": 70, 3193 | "legend": { 3194 | "alignAsTable": true, 3195 | "avg": true, 3196 | "current": true, 3197 | "max": true, 3198 | "min": true, 3199 | "show": true, 3200 | "total": false, 3201 | "values": true 3202 | }, 3203 | "lines": true, 3204 | "linewidth": 2, 3205 | "links": [], 3206 | "nullPointMode": "null", 3207 | "percentage": false, 3208 | "pointradius": 5, 3209 | "points": false, 3210 | "renderer": "flot", 3211 | "seriesOverrides": [], 3212 | "spaceLength": 10, 3213 | "stack": true, 3214 | "steppedLine": false, 3215 | "targets": [ 3216 | { 3217 | "expr": "rate(prometheus_rule_group_iterations_missed_total[1m]) ", 3218 | "format": "time_series", 3219 | "intervalFactor": 2, 3220 | "legendFormat": "Rule group missed", 3221 | "metric": "prometheus_local_storage_memory_chunkdescs", 3222 | "refId": "B", 3223 | "step": 10 3224 | }, 3225 | { 3226 | "expr": "rate(prometheus_rule_evaluation_failures_total[1m])", 3227 | "format": "time_series", 3228 | "intervalFactor": 2, 3229 | "legendFormat": "Rule evals failed", 3230 | "metric": "prometheus_local_storage_memory_chunkdescs", 3231 | "refId": "C", 3232 | "step": 10 3233 | } 3234 | ], 3235 | "thresholds": [], 3236 | "timeFrom": null, 3237 | "timeShift": null, 3238 | "title": "Rule group evaulation problems", 3239 | "tooltip": { 3240 | "msResolution": false, 3241 | "shared": true, 3242 | "sort": 0, 3243 | "value_type": "individual" 3244 | }, 3245 | "type": "graph", 3246 | "xaxis": { 3247 | "buckets": null, 3248 | "mode": "time", 3249 | "name": null, 3250 | "show": true, 3251 | "values": [] 3252 | }, 3253 | "yaxes": [ 3254 | { 3255 | "format": "short", 3256 | "label": null, 3257 | "logBase": 1, 3258 | "max": null, 3259 | "min": "0", 3260 | "show": true 3261 | }, 3262 | { 3263 | "format": "short", 3264 | "label": null, 3265 | "logBase": 1, 3266 | "max": null, 3267 | "min": null, 3268 | "show": false 3269 | } 3270 | ], 3271 | "yaxis": { 3272 | "align": false, 3273 | "alignLevel": null 3274 | } 3275 | }, 3276 | { 3277 | "aliasColors": { 3278 | "Chunks": "#1F78C1", 3279 | "Chunks to persist": "#508642", 3280 | "Max chunks": "#052B51", 3281 | "Max to persist": "#3F6833" 3282 | }, 3283 | "bars": false, 3284 | "dashLength": 10, 3285 | "dashes": false, 3286 | "datasource": "Prometheus", 3287 | "editable": true, 3288 | "error": false, 3289 | "fill": 1, 3290 | "gridPos": { 3291 | "h": 7, 3292 | "w": 8, 3293 | "x": 16, 3294 | "y": 81 3295 | }, 3296 | "id": 72, 3297 | "legend": { 3298 | "alignAsTable": true, 3299 | "avg": true, 3300 | "current": true, 3301 | "max": true, 3302 | "min": true, 3303 | "show": true, 3304 | "total": false, 3305 | "values": true 3306 | }, 3307 | "lines": true, 3308 | "linewidth": 2, 3309 | "links": [], 3310 | "nullPointMode": "null", 3311 | "percentage": false, 3312 | "pointradius": 5, 3313 | "points": false, 3314 | "renderer": "flot", 3315 | "seriesOverrides": [], 3316 | "spaceLength": 10, 3317 | "stack": false, 3318 | "steppedLine": false, 3319 | "targets": [ 3320 | { 3321 | "expr": "rate(prometheus_rule_group_duration_seconds_sum[1m])", 3322 | "format": "time_series", 3323 | "intervalFactor": 2, 3324 | "legendFormat": "Rule evaluation duration", 3325 | "metric": "prometheus_local_storage_memory_chunkdescs", 3326 | "refId": "A", 3327 | "step": 10 3328 | } 3329 | ], 3330 | "thresholds": [], 3331 | "timeFrom": null, 3332 | "timeShift": null, 3333 | "title": "Evaluation time of rule groups", 3334 | "tooltip": { 3335 | "msResolution": false, 3336 | "shared": true, 3337 | "sort": 0, 3338 | "value_type": "individual" 3339 | }, 3340 | "type": "graph", 3341 | "xaxis": { 3342 | "buckets": null, 3343 | "mode": "time", 3344 | "name": null, 3345 | "show": true, 3346 | "values": [] 3347 | }, 3348 | "yaxes": [ 3349 | { 3350 | "format": "s", 3351 | "label": null, 3352 | "logBase": 1, 3353 | "max": null, 3354 | "min": "0", 3355 | "show": true 3356 | }, 3357 | { 3358 | "format": "short", 3359 | "label": null, 3360 | "logBase": 1, 3361 | "max": null, 3362 | "min": null, 3363 | "show": false 3364 | } 3365 | ], 3366 | "yaxis": { 3367 | "align": false, 3368 | "alignLevel": null 3369 | } 3370 | } 3371 | ], 3372 | "time": { 3373 | "from": "now-15m", 3374 | "to": "now" 3375 | }, 3376 | "timepicker": { 3377 | "refresh_intervals": [ 3378 | "5s", 3379 | "10s", 3380 | "30s", 3381 | "1m", 3382 | "5m", 3383 | "15m", 3384 | "30m", 3385 | "1h", 3386 | "2h", 3387 | "1d" 3388 | ], 3389 | "time_options": [ 3390 | "5m", 3391 | "15m", 3392 | "1h", 3393 | "6h", 3394 | "12h", 3395 | "24h", 3396 | "2d", 3397 | "7d", 3398 | "30d" 3399 | ] 3400 | }, 3401 | "templating": { 3402 | "list": [] 3403 | }, 3404 | "annotations": { 3405 | "list": [] 3406 | }, 3407 | "refresh": "10s", 3408 | "schemaVersion": 12, 3409 | "version": 22, 3410 | "links": [], 3411 | "gnetId": null 3412 | }} -------------------------------------------------------------------------------- /docker/grafana/dashboards/nginx_container.json: -------------------------------------------------------------------------------- 1 | {"dashboard": { 2 | "id": null, 3 | "title": "Nginx", 4 | "description": "Nginx exporter metrics", 5 | "tags": [ 6 | "nginx" 7 | ], 8 | "style": "dark", 9 | "timezone": "browser", 10 | "editable": true, 11 | "hideControls": false, 12 | "sharedCrosshair": true, 13 | "rows": [ 14 | { 15 | "collapse": false, 16 | "editable": true, 17 | "height": "250px", 18 | "panels": [ 19 | { 20 | "aliasColors": {}, 21 | "bars": false, 22 | "datasource": "Prometheus", 23 | "decimals": 2, 24 | "editable": true, 25 | "error": false, 26 | "fill": 1, 27 | "grid": { 28 | "threshold1": null, 29 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 30 | "threshold2": null, 31 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 32 | }, 33 | "id": 3, 34 | "isNew": true, 35 | "legend": { 36 | "alignAsTable": true, 37 | "avg": true, 38 | "current": true, 39 | "max": true, 40 | "min": true, 41 | "rightSide": true, 42 | "show": true, 43 | "total": false, 44 | "values": true 45 | }, 46 | "lines": true, 47 | "linewidth": 2, 48 | "links": [], 49 | "nullPointMode": "connected", 50 | "percentage": false, 51 | "pointradius": 5, 52 | "points": false, 53 | "renderer": "flot", 54 | "seriesOverrides": [], 55 | "span": 12, 56 | "stack": false, 57 | "steppedLine": false, 58 | "targets": [ 59 | { 60 | "expr": "sum(irate(nginx_connections_processed_total{stage=\"any\"}[5m])) by (stage)", 61 | "hide": false, 62 | "interval": "", 63 | "intervalFactor": 10, 64 | "legendFormat": "requests", 65 | "metric": "", 66 | "refId": "B", 67 | "step": 10 68 | } 69 | ], 70 | "timeFrom": null, 71 | "timeShift": null, 72 | "title": "Requests/sec", 73 | "tooltip": { 74 | "msResolution": false, 75 | "shared": true, 76 | "sort": 0, 77 | "value_type": "cumulative" 78 | }, 79 | "type": "graph", 80 | "xaxis": { 81 | "show": true 82 | }, 83 | "yaxes": [ 84 | { 85 | "format": "short", 86 | "label": null, 87 | "logBase": 1, 88 | "max": null, 89 | "min": 0, 90 | "show": true 91 | }, 92 | { 93 | "format": "short", 94 | "label": null, 95 | "logBase": 1, 96 | "max": null, 97 | "min": null, 98 | "show": true 99 | } 100 | ] 101 | }, 102 | { 103 | "aliasColors": {}, 104 | "bars": false, 105 | "datasource": "Prometheus", 106 | "decimals": 2, 107 | "editable": true, 108 | "error": false, 109 | "fill": 1, 110 | "grid": { 111 | "threshold1": null, 112 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 113 | "threshold2": null, 114 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 115 | }, 116 | "id": 2, 117 | "isNew": true, 118 | "legend": { 119 | "alignAsTable": true, 120 | "avg": true, 121 | "current": true, 122 | "max": true, 123 | "min": true, 124 | "rightSide": true, 125 | "show": true, 126 | "total": false, 127 | "values": true 128 | }, 129 | "lines": true, 130 | "linewidth": 2, 131 | "links": [], 132 | "nullPointMode": "connected", 133 | "percentage": false, 134 | "pointradius": 5, 135 | "points": false, 136 | "renderer": "flot", 137 | "seriesOverrides": [], 138 | "span": 12, 139 | "stack": false, 140 | "steppedLine": false, 141 | "targets": [ 142 | { 143 | "expr": "sum(nginx_connections_current) by (state)", 144 | "interval": "", 145 | "intervalFactor": 2, 146 | "legendFormat": "{{state}}", 147 | "metric": "", 148 | "refId": "A", 149 | "step": 2 150 | } 151 | ], 152 | "timeFrom": null, 153 | "timeShift": null, 154 | "title": "Connections", 155 | "tooltip": { 156 | "msResolution": false, 157 | "shared": true, 158 | "sort": 0, 159 | "value_type": "cumulative" 160 | }, 161 | "type": "graph", 162 | "xaxis": { 163 | "show": true 164 | }, 165 | "yaxes": [ 166 | { 167 | "format": "short", 168 | "label": null, 169 | "logBase": 1, 170 | "max": null, 171 | "min": 0, 172 | "show": true 173 | }, 174 | { 175 | "format": "short", 176 | "label": null, 177 | "logBase": 1, 178 | "max": null, 179 | "min": null, 180 | "show": true 181 | } 182 | ] 183 | }, 184 | { 185 | "aliasColors": {}, 186 | "bars": false, 187 | "datasource": "Prometheus", 188 | "decimals": 2, 189 | "editable": true, 190 | "error": false, 191 | "fill": 1, 192 | "grid": { 193 | "threshold1": null, 194 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 195 | "threshold2": null, 196 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 197 | }, 198 | "id": 1, 199 | "isNew": true, 200 | "legend": { 201 | "alignAsTable": true, 202 | "avg": true, 203 | "current": true, 204 | "max": true, 205 | "min": true, 206 | "rightSide": true, 207 | "show": true, 208 | "total": false, 209 | "values": true 210 | }, 211 | "lines": true, 212 | "linewidth": 2, 213 | "links": [], 214 | "nullPointMode": "connected", 215 | "percentage": false, 216 | "pointradius": 5, 217 | "points": false, 218 | "renderer": "flot", 219 | "seriesOverrides": [], 220 | "span": 12, 221 | "stack": false, 222 | "steppedLine": false, 223 | "targets": [ 224 | { 225 | "expr": "sum(irate(nginx_connections_processed_total{stage!=\"any\"}[5m])) by (stage)", 226 | "hide": false, 227 | "interval": "", 228 | "intervalFactor": 10, 229 | "legendFormat": "{{stage}}", 230 | "metric": "", 231 | "refId": "B", 232 | "step": 10 233 | } 234 | ], 235 | "timeFrom": null, 236 | "timeShift": null, 237 | "title": "Connections rate", 238 | "tooltip": { 239 | "msResolution": false, 240 | "shared": true, 241 | "sort": 0, 242 | "value_type": "cumulative" 243 | }, 244 | "type": "graph", 245 | "xaxis": { 246 | "show": true 247 | }, 248 | "yaxes": [ 249 | { 250 | "format": "short", 251 | "label": null, 252 | "logBase": 1, 253 | "max": null, 254 | "min": 0, 255 | "show": true 256 | }, 257 | { 258 | "format": "short", 259 | "label": null, 260 | "logBase": 1, 261 | "max": null, 262 | "min": null, 263 | "show": true 264 | } 265 | ] 266 | } 267 | ], 268 | "title": "Nginx exporter metrics" 269 | }, 270 | { 271 | "collapse": false, 272 | "editable": true, 273 | "height": "250px", 274 | "panels": [ 275 | { 276 | "aliasColors": {}, 277 | "bars": false, 278 | "datasource": null, 279 | "editable": true, 280 | "error": false, 281 | "fill": 1, 282 | "grid": { 283 | "threshold1": null, 284 | "threshold1Color": "rgba(216, 200, 27, 0.27)", 285 | "threshold2": null, 286 | "threshold2Color": "rgba(234, 112, 112, 0.22)" 287 | }, 288 | "id": 4, 289 | "isNew": true, 290 | "legend": { 291 | "alignAsTable": true, 292 | "avg": true, 293 | "current": true, 294 | "max": true, 295 | "min": true, 296 | "rightSide": true, 297 | "show": true, 298 | "total": false, 299 | "values": true 300 | }, 301 | "lines": true, 302 | "linewidth": 2, 303 | "links": [], 304 | "nullPointMode": "connected", 305 | "percentage": false, 306 | "pointradius": 5, 307 | "points": false, 308 | "renderer": "flot", 309 | "seriesOverrides": [], 310 | "span": 12, 311 | "stack": false, 312 | "steppedLine": false, 313 | "targets": [ 314 | { 315 | "expr": "sum(rate(container_cpu_usage_seconds_total{name=~\"nginx\"}[5m])) / count(node_cpu_seconds_total{mode=\"system\"}) * 100", 316 | "intervalFactor": 2, 317 | "legendFormat": "nginx", 318 | "refId": "A", 319 | "step": 2 320 | } 321 | ], 322 | "timeFrom": null, 323 | "timeShift": null, 324 | "title": "CPU usage", 325 | "tooltip": { 326 | "msResolution": false, 327 | "shared": true, 328 | "sort": 0, 329 | "value_type": "cumulative" 330 | }, 331 | "type": "graph", 332 | "xaxis": { 333 | "show": true 334 | }, 335 | "yaxes": [ 336 | { 337 | "format": "short", 338 | "label": null, 339 | "logBase": 1, 340 | "max": null, 341 | "min": null, 342 | "show": true 343 | }, 344 | { 345 | "format": "short", 346 | "label": null, 347 | "logBase": 1, 348 | "max": null, 349 | "min": null, 350 | "show": true 351 | } 352 | ] 353 | } 354 | ], 355 | "title": "Nginx container metrics" 356 | } 357 | ], 358 | "time": { 359 | "from": "now-15m", 360 | "to": "now" 361 | }, 362 | "timepicker": { 363 | "refresh_intervals": [ 364 | "5s", 365 | "10s", 366 | "30s", 367 | "1m", 368 | "5m", 369 | "15m", 370 | "30m", 371 | "1h", 372 | "2h", 373 | "1d" 374 | ], 375 | "time_options": [ 376 | "5m", 377 | "15m", 378 | "1h", 379 | "6h", 380 | "12h", 381 | "24h", 382 | "2d", 383 | "7d", 384 | "30d" 385 | ] 386 | }, 387 | "templating": { 388 | "list": [] 389 | }, 390 | "annotations": { 391 | "list": [] 392 | }, 393 | "refresh": "10s", 394 | "schemaVersion": 12, 395 | "version": 9, 396 | "links": [], 397 | "gnetId": null 398 | }} 399 | -------------------------------------------------------------------------------- /docker/grafana/datasources/Prometheus.json: -------------------------------------------------------------------------------- 1 | { 2 | "name":"Prometheus", 3 | "type":"prometheus", 4 | "url":"http://prometheus:9090", 5 | "access":"proxy", 6 | "basicAuth":false 7 | } -------------------------------------------------------------------------------- /docker/grafana/setup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | # Taken from https://github.com/grafana/grafana-docker/issues/74 4 | 5 | # Script to configure grafana datasources and dashboards. 6 | # Intended to be run before grafana entrypoint... 7 | # Image: grafana/grafana:4.1.2 8 | # ENTRYPOINT [\"/run.sh\"]" 9 | 10 | GRAFANA_URL=${GRAFANA_URL:-http://$GF_SECURITY_ADMIN_USER:$GF_SECURITY_ADMIN_PASSWORD@localhost:3000} 11 | #GRAFANA_URL=http://grafana-plain.k8s.playground1.aws.ad.zopa.com 12 | DATASOURCES_PATH=${DATASOURCES_PATH:-/etc/grafana/datasources} 13 | DASHBOARDS_PATH=${DASHBOARDS_PATH:-/etc/grafana/dashboards} 14 | 15 | # Generic function to call the Vault API 16 | grafana_api() { 17 | local verb=$1 18 | local url=$2 19 | local params=$3 20 | local bodyfile=$4 21 | local response 22 | local cmd 23 | 24 | cmd="curl -L -s --fail -H \"Accept: application/json\" -H \"Content-Type: application/json\" -X ${verb} -k ${GRAFANA_URL}${url}" 25 | [[ -n "${params}" ]] && cmd="${cmd} -d \"${params}\"" 26 | [[ -n "${bodyfile}" ]] && cmd="${cmd} --data @${bodyfile}" 27 | echo "Running ${cmd}" 28 | eval ${cmd} || return 1 29 | return 0 30 | } 31 | 32 | wait_for_api() { 33 | while ! grafana_api GET /api/user/preferences 34 | do 35 | sleep 5 36 | done 37 | } 38 | 39 | install_datasources() { 40 | local datasource 41 | 42 | for datasource in ${DATASOURCES_PATH}/*.json 43 | do 44 | if [[ -f "${datasource}" ]]; then 45 | echo "Installing datasource ${datasource}" 46 | if grafana_api POST /api/datasources "" "${datasource}"; then 47 | echo "installed ok" 48 | else 49 | echo "install failed" 50 | fi 51 | fi 52 | done 53 | } 54 | 55 | install_dashboards() { 56 | local dashboard 57 | 58 | for dashboard in ${DASHBOARDS_PATH}/*.json 59 | do 60 | if [[ -f "${dashboard}" ]]; then 61 | echo "Installing dashboard ${dashboard}" 62 | 63 | if grafana_api POST /api/dashboards/db "" "${dashboard}"; then 64 | echo "installed ok" 65 | else 66 | echo "install failed" 67 | fi 68 | 69 | fi 70 | done 71 | } 72 | 73 | configure_grafana() { 74 | wait_for_api 75 | install_datasources 76 | install_dashboards 77 | } 78 | 79 | echo "Running configure_grafana in the background..." 80 | configure_grafana & 81 | /run.sh 82 | exit 0 -------------------------------------------------------------------------------- /docker/kibana/config.yml: -------------------------------------------------------------------------------- 1 | # Kibana is served by a back end server. This setting specifies the port to use. 2 | server.port: 5601 3 | 4 | # Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values. 5 | # The default is 'localhost', which usually means remote machines will not be able to connect. 6 | # To allow connections from remote users, set this parameter to a non-loopback address. 7 | #server.host: "localhost" 8 | 9 | # Enables you to specify a path to mount Kibana at if you are running behind a proxy. This only affects 10 | # the URLs generated by Kibana, your proxy is expected to remove the basePath value before forwarding requests 11 | # to Kibana. This setting cannot end in a slash. 12 | #server.basePath: "" 13 | 14 | # The maximum payload size in bytes for incoming server requests. 15 | #server.maxPayloadBytes: 1048576 16 | 17 | # The Kibana server's name. This is used for display purposes. 18 | server.name: "kibana" 19 | 20 | # The URL of the Elasticsearch instance to use for all your queries. 21 | #elasticsearch.url: "http://192.16.0.22:9200" 22 | #elasticsearch.url: "http://elasticsearch:9200" 23 | elasticsearch.hosts: [ "http://elasticsearch:9200" ] 24 | 25 | # When this setting's value is true Kibana uses the hostname specified in the server.host 26 | # setting. When the value of this setting is false, Kibana uses the hostname of the host 27 | # that connects to this Kibana instance. 28 | #elasticsearch.preserveHost: true 29 | 30 | # Kibana uses an index in Elasticsearch to store saved searches, visualizations and 31 | # dashboards. Kibana creates a new index if the index doesn't already exist. 32 | kibana.index: ".kibana" 33 | 34 | # The default application to load. 35 | #kibana.defaultAppId: "discover" 36 | 37 | # If your Elasticsearch is protected with basic authentication, these settings provide 38 | # the username and password that the Kibana server uses to perform maintenance on the Kibana 39 | # index at startup. Your Kibana users still need to authenticate with Elasticsearch, which 40 | # is proxied through the Kibana server. 41 | #elasticsearch.username: "user" 42 | #elasticsearch.password: "pass" 43 | 44 | # Paths to the PEM-format SSL certificate and SSL key files, respectively. These 45 | # files enable SSL for outgoing requests from the Kibana server to the browser. 46 | #server.ssl.cert: /path/to/your/server.crt 47 | #server.ssl.key: /path/to/your/server.key 48 | 49 | # Optional settings that provide the paths to the PEM-format SSL certificate and key files. 50 | # These files validate that your Elasticsearch backend uses the same key files. 51 | #elasticsearch.ssl.cert: /path/to/your/client.crt 52 | #elasticsearch.ssl.key: /path/to/your/client.key 53 | 54 | # Optional setting that enables you to specify a path to the PEM file for the certificate 55 | # authority for your Elasticsearch instance. 56 | #elasticsearch.ssl.ca: /path/to/your/CA.pem 57 | 58 | # To disregard the validity of SSL certificates, change this setting's value to false. 59 | #elasticsearch.ssl.verify: true 60 | 61 | # Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of 62 | # the elasticsearch.requestTimeout setting. 63 | #elasticsearch.pingTimeout: 1500 64 | 65 | # Time in milliseconds to wait for responses from the back end or Elasticsearch. This value 66 | # must be a positive integer. 67 | #elasticsearch.requestTimeout: 30000 68 | 69 | # List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side 70 | # headers, set this value to [] (an empty list). 71 | #elasticsearch.requestHeadersWhitelist: [ authorization ] 72 | 73 | # Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten 74 | # by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration. 75 | #elasticsearch.customHeaders: {} 76 | 77 | # Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable. 78 | #elasticsearch.shardTimeout: 0 79 | 80 | # Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying. 81 | #elasticsearch.startupTimeout: 5000 82 | 83 | # Specifies the path where Kibana creates the process ID file. 84 | #pid.file: /var/run/kibana.pid 85 | 86 | # Enables you specify a file where Kibana stores log output. 87 | #logging.dest: stdout 88 | 89 | # Set the value of this setting to true to suppress all logging output. 90 | #logging.silent: false 91 | 92 | # Set the value of this setting to true to suppress all logging output other than error messages. 93 | #logging.quiet: false 94 | 95 | # Set the value of this setting to true to log all events, including system usage information 96 | # and all requests. 97 | #logging.verbose: false 98 | 99 | # Set the interval in milliseconds to sample system and process performance 100 | # metrics. Minimum is 100ms. Defaults to 5000. 101 | #ops.interval: 5000 102 | 103 | # The default locale. This locale can be used in certain circumstances to substitute any missing 104 | # translations. 105 | #i18n.defaultLocale: "en" 106 | -------------------------------------------------------------------------------- /docker/logstash/config/logstash.yml: -------------------------------------------------------------------------------- 1 | --- 2 | ## Default Logstash configuration from Logstash base image. 3 | ## https://github.com/elastic/logstash/blob/master/docker/data/logstash/config/logstash-full.yml 4 | # 5 | http.host: "0.0.0.0" 6 | xpack.monitoring.elasticsearch.hosts: [ "http://elasticsearch:9200" ] 7 | 8 | ## X-Pack security credentials 9 | # 10 | xpack.monitoring.enabled: false 11 | # xpack.monitoring.elasticsearch.username: elastic 12 | # xpack.monitoring.elasticsearch.password: admin 13 | -------------------------------------------------------------------------------- /docker/logstash/pipeline/logstash.conf: -------------------------------------------------------------------------------- 1 | input { 2 | tcp { 3 | port => 5000 4 | } 5 | } 6 | 7 | ## Add your filters / logstash plugins configuration here 8 | 9 | output { 10 | elasticsearch { 11 | hosts => "elasticsearch:9200" 12 | #user => "elastic" 13 | #password => "changeme" 14 | } 15 | } 16 | -------------------------------------------------------------------------------- /docker/prometheus/alert.rules: -------------------------------------------------------------------------------- 1 | groups: 2 | - name: targets 3 | rules: 4 | - alert: monitor_service_down 5 | expr: up == 0 6 | for: 30s 7 | labels: 8 | severity: critical 9 | annotations: 10 | summary: "Monitor service non-operational" 11 | description: "Service {{ $labels.instance }} is down." 12 | 13 | - name: host 14 | rules: 15 | - alert: high_cpu_load 16 | expr: node_load1 > 1.5 17 | for: 30s 18 | labels: 19 | severity: warning 20 | annotations: 21 | summary: "Server under high load" 22 | description: "Docker host is under high load, the avg load 1m is at {{ $value}}. Reported by instance {{ $labels.instance }} of job {{ $labels.job }}." 23 | 24 | - alert: high_memory_load 25 | expr: (sum(node_memory_MemTotal_bytes) - sum(node_memory_MemFree_bytes + node_memory_Buffers_bytes + node_memory_Cached_bytes) ) / sum(node_memory_MemTotal_bytes) * 100 > 85 26 | for: 30s 27 | labels: 28 | severity: warning 29 | annotations: 30 | summary: "Server memory is almost full" 31 | description: "Docker host memory usage is {{ humanize $value}}%. Reported by instance {{ $labels.instance }} of job {{ $labels.job }}." 32 | 33 | - alert: high_storage_load 34 | expr: (node_filesystem_size_bytes{fstype="aufs"} - node_filesystem_free_bytes{fstype="aufs"}) / node_filesystem_size_bytes{fstype="aufs"} * 100 > 85 35 | for: 30s 36 | labels: 37 | severity: warning 38 | annotations: 39 | summary: "Server storage is almost full" 40 | description: "Docker host storage usage is {{ humanize $value}}%. Reported by instance {{ $labels.instance }} of job {{ $labels.job }}." 41 | 42 | - name: containers 43 | rules: 44 | - alert: jenkins_down 45 | expr: absent(container_memory_usage_bytes{name="jenkins"}) 46 | for: 30s 47 | labels: 48 | severity: critical 49 | annotations: 50 | summary: "Jenkins down" 51 | description: "Jenkins container is down for more than 30 seconds." 52 | 53 | - alert: jenkins_high_cpu 54 | expr: sum(rate(container_cpu_usage_seconds_total{name="jenkins"}[1m])) / count(node_cpu_seconds_total{mode="system"}) * 100 > 10 55 | for: 30s 56 | labels: 57 | severity: warning 58 | annotations: 59 | summary: "Jenkins high CPU usage" 60 | description: "Jenkins CPU usage is {{ humanize $value}}%." 61 | 62 | - alert: jenkins_high_memory 63 | expr: sum(container_memory_usage_bytes{name="jenkins"}) > 1200000000 64 | for: 30s 65 | labels: 66 | severity: warning 67 | annotations: 68 | summary: "Jenkins high memory usage" 69 | description: "Jenkins memory consumption is at {{ humanize $value}}." 70 | 71 | -------------------------------------------------------------------------------- /docker/prometheus/prometheus.yml: -------------------------------------------------------------------------------- 1 | global: 2 | scrape_interval: 15s 3 | evaluation_interval: 15s 4 | 5 | # Attach these labels to any time series or alerts when communicating with 6 | # external systems (federation, remote storage, Alertmanager). 7 | external_labels: 8 | monitor: 'docker-host-alpha' 9 | 10 | # Load and evaluate rules in this file every 'evaluation_interval' seconds. 11 | rule_files: 12 | - "alert.rules" 13 | 14 | # A scrape configuration containing exactly one endpoint to scrape. 15 | scrape_configs: 16 | - job_name: 'nodeexporter' 17 | scrape_interval: 5s 18 | static_configs: 19 | - targets: ['nodeexporter:9100'] 20 | 21 | - job_name: 'cadvisor' 22 | scrape_interval: 5s 23 | static_configs: 24 | - targets: ['cadvisor:8080'] 25 | 26 | - job_name: 'alertmanager' 27 | scrape_interval: 5s 28 | static_configs: 29 | - targets: ['alertmanager:9093'] 30 | 31 | - job_name: 'prometheus' 32 | scrape_interval: 10s 33 | static_configs: 34 | - targets: ['localhost:9090'] 35 | 36 | - job_name: 'pushgateway' 37 | scrape_interval: 10s 38 | honor_labels: true 39 | static_configs: 40 | - targets: ['pushgateway:9091'] 41 | 42 | 43 | - job_name: 'elasticsearch_exporter' 44 | scrape_interval: 10s 45 | honor_labels: true 46 | static_configs: 47 | - targets: ['elasticsearch_exporter:9114'] 48 | 49 | alerting: 50 | alertmanagers: 51 | - scheme: http 52 | static_configs: 53 | - targets: 54 | - 'alertmanager:9093' 55 | 56 | # - job_name: 'nginx' 57 | # scrape_interval: 10s 58 | # static_configs: 59 | # - targets: ['nginxexporter:9113'] 60 | 61 | # - job_name: 'aspnetcore' 62 | # scrape_interval: 10s 63 | # static_configs: 64 | # - targets: ['eventlog-proxy:5000', 'eventlog:5000'] 65 | -------------------------------------------------------------------------------- /docs/architecture_diagram.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/silveiralexf/monitoring-stack/d0d37baed30ddc017e4b94888d5b13880171085a/docs/architecture_diagram.png -------------------------------------------------------------------------------- /k8s/.keep: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/silveiralexf/monitoring-stack/d0d37baed30ddc017e4b94888d5b13880171085a/k8s/.keep --------------------------------------------------------------------------------