├── .gitignore ├── README.md ├── docker-compose.yml ├── grafana ├── Dockerfile.template └── conf │ ├── dashboard.yml │ ├── dashboard_node_exporter.json │ └── datasources.yml ├── node_exporter └── Dockerfile.template └── prometheus ├── Dockerfile.template ├── prometheus.yml └── start.sh /.gitignore: -------------------------------------------------------------------------------- 1 | token 2 | prometheus/targets 3 | 4 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | balenaCloud Prometheus and Grafana 2 | =================================== 3 | 4 | ### Introduction 5 | This project creates an application running Prometheus, Grafana and Node Exporter and can self-monitor. To add additional node targets, simply add the _node_exporter_ folder to any other project and add a Device Environment variable via the balenaCloud dashboard to update the _docker-compose.yml_ file, as shown in __Monitor other devices__ below. 6 | 7 | ![Dashboard](http://tonellolabs.com/images/grafana_dashboard.png) 8 | 9 | 10 | ``` 11 | /balena-prometheus-grafana/ 12 | ├── grafana 13 | │ ├── Dockerfile.template 14 | │ └── conf 15 | │ └── dashboard.yml 16 | │ └── dashboard_node_exporter.json 17 | │ └── datasources.yml 18 | ├── node_exporter 19 | │ └── Dockerfile.template 20 | ├── prometheus 21 | │ ├── Dockerfile.template 22 | │ └── prometheus.yml 23 | │ └── shart.sh 24 | ├── docker-compose.yml 25 | ├── README.md 26 | ``` 27 | 28 | In order to work properly, Prometheus needs at least a minimal _prometheus.yml_ file with basic _global_ and _scrape_config_ entries. The following is used in this project: 29 | 30 | ``` 31 | global: 32 | scrape_interval: 15s 33 | evaluation_interval: 15s 34 | scrape_timeout: 10s 35 | 36 | scrape_configs: 37 | - job_name: prometheus 38 | honor_timestamps: true 39 | scrape_interval: 15s 40 | scrape_timeout: 10s 41 | metrics_path: /metrics 42 | scheme: http 43 | static_configs: 44 | - targets: 45 | - prometheus:9090 46 | - job_name: node_exporter 47 | honor_timestamps: true 48 | scrape_interval: 15s 49 | scrape_timeout: 10s 50 | metrics_path: /metrics 51 | scheme: http 52 | static_configs: 53 | - targets: ['node_exporter:9100'] 54 | ``` 55 | 56 | Two networks are used. The _frontend_ network enables external connections to ports 3000, 9090 and 9100 on the Prometheus service to enable requests to and from Grafana (3000), Prometheus (9090) and Node Exporter (9100). The _backend_ network enables the local name resolution of _node_exporter_. However, all networks could be set as _network_mode: host_ for simplicity, and on other devices that aren't resolvable on the Prometheus/Grafana node. 57 | 58 | Note that the architecture of Prometheus is set as an "ARCH" argument defined in each _Dockerfile.template_. This should be updated with the appropriate version for your architecture. RPi 3 requires _armv7_. Pi4 and NUC devices should use _arm64_. 59 | 60 | ### Monitor other devices 61 | In order to add monitoring of other devices, add the following to its separate _docker-compose.yml_ file: 62 | 63 | ``` 64 | # Add Node Exporter to this app 65 | node_exporter: 66 | build: ./node_exporter 67 | ports: 68 | - "9100:9100" 69 | network_mode: host 70 | container_name: node_exporter 71 | ``` 72 | Secondly, add the ./node_exporter folder to that application, with a Dockerfile.template that looks like this: 73 | 74 | ``` 75 | FROM balenalib/%%BALENA_MACHINE_NAME%%-alpine 76 | WORKDIR /app 77 | 78 | RUN echo "http://dl-cdn.alpinelinux.org/alpine/edge/testing/" >> /etc/apk/repositories && \ 79 | apk add --no-cache prometheus-node-exporter 80 | 81 | # Expose the port Prometheus uses 82 | EXPOSE 9100 83 | 84 | # Start Node Exporter 85 | CMD [ "/usr/bin/node_exporter" ] 86 | ``` 87 | 88 | Finally, add a new entry in the balenaCloud Device Variables for the application running Prometheus (and Grafana). Add the variable _TARGETS_ and enter values that are the host IP addresses and ports of the devices you want to monitor, such as _10.128.1.134:9100, 10.128.1.211:9100_, etc. Spaces are optional. 89 | 90 | ![Add a device variable](http://tonellolabs.com/images/grafana_env_var.png) 91 | 92 | Each target address will show up in your Prometheus targets, [http://prometheus-IP/targets](#). 93 | 94 | ### Deploy 95 | Clone this repository, change into the balena-prometheus-grafana directory and push to your application: 96 | 97 | ``` 98 | $ git clone git@github.com:balenalabs-incubator/balena-prometheus-grafana.git 99 | $ cd balena-grafana 100 | $ balena push 101 | ``` 102 | ### View data in the dashboard 103 | In order to start viewing system data, log in to the Grafana dashboard at _http://device-host-ip:3000_ with admin/admin. Set a password. Since this deployment automatically sets up the Prometheus datasource and the Node Exporter dashboard, you're done. Once you login, you can view graphical data by clicking on the Dashboard menu item and then on "Node Exporter". 104 | 105 | ### Helpful references 106 | * [How to Integrate Grafana with Prometheus for Monitoring](https://www.linuxtechi.com/integrate-grafana-prometheus-monitoring/) 107 | * [How to Install Prometheus](https://www.linuxtechi.com/install-prometheus-monitoring-tool-centos-8-rhel-8/) 108 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: "2.0" 2 | services: 3 | prometheus: 4 | build: ./prometheus 5 | ports: 6 | - "9090:9090" 7 | - "9100:9100" 8 | networks: 9 | - frontend 10 | - backend 11 | depends_on: 12 | - node_exporter 13 | container_name: prometheus 14 | grafana: 15 | build: ./grafana 16 | ports: 17 | - "3000:3000" 18 | networks: 19 | - frontend 20 | - backend 21 | volumes: 22 | - grafana_etc:/etc/grafana 23 | - grafana_usr:/usr/share/grafana 24 | - grafana_var:/var/lib/grafana 25 | container_name: grafana 26 | node_exporter: 27 | build: ./node_exporter 28 | networks: 29 | - backend 30 | container_name: node_exporter 31 | 32 | networks: 33 | frontend: 34 | backend: 35 | 36 | volumes: 37 | grafana_etc: 38 | grafana_usr: 39 | grafana_var: -------------------------------------------------------------------------------- /grafana/Dockerfile.template: -------------------------------------------------------------------------------- 1 | FROM grafana/grafana 2 | 3 | WORKDIR /usr/src/app 4 | 5 | COPY ./conf/datasources.yml /etc/grafana/provisioning/datasources/ 6 | 7 | COPY ./conf/dash* /etc/grafana/provisioning/dashboards/ 8 | 9 | EXPOSE 3000 10 | 11 | CMD [ "/usr/share/grafana/bin/grafana-server", "--homepath=/usr/share/grafana --config=/etc/grafana/grafana.ini --pac] -------------------------------------------------------------------------------- /grafana/conf/dashboard.yml: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | 3 | providers: 4 | - name: 'Prometheus' 5 | orgId: 1 6 | folder: '' 7 | type: file 8 | disableDeletion: false 9 | editable: true 10 | options: 11 | path: /etc/grafana/provisioning/dashboards -------------------------------------------------------------------------------- /grafana/conf/dashboard_node_exporter.json: -------------------------------------------------------------------------------- 1 | { 2 | "annotations": { 3 | "list": [ 4 | { 5 | "builtIn": 1, 6 | "datasource": "-- Grafana --", 7 | "enable": true, 8 | "hide": true, 9 | "iconColor": "rgba(0, 211, 255, 1)", 10 | "name": "Annotations & Alerts", 11 | "type": "dashboard" 12 | } 13 | ] 14 | }, 15 | "description": "Support Node Exporter v0.16 and above.Optimize the main metrics display. Includes: CPU, memory, disk IO, network, temperature and other monitoring metrics。https://github.com/starsliao/Prometheus", 16 | "editable": true, 17 | "gnetId": 11074, 18 | "graphTooltip": 0, 19 | "id": 1, 20 | "iteration": 1580136797277, 21 | "links": [ 22 | { 23 | "asDropdown": false, 24 | "icon": "info", 25 | "includeVars": false, 26 | "tags": [ 27 | "$node" 28 | ], 29 | "targetBlank": true, 30 | "title": "Server IP：$node", 31 | "type": "link", 32 | "url": "" 33 | }, 34 | { 35 | "icon": "external link", 36 | "tags": [], 37 | "targetBlank": true, 38 | "title": "Update node_exporter", 39 | "tooltip": "", 40 | "type": "link", 41 | "url": "https://github.com/prometheus/node_exporter/releases" 42 | }, 43 | { 44 | "icon": "external link", 45 | "tags": [], 46 | "targetBlank": true, 47 | "title": "Update Dashboards", 48 | "tooltip": "", 49 | "type": "link", 50 | "url": "https://grafana.com/dashboards/11074" 51 | }, 52 | { 53 | "icon": "external link", 54 | "tags": [], 55 | "targetBlank": true, 56 | "title": "StarsL's Blog", 57 | "tooltip": "", 58 | "type": "link", 59 | "url": "https://starsl.cn" 60 | } 61 | ], 62 | "panels": [ 63 | { 64 | "cacheTimeout": null, 65 | "colorBackground": false, 66 | "colorPostfix": false, 67 | "colorPrefix": false, 68 | "colorValue": true, 69 | "colors": [ 70 | "rgba(245, 54, 54, 0.9)", 71 | "rgba(237, 129, 40, 0.89)", 72 | "rgba(50, 172, 45, 0.97)" 73 | ], 74 | "datasource": "Prometheus", 75 | "decimals": 1, 76 | "description": "", 77 | "format": "s", 78 | "gauge": { 79 | "maxValue": 100, 80 | "minValue": 0, 81 | "show": false, 82 | "thresholdLabels": false, 83 | "thresholdMarkers": true 84 | }, 85 | "gridPos": { 86 | "h": 3, 87 | "w": 2, 88 | "x": 0, 89 | "y": 0 90 | }, 91 | "hideTimeOverride": true, 92 | "id": 15, 93 | "interval": null, 94 | "links": [], 95 | "mappingType": 1, 96 | "mappingTypes": [ 97 | { 98 | "name": "value to text", 99 | "value": 1 100 | }, 101 | { 102 | "name": "range to text", 103 | "value": 2 104 | } 105 | ], 106 | "maxDataPoints": 100, 107 | "nullPointMode": "null", 108 | "nullText": null, 109 | "options": {}, 110 | "pluginVersion": "6.4.2", 111 | "postfix": "", 112 | "postfixFontSize": "50%", 113 | "prefix": "", 114 | "prefixFontSize": "50%", 115 | "rangeMaps": [ 116 | { 117 | "from": "null", 118 | "text": "N/A", 119 | "to": "null" 120 | } 121 | ], 122 | "sparkline": { 123 | "fillColor": "rgba(31, 118, 189, 0.18)", 124 | "full": false, 125 | "lineColor": "rgb(31, 120, 193)", 126 | "show": false 127 | }, 128 | "tableColumn": "", 129 | "targets": [ 130 | { 131 | "expr": "sum(time() - node_boot_time_seconds{instance=~\"$node\"})", 132 | "format": "time_series", 133 | "hide": false, 134 | "instant": true, 135 | "intervalFactor": 1, 136 | "refId": "A", 137 | "step": 40 138 | } 139 | ], 140 | "thresholds": "1,2", 141 | "title": "System Uptime", 142 | "type": "singlestat", 143 | "valueFontSize": "100%", 144 | "valueMaps": [ 145 | { 146 | "op": "=", 147 | "text": "N/A", 148 | "value": "null" 149 | } 150 | ], 151 | "valueName": "current" 152 | }, 153 | { 154 | "cacheTimeout": null, 155 | "colorBackground": false, 156 | "colorValue": true, 157 | "colors": [ 158 | "rgba(245, 54, 54, 0.9)", 159 | "rgba(237, 129, 40, 0.89)", 160 | "rgba(50, 172, 45, 0.97)" 161 | ], 162 | "datasource": "Prometheus", 163 | "decimals": 2, 164 | "description": "", 165 | "format": "bytes", 166 | "gauge": { 167 | "maxValue": 100, 168 | "minValue": 0, 169 | "show": false, 170 | "thresholdLabels": false, 171 | "thresholdMarkers": true 172 | }, 173 | "gridPos": { 174 | "h": 3, 175 | "w": 2, 176 | "x": 2, 177 | "y": 0 178 | }, 179 | "id": 75, 180 | "interval": null, 181 | "links": [], 182 | "mappingType": 1, 183 | "mappingTypes": [ 184 | { 185 | "name": "value to text", 186 | "value": 1 187 | }, 188 | { 189 | "name": "range to text", 190 | "value": 2 191 | } 192 | ], 193 | "maxDataPoints": 100, 194 | "maxPerRow": 6, 195 | "nullPointMode": "null", 196 | "nullText": null, 197 | "options": {}, 198 | "postfix": "", 199 | "postfixFontSize": "70%", 200 | "prefix": "", 201 | "prefixFontSize": "50%", 202 | "rangeMaps": [ 203 | { 204 | "from": "null", 205 | "text": "N/A", 206 | "to": "null" 207 | } 208 | ], 209 | "sparkline": { 210 | "fillColor": "rgba(31, 118, 189, 0.18)", 211 | "full": false, 212 | "lineColor": "rgb(31, 120, 193)", 213 | "show": false 214 | }, 215 | "tableColumn": "", 216 | "targets": [ 217 | { 218 | "expr": "sum(node_memory_MemTotal_bytes{instance=~\"$node\"})", 219 | "format": "time_series", 220 | "instant": true, 221 | "intervalFactor": 1, 222 | "legendFormat": "{{instance}}", 223 | "refId": "A", 224 | "step": 20 225 | } 226 | ], 227 | "thresholds": "2,3", 228 | "title": "Total RAM", 229 | "type": "singlestat", 230 | "valueFontSize": "80%", 231 | "valueMaps": [ 232 | { 233 | "op": "=", 234 | "text": "N/A", 235 | "value": "null" 236 | } 237 | ], 238 | "valueName": "current" 239 | }, 240 | { 241 | "datasource": "Prometheus", 242 | "gridPos": { 243 | "h": 6, 244 | "w": 4, 245 | "x": 4, 246 | "y": 0 247 | }, 248 | "id": 177, 249 | "options": { 250 | "displayMode": "lcd", 251 | "fieldOptions": { 252 | "calcs": [ 253 | "last" 254 | ], 255 | "defaults": { 256 | "color": { 257 | "mode": "thresholds" 258 | }, 259 | "mappings": [], 260 | "max": 100, 261 | "min": 0, 262 | "thresholds": { 263 | "mode": "absolute", 264 | "steps": [ 265 | { 266 | "color": "green", 267 | "value": null 268 | }, 269 | { 270 | "color": "#EAB839", 271 | "value": 60 272 | }, 273 | { 274 | "color": "red", 275 | "value": 80 276 | } 277 | ] 278 | }, 279 | "title": "", 280 | "unit": "percent" 281 | }, 282 | "overrides": [], 283 | "values": false 284 | }, 285 | "orientation": "horizontal", 286 | "showUnfilled": true 287 | }, 288 | "pluginVersion": "6.6.0", 289 | "targets": [ 290 | { 291 | "expr": "100 - (avg(irate(node_cpu_seconds_total{instance=~\"$node\",mode=\"idle\"}[30m])) * 100)", 292 | "instant": true, 293 | "legendFormat": "CPU Busy", 294 | "refId": "A" 295 | }, 296 | { 297 | "expr": "avg(irate(node_cpu_seconds_total{instance=~\"$node\",mode=\"iowait\"}[30m])) * 100", 298 | "hide": true, 299 | "instant": true, 300 | "legendFormat": "Busy Iowait", 301 | "refId": "C" 302 | }, 303 | { 304 | "expr": "(1 - (node_memory_MemAvailable_bytes{instance=~\"$node\"} / (node_memory_MemTotal_bytes{instance=~\"$node\"})))* 100", 305 | "instant": true, 306 | "legendFormat": "Used RAM Memory", 307 | "refId": "B" 308 | }, 309 | { 310 | "expr": "100 - ((node_filesystem_avail_bytes{instance=~\"$node\",mountpoint=\"$maxmount\",fstype=~\"ext4|xfs\"} * 100) / node_filesystem_size_bytes {instance=~\"$node\",mountpoint=\"$maxmount\",fstype=~\"ext4|xfs\"})", 311 | "hide": false, 312 | "instant": true, 313 | "legendFormat": "Used Max Mount($maxmount)", 314 | "refId": "D" 315 | }, 316 | { 317 | "expr": "(1 - (node_memory_SwapFree_bytes{instance=~\"$node\"} / node_memory_SwapTotal_bytes{instance=~\"$node\"})) * 100", 318 | "instant": true, 319 | "legendFormat": "Used SWAP", 320 | "refId": "E" 321 | } 322 | ], 323 | "timeFrom": null, 324 | "timeShift": null, 325 | "title": "", 326 | "type": "bargauge" 327 | }, 328 | { 329 | "columns": [], 330 | "datasource": "Prometheus", 331 | "fontSize": "110%", 332 | "gridPos": { 333 | "h": 6, 334 | "w": 10, 335 | "x": 8, 336 | "y": 0 337 | }, 338 | "id": 164, 339 | "links": [], 340 | "options": {}, 341 | "pageSize": null, 342 | "scroll": true, 343 | "showHeader": true, 344 | "sort": { 345 | "col": 6, 346 | "desc": false 347 | }, 348 | "styles": [ 349 | { 350 | "alias": "Mounted on", 351 | "align": "auto", 352 | "colorMode": null, 353 | "colors": [ 354 | "rgba(50, 172, 45, 0.97)", 355 | "rgba(237, 129, 40, 0.89)", 356 | "rgba(245, 54, 54, 0.9)" 357 | ], 358 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 359 | "decimals": 2, 360 | "mappingType": 1, 361 | "pattern": "mountpoint", 362 | "thresholds": [ 363 | "" 364 | ], 365 | "type": "string", 366 | "unit": "bytes" 367 | }, 368 | { 369 | "alias": "Avail", 370 | "align": "auto", 371 | "colorMode": "value", 372 | "colors": [ 373 | "rgba(245, 54, 54, 0.9)", 374 | "rgba(237, 129, 40, 0.89)", 375 | "rgba(50, 172, 45, 0.97)" 376 | ], 377 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 378 | "decimals": 2, 379 | "mappingType": 1, 380 | "pattern": "Value #A", 381 | "thresholds": [ 382 | "10000000000", 383 | "20000000000" 384 | ], 385 | "type": "number", 386 | "unit": "bytes" 387 | }, 388 | { 389 | "alias": "Used", 390 | "align": "auto", 391 | "colorMode": "cell", 392 | "colors": [ 393 | "rgba(50, 172, 45, 0.97)", 394 | "rgba(237, 129, 40, 0.89)", 395 | "rgba(245, 54, 54, 0.9)" 396 | ], 397 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 398 | "decimals": 2, 399 | "mappingType": 1, 400 | "pattern": "Value #B", 401 | "thresholds": [ 402 | "0.6", 403 | "0.8" 404 | ], 405 | "type": "number", 406 | "unit": "percentunit" 407 | }, 408 | { 409 | "alias": "Size", 410 | "align": "auto", 411 | "colorMode": null, 412 | "colors": [ 413 | "rgba(245, 54, 54, 0.9)", 414 | "rgba(237, 129, 40, 0.89)", 415 | "rgba(50, 172, 45, 0.97)" 416 | ], 417 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 418 | "decimals": 1, 419 | "link": false, 420 | "mappingType": 1, 421 | "pattern": "Value #C", 422 | "thresholds": [], 423 | "type": "number", 424 | "unit": "bytes" 425 | }, 426 | { 427 | "alias": "Filesystem", 428 | "align": "auto", 429 | "colorMode": null, 430 | "colors": [ 431 | "rgba(245, 54, 54, 0.9)", 432 | "rgba(237, 129, 40, 0.89)", 433 | "rgba(50, 172, 45, 0.97)" 434 | ], 435 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 436 | "decimals": 2, 437 | "link": false, 438 | "mappingType": 1, 439 | "pattern": "fstype", 440 | "thresholds": [], 441 | "type": "string", 442 | "unit": "short" 443 | }, 444 | { 445 | "alias": "IP", 446 | "align": "auto", 447 | "colorMode": null, 448 | "colors": [ 449 | "rgba(245, 54, 54, 0.9)", 450 | "rgba(237, 129, 40, 0.89)", 451 | "rgba(50, 172, 45, 0.97)" 452 | ], 453 | "dateFormat": "YYYY-MM-DD HH:mm:ss", 454 | "decimals": 2, 455 | "link": false, 456 | "mappingType": 1, 457 | "pattern": "instance", 458 | "preserveFormat": false, 459 | "sanitize": false, 460 | "thresholds": [], 461 | "type": "string", 462 | "unit": "short" 463 | }, 464 | { 465 | "alias": "", 466 | "align": "auto", 467 | "colorMode": null, 468 | "colors": [ 469 | "rgba(245, 54, 54, 0.9)", 470 | "rgba(237, 129, 40, 0.89)", 471 | "rgba(50, 172, 45, 0.97)" 472 | ], 473 | "decimals": 2, 474 | "pattern": "/.*/", 475 | "preserveFormat": true, 476 | "sanitize": false, 477 | "thresholds": [], 478 | "type": "hidden", 479 | "unit": "short" 480 | } 481 | ], 482 | "targets": [ 483 | { 484 | "expr": "node_filesystem_size_bytes{instance=~'$node',fstype=~\"ext4|xfs\"}-0", 485 | "format": "table", 486 | "hide": false, 487 | "instant": true, 488 | "intervalFactor": 1, 489 | "legendFormat": "", 490 | "refId": "C" 491 | }, 492 | { 493 | "expr": "node_filesystem_avail_bytes {instance=~'$node',fstype=~\"ext4|xfs\"}-0", 494 | "format": "table", 495 | "hide": false, 496 | "instant": true, 497 | "interval": "10s", 498 | "intervalFactor": 1, 499 | "legendFormat": "", 500 | "refId": "A" 501 | }, 502 | { 503 | "expr": "1-(node_filesystem_free_bytes{instance=~'$node',fstype=~\"ext4|xfs\"} / node_filesystem_size_bytes{instance=~'$node',fstype=~\"ext4|xfs\"})", 504 | "format": "table", 505 | "hide": false, 506 | "instant": true, 507 | "intervalFactor": 1, 508 | "legendFormat": "", 509 | "refId": "B" 510 | } 511 | ], 512 | "title": "Disk Space Used Basic(EXT4/XFS)", 513 | "transform": "table", 514 | "type": "table" 515 | }, 516 | { 517 | "aliasColors": { 518 | "filefd_192.168.200.241:9100": "super-light-green", 519 | "switches_192.168.200.241:9100": "semi-dark-red" 520 | }, 521 | "bars": false, 522 | "cacheTimeout": null, 523 | "dashLength": 10, 524 | "dashes": false, 525 | "datasource": "Prometheus", 526 | "description": "", 527 | "fill": 0, 528 | "fillGradient": 1, 529 | "gridPos": { 530 | "h": 6, 531 | "w": 6, 532 | "x": 18, 533 | "y": 0 534 | }, 535 | "hiddenSeries": false, 536 | "hideTimeOverride": false, 537 | "id": 16, 538 | "legend": { 539 | "alignAsTable": false, 540 | "avg": false, 541 | "current": true, 542 | "max": true, 543 | "min": false, 544 | "rightSide": false, 545 | "show": true, 546 | "total": false, 547 | "values": true 548 | }, 549 | "lines": true, 550 | "linewidth": 2, 551 | "links": [], 552 | "nullPointMode": "null", 553 | "options": { 554 | "dataLinks": [] 555 | }, 556 | "percentage": false, 557 | "pluginVersion": "6.4.2", 558 | "pointradius": 1, 559 | "points": false, 560 | "renderer": "flot", 561 | "seriesOverrides": [ 562 | { 563 | "alias": "/filefd_.*/", 564 | "lines": false, 565 | "pointradius": 1, 566 | "points": true 567 | }, 568 | { 569 | "alias": "/switches_.*/", 570 | "color": "#F2495C", 571 | "yaxis": 2 572 | } 573 | ], 574 | "spaceLength": 10, 575 | "stack": false, 576 | "steppedLine": false, 577 | "targets": [ 578 | { 579 | "expr": "node_filefd_allocated{instance=~\"$node\"}", 580 | "format": "time_series", 581 | "instant": false, 582 | "interval": "", 583 | "intervalFactor": 5, 584 | "legendFormat": "filefd_{{instance}}", 585 | "refId": "B" 586 | }, 587 | { 588 | "expr": "irate(node_context_switches_total{instance=~\"$node\"}[30m])", 589 | "intervalFactor": 5, 590 | "legendFormat": "switches_{{instance}}", 591 | "refId": "A" 592 | }, 593 | { 594 | "expr": "node_filefd_maximum{instance=~\"$node\"}", 595 | "hide": true, 596 | "refId": "C" 597 | } 598 | ], 599 | "thresholds": [], 600 | "timeFrom": null, 601 | "timeRegions": [], 602 | "timeShift": null, 603 | "title": "Open File Descriptor(left)/Context switches(right)", 604 | "tooltip": { 605 | "shared": true, 606 | "sort": 2, 607 | "value_type": "individual" 608 | }, 609 | "type": "graph", 610 | "xaxis": { 611 | "buckets": null, 612 | "mode": "time", 613 | "name": null, 614 | "show": true, 615 | "values": [] 616 | }, 617 | "yaxes": [ 618 | { 619 | "format": "short", 620 | "label": "", 621 | "logBase": 1, 622 | "max": null, 623 | "min": null, 624 | "show": true 625 | }, 626 | { 627 | "format": "short", 628 | "label": "context_switches", 629 | "logBase": 1, 630 | "max": null, 631 | "min": null, 632 | "show": true 633 | } 634 | ], 635 | "yaxis": { 636 | "align": false, 637 | "alignLevel": null 638 | } 639 | }, 640 | { 641 | "cacheTimeout": null, 642 | "colorBackground": false, 643 | "colorPostfix": false, 644 | "colorValue": true, 645 | "colors": [ 646 | "rgba(245, 54, 54, 0.9)", 647 | "rgba(237, 129, 40, 0.89)", 648 | "rgba(50, 172, 45, 0.97)" 649 | ], 650 | "datasource": "Prometheus", 651 | "description": "", 652 | "format": "short", 653 | "gauge": { 654 | "maxValue": 100, 655 | "minValue": 0, 656 | "show": false, 657 | "thresholdLabels": false, 658 | "thresholdMarkers": true 659 | }, 660 | "gridPos": { 661 | "h": 3, 662 | "w": 2, 663 | "x": 0, 664 | "y": 3 665 | }, 666 | "id": 14, 667 | "interval": null, 668 | "links": [], 669 | "mappingType": 1, 670 | "mappingTypes": [ 671 | { 672 | "name": "value to text", 673 | "value": 1 674 | }, 675 | { 676 | "name": "range to text", 677 | "value": 2 678 | } 679 | ], 680 | "maxDataPoints": 100, 681 | "maxPerRow": 6, 682 | "nullPointMode": "null", 683 | "nullText": null, 684 | "options": {}, 685 | "postfix": "", 686 | "postfixFontSize": "50%", 687 | "prefix": "", 688 | "prefixFontSize": "50%", 689 | "rangeMaps": [ 690 | { 691 | "from": "null", 692 | "text": "N/A", 693 | "to": "null" 694 | } 695 | ], 696 | "sparkline": { 697 | "fillColor": "rgba(31, 118, 189, 0.18)", 698 | "full": false, 699 | "lineColor": "rgb(31, 120, 193)", 700 | "show": false 701 | }, 702 | "tableColumn": "", 703 | "targets": [ 704 | { 705 | "expr": "sum(count(node_cpu_seconds_total{instance=~\"$node\", mode='system'}) by (cpu))", 706 | "format": "time_series", 707 | "instant": true, 708 | "intervalFactor": 1, 709 | "legendFormat": "", 710 | "refId": "A", 711 | "step": 20 712 | } 713 | ], 714 | "thresholds": "1,2", 715 | "title": "CPU Cores", 716 | "type": "singlestat", 717 | "valueFontSize": "100%", 718 | "valueMaps": [ 719 | { 720 | "op": "=", 721 | "text": "N/A", 722 | "value": "null" 723 | } 724 | ], 725 | "valueName": "current" 726 | }, 727 | { 728 | "cacheTimeout": null, 729 | "colorBackground": false, 730 | "colorValue": true, 731 | "colors": [ 732 | "#299c46", 733 | "rgba(237, 129, 40, 0.89)", 734 | "#d44a3a" 735 | ], 736 | "datasource": "Prometheus", 737 | "decimals": 2, 738 | "description": "", 739 | "format": "percent", 740 | "gauge": { 741 | "maxValue": 100, 742 | "minValue": 0, 743 | "show": false, 744 | "thresholdLabels": false, 745 | "thresholdMarkers": true 746 | }, 747 | "gridPos": { 748 | "h": 3, 749 | "w": 2, 750 | "x": 2, 751 | "y": 3 752 | }, 753 | "id": 20, 754 | "interval": null, 755 | "links": [], 756 | "mappingType": 1, 757 | "mappingTypes": [ 758 | { 759 | "name": "value to text", 760 | "value": 1 761 | }, 762 | { 763 | "name": "range to text", 764 | "value": 2 765 | } 766 | ], 767 | "maxDataPoints": 100, 768 | "nullPointMode": "connected", 769 | "nullText": null, 770 | "options": {}, 771 | "pluginVersion": "6.4.2", 772 | "postfix": "", 773 | "postfixFontSize": "50%", 774 | "prefix": "", 775 | "prefixFontSize": "50%", 776 | "rangeMaps": [ 777 | { 778 | "from": "null", 779 | "text": "N/A", 780 | "to": "null" 781 | } 782 | ], 783 | "sparkline": { 784 | "fillColor": "rgba(31, 118, 189, 0.18)", 785 | "full": false, 786 | "lineColor": "#3274D9", 787 | "show": true, 788 | "ymax": null, 789 | "ymin": null 790 | }, 791 | "tableColumn": "", 792 | "targets": [ 793 | { 794 | "expr": "avg(irate(node_cpu_seconds_total{instance=~\"$node\",mode=\"iowait\"}[30m])) * 100", 795 | "format": "time_series", 796 | "hide": false, 797 | "instant": false, 798 | "interval": "", 799 | "intervalFactor": 1, 800 | "legendFormat": "", 801 | "refId": "A", 802 | "step": 20 803 | } 804 | ], 805 | "thresholds": "20,50", 806 | "timeFrom": null, 807 | "timeShift": null, 808 | "title": "CPU IOwait", 809 | "type": "singlestat", 810 | "valueFontSize": "100%", 811 | "valueMaps": [ 812 | { 813 | "op": "=", 814 | "text": "N/A", 815 | "value": "null" 816 | } 817 | ], 818 | "valueName": "avg" 819 | }, 820 | { 821 | "aliasColors": { 822 | "15分钟": "#6ED0E0", 823 | "1分钟": "#BF1B00", 824 | "5分钟": "#CCA300" 825 | }, 826 | "bars": false, 827 | "dashLength": 10, 828 | "dashes": false, 829 | "datasource": "Prometheus", 830 | "editable": true, 831 | "error": false, 832 | "fill": 1, 833 | "fillGradient": 1, 834 | "grid": {}, 835 | "gridPos": { 836 | "h": 8, 837 | "w": 8, 838 | "x": 0, 839 | "y": 6 840 | }, 841 | "height": "300", 842 | "hiddenSeries": false, 843 | "id": 13, 844 | "legend": { 845 | "alignAsTable": true, 846 | "avg": true, 847 | "current": true, 848 | "max": true, 849 | "min": false, 850 | "rightSide": false, 851 | "show": true, 852 | "total": false, 853 | "values": true 854 | }, 855 | "lines": true, 856 | "linewidth": 2, 857 | "links": [], 858 | "maxPerRow": 6, 859 | "nullPointMode": "null as zero", 860 | "options": { 861 | "dataLinks": [] 862 | }, 863 | "percentage": false, 864 | "pointradius": 5, 865 | "points": false, 866 | "renderer": "flot", 867 | "repeat": null, 868 | "seriesOverrides": [], 869 | "spaceLength": 10, 870 | "stack": false, 871 | "steppedLine": false, 872 | "targets": [ 873 | { 874 | "expr": "node_load1{instance=~\"$node\"}", 875 | "format": "time_series", 876 | "instant": false, 877 | "interval": "", 878 | "intervalFactor": 1, 879 | "legendFormat": "{{instance}}_1m", 880 | "metric": "", 881 | "refId": "A", 882 | "step": 20, 883 | "target": "" 884 | }, 885 | { 886 | "expr": "node_load5{instance=~\"$node\"}", 887 | "format": "time_series", 888 | "instant": false, 889 | "interval": "", 890 | "intervalFactor": 1, 891 | "legendFormat": "{{instance}}_5m", 892 | "refId": "B", 893 | "step": 20 894 | }, 895 | { 896 | "expr": "node_load15{instance=~\"$node\"}", 897 | "format": "time_series", 898 | "instant": false, 899 | "interval": "", 900 | "intervalFactor": 1, 901 | "legendFormat": "{{instance}}_15m", 902 | "refId": "C", 903 | "step": 20 904 | } 905 | ], 906 | "thresholds": [], 907 | "timeFrom": null, 908 | "timeRegions": [], 909 | "timeShift": null, 910 | "title": "System Load", 911 | "tooltip": { 912 | "msResolution": false, 913 | "shared": true, 914 | "sort": 2, 915 | "value_type": "cumulative" 916 | }, 917 | "type": "graph", 918 | "xaxis": { 919 | "buckets": null, 920 | "mode": "time", 921 | "name": null, 922 | "show": true, 923 | "values": [] 924 | }, 925 | "yaxes": [ 926 | { 927 | "format": "short", 928 | "logBase": 1, 929 | "max": null, 930 | "min": null, 931 | "show": true 932 | }, 933 | { 934 | "format": "short", 935 | "logBase": 1, 936 | "max": null, 937 | "min": null, 938 | "show": true 939 | } 940 | ], 941 | "yaxis": { 942 | "align": false, 943 | "alignLevel": null 944 | } 945 | }, 946 | { 947 | "aliasColors": { 948 | "192.168.200.241:9100_Total": "dark-red", 949 | "Idle - Waiting for something to happen": "#052B51", 950 | "guest": "#9AC48A", 951 | "idle": "#052B51", 952 | "iowait": "#EAB839", 953 | "irq": "#BF1B00", 954 | "nice": "#C15C17", 955 | "sdb_每秒I/O操作%": "#d683ce", 956 | "softirq": "#E24D42", 957 | "steal": "#FCE2DE", 958 | "system": "#508642", 959 | "user": "#5195CE", 960 | "磁盘花费在I/O操作占比": "#ba43a9" 961 | }, 962 | "bars": false, 963 | "dashLength": 10, 964 | "dashes": false, 965 | "datasource": "Prometheus", 966 | "decimals": 2, 967 | "description": "", 968 | "fill": 1, 969 | "fillGradient": 0, 970 | "gridPos": { 971 | "h": 8, 972 | "w": 8, 973 | "x": 8, 974 | "y": 6 975 | }, 976 | "hiddenSeries": false, 977 | "id": 7, 978 | "legend": { 979 | "alignAsTable": true, 980 | "avg": true, 981 | "current": true, 982 | "hideEmpty": true, 983 | "hideZero": true, 984 | "max": true, 985 | "min": false, 986 | "rightSide": false, 987 | "show": true, 988 | "sideWidth": null, 989 | "sort": "current", 990 | "sortDesc": true, 991 | "total": false, 992 | "values": true 993 | }, 994 | "lines": true, 995 | "linewidth": 2, 996 | "links": [], 997 | "maxPerRow": 6, 998 | "nullPointMode": "null", 999 | "options": { 1000 | "dataLinks": [] 1001 | }, 1002 | "percentage": false, 1003 | "pointradius": 5, 1004 | "points": false, 1005 | "renderer": "flot", 1006 | "repeat": null, 1007 | "seriesOverrides": [ 1008 | { 1009 | "alias": "/.*_Total/", 1010 | "color": "#C4162A", 1011 | "fill": 0 1012 | } 1013 | ], 1014 | "spaceLength": 10, 1015 | "stack": false, 1016 | "steppedLine": false, 1017 | "targets": [ 1018 | { 1019 | "expr": "avg(irate(node_cpu_seconds_total{instance=~\"$node\",mode=\"system\"}[30m])) by (instance)", 1020 | "format": "time_series", 1021 | "hide": false, 1022 | "instant": false, 1023 | "interval": "", 1024 | "intervalFactor": 1, 1025 | "legendFormat": "{{instance}}_System", 1026 | "refId": "A", 1027 | "step": 20 1028 | }, 1029 | { 1030 | "expr": "avg(irate(node_cpu_seconds_total{instance=~\"$node\",mode=\"user\"}[30m])) by (instance)", 1031 | "format": "time_series", 1032 | "hide": false, 1033 | "intervalFactor": 1, 1034 | "legendFormat": "{{instance}}_User", 1035 | "refId": "B", 1036 | "step": 240 1037 | }, 1038 | { 1039 | "expr": "avg(irate(node_cpu_seconds_total{instance=~\"$node\",mode=\"iowait\"}[30m])) by (instance)", 1040 | "format": "time_series", 1041 | "hide": false, 1042 | "instant": false, 1043 | "intervalFactor": 1, 1044 | "legendFormat": "{{instance}}_Iowait", 1045 | "refId": "D", 1046 | "step": 240 1047 | }, 1048 | { 1049 | "expr": "1 - avg(irate(node_cpu_seconds_total{instance=~\"$node\",mode=\"idle\"}[30m])) by (instance)", 1050 | "format": "time_series", 1051 | "hide": false, 1052 | "intervalFactor": 1, 1053 | "legendFormat": "{{instance}}_Total", 1054 | "refId": "F", 1055 | "step": 240 1056 | } 1057 | ], 1058 | "thresholds": [], 1059 | "timeFrom": null, 1060 | "timeRegions": [], 1061 | "timeShift": null, 1062 | "title": "CPU Basic", 1063 | "tooltip": { 1064 | "shared": true, 1065 | "sort": 2, 1066 | "value_type": "individual" 1067 | }, 1068 | "type": "graph", 1069 | "xaxis": { 1070 | "buckets": null, 1071 | "mode": "time", 1072 | "name": null, 1073 | "show": true, 1074 | "values": [] 1075 | }, 1076 | "yaxes": [ 1077 | { 1078 | "decimals": 2, 1079 | "format": "percentunit", 1080 | "label": "", 1081 | "logBase": 1, 1082 | "max": null, 1083 | "min": null, 1084 | "show": true 1085 | }, 1086 | { 1087 | "format": "short", 1088 | "label": null, 1089 | "logBase": 1, 1090 | "max": null, 1091 | "min": null, 1092 | "show": false 1093 | } 1094 | ], 1095 | "yaxis": { 1096 | "align": false, 1097 | "alignLevel": null 1098 | } 1099 | }, 1100 | { 1101 | "aliasColors": { 1102 | "192.168.10.227:9100_em1_in下载": "super-light-green", 1103 | "192.168.10.227:9100_em1_out上传": "dark-blue" 1104 | }, 1105 | "bars": false, 1106 | "dashLength": 10, 1107 | "dashes": false, 1108 | "datasource": "Prometheus", 1109 | "fill": 1, 1110 | "fillGradient": 3, 1111 | "gridPos": { 1112 | "h": 8, 1113 | "w": 8, 1114 | "x": 16, 1115 | "y": 6 1116 | }, 1117 | "height": "300", 1118 | "hiddenSeries": false, 1119 | "id": 157, 1120 | "legend": { 1121 | "alignAsTable": true, 1122 | "avg": false, 1123 | "current": true, 1124 | "hideEmpty": true, 1125 | "hideZero": true, 1126 | "max": true, 1127 | "min": false, 1128 | "rightSide": false, 1129 | "show": true, 1130 | "sort": "current", 1131 | "sortDesc": true, 1132 | "total": false, 1133 | "values": true 1134 | }, 1135 | "lines": true, 1136 | "linewidth": 2, 1137 | "links": [], 1138 | "nullPointMode": "null", 1139 | "options": { 1140 | "dataLinks": [] 1141 | }, 1142 | "percentage": false, 1143 | "pointradius": 2, 1144 | "points": false, 1145 | "renderer": "flot", 1146 | "seriesOverrides": [ 1147 | { 1148 | "alias": "/.*_transmit$/", 1149 | "transform": "negative-Y" 1150 | } 1151 | ], 1152 | "spaceLength": 10, 1153 | "stack": false, 1154 | "steppedLine": false, 1155 | "targets": [ 1156 | { 1157 | "expr": "irate(node_network_receive_bytes_total{instance=~'$node',device!~'tap.*|veth.*|br.*|docker.*|virbr*|lo*'}[30m])*8", 1158 | "format": "time_series", 1159 | "intervalFactor": 1, 1160 | "legendFormat": "{{instance}}_{{device}}_receive", 1161 | "refId": "A", 1162 | "step": 4 1163 | }, 1164 | { 1165 | "expr": "irate(node_network_transmit_bytes_total{instance=~'$node',device!~'tap.*|veth.*|br.*|docker.*|virbr*|lo*'}[30m])*8", 1166 | "format": "time_series", 1167 | "intervalFactor": 1, 1168 | "legendFormat": "{{instance}}_{{device}}_transmit", 1169 | "refId": "B", 1170 | "step": 4 1171 | } 1172 | ], 1173 | "thresholds": [], 1174 | "timeFrom": null, 1175 | "timeRegions": [], 1176 | "timeShift": null, 1177 | "title": "Network Traffic Basic", 1178 | "tooltip": { 1179 | "shared": true, 1180 | "sort": 2, 1181 | "value_type": "individual" 1182 | }, 1183 | "type": "graph", 1184 | "xaxis": { 1185 | "buckets": null, 1186 | "mode": "time", 1187 | "name": null, 1188 | "show": true, 1189 | "values": [] 1190 | }, 1191 | "yaxes": [ 1192 | { 1193 | "format": "bps", 1194 | "label": "transmit（-）/receive（+）", 1195 | "logBase": 1, 1196 | "max": null, 1197 | "min": null, 1198 | "show": true 1199 | }, 1200 | { 1201 | "format": "short", 1202 | "label": null, 1203 | "logBase": 1, 1204 | "max": null, 1205 | "min": null, 1206 | "show": false 1207 | } 1208 | ], 1209 | "yaxis": { 1210 | "align": false, 1211 | "alignLevel": null 1212 | } 1213 | }, 1214 | { 1215 | "aliasColors": {}, 1216 | "bars": false, 1217 | "dashLength": 10, 1218 | "dashes": false, 1219 | "datasource": "Prometheus", 1220 | "fill": 1, 1221 | "fillGradient": 3, 1222 | "gridPos": { 1223 | "h": 8, 1224 | "w": 8, 1225 | "x": 0, 1226 | "y": 14 1227 | }, 1228 | "hiddenSeries": false, 1229 | "id": 174, 1230 | "legend": { 1231 | "alignAsTable": true, 1232 | "avg": false, 1233 | "current": true, 1234 | "hideEmpty": false, 1235 | "hideZero": false, 1236 | "max": false, 1237 | "min": false, 1238 | "rightSide": false, 1239 | "show": true, 1240 | "sideWidth": null, 1241 | "total": false, 1242 | "values": true 1243 | }, 1244 | "lines": true, 1245 | "linewidth": 2, 1246 | "links": [], 1247 | "nullPointMode": "null", 1248 | "options": { 1249 | "dataLinks": [] 1250 | }, 1251 | "percentage": false, 1252 | "pointradius": 5, 1253 | "points": false, 1254 | "renderer": "flot", 1255 | "seriesOverrides": [ 1256 | { 1257 | "alias": "/Inodes.*/", 1258 | "yaxis": 2 1259 | } 1260 | ], 1261 | "spaceLength": 10, 1262 | "stack": false, 1263 | "steppedLine": false, 1264 | "targets": [ 1265 | { 1266 | "expr": "1-(node_filesystem_free_bytes{instance=~'$node',fstype=~\"ext4|xfs\"} / node_filesystem_size_bytes{instance=~'$node',fstype=~\"ext4|xfs\"})", 1267 | "format": "time_series", 1268 | "instant": false, 1269 | "intervalFactor": 1, 1270 | "legendFormat": "{{instance}}：{{mountpoint}}", 1271 | "refId": "A" 1272 | }, 1273 | { 1274 | "expr": "node_filesystem_files_free{instance=~'$node',fstype=~\"ext4|xfs\"} / node_filesystem_files{instance=~'$node',fstype=~\"ext4|xfs\"}", 1275 | "hide": true, 1276 | "legendFormat": "Inodes：{{instance}}：{{mountpoint}}", 1277 | "refId": "B" 1278 | } 1279 | ], 1280 | "thresholds": [], 1281 | "timeFrom": null, 1282 | "timeRegions": [], 1283 | "timeShift": null, 1284 | "title": "Disk Space Used Basic", 1285 | "tooltip": { 1286 | "shared": true, 1287 | "sort": 2, 1288 | "value_type": "individual" 1289 | }, 1290 | "type": "graph", 1291 | "xaxis": { 1292 | "buckets": null, 1293 | "mode": "time", 1294 | "name": null, 1295 | "show": true, 1296 | "values": [] 1297 | }, 1298 | "yaxes": [ 1299 | { 1300 | "decimals": 2, 1301 | "format": "percentunit", 1302 | "label": "", 1303 | "logBase": 1, 1304 | "max": null, 1305 | "min": null, 1306 | "show": true 1307 | }, 1308 | { 1309 | "decimals": 2, 1310 | "format": "percentunit", 1311 | "label": null, 1312 | "logBase": 1, 1313 | "max": "1", 1314 | "min": null, 1315 | "show": true 1316 | } 1317 | ], 1318 | "yaxis": { 1319 | "align": false, 1320 | "alignLevel": null 1321 | } 1322 | }, 1323 | { 1324 | "aliasColors": { 1325 | "192.168.200.241:9100_总内存": "dark-red", 1326 | "内存_Avaliable": "#6ED0E0", 1327 | "内存_Cached": "#EF843C", 1328 | "内存_Free": "#629E51", 1329 | "内存_Total": "#6d1f62", 1330 | "内存_Used": "#eab839", 1331 | "可用": "#9ac48a", 1332 | "总内存": "#bf1b00" 1333 | }, 1334 | "bars": false, 1335 | "dashLength": 10, 1336 | "dashes": false, 1337 | "datasource": "Prometheus", 1338 | "decimals": 2, 1339 | "fill": 1, 1340 | "fillGradient": 0, 1341 | "gridPos": { 1342 | "h": 8, 1343 | "w": 8, 1344 | "x": 8, 1345 | "y": 14 1346 | }, 1347 | "height": "300", 1348 | "hiddenSeries": false, 1349 | "id": 156, 1350 | "legend": { 1351 | "alignAsTable": true, 1352 | "avg": false, 1353 | "current": true, 1354 | "max": false, 1355 | "min": false, 1356 | "rightSide": false, 1357 | "show": true, 1358 | "sort": "current", 1359 | "sortDesc": true, 1360 | "total": false, 1361 | "values": true 1362 | }, 1363 | "lines": true, 1364 | "linewidth": 2, 1365 | "links": [], 1366 | "nullPointMode": "null", 1367 | "options": { 1368 | "dataLinks": [] 1369 | }, 1370 | "percentage": false, 1371 | "pointradius": 5, 1372 | "points": false, 1373 | "renderer": "flot", 1374 | "seriesOverrides": [ 1375 | { 1376 | "alias": "/.*_Total/", 1377 | "color": "#C4162A", 1378 | "fill": 0 1379 | } 1380 | ], 1381 | "spaceLength": 10, 1382 | "stack": false, 1383 | "steppedLine": false, 1384 | "targets": [ 1385 | { 1386 | "expr": "node_memory_MemTotal_bytes{instance=~\"$node\"}", 1387 | "format": "time_series", 1388 | "hide": false, 1389 | "instant": false, 1390 | "intervalFactor": 1, 1391 | "legendFormat": "{{instance}}_Total", 1392 | "refId": "A", 1393 | "step": 4 1394 | }, 1395 | { 1396 | "expr": "node_memory_MemTotal_bytes{instance=~\"$node\"} - node_memory_MemAvailable_bytes{instance=~\"$node\"}", 1397 | "format": "time_series", 1398 | "hide": false, 1399 | "intervalFactor": 1, 1400 | "legendFormat": "{{instance}}_Used", 1401 | "refId": "B", 1402 | "step": 4 1403 | }, 1404 | { 1405 | "expr": "node_memory_MemAvailable_bytes{instance=~\"$node\"}", 1406 | "format": "time_series", 1407 | "hide": false, 1408 | "interval": "", 1409 | "intervalFactor": 1, 1410 | "legendFormat": "{{instance}}_Avaliable", 1411 | "refId": "F", 1412 | "step": 4 1413 | } 1414 | ], 1415 | "thresholds": [], 1416 | "timeFrom": null, 1417 | "timeRegions": [], 1418 | "timeShift": null, 1419 | "title": "Memory Basic", 1420 | "tooltip": { 1421 | "shared": true, 1422 | "sort": 2, 1423 | "value_type": "individual" 1424 | }, 1425 | "type": "graph", 1426 | "xaxis": { 1427 | "buckets": null, 1428 | "mode": "time", 1429 | "name": null, 1430 | "show": true, 1431 | "values": [] 1432 | }, 1433 | "yaxes": [ 1434 | { 1435 | "format": "bytes", 1436 | "label": null, 1437 | "logBase": 1, 1438 | "max": null, 1439 | "min": "0", 1440 | "show": true 1441 | }, 1442 | { 1443 | "format": "short", 1444 | "label": null, 1445 | "logBase": 1, 1446 | "max": null, 1447 | "min": null, 1448 | "show": true 1449 | } 1450 | ], 1451 | "yaxis": { 1452 | "align": false, 1453 | "alignLevel": null 1454 | } 1455 | }, 1456 | { 1457 | "aliasColors": { 1458 | "Idle - Waiting for something to happen": "#052B51", 1459 | "guest": "#9AC48A", 1460 | "idle": "#052B51", 1461 | "iowait": "#EAB839", 1462 | "irq": "#BF1B00", 1463 | "nice": "#C15C17", 1464 | "sdb_每秒I/O操作%": "#d683ce", 1465 | "softirq": "#E24D42", 1466 | "steal": "#FCE2DE", 1467 | "system": "#508642", 1468 | "user": "#5195CE", 1469 | "磁盘花费在I/O操作占比": "#ba43a9" 1470 | }, 1471 | "bars": false, 1472 | "dashLength": 10, 1473 | "dashes": false, 1474 | "datasource": "Prometheus", 1475 | "decimals": null, 1476 | "description": "The time spent on I/O in the natural time of each second.（wall-clock time）", 1477 | "fill": 1, 1478 | "fillGradient": 5, 1479 | "gridPos": { 1480 | "h": 8, 1481 | "w": 8, 1482 | "x": 16, 1483 | "y": 14 1484 | }, 1485 | "hiddenSeries": false, 1486 | "id": 175, 1487 | "legend": { 1488 | "alignAsTable": true, 1489 | "avg": true, 1490 | "current": true, 1491 | "hideEmpty": true, 1492 | "hideZero": true, 1493 | "max": true, 1494 | "min": false, 1495 | "rightSide": false, 1496 | "show": true, 1497 | "sideWidth": null, 1498 | "sort": null, 1499 | "sortDesc": null, 1500 | "total": false, 1501 | "values": true 1502 | }, 1503 | "lines": true, 1504 | "linewidth": 2, 1505 | "links": [], 1506 | "maxPerRow": 6, 1507 | "nullPointMode": "null", 1508 | "options": { 1509 | "dataLinks": [] 1510 | }, 1511 | "percentage": false, 1512 | "pointradius": 5, 1513 | "points": false, 1514 | "renderer": "flot", 1515 | "seriesOverrides": [], 1516 | "spaceLength": 10, 1517 | "stack": false, 1518 | "steppedLine": false, 1519 | "targets": [ 1520 | { 1521 | "expr": "irate(node_disk_io_time_seconds_total{instance=~\"$node\"}[30m])", 1522 | "format": "time_series", 1523 | "intervalFactor": 1, 1524 | "legendFormat": "{{instance}}_{{device}}_ IO time", 1525 | "refId": "C" 1526 | } 1527 | ], 1528 | "thresholds": [], 1529 | "timeFrom": null, 1530 | "timeRegions": [], 1531 | "timeShift": null, 1532 | "title": "Time Spent Doing I/Os", 1533 | "tooltip": { 1534 | "shared": true, 1535 | "sort": 2, 1536 | "value_type": "individual" 1537 | }, 1538 | "type": "graph", 1539 | "xaxis": { 1540 | "buckets": null, 1541 | "mode": "time", 1542 | "name": null, 1543 | "show": true, 1544 | "values": [] 1545 | }, 1546 | "yaxes": [ 1547 | { 1548 | "decimals": null, 1549 | "format": "s", 1550 | "label": "", 1551 | "logBase": 1, 1552 | "max": null, 1553 | "min": null, 1554 | "show": true 1555 | }, 1556 | { 1557 | "format": "short", 1558 | "label": null, 1559 | "logBase": 1, 1560 | "max": null, 1561 | "min": null, 1562 | "show": false 1563 | } 1564 | ], 1565 | "yaxis": { 1566 | "align": false, 1567 | "alignLevel": null 1568 | } 1569 | }, 1570 | { 1571 | "aliasColors": { 1572 | "vda_write": "#6ED0E0" 1573 | }, 1574 | "bars": false, 1575 | "dashLength": 10, 1576 | "dashes": false, 1577 | "datasource": "Prometheus", 1578 | "description": "Read/write completions per second", 1579 | "fill": 1, 1580 | "fillGradient": 1, 1581 | "gridPos": { 1582 | "h": 9, 1583 | "w": 8, 1584 | "x": 0, 1585 | "y": 22 1586 | }, 1587 | "height": "300", 1588 | "hiddenSeries": false, 1589 | "id": 161, 1590 | "legend": { 1591 | "alignAsTable": true, 1592 | "avg": true, 1593 | "current": true, 1594 | "hideEmpty": true, 1595 | "hideZero": true, 1596 | "max": true, 1597 | "min": false, 1598 | "show": true, 1599 | "sort": "current", 1600 | "sortDesc": true, 1601 | "total": false, 1602 | "values": true 1603 | }, 1604 | "lines": true, 1605 | "linewidth": 2, 1606 | "links": [], 1607 | "nullPointMode": "null", 1608 | "options": { 1609 | "dataLinks": [] 1610 | }, 1611 | "percentage": false, 1612 | "pointradius": 5, 1613 | "points": false, 1614 | "renderer": "flot", 1615 | "seriesOverrides": [ 1616 | { 1617 | "alias": "/.*_Reads completed$/", 1618 | "transform": "negative-Y" 1619 | } 1620 | ], 1621 | "spaceLength": 10, 1622 | "stack": false, 1623 | "steppedLine": false, 1624 | "targets": [ 1625 | { 1626 | "expr": "irate(node_disk_reads_completed_total{instance=~\"$node\"}[30m])", 1627 | "format": "time_series", 1628 | "hide": false, 1629 | "interval": "", 1630 | "intervalFactor": 1, 1631 | "legendFormat": "{{instance}}_{{device}}_Reads completed", 1632 | "refId": "A", 1633 | "step": 10 1634 | }, 1635 | { 1636 | "expr": "irate(node_disk_writes_completed_total{instance=~\"$node\"}[30m])", 1637 | "format": "time_series", 1638 | "hide": false, 1639 | "intervalFactor": 1, 1640 | "legendFormat": "{{instance}}_{{device}}_Writes completed", 1641 | "refId": "B", 1642 | "step": 10 1643 | } 1644 | ], 1645 | "thresholds": [], 1646 | "timeFrom": null, 1647 | "timeRegions": [], 1648 | "timeShift": null, 1649 | "title": "Disk IOps Completed", 1650 | "tooltip": { 1651 | "shared": true, 1652 | "sort": 2, 1653 | "value_type": "individual" 1654 | }, 1655 | "type": "graph", 1656 | "xaxis": { 1657 | "buckets": null, 1658 | "mode": "time", 1659 | "name": null, 1660 | "show": true, 1661 | "values": [] 1662 | }, 1663 | "yaxes": [ 1664 | { 1665 | "decimals": null, 1666 | "format": "iops", 1667 | "label": "IO read (-) / write (+)", 1668 | "logBase": 1, 1669 | "max": null, 1670 | "min": null, 1671 | "show": true 1672 | }, 1673 | { 1674 | "format": "short", 1675 | "label": null, 1676 | "logBase": 1, 1677 | "max": null, 1678 | "min": null, 1679 | "show": true 1680 | } 1681 | ], 1682 | "yaxis": { 1683 | "align": false, 1684 | "alignLevel": null 1685 | } 1686 | }, 1687 | { 1688 | "aliasColors": { 1689 | "vda_write": "#6ED0E0" 1690 | }, 1691 | "bars": false, 1692 | "dashLength": 10, 1693 | "dashes": false, 1694 | "datasource": "Prometheus", 1695 | "description": "Per second read / write bytes ", 1696 | "fill": 1, 1697 | "fillGradient": 1, 1698 | "gridPos": { 1699 | "h": 9, 1700 | "w": 8, 1701 | "x": 8, 1702 | "y": 22 1703 | }, 1704 | "height": "300", 1705 | "hiddenSeries": false, 1706 | "id": 168, 1707 | "legend": { 1708 | "alignAsTable": true, 1709 | "avg": true, 1710 | "current": true, 1711 | "hideEmpty": true, 1712 | "hideZero": true, 1713 | "max": true, 1714 | "min": false, 1715 | "show": true, 1716 | "sort": "current", 1717 | "sortDesc": true, 1718 | "total": false, 1719 | "values": true 1720 | }, 1721 | "lines": true, 1722 | "linewidth": 2, 1723 | "links": [], 1724 | "nullPointMode": "null", 1725 | "options": { 1726 | "dataLinks": [] 1727 | }, 1728 | "percentage": false, 1729 | "pointradius": 5, 1730 | "points": false, 1731 | "renderer": "flot", 1732 | "seriesOverrides": [ 1733 | { 1734 | "alias": "/.*_Read bytes$/", 1735 | "transform": "negative-Y" 1736 | } 1737 | ], 1738 | "spaceLength": 10, 1739 | "stack": false, 1740 | "steppedLine": false, 1741 | "targets": [ 1742 | { 1743 | "expr": "irate(node_disk_read_bytes_total{instance=~\"$node\"}[30m])", 1744 | "format": "time_series", 1745 | "interval": "", 1746 | "intervalFactor": 1, 1747 | "legendFormat": "{{instance}}_{{device}}_Read bytes", 1748 | "refId": "A", 1749 | "step": 10 1750 | }, 1751 | { 1752 | "expr": "irate(node_disk_written_bytes_total{instance=~\"$node\"}[30m])", 1753 | "format": "time_series", 1754 | "hide": false, 1755 | "intervalFactor": 1, 1756 | "legendFormat": "{{instance}}_{{device}}_Written bytes", 1757 | "refId": "B", 1758 | "step": 10 1759 | } 1760 | ], 1761 | "thresholds": [], 1762 | "timeFrom": null, 1763 | "timeRegions": [], 1764 | "timeShift": null, 1765 | "title": "Disk R/W Data", 1766 | "tooltip": { 1767 | "shared": true, 1768 | "sort": 2, 1769 | "value_type": "individual" 1770 | }, 1771 | "type": "graph", 1772 | "xaxis": { 1773 | "buckets": null, 1774 | "mode": "time", 1775 | "name": null, 1776 | "show": true, 1777 | "values": [] 1778 | }, 1779 | "yaxes": [ 1780 | { 1781 | "decimals": null, 1782 | "format": "Bps", 1783 | "label": "Bytes read (-) / write (+)", 1784 | "logBase": 1, 1785 | "max": null, 1786 | "min": null, 1787 | "show": true 1788 | }, 1789 | { 1790 | "format": "short", 1791 | "label": null, 1792 | "logBase": 1, 1793 | "max": null, 1794 | "min": null, 1795 | "show": false 1796 | } 1797 | ], 1798 | "yaxis": { 1799 | "align": false, 1800 | "alignLevel": null 1801 | } 1802 | }, 1803 | { 1804 | "aliasColors": { 1805 | "vda": "#6ED0E0" 1806 | }, 1807 | "bars": false, 1808 | "dashLength": 10, 1809 | "dashes": false, 1810 | "datasource": "Prometheus", 1811 | "description": "Time spent on each read/write operation", 1812 | "fill": 1, 1813 | "fillGradient": 1, 1814 | "gridPos": { 1815 | "h": 9, 1816 | "w": 8, 1817 | "x": 16, 1818 | "y": 22 1819 | }, 1820 | "height": "300", 1821 | "hiddenSeries": false, 1822 | "id": 160, 1823 | "legend": { 1824 | "alignAsTable": true, 1825 | "avg": true, 1826 | "current": true, 1827 | "hideEmpty": true, 1828 | "hideZero": true, 1829 | "max": true, 1830 | "min": false, 1831 | "show": true, 1832 | "sort": "current", 1833 | "sortDesc": true, 1834 | "total": false, 1835 | "values": true 1836 | }, 1837 | "lines": true, 1838 | "linewidth": 2, 1839 | "links": [], 1840 | "nullPointMode": "null as zero", 1841 | "options": { 1842 | "dataLinks": [] 1843 | }, 1844 | "percentage": false, 1845 | "pointradius": 5, 1846 | "points": false, 1847 | "renderer": "flot", 1848 | "seriesOverrides": [ 1849 | { 1850 | "alias": "/,*_Read time$/", 1851 | "transform": "negative-Y" 1852 | } 1853 | ], 1854 | "spaceLength": 10, 1855 | "stack": false, 1856 | "steppedLine": false, 1857 | "targets": [ 1858 | { 1859 | "expr": "irate(node_disk_read_time_seconds_total{instance=~\"$node\"}[30m]) / irate(node_disk_reads_completed_total{instance=~\"$node\"}[30m])", 1860 | "format": "time_series", 1861 | "hide": false, 1862 | "instant": false, 1863 | "interval": "", 1864 | "intervalFactor": 1, 1865 | "legendFormat": "{{instance}}_{{device}}_Read time", 1866 | "refId": "B" 1867 | }, 1868 | { 1869 | "expr": "irate(node_disk_write_time_seconds_total{instance=~\"$node\"}[30m]) / irate(node_disk_writes_completed_total{instance=~\"$node\"}[30m])", 1870 | "format": "time_series", 1871 | "hide": false, 1872 | "instant": false, 1873 | "intervalFactor": 1, 1874 | "legendFormat": "{{instance}}_{{device}}_Write time", 1875 | "refId": "C" 1876 | } 1877 | ], 1878 | "thresholds": [], 1879 | "timeFrom": null, 1880 | "timeRegions": [], 1881 | "timeShift": null, 1882 | "title": "Disk R/W Time(Reference: less than 100ms)(beta)", 1883 | "tooltip": { 1884 | "shared": true, 1885 | "sort": 2, 1886 | "value_type": "individual" 1887 | }, 1888 | "type": "graph", 1889 | "xaxis": { 1890 | "buckets": null, 1891 | "mode": "time", 1892 | "name": null, 1893 | "show": true, 1894 | "values": [] 1895 | }, 1896 | "yaxes": [ 1897 | { 1898 | "format": "s", 1899 | "label": "Time. read (-) / write (+)", 1900 | "logBase": 1, 1901 | "max": null, 1902 | "min": null, 1903 | "show": true 1904 | }, 1905 | { 1906 | "format": "short", 1907 | "label": null, 1908 | "logBase": 1, 1909 | "max": null, 1910 | "min": null, 1911 | "show": false 1912 | } 1913 | ], 1914 | "yaxis": { 1915 | "align": false, 1916 | "alignLevel": null 1917 | } 1918 | }, 1919 | { 1920 | "aliasColors": { 1921 | "TCP": "#6ED0E0" 1922 | }, 1923 | "bars": false, 1924 | "dashLength": 10, 1925 | "dashes": false, 1926 | "datasource": "Prometheus", 1927 | "description": "TCP_alloc - Allocated sockets\n\nCurrEstab - TCP connections for which the current state is either ESTABLISHED or CLOSE- WAIT\n\nTCP_tw - Sockets wating close\n\nUDP_inuse - Udp sockets currently in use\n\nSockets_used - Sockets currently in use", 1928 | "fill": 1, 1929 | "fillGradient": 0, 1930 | "gridPos": { 1931 | "h": 12, 1932 | "w": 12, 1933 | "x": 0, 1934 | "y": 31 1935 | }, 1936 | "height": "300", 1937 | "hiddenSeries": false, 1938 | "id": 158, 1939 | "interval": "", 1940 | "legend": { 1941 | "alignAsTable": true, 1942 | "avg": true, 1943 | "current": true, 1944 | "hideEmpty": true, 1945 | "hideZero": true, 1946 | "max": true, 1947 | "min": false, 1948 | "rightSide": false, 1949 | "show": true, 1950 | "sort": "current", 1951 | "sortDesc": true, 1952 | "total": false, 1953 | "values": true 1954 | }, 1955 | "lines": true, 1956 | "linewidth": 2, 1957 | "links": [], 1958 | "nullPointMode": "null", 1959 | "options": { 1960 | "dataLinks": [] 1961 | }, 1962 | "percentage": false, 1963 | "pointradius": 5, 1964 | "points": false, 1965 | "renderer": "flot", 1966 | "seriesOverrides": [ 1967 | { 1968 | "alias": "/.*_Sockets_used/", 1969 | "color": "#C4162A", 1970 | "fill": 0 1971 | } 1972 | ], 1973 | "spaceLength": 10, 1974 | "stack": false, 1975 | "steppedLine": false, 1976 | "targets": [ 1977 | { 1978 | "expr": "node_netstat_Tcp_CurrEstab{instance=~'$node'}", 1979 | "format": "time_series", 1980 | "hide": false, 1981 | "instant": false, 1982 | "interval": "", 1983 | "intervalFactor": 1, 1984 | "legendFormat": "{{instance}}_CurrEstab", 1985 | "refId": "A", 1986 | "step": 20 1987 | }, 1988 | { 1989 | "expr": "node_sockstat_TCP_tw{instance=~'$node'}", 1990 | "format": "time_series", 1991 | "intervalFactor": 1, 1992 | "legendFormat": "{{instance}}_TCP_tw", 1993 | "refId": "D" 1994 | }, 1995 | { 1996 | "expr": "node_sockstat_sockets_used{instance=~'$node'}", 1997 | "legendFormat": "{{instance}}_Sockets_used", 1998 | "refId": "B" 1999 | }, 2000 | { 2001 | "expr": "node_sockstat_UDP_inuse{instance=~'$node'}", 2002 | "legendFormat": "{{instance}}_UDP_inuse", 2003 | "refId": "C" 2004 | }, 2005 | { 2006 | "expr": "node_sockstat_TCP_alloc{instance=~'$node'}", 2007 | "legendFormat": "{{instance}}_TCP_alloc", 2008 | "refId": "E" 2009 | } 2010 | ], 2011 | "thresholds": [], 2012 | "timeFrom": null, 2013 | "timeRegions": [], 2014 | "timeShift": null, 2015 | "title": "Network Sockstat", 2016 | "tooltip": { 2017 | "shared": true, 2018 | "sort": 2, 2019 | "value_type": "individual" 2020 | }, 2021 | "type": "graph", 2022 | "xaxis": { 2023 | "buckets": null, 2024 | "mode": "time", 2025 | "name": null, 2026 | "show": true, 2027 | "values": [] 2028 | }, 2029 | "yaxes": [ 2030 | { 2031 | "format": "short", 2032 | "label": null, 2033 | "logBase": 1, 2034 | "max": null, 2035 | "min": null, 2036 | "show": true 2037 | }, 2038 | { 2039 | "format": "short", 2040 | "label": null, 2041 | "logBase": 1, 2042 | "max": null, 2043 | "min": null, 2044 | "show": true 2045 | } 2046 | ], 2047 | "yaxis": { 2048 | "align": false, 2049 | "alignLevel": null 2050 | } 2051 | }, 2052 | { 2053 | "aliasColors": {}, 2054 | "bars": false, 2055 | "dashLength": 10, 2056 | "dashes": false, 2057 | "datasource": "Prometheus", 2058 | "description": "", 2059 | "fill": 0, 2060 | "fillGradient": 1, 2061 | "gridPos": { 2062 | "h": 12, 2063 | "w": 12, 2064 | "x": 12, 2065 | "y": 31 2066 | }, 2067 | "hiddenSeries": false, 2068 | "id": 169, 2069 | "legend": { 2070 | "alignAsTable": true, 2071 | "avg": true, 2072 | "current": true, 2073 | "hideEmpty": true, 2074 | "hideZero": true, 2075 | "max": true, 2076 | "min": false, 2077 | "rightSide": false, 2078 | "show": true, 2079 | "sideWidth": null, 2080 | "sort": "current", 2081 | "sortDesc": true, 2082 | "total": false, 2083 | "values": true 2084 | }, 2085 | "lines": true, 2086 | "linewidth": 2, 2087 | "links": [], 2088 | "nullPointMode": "null", 2089 | "options": { 2090 | "dataLinks": [] 2091 | }, 2092 | "percentage": false, 2093 | "pointradius": 0.5, 2094 | "points": false, 2095 | "renderer": "flot", 2096 | "seriesOverrides": [], 2097 | "spaceLength": 10, 2098 | "stack": false, 2099 | "steppedLine": false, 2100 | "targets": [ 2101 | { 2102 | "expr": "node_hwmon_temp_celsius{instance=~'$node'}", 2103 | "format": "time_series", 2104 | "intervalFactor": 10, 2105 | "legendFormat": "{{instance}}_{{chip}}_{{sensor}}", 2106 | "refId": "A" 2107 | } 2108 | ], 2109 | "thresholds": [], 2110 | "timeFrom": null, 2111 | "timeRegions": [], 2112 | "timeShift": null, 2113 | "title": "Hardware Temperature(VM may not display the metrics)", 2114 | "tooltip": { 2115 | "shared": true, 2116 | "sort": 2, 2117 | "value_type": "individual" 2118 | }, 2119 | "type": "graph", 2120 | "xaxis": { 2121 | "buckets": null, 2122 | "mode": "time", 2123 | "name": null, 2124 | "show": true, 2125 | "values": [] 2126 | }, 2127 | "yaxes": [ 2128 | { 2129 | "format": "celsius", 2130 | "label": null, 2131 | "logBase": 1, 2132 | "max": null, 2133 | "min": null, 2134 | "show": true 2135 | }, 2136 | { 2137 | "format": "short", 2138 | "label": null, 2139 | "logBase": 1, 2140 | "max": null, 2141 | "min": null, 2142 | "show": true 2143 | } 2144 | ], 2145 | "yaxis": { 2146 | "align": false, 2147 | "alignLevel": null 2148 | } 2149 | } 2150 | ], 2151 | "refresh": false, 2152 | "schemaVersion": 22, 2153 | "style": "dark", 2154 | "tags": [ 2155 | "Prometheus", 2156 | "node_exporter", 2157 | "StarsL.cn" 2158 | ], 2159 | "templating": { 2160 | "list": [ 2161 | { 2162 | "allValue": null, 2163 | "current": { 2164 | "text": "node_exporter", 2165 | "value": "node_exporter" 2166 | }, 2167 | "datasource": "Prometheus", 2168 | "definition": "label_values(node_uname_info, job)", 2169 | "hide": 0, 2170 | "includeAll": false, 2171 | "label": "JOB", 2172 | "multi": false, 2173 | "name": "job", 2174 | "options": [], 2175 | "query": "label_values(node_uname_info, job)", 2176 | "refresh": 1, 2177 | "regex": "", 2178 | "skipUrlSync": false, 2179 | "sort": 1, 2180 | "tagValuesQuery": "", 2181 | "tags": [], 2182 | "tagsQuery": "", 2183 | "type": "query", 2184 | "useTags": false 2185 | }, 2186 | { 2187 | "allValue": null, 2188 | "current": { 2189 | "selected": false, 2190 | "text": "All", 2191 | "value": "$__all" 2192 | }, 2193 | "datasource": "Prometheus", 2194 | "definition": "label_values(node_uname_info{job=~\"$job\"}, nodename)", 2195 | "hide": 0, 2196 | "includeAll": true, 2197 | "label": "Host", 2198 | "multi": true, 2199 | "name": "hostname", 2200 | "options": [], 2201 | "query": "label_values(node_uname_info{job=~\"$job\"}, nodename)", 2202 | "refresh": 1, 2203 | "regex": "", 2204 | "skipUrlSync": false, 2205 | "sort": 0, 2206 | "tagValuesQuery": "", 2207 | "tags": [], 2208 | "tagsQuery": "", 2209 | "type": "query", 2210 | "useTags": false 2211 | }, 2212 | { 2213 | "allFormat": "glob", 2214 | "allValue": null, 2215 | "current": { 2216 | "selected": false, 2217 | "text": "All", 2218 | "value": "$__all" 2219 | }, 2220 | "datasource": "Prometheus", 2221 | "definition": "label_values(node_uname_info{nodename=~\"$hostname\"},instance)", 2222 | "hide": 0, 2223 | "includeAll": true, 2224 | "label": "IP", 2225 | "multi": false, 2226 | "multiFormat": "regex values", 2227 | "name": "node", 2228 | "options": [], 2229 | "query": "label_values(node_uname_info{nodename=~\"$hostname\"},instance)", 2230 | "refresh": 2, 2231 | "regex": "", 2232 | "skipUrlSync": false, 2233 | "sort": 1, 2234 | "tagValuesQuery": "", 2235 | "tags": [], 2236 | "tagsQuery": "", 2237 | "type": "query", 2238 | "useTags": false 2239 | }, 2240 | { 2241 | "allValue": null, 2242 | "current": { 2243 | "text": "/etc/hostname", 2244 | "value": "/etc/hostname" 2245 | }, 2246 | "datasource": "Prometheus", 2247 | "definition": "", 2248 | "hide": 2, 2249 | "includeAll": false, 2250 | "label": "", 2251 | "multi": false, 2252 | "name": "maxmount", 2253 | "options": [], 2254 | "query": "query_result(topk(1,sort_desc (max(node_filesystem_size_bytes{instance=~'$node',fstype=~\"ext4|xfs\"}) by (mountpoint))))", 2255 | "refresh": 2, 2256 | "regex": "/.*\\\"(.*)\\\".*/", 2257 | "skipUrlSync": false, 2258 | "sort": 0, 2259 | "tagValuesQuery": "", 2260 | "tags": [], 2261 | "tagsQuery": "", 2262 | "type": "query", 2263 | "useTags": false 2264 | }, 2265 | { 2266 | "allFormat": "glob", 2267 | "allValue": null, 2268 | "current": { 2269 | "isNone": true, 2270 | "selected": false, 2271 | "text": "None", 2272 | "value": "" 2273 | }, 2274 | "datasource": "Prometheus", 2275 | "definition": "", 2276 | "hide": 2, 2277 | "includeAll": false, 2278 | "label": "环境", 2279 | "multi": false, 2280 | "multiFormat": "regex values", 2281 | "name": "env", 2282 | "options": [], 2283 | "query": "label_values(node_exporter_build_info,env)", 2284 | "refresh": 2, 2285 | "regex": "", 2286 | "skipUrlSync": false, 2287 | "sort": 1, 2288 | "tagValuesQuery": "", 2289 | "tags": [], 2290 | "tagsQuery": "", 2291 | "type": "query", 2292 | "useTags": false 2293 | }, 2294 | { 2295 | "allFormat": "glob", 2296 | "allValue": "", 2297 | "current": { 2298 | "isNone": true, 2299 | "selected": false, 2300 | "text": "None", 2301 | "value": "" 2302 | }, 2303 | "datasource": "Prometheus", 2304 | "definition": "label_values(node_exporter_build_info{env=~'$env'},name)", 2305 | "hide": 2, 2306 | "includeAll": false, 2307 | "label": "名称", 2308 | "multi": true, 2309 | "multiFormat": "regex values", 2310 | "name": "name", 2311 | "options": [], 2312 | "query": "label_values(node_exporter_build_info{env=~'$env'},name)", 2313 | "refresh": 2, 2314 | "regex": "", 2315 | "skipUrlSync": false, 2316 | "sort": 1, 2317 | "tagValuesQuery": "/.*/", 2318 | "tags": [], 2319 | "tagsQuery": "", 2320 | "type": "query", 2321 | "useTags": false 2322 | } 2323 | ] 2324 | }, 2325 | "time": { 2326 | "from": "now-2d", 2327 | "to": "now" 2328 | }, 2329 | "timepicker": { 2330 | "now": true, 2331 | "refresh_intervals": [ 2332 | "5s", 2333 | "10s", 2334 | "30s", 2335 | "1m", 2336 | "5m", 2337 | "15m", 2338 | "30m", 2339 | "1h", 2340 | "2h", 2341 | "1d" 2342 | ], 2343 | "time_options": [ 2344 | "5m", 2345 | "15m", 2346 | "1h", 2347 | "6h", 2348 | "12h", 2349 | "24h", 2350 | "2d", 2351 | "7d", 2352 | "30d" 2353 | ] 2354 | }, 2355 | "timezone": "browser", 2356 | "title": "Node Exporter", 2357 | "uid": "hb7fSE0Zz", 2358 | "version": 1 2359 | } -------------------------------------------------------------------------------- /grafana/conf/datasources.yml: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | 3 | datasources: 4 | - name: 'Prometheus' 5 | type: 'prometheus' 6 | url: 'http://prometheus:9090' 7 | access: proxy 8 | isDefault: true 9 | version: 1 10 | editable: true 11 | jsonData: 12 | timeInterval: '10s' -------------------------------------------------------------------------------- /node_exporter/Dockerfile.template: -------------------------------------------------------------------------------- 1 | FROM balenalib/%%BALENA_MACHINE_NAME%%-alpine 2 | WORKDIR /app 3 | 4 | RUN echo "http://dl-cdn.alpinelinux.org/alpine/edge/testing/" >> /etc/apk/repositories && \ 5 | apk add --no-cache prometheus-node-exporter 6 | 7 | # Expose the port Prometheus uses 8 | EXPOSE 9100 9 | 10 | # Start Node Exporter 11 | CMD [ "/usr/bin/node_exporter" ] 12 | -------------------------------------------------------------------------------- /prometheus/Dockerfile.template: -------------------------------------------------------------------------------- 1 | FROM balenalib/%%BALENA_MACHINE_NAME%%-debian:stretch 2 | 3 | # Set the correct architecture for your device 4 | ARG ARCH="armv7" 5 | 6 | # Install Prometheus 7 | RUN export DEBIAN_FRONTEND=noninteractive && \ 8 | install_packages apt-utils wget vim tar gzip && \ 9 | wget -qO - https://github.com/prometheus/prometheus/releases/download/v2.15.2/prometheus-2.15.2.linux-${ARCH}.tar.gz > prometheus.tar.gz && \ 10 | tar zxf prometheus.tar.gz && \ 11 | mv prometheus-* prometheus && \ 12 | rm prometheus.tar.gz 13 | 14 | # Create a peristent volume and copy config file into it 15 | VOLUME /etc/prometheus 16 | COPY prometheus.yml /etc/prometheus/prometheus.yml 17 | 18 | # Start Prometheus 19 | COPY start.sh ./ 20 | ENTRYPOINT [ "/bin/bash", "./start.sh" ] 21 | -------------------------------------------------------------------------------- /prometheus/prometheus.yml: -------------------------------------------------------------------------------- 1 | global: 2 | scrape_interval: 15s 3 | evaluation_interval: 15s 4 | scrape_timeout: 10s 5 | 6 | scrape_configs: 7 | - job_name: prometheus 8 | honor_timestamps: true 9 | scrape_interval: 15s 10 | scrape_timeout: 10s 11 | metrics_path: /metrics 12 | scheme: http 13 | static_configs: 14 | - targets: 15 | - prometheus:9090 16 | - job_name: node_exporter 17 | honor_timestamps: true 18 | scrape_interval: 15s 19 | scrape_timeout: 10s 20 | metrics_path: /metrics 21 | scheme: http 22 | static_configs: 23 | - targets: ['node_exporter:9100'] 24 | 25 | 26 | -------------------------------------------------------------------------------- /prometheus/start.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | if [[ ! -z $TARGETS ]] 4 | then 5 | # Copy the environment variable to a file 6 | echo $TARGETS > targets 7 | 8 | # Clean up the entries to parse correctly 9 | sed -i "s/ //g" targets 10 | sed -ie "s/^/\\\'/g" targets 11 | sed -ie "s/$/\\\'/g" targets 12 | sed -ie "s/,/\\\',\\\'/g" targets 13 | 14 | # Replace the default node_exporter entry with all the targets 15 | sed -ie "s/\['node_exporter:9100'\]/TARGETS/g" /etc/prometheus/prometheus.yml 16 | sed -ie "s/TARGETS/\['node_exporter:9100',$(cat targets)\]/g" /etc/prometheus/prometheus.yml 17 | fi 18 | 19 | # Start Prometheus with the updated config file 20 | chmod a+x ./prometheus/prometheus 21 | ./prometheus/prometheus --config.file=/etc/prometheus/prometheus.yml --------------------------------------------------------------------------------