├── LICENSE ├── README.md ├── config ├── docker-compose.yml ├── samples ├── Dockerfile ├── accounting-dashboard-kibana.json ├── grafana │ ├── dashboards │ │ ├── accounting.json │ │ └── accounting.yaml │ └── datasources │ │ └── datasource.yaml ├── http.conf ├── logstash │ ├── es_template_accounting.json │ └── logstash.conf ├── nginx.conf └── stream.conf └── src ├── http ├── ngx_http_accounting_module.c ├── ngx_http_accounting_module.h └── ngx_http_accounting_statuses.c ├── ngx_traffic_accounting.h ├── ngx_traffic_accounting_log.c ├── ngx_traffic_accounting_module.c ├── ngx_traffic_accounting_module.h ├── ngx_traffic_accounting_module_conf.c ├── ngx_traffic_accounting_period_metrics.c ├── ngx_traffic_accounting_statuses.c └── stream ├── ngx_stream_accounting_module.c ├── ngx_stream_accounting_module.h └── ngx_stream_accounting_statuses.c /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 2-Clause License 2 | 3 | Copyright (C) 2010-2018, Liu Lantao 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | * Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | * Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 17 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 18 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 20 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 21 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 22 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 23 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 24 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 25 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # traffic-accounting-nginx-module 2 | 3 | Monitor the incoming and outgoing traffic metrics in realtime for `NGINX`. 4 | 5 | **Now accounting module supports both `HTTP` and `STREAM` subsystems** 6 | 7 | A realtime traffic and status code monitor solution for NGINX, 8 | which needs less memory and cpu than other realtime log analyzing solutions. 9 | Useful for traffic accounting based on NGINX config logic (by location / server / user-defined-variables). 10 | 11 | [![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2FLax%2Ftraffic-accounting-nginx-module.svg?type=shield)](https://app.fossa.io/projects/git%2Bgithub.com%2FLax%2Ftraffic-accounting-nginx-module?ref=badge_shield) 12 | [![Financial Contributors on Open Collective](https://opencollective.com/traffic-acctiong-nginx-module/all/badge.svg?label=financial+contributors)](https://opencollective.com/traffic-acctiong-nginx-module) 13 | 14 | ## Why? 15 | 16 | Realtime log analysis solutions, 17 | which requires multiple machines for storage and analysis, 18 | are too heavy for application monitoring. 19 | 20 | An cost-effective solution is in need to monitor the traffic metrics/status of application requests. 21 | This solution should be accurate, sensitive, robust, light weight enough, and not affected by traffic peaks. 22 | 23 | ## How it works? 24 | 25 | This module keeps a list of **metrics** identified by `accounting_id` in its context. 26 | 27 | When a new **request** hits the server, the module will try to find its `accounting_id`, calculate statistics, and **aggregate** them into the corresponding metrics by `accounting_id`. 28 | 29 | For each time period (defined by `interval`), a timer event is triggered, those metrics are rotated and exported to log files or sent to remote log servers. 30 | 31 | 32 | --- 33 | 34 | # Quickstart 35 | 36 | Download pre-build binaries from [Releases](https://github.com/Lax/traffic-accounting-nginx-module/releases), 37 | place them into `./modules` sub-directory of `nginx`. 38 | 39 | Add following lines at the beginning of `nginx.conf`: 40 | 41 | ``` 42 | load_module modules/ngx_http_accounting_module.so; 43 | ``` 44 | 45 | Reload nginx config with `nginx -s reload`. *Done!* 46 | 47 | *Alternatively, you can install this module manually with the Nginx source, see the [installation instructions](#Installation)* 48 | 49 | --- 50 | 51 | ## Dashboard 52 | 53 | **Dashboard - Visualize with Grafana** 54 | ![Accounting Dashboard](http://lax.github.io/traffic-accounting-nginx-module/images/accounting-dashboard.png) 55 | 56 | --- 57 | 58 | # Configuration 59 | 60 | Edit your nginx.conf. 61 | 62 | Example: 63 | 64 | ```nginx 65 | http{ 66 | # turn on accounting function 67 | accounting on; 68 | accounting_log logs/http-accounting.log; 69 | ... 70 | server { 71 | server_name example.com; 72 | 73 | accounting_id $http_host; # set accounting_id string by variable 74 | 75 | location / { 76 | accounting_id accounting_id_str; # set accounting_id string by location 77 | 78 | ... 79 | } 80 | 81 | location /api { 82 | accounting_id API_PC; # for pc 83 | 84 | if ($http_user_agent ~* '(Android|webOS|iPhone|iPod|BlackBerry)') { 85 | accounting_id API_MOBILE; # for mobile 86 | } 87 | 88 | ... 89 | } 90 | } 91 | 92 | } 93 | ``` 94 | 95 | # Directives 96 | 97 | accounting 98 | -------------------- 99 | **syntax:** *accounting on | off* 100 | 101 | **default:** *accounting off* 102 | 103 | **context:** *http, stream* 104 | 105 | accounting_log 106 | -------------------- 107 | **syntax:** *accounting_log \ \[level]* 108 | 109 | **default:** *-* 110 | 111 | **context:** *http, stream* 112 | 113 | Configures logging. 114 | 115 | Support both local `file` path, or `stderr`, or `syslog:`. 116 | The second parameter is the log level. 117 | For more details of supported params, refer to [this page from nginx.org](http://nginx.org/en/docs/ngx_core_module.html#error_log). 118 | 119 | If not specified, accounting log will be written to `/dev/log`. 120 | 121 | accounting_id 122 | -------------------- 123 | **syntax:** *accounting_id \* 124 | 125 | **default:** *accounting_id default* 126 | 127 | **context:** *http, stream, server, location, if in location* 128 | 129 | Sets the `accounting_id` string by user defined variable. 130 | 131 | This string is used to determine which `metrics` a request/session should be aggregated to. 132 | 133 | accounting_interval 134 | ------------------------ 135 | **syntax:** *accounting_interval \* 136 | 137 | **default:** *accounting_interval 60* 138 | 139 | **context:** *http, stream* 140 | 141 | Specifies the reporting interval. Defaults to 60 seconds. 142 | 143 | accounting_perturb 144 | ------------------------ 145 | **syntax:** *accounting_perturb on | off* 146 | 147 | **default:** *accounting_perturb off* 148 | 149 | **context:** *http, stream* 150 | 151 | Randomly staggers the reporting interval by 20% from the usual time. 152 | 153 | # Usage 154 | 155 | This module can be configured to writes metrics to local file, remote log server or local syslog device. 156 | 157 | Open-source log-aggregation software such as logstash also support syslog input, which will help you establish a central log server. 158 | See [samples/logstash/](samples/logstash/) for examples. [**Recommended**] 159 | 160 | To collect logs with local syslog, 161 | refer [Lax/ngx_http_accounting_module-utils](http://github.com/Lax/ngx_http_accounting_module-utils) to for sample configuration / utils. 162 | 163 | ## docker / docker-compose 164 | To demonstrate with docker-compose, run 165 | 166 | ``` 167 | docker-compose build 168 | docker-compose up -d 169 | ``` 170 | 171 | Open Grafana (address: `http://localhost:3000`) in your browser. 172 | 173 | Create and configurate elasticsearch datasource with options: 174 | ``` 175 | Type: elasticsearch 176 | URL: http://elasticsearch:9200 177 | Version: 5.6+ 178 | Min time interval: 1m 179 | ``` 180 | 181 | Then import accounting dashboard from [`samples/accounting-dashboard-grafana.json`](samples/accounting-dashboard-grafana.json). 182 | 183 | 184 | ## Metrics log format 185 | 186 | ``` 187 | # HTTP 188 | 2018/05/14 14:18:18 [notice] 5#0: pid:5|from:1526278638|to:1526278659|accounting_id:HTTP_ECHO_HELLO|requests:4872|bytes_in:438480|bytes_out:730800|latency_ms:0|upstream_latency_ms:0|200:4872 189 | 2018/05/14 14:18:18 [notice] 5#0: pid:5|from:1526278638|to:1526278659|accounting_id:INDEX|requests:4849|bytes_in:421863|bytes_out:1857167|latency_ms:0|upstream_latency_ms:0|301:4849 190 | 191 | # Stream 192 | 2018/05/14 14:18:22 [notice] 5#0: pid:5|from:1526278642|to:1526278659|accounting_id:TCP_PROXY_ECHO|sessions:9723|bytes_in:860343|bytes_out:2587967|latency_ms:4133|upstream_latency_ms:3810|200:9723 193 | ``` 194 | 195 | Each line of the log output contains `metrics` for a particular `accounting_id`, 196 | which contains a list of key-values. 197 | 198 | | key name | meanings of values | 199 | |-----------------|---------------------| 200 | | `pid` | pid of nginx worker process | 201 | | `from` / `to` | metric was collected from the `period` between these timestamps | 202 | | `accounting_id` | identify for the accounting unit, set by `accounting_id` directive | 203 | | `requests` | count of total requests processed in current period (HTTP module only) | 204 | | `sessions` | count of total sessions processed in current period (Stream module only) | 205 | | `bytes_in` | total bytes received by the server | 206 | | `bytes_out` | total bytes send out by the server | 207 | | `latency_ms` | sum of all requests/sessions' `$session_time`, in `millisecond` | 208 | | `upstream_latency_ms` | sum of `$upstream_response_time`, in `millisecond` | 209 | | `200` / `302` / `400` / `404` / `500` ... | count of requests/sessions with status code `200`/`302`/`400`/`404`/`500`, etc. Notice the differences between http codes and stream codes | 210 | 211 | 212 | --- 213 | ## Installation 214 | 215 | ### Step 1 216 | 217 | There are several ways to integrate traffic accounting functions into NGINX. 218 | 219 | * Download pre-build binaries from [Releases](https://github.com/Lax/traffic-accounting-nginx-module/releases). 220 | 221 | * Build the binaries from sources 222 | 223 | ``` 224 | # grab nginx source code from nginx.org, then cd to /path/to/nginx-src/ 225 | git clone https://github.com/Lax/traffic-accounting-nginx-module.git 226 | 227 | # to build as `static` module 228 | ./configure --prefix=/opt/nginx --with-stream --add-module=traffic-accounting-nginx-module 229 | make && make install 230 | 231 | 232 | # to build as `dynamic` module 233 | # both HTTP and STREAM module, target module file name is ngx_http_accounting_module.so 234 | ./configure --prefix=/opt/nginx --with-stream --add-dynamic-module=traffic-accounting-nginx-module 235 | 236 | # only HTTP module, target module file name is ngx_http_accounting_module.so 237 | #./configure --prefix=/opt/nginx --add-dynamic-module=traffic-accounting-nginx-module 238 | 239 | # only STREAM module, target module file name is ngx_stream_accounting_module.so 240 | #./configure --prefix=/opt/nginx --without-http --add-dynamic-module=traffic-accounting-nginx-module 241 | 242 | make modules 243 | ``` 244 | 245 | ### Step 2 (dynamic module only) 246 | 247 | Add the following lines at the beginning of `nginx.conf`: 248 | 249 | ``` 250 | load_module modules/ngx_http_accounting_module.so; 251 | 252 | # for STREAM only build 253 | #load_module modules/ngx_stream_accounting_module.so; 254 | ``` 255 | 256 | ### Step 3 257 | 258 | ``` 259 | http { 260 | accounting on; 261 | accounting_log logs/http-accounting.log; 262 | accounting_id $hostname; 263 | 264 | ... 265 | } 266 | 267 | stream { 268 | accounting on; 269 | accounting_log logs/stream-accounting.log; 270 | accounting_id $hostname; 271 | 272 | ... 273 | } 274 | ``` 275 | 276 | ## Visualization 277 | 278 | Visualization with `Kibana` or `Grafana` is easy. 279 | See [samples/](samples/) for examples. 280 | 281 | --- 282 | 283 | # Branches 284 | 285 | * master : main development branch. 286 | * tag v0.1 or v2-freeze-20110526 : legacy release. works with nginx version(0.7.xx, 0.8.xx), nginx 0.9 is not tested. didn't work with nginx above 1.0.x. 287 | 288 | # Contributing 289 | 290 | 1. Fork it ( https://github.com/Lax/traffic-accounting-nginx-module/fork ) 291 | 2. Create your feature branch (`git checkout -b my-new-feature`) 292 | 3. Commit your changes (`git commit -am 'Add some feature'`) 293 | 4. Push to the branch (`git push origin my-new-feature`) 294 | 5. Create a new Pull Request 295 | 296 | [Known issues](https://github.com/Lax/traffic-accounting-nginx-module/issues?q=) 297 | 298 | # Author 299 | 300 | Liu Lantao [Github@Lax](https://github.com/Lax) 301 | 302 | [Contributors](https://github.com/Lax/traffic-accounting-nginx-module/graphs/contributors) 303 | 304 | ## Contributors 305 | 306 | ### Code Contributors 307 | 308 | This project exists thanks to all the people who contribute. [[Contribute](CONTRIBUTING.md)]. 309 | 310 | 311 | ### Financial Contributors 312 | 313 | Become a financial contributor and help us sustain our community. [[Contribute](https://opencollective.com/traffic-acctiong-nginx-module/contribute)] 314 | 315 | #### Individuals 316 | 317 | 318 | 319 | #### Organizations 320 | 321 | Support this project with your organization. Your logo will show up here with a link to your website. [[Contribute](https://opencollective.com/traffic-acctiong-nginx-module/contribute)] 322 | 323 | 324 | 325 | 326 | 327 | 328 | 329 | 330 | 331 | 332 | 333 | 334 | # License 335 | 336 | [BSD-2-Clause](LICENSE) 337 | 338 | 339 | [![FOSSA Status](https://app.fossa.io/api/projects/git%2Bgithub.com%2FLax%2Ftraffic-accounting-nginx-module.svg?type=large)](https://app.fossa.io/projects/git%2Bgithub.com%2FLax%2Ftraffic-accounting-nginx-module?ref=badge_large) 340 | -------------------------------------------------------------------------------- /config: -------------------------------------------------------------------------------- 1 | ngx_addon_name="traffic-accounting-nginx-module" 2 | 3 | TRAFFIC_ACCOUNTING_DEPS=" \ 4 | $ngx_addon_dir/src/ngx_traffic_accounting.h \ 5 | $ngx_addon_dir/src/ngx_traffic_accounting_module.h \ 6 | " 7 | TRAFFIC_ACCOUNTING_SRCS=" \ 8 | $ngx_addon_dir/src/ngx_traffic_accounting_log.c \ 9 | $ngx_addon_dir/src/ngx_traffic_accounting_module.c \ 10 | $ngx_addon_dir/src/ngx_traffic_accounting_module_conf.c \ 11 | $ngx_addon_dir/src/ngx_traffic_accounting_period_metrics.c \ 12 | $ngx_addon_dir/src/ngx_traffic_accounting_statuses.c \ 13 | " 14 | 15 | HTTP_ACCOUNTING_DEPS=" \ 16 | $ngx_addon_dir/src/http/ngx_http_accounting_module.h \ 17 | " 18 | HTTP_ACCOUNTING_SRCS=" \ 19 | $ngx_addon_dir/src/http/ngx_http_accounting_module.c \ 20 | $ngx_addon_dir/src/http/ngx_http_accounting_statuses.c \ 21 | " 22 | 23 | STREAM_ACCOUNTING_DEPS=" \ 24 | $ngx_addon_dir/src/stream/ngx_stream_accounting_module.h \ 25 | " 26 | STREAM_ACCOUNTING_SRCS=" \ 27 | $ngx_addon_dir/src/stream/ngx_stream_accounting_module.c \ 28 | $ngx_addon_dir/src/stream/ngx_stream_accounting_statuses.c \ 29 | " 30 | 31 | ngx_module_type= 32 | ngx_module_name= 33 | ngx_module_incs=$ngx_addon_dir 34 | ngx_module_deps=$TRAFFIC_ACCOUNTING_DEPS 35 | ngx_module_srcs=$TRAFFIC_ACCOUNTING_SRCS 36 | ngx_module_libs= 37 | 38 | if [ $HTTP != NO ] 39 | then 40 | ngx_module_type=HTTP 41 | ngx_module_name="ngx_http_accounting_module" 42 | ngx_module_deps="$ngx_module_deps $HTTP_ACCOUNTING_DEPS" 43 | ngx_module_srcs="$ngx_module_srcs $HTTP_ACCOUNTING_SRCS" 44 | fi 45 | 46 | if [ $STREAM != NO ] 47 | then 48 | ngx_module_type=STREAM 49 | if test -n "$ngx_module_name"; then 50 | ngx_module_name="ngx_http_accounting_module ngx_stream_accounting_module" 51 | else 52 | ngx_module_name="ngx_stream_accounting_module" 53 | fi 54 | ngx_module_deps="$ngx_module_deps $STREAM_ACCOUNTING_DEPS" 55 | ngx_module_srcs="$ngx_module_srcs $STREAM_ACCOUNTING_SRCS" 56 | fi 57 | 58 | . auto/module 59 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3.6' 2 | 3 | services: 4 | elasticsearch: 5 | labels: 6 | com.example.service: "es" 7 | com.example.description: "For searching and indexing data" 8 | image: elasticsearch 9 | networks: 10 | - elk 11 | volumes: 12 | - type: volume 13 | source: esdata 14 | target: /usr/share/elasticsearch/data/ 15 | ports: 16 | - "9200:9200" 17 | 18 | logstash: 19 | labels: 20 | com.example.service: "logstash" 21 | com.example.description: "For logging data" 22 | image: logstash 23 | networks: 24 | - elk 25 | volumes: 26 | - ./samples/logstash:/etc/logstash 27 | command: logstash -f /etc/logstash/logstash.conf 28 | depends_on: 29 | - elasticsearch 30 | 31 | kibana: 32 | labels: 33 | com.example.service: "kibana" 34 | com.example.description: "Data visualisation and for log aggregation" 35 | image: kibana 36 | networks: 37 | - elk 38 | ports: 39 | - "5601:5601" 40 | environment: 41 | - ELASTICSEARCH_URL=http://elasticsearch:9200 42 | depends_on: 43 | - elasticsearch 44 | 45 | grafana: 46 | labels: 47 | com.example.service: "grafana" 48 | com.example.description: "Data visualisation" 49 | image: grafana/grafana 50 | networks: 51 | - elk 52 | ports: 53 | - "3000:3000" 54 | environment: 55 | - GF_SECURITY_ADMIN_PASSWORD=admin 56 | volumes: 57 | - ./samples/grafana:/etc/grafana/provisioning 58 | depends_on: 59 | - elasticsearch 60 | 61 | nginx: 62 | container_name: nginx_accounting 63 | build: 64 | context: . 65 | dockerfile: samples/Dockerfile 66 | networks: 67 | - nginx 68 | - elk 69 | ports: 70 | - 8080:8080 71 | - 8888:8888 72 | - 9999:9999 73 | depends_on: 74 | - logstash 75 | 76 | 77 | networks: 78 | nginx: 79 | elk: 80 | 81 | volumes: 82 | esdata: 83 | -------------------------------------------------------------------------------- /samples/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM centos as builder 2 | 3 | RUN yum install gcc make pcre-devel zlib-devel openssl-devel -y \ 4 | && yum clean all 5 | 6 | ENV PREFIX /opt/nginx 7 | ENV NGX_VER 1.16.0 8 | 9 | ENV WORKDIR /src 10 | ENV NGX_SRC_DIR ${WORKDIR}/nginx-${NGX_VER} 11 | ENV NGX_URL http://nginx.org/download/nginx-${NGX_VER}.tar.gz 12 | ENV NGX_HTTP_ECHO_URL https://github.com/openresty/echo-nginx-module/archive/master.tar.gz 13 | 14 | WORKDIR ${WORKDIR} 15 | 16 | RUN tar zxf `curl -SLOs -w'%{filename_effective}' ${NGX_URL}` -C ${WORKDIR} \ 17 | && tar zxf `curl -SLJOs -w'%{filename_effective}' ${NGX_HTTP_ECHO_URL}` -C ${NGX_SRC_DIR} 18 | 19 | WORKDIR ${NGX_SRC_DIR} 20 | ADD . traffic-accounting-nginx-module 21 | RUN ./configure --prefix=${PREFIX} \ 22 | --with-stream \ 23 | --add-dynamic-module=traffic-accounting-nginx-module \ 24 | --add-dynamic-module=echo-nginx-module-master \ 25 | --http-log-path=/dev/stdout \ 26 | --error-log-path=/dev/stderr \ 27 | && make -s && make -s install 28 | 29 | 30 | FROM centos 31 | 32 | ENV PREFIX /opt/nginx 33 | ENV CONFIG_VER $(date) 34 | 35 | COPY --from=builder ${PREFIX} ${PREFIX} 36 | 37 | WORKDIR ${PREFIX} 38 | 39 | RUN ln -sf /dev/stdout ${PREFIX}/logs/access.log \ 40 | && ln -sf /dev/stderr ${PREFIX}/logs/http-accounting.log \ 41 | && ln -sf /dev/stderr ${PREFIX}/logs/stream-accounting.log \ 42 | && ln -sf /dev/stderr ${PREFIX}/logs/error.log \ 43 | && ln -sf ../usr/share/zoneinfo/Asia/Shanghai /etc/localtime 44 | 45 | ADD samples/nginx.conf ${PREFIX}/conf/nginx.conf 46 | ADD samples/http.conf ${PREFIX}/conf/http.conf 47 | ADD samples/stream.conf ${PREFIX}/conf/stream.conf 48 | 49 | EXPOSE 8080 50 | EXPOSE 8888 51 | EXPOSE 9999 52 | STOPSIGNAL SIGTERM 53 | ENTRYPOINT ["./sbin/nginx", "-g", "daemon off;"] 54 | -------------------------------------------------------------------------------- /samples/accounting-dashboard-kibana.json: -------------------------------------------------------------------------------- 1 | [ 2 | { 3 | "_id": "AWM_y8MHau6ioD98d01F", 4 | "_type": "dashboard", 5 | "_source": { 6 | "title": "Accounting", 7 | "hits": 0, 8 | "description": "", 9 | "panelsJSON": "[{\"col\":1,\"id\":\"AWM_yzSpau6ioD98d01A\",\"panelIndex\":1,\"row\":1,\"size_x\":6,\"size_y\":3,\"type\":\"visualization\"},{\"col\":1,\"id\":\"AWNAXk7_au6ioD98d09W\",\"panelIndex\":2,\"row\":4,\"size_x\":12,\"size_y\":6,\"type\":\"visualization\"},{\"col\":7,\"id\":\"AWNC7Exeau6ioD98d1a0\",\"panelIndex\":3,\"row\":1,\"size_x\":6,\"size_y\":3,\"type\":\"visualization\"}]", 10 | "optionsJSON": "{\"darkTheme\":false}", 11 | "uiStateJSON": "{\"P-1\":{\"vis\":{\"legendOpen\":false}},\"P-2\":{\"vis\":{\"legendOpen\":false}},\"P-3\":{\"vis\":{\"legendOpen\":false}}}", 12 | "version": 1, 13 | "timeRestore": false, 14 | "kibanaSavedObjectMeta": { 15 | "searchSourceJSON": "{\"filter\":[{\"query\":{\"match_all\":{}}}],\"highlightAll\":true,\"version\":true}" 16 | } 17 | } 18 | }, 19 | { 20 | "_id": "AWNC7Exeau6ioD98d1a0", 21 | "_type": "visualization", 22 | "_source": { 23 | "title": "Bytes In/Out", 24 | "visState": "{\"title\":\"Bytes In/Out\",\"type\":\"area\",\"params\":{\"addLegend\":true,\"addTimeMarker\":true,\"addTooltip\":true,\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"labels\":{\"show\":true,\"truncate\":100},\"position\":\"bottom\",\"scale\":{\"type\":\"linear\"},\"show\":true,\"style\":{},\"title\":{\"text\":\"NR Entries\"},\"type\":\"category\"}],\"grid\":{\"categoryLines\":false,\"style\":{\"color\":\"#eee\"},\"valueAxis\":null},\"legendPosition\":\"top\",\"seriesParams\":[{\"data\":{\"id\":\"1\",\"label\":\"Sum of in_bytes\"},\"drawLinesBetweenPoints\":true,\"interpolate\":\"step-after\",\"mode\":\"stacked\",\"show\":\"true\",\"showCircles\":true,\"type\":\"area\",\"valueAxis\":\"ValueAxis-1\"},{\"data\":{\"id\":\"5\",\"label\":\"Sum of out_bytes\"},\"drawLinesBetweenPoints\":true,\"interpolate\":\"step-after\",\"mode\":\"stacked\",\"show\":true,\"showCircles\":true,\"type\":\"area\",\"valueAxis\":\"ValueAxis-1\"}],\"times\":[],\"type\":\"line\",\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"labels\":{\"filter\":false,\"rotate\":0,\"show\":true,\"truncate\":100},\"name\":\"LeftAxis-1\",\"position\":\"left\",\"scale\":{\"mode\":\"normal\",\"type\":\"linear\"},\"show\":true,\"style\":{},\"title\":{\"text\":\"bytes(in/out)\"},\"type\":\"value\"}]},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"sum\",\"schema\":\"metric\",\"params\":{\"field\":\"in_bytes\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"m\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{},\"customLabel\":\"NR Entries\"}},{\"id\":\"4\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"group\",\"params\":{\"field\":\"accounting_id\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"split\",\"params\":{\"field\":\"entry_type\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"row\":true}},{\"id\":\"5\",\"enabled\":true,\"type\":\"sum\",\"schema\":\"metric\",\"params\":{\"field\":\"out_bytes\"}}],\"listeners\":{}}", 25 | "uiStateJSON": "{}", 26 | "description": "", 27 | "version": 1, 28 | "kibanaSavedObjectMeta": { 29 | "searchSourceJSON": "{\"index\":\"AWM_a3AQ_Zm79t8AWN07\",\"query\":{\"match_all\":{}},\"filter\":[]}" 30 | } 31 | } 32 | }, 33 | { 34 | "_id": "AWM_yzSpau6ioD98d01A", 35 | "_type": "visualization", 36 | "_source": { 37 | "title": "NR_Entries", 38 | "visState": "{\"title\":\"NR_Entries\",\"type\":\"histogram\",\"params\":{\"grid\":{\"categoryLines\":false,\"style\":{\"color\":\"#eee\"},\"valueAxis\":null},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"truncate\":100},\"title\":{\"text\":\"NR Entries\"}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\"},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":false,\"truncate\":100},\"title\":{\"text\":\"nr_entries\"}}],\"seriesParams\":[{\"show\":\"true\",\"type\":\"histogram\",\"mode\":\"stacked\",\"data\":{\"label\":\"Sum of nr_entries\",\"id\":\"1\"},\"valueAxis\":\"ValueAxis-1\",\"drawLinesBetweenPoints\":true,\"showCircles\":true,\"interpolate\":\"step-after\"}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"top\",\"times\":[],\"addTimeMarker\":true,\"type\":\"line\"},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"sum\",\"schema\":\"metric\",\"params\":{\"field\":\"nr_entries\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"m\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{},\"customLabel\":\"NR Entries\"}},{\"id\":\"4\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"group\",\"params\":{\"field\":\"accounting_id\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\"}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"split\",\"params\":{\"field\":\"entry_type\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"1\",\"row\":true}}],\"listeners\":{}}", 39 | "uiStateJSON": "{}", 40 | "description": "", 41 | "version": 1, 42 | "kibanaSavedObjectMeta": { 43 | "searchSourceJSON": "{\"index\":\"AWM_a3AQ_Zm79t8AWN07\",\"query\":{\"match_all\":{}},\"filter\":[]}" 44 | } 45 | } 46 | }, 47 | { 48 | "_id": "AWNAXk7_au6ioD98d09W", 49 | "_type": "visualization", 50 | "_source": { 51 | "title": "NR_Status", 52 | "visState": "{\"title\":\"NR_Status\",\"type\":\"histogram\",\"params\":{\"grid\":{\"categoryLines\":false,\"style\":{\"color\":\"#eee\"},\"valueAxis\":null},\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"type\":\"category\",\"position\":\"bottom\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\"},\"labels\":{\"show\":true,\"truncate\":100},\"title\":{\"text\":\"NR Entries\"}}],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"name\":\"LeftAxis-1\",\"type\":\"value\",\"position\":\"left\",\"show\":true,\"style\":{},\"scale\":{\"type\":\"linear\",\"mode\":\"normal\",\"defaultYExtents\":false,\"setYExtents\":false},\"labels\":{\"show\":true,\"rotate\":0,\"filter\":true,\"truncate\":100},\"title\":{\"text\":\"@nr_status\"}}],\"seriesParams\":[{\"show\":true,\"mode\":\"stacked\",\"type\":\"line\",\"drawLinesBetweenPoints\":true,\"showCircles\":true,\"interpolate\":\"step-after\",\"data\":{\"id\":\"5\",\"label\":\"200\"},\"valueAxis\":\"ValueAxis-1\"},{\"show\":\"true\",\"type\":\"area\",\"mode\":\"stacked\",\"data\":{\"label\":\"301\",\"id\":\"1\"},\"valueAxis\":\"ValueAxis-1\",\"drawLinesBetweenPoints\":true,\"showCircles\":true,\"interpolate\":\"step-after\"},{\"show\":true,\"mode\":\"stacked\",\"type\":\"line\",\"drawLinesBetweenPoints\":true,\"showCircles\":true,\"interpolate\":\"step-after\",\"data\":{\"id\":\"7\",\"label\":\"404\"},\"valueAxis\":\"ValueAxis-1\"},{\"show\":true,\"mode\":\"stacked\",\"type\":\"area\",\"drawLinesBetweenPoints\":true,\"showCircles\":true,\"interpolate\":\"step-after\",\"data\":{\"id\":\"6\",\"label\":\"502\"},\"valueAxis\":\"ValueAxis-1\"}],\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"top\",\"times\":[],\"addTimeMarker\":true,\"type\":\"line\"},\"aggs\":[{\"id\":\"2\",\"enabled\":true,\"type\":\"date_histogram\",\"schema\":\"segment\",\"params\":{\"field\":\"@timestamp\",\"interval\":\"m\",\"customInterval\":\"2h\",\"min_doc_count\":1,\"extended_bounds\":{},\"customLabel\":\"NR Entries\"}},{\"id\":\"4\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"group\",\"params\":{\"field\":\"accounting_id\",\"size\":5,\"order\":\"desc\",\"orderBy\":\"_term\",\"customLabel\":\"\"}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"split\",\"params\":{\"field\":\"accounting_id\",\"size\":5,\"order\":\"asc\",\"orderBy\":\"_term\",\"customLabel\":\"Status\",\"row\":true}},{\"id\":\"5\",\"enabled\":true,\"type\":\"sum\",\"schema\":\"metric\",\"params\":{\"field\":\"@nr_status.200\",\"customLabel\":\"200\"}},{\"id\":\"1\",\"enabled\":true,\"type\":\"sum\",\"schema\":\"metric\",\"params\":{\"field\":\"@nr_status.301\",\"customLabel\":\"301\"}},{\"id\":\"7\",\"enabled\":true,\"type\":\"sum\",\"schema\":\"metric\",\"params\":{\"field\":\"@nr_status.404\",\"customLabel\":\"404\"}},{\"id\":\"6\",\"enabled\":true,\"type\":\"sum\",\"schema\":\"metric\",\"params\":{\"field\":\"@nr_status.502\",\"customLabel\":\"502\"}}],\"listeners\":{}}", 53 | "uiStateJSON": "{}", 54 | "description": "", 55 | "version": 1, 56 | "kibanaSavedObjectMeta": { 57 | "searchSourceJSON": "{\"index\":\"AWM_a3AQ_Zm79t8AWN07\",\"query\":{\"match_all\":{}},\"filter\":[]}" 58 | } 59 | } 60 | } 61 | ] -------------------------------------------------------------------------------- /samples/grafana/dashboards/accounting.json: -------------------------------------------------------------------------------- 1 | { 2 | "__inputs": [ 3 | { 4 | "name": "accounting_es", 5 | "label": "accounting_es", 6 | "description": "", 7 | "type": "datasource", 8 | "pluginId": "elasticsearch", 9 | "pluginName": "Elasticsearch" 10 | } 11 | ], 12 | "__requires": [ 13 | { 14 | "type": "datasource", 15 | "id": "elasticsearch", 16 | "name": "Elasticsearch", 17 | "version": "5.0.0" 18 | }, 19 | { 20 | "type": "grafana", 21 | "id": "grafana", 22 | "name": "Grafana", 23 | "version": "5.1.3" 24 | }, 25 | { 26 | "type": "panel", 27 | "id": "graph", 28 | "name": "Graph", 29 | "version": "5.0.0" 30 | }, 31 | { 32 | "type": "panel", 33 | "id": "singlestat", 34 | "name": "Singlestat", 35 | "version": "5.0.0" 36 | } 37 | ], 38 | "annotations": { 39 | "list": [ 40 | { 41 | "builtIn": 1, 42 | "datasource": "-- Grafana --", 43 | "enable": true, 44 | "hide": true, 45 | "iconColor": "rgba(0, 211, 255, 1)", 46 | "name": "Annotations & Alerts", 47 | "type": "dashboard" 48 | } 49 | ] 50 | }, 51 | "editable": false, 52 | "gnetId": null, 53 | "graphTooltip": 0, 54 | "id": null, 55 | "iteration": 1526568107006, 56 | "links": [], 57 | "panels": [ 58 | { 59 | "collapsed": false, 60 | "gridPos": { 61 | "h": 1, 62 | "w": 24, 63 | "x": 0, 64 | "y": 0 65 | }, 66 | "id": 41, 67 | "panels": [], 68 | "repeat": "entry_type", 69 | "title": "Subsystem Summary: [[entry_type]]", 70 | "type": "row" 71 | }, 72 | { 73 | "aliasColors": {}, 74 | "bars": false, 75 | "dashLength": 10, 76 | "dashes": false, 77 | "datasource": "accounting_es", 78 | "decimals": null, 79 | "fill": 0, 80 | "gridPos": { 81 | "h": 6, 82 | "w": 10, 83 | "x": 0, 84 | "y": 1 85 | }, 86 | "id": 2, 87 | "legend": { 88 | "alignAsTable": true, 89 | "avg": false, 90 | "current": true, 91 | "hideEmpty": false, 92 | "hideZero": false, 93 | "max": true, 94 | "min": false, 95 | "show": false, 96 | "total": false, 97 | "values": true 98 | }, 99 | "lines": true, 100 | "linewidth": 1, 101 | "links": [], 102 | "minSpan": 12, 103 | "nullPointMode": "null", 104 | "percentage": false, 105 | "pointradius": 0.5, 106 | "points": true, 107 | "renderer": "flot", 108 | "repeat": null, 109 | "repeatDirection": "h", 110 | "seriesOverrides": [ 111 | { 112 | "alias": "in_bytes", 113 | "color": "#6ed0e0", 114 | "fill": 5, 115 | "linewidth": 0, 116 | "points": false, 117 | "stack": "A", 118 | "steppedLine": true 119 | }, 120 | { 121 | "alias": "out_bytes", 122 | "color": "#508642", 123 | "fill": 5, 124 | "linewidth": 0, 125 | "points": false, 126 | "stack": "A", 127 | "steppedLine": true 128 | }, 129 | { 130 | "alias": "nr_entries", 131 | "color": "#ea6460", 132 | "yaxis": 2 133 | } 134 | ], 135 | "spaceLength": 10, 136 | "stack": false, 137 | "steppedLine": false, 138 | "targets": [ 139 | { 140 | "alias": "{{field}}", 141 | "bucketAggs": [ 142 | { 143 | "field": "@timestamp", 144 | "id": "2", 145 | "settings": { 146 | "interval": "auto", 147 | "min_doc_count": 0, 148 | "trimEdges": 1 149 | }, 150 | "type": "date_histogram" 151 | } 152 | ], 153 | "metrics": [ 154 | { 155 | "field": "in_bytes", 156 | "id": "1", 157 | "meta": {}, 158 | "settings": {}, 159 | "type": "sum" 160 | }, 161 | { 162 | "field": "out_bytes", 163 | "id": "3", 164 | "meta": {}, 165 | "settings": {}, 166 | "type": "sum" 167 | }, 168 | { 169 | "field": "nr_entries", 170 | "id": "4", 171 | "meta": {}, 172 | "settings": {}, 173 | "type": "sum" 174 | } 175 | ], 176 | "query": "entry_type: [[entry_type]]", 177 | "refId": "A", 178 | "timeField": "@timestamp" 179 | } 180 | ], 181 | "thresholds": [], 182 | "timeFrom": null, 183 | "timeShift": null, 184 | "title": "Subsystem: [[entry_type]]", 185 | "tooltip": { 186 | "shared": true, 187 | "sort": 0, 188 | "value_type": "individual" 189 | }, 190 | "transparent": true, 191 | "type": "graph", 192 | "xaxis": { 193 | "buckets": null, 194 | "mode": "time", 195 | "name": null, 196 | "show": true, 197 | "values": [] 198 | }, 199 | "yaxes": [ 200 | { 201 | "decimals": null, 202 | "format": "bytes", 203 | "label": "", 204 | "logBase": 1, 205 | "max": null, 206 | "min": "0", 207 | "show": true 208 | }, 209 | { 210 | "decimals": null, 211 | "format": "short", 212 | "label": null, 213 | "logBase": 1, 214 | "max": null, 215 | "min": "0", 216 | "show": true 217 | } 218 | ], 219 | "yaxis": { 220 | "align": false, 221 | "alignLevel": null 222 | } 223 | }, 224 | { 225 | "aliasColors": {}, 226 | "bars": false, 227 | "dashLength": 10, 228 | "dashes": false, 229 | "datasource": "accounting_es", 230 | "decimals": null, 231 | "fill": 0, 232 | "gridPos": { 233 | "h": 6, 234 | "w": 10, 235 | "x": 10, 236 | "y": 1 237 | }, 238 | "id": 168, 239 | "legend": { 240 | "alignAsTable": true, 241 | "avg": false, 242 | "current": true, 243 | "hideEmpty": false, 244 | "hideZero": false, 245 | "max": true, 246 | "min": false, 247 | "show": false, 248 | "total": false, 249 | "values": true 250 | }, 251 | "lines": true, 252 | "linewidth": 1, 253 | "links": [], 254 | "minSpan": 12, 255 | "nullPointMode": "null", 256 | "percentage": false, 257 | "pointradius": 0.5, 258 | "points": true, 259 | "renderer": "flot", 260 | "repeatDirection": "h", 261 | "seriesOverrides": [], 262 | "spaceLength": 10, 263 | "stack": false, 264 | "steppedLine": true, 265 | "targets": [ 266 | { 267 | "alias": "{{field}}", 268 | "bucketAggs": [ 269 | { 270 | "field": "@timestamp", 271 | "id": "2", 272 | "settings": { 273 | "interval": "auto", 274 | "min_doc_count": 0, 275 | "trimEdges": 1 276 | }, 277 | "type": "date_histogram" 278 | } 279 | ], 280 | "metrics": [ 281 | { 282 | "field": "@agg_status.2xx", 283 | "id": "1", 284 | "meta": {}, 285 | "settings": {}, 286 | "type": "sum" 287 | }, 288 | { 289 | "field": "@agg_status.3xx", 290 | "id": "3", 291 | "meta": {}, 292 | "settings": {}, 293 | "type": "sum" 294 | }, 295 | { 296 | "field": "@agg_status.4xx", 297 | "id": "4", 298 | "meta": {}, 299 | "settings": {}, 300 | "type": "sum" 301 | }, 302 | { 303 | "field": "@agg_status.5xx", 304 | "id": "5", 305 | "meta": {}, 306 | "settings": {}, 307 | "type": "sum" 308 | } 309 | ], 310 | "query": "entry_type: [[entry_type]]", 311 | "refId": "A", 312 | "timeField": "@timestamp" 313 | } 314 | ], 315 | "thresholds": [], 316 | "timeFrom": null, 317 | "timeShift": null, 318 | "title": "[[entry_type]] - Status", 319 | "tooltip": { 320 | "shared": true, 321 | "sort": 0, 322 | "value_type": "individual" 323 | }, 324 | "transparent": true, 325 | "type": "graph", 326 | "xaxis": { 327 | "buckets": null, 328 | "mode": "time", 329 | "name": null, 330 | "show": true, 331 | "values": [] 332 | }, 333 | "yaxes": [ 334 | { 335 | "format": "short", 336 | "label": "", 337 | "logBase": 1, 338 | "max": null, 339 | "min": "0", 340 | "show": true 341 | }, 342 | { 343 | "decimals": null, 344 | "format": "short", 345 | "label": null, 346 | "logBase": 1, 347 | "max": null, 348 | "min": "0", 349 | "show": true 350 | } 351 | ], 352 | "yaxis": { 353 | "align": false, 354 | "alignLevel": null 355 | } 356 | }, 357 | { 358 | "cacheTimeout": null, 359 | "colorBackground": false, 360 | "colorValue": true, 361 | "colors": [ 362 | "#299c46", 363 | "rgba(237, 129, 40, 0.89)", 364 | "#d44a3a" 365 | ], 366 | "datasource": "accounting_es", 367 | "decimals": null, 368 | "format": "none", 369 | "gauge": { 370 | "maxValue": 100, 371 | "minValue": 0, 372 | "show": false, 373 | "thresholdLabels": false, 374 | "thresholdMarkers": true 375 | }, 376 | "gridPos": { 377 | "h": 2, 378 | "w": 4, 379 | "x": 20, 380 | "y": 1 381 | }, 382 | "hideTimeOverride": false, 383 | "id": 150, 384 | "interval": null, 385 | "links": [], 386 | "mappingType": 1, 387 | "mappingTypes": [ 388 | { 389 | "name": "value to text", 390 | "value": 1 391 | }, 392 | { 393 | "name": "range to text", 394 | "value": 2 395 | } 396 | ], 397 | "maxDataPoints": 100, 398 | "minSpan": 1, 399 | "nullPointMode": "connected", 400 | "nullText": null, 401 | "postfix": "", 402 | "postfixFontSize": "50%", 403 | "prefix": "", 404 | "prefixFontSize": "50%", 405 | "rangeMaps": [ 406 | { 407 | "from": "null", 408 | "text": "N/A", 409 | "to": "null" 410 | } 411 | ], 412 | "repeatDirection": "h", 413 | "sparkline": { 414 | "fillColor": "rgba(31, 118, 189, 0.18)", 415 | "full": false, 416 | "lineColor": "rgb(31, 120, 193)", 417 | "show": true 418 | }, 419 | "tableColumn": "", 420 | "targets": [ 421 | { 422 | "alias": "{{field}}", 423 | "bucketAggs": [ 424 | { 425 | "field": "@timestamp", 426 | "id": "2", 427 | "settings": { 428 | "interval": "auto", 429 | "min_doc_count": 0, 430 | "trimEdges": 1 431 | }, 432 | "type": "date_histogram" 433 | } 434 | ], 435 | "metrics": [ 436 | { 437 | "field": "nr_entries", 438 | "id": "1", 439 | "meta": {}, 440 | "settings": {}, 441 | "type": "sum" 442 | } 443 | ], 444 | "query": "entry_type: [[entry_type]]", 445 | "refId": "A", 446 | "timeField": "@timestamp" 447 | } 448 | ], 449 | "thresholds": "", 450 | "title": "entries", 451 | "transparent": true, 452 | "type": "singlestat", 453 | "valueFontSize": "50%", 454 | "valueMaps": [ 455 | { 456 | "op": "=", 457 | "text": "N/A", 458 | "value": "null" 459 | } 460 | ], 461 | "valueName": "current" 462 | }, 463 | { 464 | "cacheTimeout": null, 465 | "colorBackground": false, 466 | "colorValue": true, 467 | "colors": [ 468 | "#299c46", 469 | "rgba(237, 129, 40, 0.89)", 470 | "#d44a3a" 471 | ], 472 | "datasource": "accounting_es", 473 | "decimals": null, 474 | "format": "bytes", 475 | "gauge": { 476 | "maxValue": 100, 477 | "minValue": 0, 478 | "show": false, 479 | "thresholdLabels": false, 480 | "thresholdMarkers": true 481 | }, 482 | "gridPos": { 483 | "h": 2, 484 | "w": 4, 485 | "x": 20, 486 | "y": 3 487 | }, 488 | "hideTimeOverride": true, 489 | "id": 81, 490 | "interval": null, 491 | "links": [], 492 | "mappingType": 1, 493 | "mappingTypes": [ 494 | { 495 | "name": "value to text", 496 | "value": 1 497 | }, 498 | { 499 | "name": "range to text", 500 | "value": 2 501 | } 502 | ], 503 | "maxDataPoints": 100, 504 | "minSpan": 1, 505 | "nullPointMode": "connected", 506 | "nullText": null, 507 | "postfix": "", 508 | "postfixFontSize": "30%", 509 | "prefix": "", 510 | "prefixFontSize": "50%", 511 | "rangeMaps": [ 512 | { 513 | "from": "null", 514 | "text": "N/A", 515 | "to": "null" 516 | } 517 | ], 518 | "repeatDirection": "h", 519 | "sparkline": { 520 | "fillColor": "rgba(31, 118, 189, 0.18)", 521 | "full": false, 522 | "lineColor": "rgb(31, 120, 193)", 523 | "show": true 524 | }, 525 | "tableColumn": "", 526 | "targets": [ 527 | { 528 | "alias": "{{field}}", 529 | "bucketAggs": [ 530 | { 531 | "field": "@timestamp", 532 | "id": "2", 533 | "settings": { 534 | "interval": "auto", 535 | "min_doc_count": 0, 536 | "trimEdges": 1 537 | }, 538 | "type": "date_histogram" 539 | } 540 | ], 541 | "metrics": [ 542 | { 543 | "field": "in_bytes", 544 | "id": "1", 545 | "meta": {}, 546 | "settings": { 547 | "missing": null 548 | }, 549 | "type": "sum" 550 | } 551 | ], 552 | "query": "entry_type: [[entry_type]]", 553 | "refId": "A", 554 | "timeField": "@timestamp" 555 | } 556 | ], 557 | "thresholds": "", 558 | "timeShift": null, 559 | "title": "Traffic Inbound", 560 | "transparent": true, 561 | "type": "singlestat", 562 | "valueFontSize": "50%", 563 | "valueMaps": [ 564 | { 565 | "op": "=", 566 | "text": "N/A", 567 | "value": "null" 568 | } 569 | ], 570 | "valueName": "current" 571 | }, 572 | { 573 | "cacheTimeout": null, 574 | "colorBackground": false, 575 | "colorValue": true, 576 | "colors": [ 577 | "#299c46", 578 | "rgba(237, 129, 40, 0.89)", 579 | "#d44a3a" 580 | ], 581 | "datasource": "accounting_es", 582 | "decimals": null, 583 | "format": "bytes", 584 | "gauge": { 585 | "maxValue": 100, 586 | "minValue": 0, 587 | "show": false, 588 | "thresholdLabels": false, 589 | "thresholdMarkers": true 590 | }, 591 | "gridPos": { 592 | "h": 2, 593 | "w": 4, 594 | "x": 20, 595 | "y": 5 596 | }, 597 | "hideTimeOverride": true, 598 | "id": 82, 599 | "interval": null, 600 | "links": [], 601 | "mappingType": 1, 602 | "mappingTypes": [ 603 | { 604 | "name": "value to text", 605 | "value": 1 606 | }, 607 | { 608 | "name": "range to text", 609 | "value": 2 610 | } 611 | ], 612 | "maxDataPoints": 100, 613 | "minSpan": 1, 614 | "nullPointMode": "connected", 615 | "nullText": null, 616 | "postfix": "", 617 | "postfixFontSize": "50%", 618 | "prefix": "", 619 | "prefixFontSize": "50%", 620 | "rangeMaps": [ 621 | { 622 | "from": "null", 623 | "text": "N/A", 624 | "to": "null" 625 | } 626 | ], 627 | "repeatDirection": "h", 628 | "sparkline": { 629 | "fillColor": "rgba(31, 118, 189, 0.18)", 630 | "full": false, 631 | "lineColor": "rgb(31, 120, 193)", 632 | "show": true 633 | }, 634 | "tableColumn": "", 635 | "targets": [ 636 | { 637 | "alias": "{{field}}", 638 | "bucketAggs": [ 639 | { 640 | "field": "@timestamp", 641 | "id": "2", 642 | "settings": { 643 | "interval": "auto", 644 | "min_doc_count": 0, 645 | "trimEdges": 1 646 | }, 647 | "type": "date_histogram" 648 | } 649 | ], 650 | "metrics": [ 651 | { 652 | "field": "out_bytes", 653 | "id": "1", 654 | "meta": {}, 655 | "settings": { 656 | "missing": null 657 | }, 658 | "type": "sum" 659 | } 660 | ], 661 | "query": "entry_type: [[entry_type]]", 662 | "refId": "A", 663 | "timeField": "@timestamp" 664 | } 665 | ], 666 | "thresholds": "", 667 | "timeFrom": null, 668 | "timeShift": null, 669 | "title": "Traffic Outbound", 670 | "transparent": true, 671 | "type": "singlestat", 672 | "valueFontSize": "50%", 673 | "valueMaps": [ 674 | { 675 | "op": "=", 676 | "text": "N/A", 677 | "value": "null" 678 | } 679 | ], 680 | "valueName": "current" 681 | }, 682 | { 683 | "collapsed": false, 684 | "gridPos": { 685 | "h": 1, 686 | "w": 24, 687 | "x": 0, 688 | "y": 7 689 | }, 690 | "id": 31, 691 | "panels": [], 692 | "repeat": "accounting_id", 693 | "title": "Channel Details - [[accounting_id]]", 694 | "type": "row" 695 | }, 696 | { 697 | "aliasColors": {}, 698 | "bars": false, 699 | "dashLength": 10, 700 | "dashes": false, 701 | "datasource": "accounting_es", 702 | "fill": 0, 703 | "gridPos": { 704 | "h": 6, 705 | "w": 10, 706 | "x": 0, 707 | "y": 8 708 | }, 709 | "id": 10, 710 | "legend": { 711 | "alignAsTable": true, 712 | "avg": true, 713 | "current": true, 714 | "max": true, 715 | "min": false, 716 | "show": false, 717 | "total": false, 718 | "values": true 719 | }, 720 | "lines": true, 721 | "linewidth": 1, 722 | "links": [], 723 | "minSpan": 12, 724 | "nullPointMode": "null", 725 | "percentage": false, 726 | "pointradius": 0.5, 727 | "points": true, 728 | "renderer": "flot", 729 | "repeat": null, 730 | "repeatDirection": "v", 731 | "seriesOverrides": [ 732 | { 733 | "alias": "nr_entries", 734 | "yaxis": 2 735 | }, 736 | { 737 | "alias": "nr_open_entries", 738 | "yaxis": 2 739 | }, 740 | { 741 | "alias": "nr_close_entries", 742 | "yaxis": 2 743 | }, 744 | { 745 | "alias": "in_bytes", 746 | "color": "#6ed0e0", 747 | "fill": 5, 748 | "linewidth": 0, 749 | "points": false, 750 | "stack": true, 751 | "steppedLine": true 752 | }, 753 | { 754 | "alias": "out_bytes", 755 | "color": "#508642", 756 | "fill": 5, 757 | "linewidth": 0, 758 | "points": false, 759 | "stack": true, 760 | "steppedLine": true 761 | } 762 | ], 763 | "spaceLength": 10, 764 | "stack": false, 765 | "steppedLine": false, 766 | "targets": [ 767 | { 768 | "alias": "{{field}}", 769 | "bucketAggs": [ 770 | { 771 | "field": "@timestamp", 772 | "id": "2", 773 | "settings": { 774 | "interval": "auto", 775 | "min_doc_count": 0, 776 | "trimEdges": 1 777 | }, 778 | "type": "date_histogram" 779 | } 780 | ], 781 | "metrics": [ 782 | { 783 | "field": "in_bytes", 784 | "id": "1", 785 | "meta": {}, 786 | "settings": {}, 787 | "type": "sum" 788 | }, 789 | { 790 | "field": "out_bytes", 791 | "id": "3", 792 | "meta": {}, 793 | "settings": {}, 794 | "type": "sum" 795 | }, 796 | { 797 | "field": "nr_entries", 798 | "id": "4", 799 | "meta": {}, 800 | "settings": {}, 801 | "type": "sum" 802 | }, 803 | { 804 | "field": "nr_open_entries", 805 | "id": "5", 806 | "meta": {}, 807 | "settings": {}, 808 | "type": "sum" 809 | }, 810 | { 811 | "field": "nr_close_entries", 812 | "id": "6", 813 | "meta": {}, 814 | "settings": {}, 815 | "type": "sum" 816 | } 817 | ], 818 | "query": "accounting_id: [[accounting_id]]", 819 | "refId": "A", 820 | "timeField": "@timestamp" 821 | } 822 | ], 823 | "thresholds": [], 824 | "timeFrom": null, 825 | "timeShift": null, 826 | "title": "Channel: [[accounting_id]]", 827 | "tooltip": { 828 | "shared": true, 829 | "sort": 0, 830 | "value_type": "individual" 831 | }, 832 | "type": "graph", 833 | "xaxis": { 834 | "buckets": null, 835 | "mode": "time", 836 | "name": null, 837 | "show": true, 838 | "values": [] 839 | }, 840 | "yaxes": [ 841 | { 842 | "format": "bytes", 843 | "label": null, 844 | "logBase": 1, 845 | "max": null, 846 | "min": "0", 847 | "show": true 848 | }, 849 | { 850 | "format": "short", 851 | "label": null, 852 | "logBase": 1, 853 | "max": null, 854 | "min": "0", 855 | "show": true 856 | } 857 | ], 858 | "yaxis": { 859 | "align": false, 860 | "alignLevel": null 861 | } 862 | }, 863 | { 864 | "aliasColors": {}, 865 | "bars": false, 866 | "dashLength": 10, 867 | "dashes": false, 868 | "datasource": "accounting_es", 869 | "fill": 1, 870 | "gridPos": { 871 | "h": 6, 872 | "w": 10, 873 | "x": 10, 874 | "y": 8 875 | }, 876 | "id": 5, 877 | "interval": "", 878 | "legend": { 879 | "alignAsTable": true, 880 | "avg": true, 881 | "current": true, 882 | "hideEmpty": true, 883 | "hideZero": true, 884 | "max": true, 885 | "min": false, 886 | "show": false, 887 | "total": false, 888 | "values": true 889 | }, 890 | "lines": true, 891 | "linewidth": 1, 892 | "links": [], 893 | "minSpan": 12, 894 | "nullPointMode": "null", 895 | "percentage": false, 896 | "pointradius": 0.5, 897 | "points": true, 898 | "renderer": "flot", 899 | "repeat": null, 900 | "repeatDirection": "v", 901 | "seriesOverrides": [], 902 | "spaceLength": 10, 903 | "stack": true, 904 | "steppedLine": true, 905 | "targets": [ 906 | { 907 | "alias": "{{field}}", 908 | "bucketAggs": [ 909 | { 910 | "field": "@timestamp", 911 | "id": "2", 912 | "settings": { 913 | "interval": "auto", 914 | "min_doc_count": 0, 915 | "trimEdges": 1 916 | }, 917 | "type": "date_histogram" 918 | } 919 | ], 920 | "metrics": [ 921 | { 922 | "field": "@nr_status.200", 923 | "id": "1", 924 | "meta": {}, 925 | "settings": {}, 926 | "type": "sum" 927 | }, 928 | { 929 | "field": "@nr_status.301", 930 | "id": "3", 931 | "meta": {}, 932 | "settings": {}, 933 | "type": "sum" 934 | }, 935 | { 936 | "field": "@nr_status.302", 937 | "id": "4", 938 | "meta": {}, 939 | "settings": {}, 940 | "type": "sum" 941 | }, 942 | { 943 | "field": "@nr_status.403", 944 | "id": "5", 945 | "meta": {}, 946 | "settings": {}, 947 | "type": "sum" 948 | }, 949 | { 950 | "field": "@nr_status.404", 951 | "id": "6", 952 | "meta": {}, 953 | "settings": {}, 954 | "type": "sum" 955 | }, 956 | { 957 | "field": "@nr_status.499", 958 | "id": "7", 959 | "meta": {}, 960 | "settings": {}, 961 | "type": "sum" 962 | }, 963 | { 964 | "field": "@nr_status.500", 965 | "id": "8", 966 | "meta": {}, 967 | "settings": {}, 968 | "type": "sum" 969 | }, 970 | { 971 | "field": "@nr_status.503", 972 | "id": "9", 973 | "meta": {}, 974 | "settings": {}, 975 | "type": "sum" 976 | } 977 | ], 978 | "query": "accounting_id: [[accounting_id]]", 979 | "refId": "A", 980 | "timeField": "@timestamp" 981 | } 982 | ], 983 | "thresholds": [], 984 | "timeFrom": null, 985 | "timeShift": null, 986 | "title": "Status: [[accounting_id]]", 987 | "tooltip": { 988 | "shared": true, 989 | "sort": 0, 990 | "value_type": "individual" 991 | }, 992 | "type": "graph", 993 | "xaxis": { 994 | "buckets": null, 995 | "mode": "time", 996 | "name": null, 997 | "show": true, 998 | "values": [] 999 | }, 1000 | "yaxes": [ 1001 | { 1002 | "format": "short", 1003 | "label": null, 1004 | "logBase": 1, 1005 | "max": null, 1006 | "min": "0", 1007 | "show": true 1008 | }, 1009 | { 1010 | "format": "short", 1011 | "label": null, 1012 | "logBase": 1, 1013 | "max": null, 1014 | "min": "0", 1015 | "show": true 1016 | } 1017 | ], 1018 | "yaxis": { 1019 | "align": true, 1020 | "alignLevel": null 1021 | } 1022 | }, 1023 | { 1024 | "cacheTimeout": null, 1025 | "colorBackground": false, 1026 | "colorValue": true, 1027 | "colors": [ 1028 | "#299c46", 1029 | "rgba(237, 129, 40, 0.89)", 1030 | "#d44a3a" 1031 | ], 1032 | "datasource": "accounting_es", 1033 | "format": "none", 1034 | "gauge": { 1035 | "maxValue": 100, 1036 | "minValue": 0, 1037 | "show": false, 1038 | "thresholdLabels": false, 1039 | "thresholdMarkers": true 1040 | }, 1041 | "gridPos": { 1042 | "h": 2, 1043 | "w": 4, 1044 | "x": 20, 1045 | "y": 8 1046 | }, 1047 | "id": 206, 1048 | "interval": null, 1049 | "links": [], 1050 | "mappingType": 1, 1051 | "mappingTypes": [ 1052 | { 1053 | "name": "value to text", 1054 | "value": 1 1055 | }, 1056 | { 1057 | "name": "range to text", 1058 | "value": 2 1059 | } 1060 | ], 1061 | "maxDataPoints": 100, 1062 | "nullPointMode": "connected", 1063 | "nullText": null, 1064 | "postfix": "", 1065 | "postfixFontSize": "50%", 1066 | "prefix": "", 1067 | "prefixFontSize": "50%", 1068 | "rangeMaps": [ 1069 | { 1070 | "from": "null", 1071 | "text": "N/A", 1072 | "to": "null" 1073 | } 1074 | ], 1075 | "repeatDirection": "h", 1076 | "sparkline": { 1077 | "fillColor": "rgba(31, 118, 189, 0.18)", 1078 | "full": false, 1079 | "lineColor": "rgb(31, 120, 193)", 1080 | "show": true 1081 | }, 1082 | "tableColumn": "", 1083 | "targets": [ 1084 | { 1085 | "alias": "{{field}}", 1086 | "bucketAggs": [ 1087 | { 1088 | "field": "@timestamp", 1089 | "id": "2", 1090 | "settings": { 1091 | "interval": "auto", 1092 | "min_doc_count": 0, 1093 | "trimEdges": 1 1094 | }, 1095 | "type": "date_histogram" 1096 | } 1097 | ], 1098 | "metrics": [ 1099 | { 1100 | "field": "nr_entries", 1101 | "id": "1", 1102 | "meta": {}, 1103 | "settings": {}, 1104 | "type": "sum" 1105 | } 1106 | ], 1107 | "query": "accounting_id: [[accounting_id]]", 1108 | "refId": "A", 1109 | "timeField": "@timestamp" 1110 | } 1111 | ], 1112 | "thresholds": "", 1113 | "title": "entries", 1114 | "transparent": true, 1115 | "type": "singlestat", 1116 | "valueFontSize": "50%", 1117 | "valueMaps": [ 1118 | { 1119 | "op": "=", 1120 | "text": "N/A", 1121 | "value": "null" 1122 | } 1123 | ], 1124 | "valueName": "current" 1125 | }, 1126 | { 1127 | "cacheTimeout": null, 1128 | "colorBackground": false, 1129 | "colorValue": true, 1130 | "colors": [ 1131 | "#299c46", 1132 | "rgba(237, 129, 40, 0.89)", 1133 | "#d44a3a" 1134 | ], 1135 | "datasource": "accounting_es", 1136 | "format": "bytes", 1137 | "gauge": { 1138 | "maxValue": 100, 1139 | "minValue": 0, 1140 | "show": false, 1141 | "thresholdLabels": false, 1142 | "thresholdMarkers": true 1143 | }, 1144 | "gridPos": { 1145 | "h": 2, 1146 | "w": 4, 1147 | "x": 20, 1148 | "y": 10 1149 | }, 1150 | "id": 248, 1151 | "interval": null, 1152 | "links": [], 1153 | "mappingType": 1, 1154 | "mappingTypes": [ 1155 | { 1156 | "name": "value to text", 1157 | "value": 1 1158 | }, 1159 | { 1160 | "name": "range to text", 1161 | "value": 2 1162 | } 1163 | ], 1164 | "maxDataPoints": 100, 1165 | "nullPointMode": "connected", 1166 | "nullText": null, 1167 | "postfix": "", 1168 | "postfixFontSize": "50%", 1169 | "prefix": "", 1170 | "prefixFontSize": "50%", 1171 | "rangeMaps": [ 1172 | { 1173 | "from": "null", 1174 | "text": "N/A", 1175 | "to": "null" 1176 | } 1177 | ], 1178 | "repeatDirection": "h", 1179 | "sparkline": { 1180 | "fillColor": "rgba(31, 118, 189, 0.18)", 1181 | "full": false, 1182 | "lineColor": "rgb(31, 120, 193)", 1183 | "show": true 1184 | }, 1185 | "tableColumn": "", 1186 | "targets": [ 1187 | { 1188 | "alias": "{{field}}", 1189 | "bucketAggs": [ 1190 | { 1191 | "field": "@timestamp", 1192 | "id": "2", 1193 | "settings": { 1194 | "interval": "auto", 1195 | "min_doc_count": 0, 1196 | "trimEdges": 1 1197 | }, 1198 | "type": "date_histogram" 1199 | } 1200 | ], 1201 | "metrics": [ 1202 | { 1203 | "field": "in_bytes", 1204 | "id": "1", 1205 | "meta": {}, 1206 | "settings": {}, 1207 | "type": "sum" 1208 | } 1209 | ], 1210 | "query": "accounting_id: [[accounting_id]]", 1211 | "refId": "A", 1212 | "timeField": "@timestamp" 1213 | } 1214 | ], 1215 | "thresholds": "", 1216 | "title": "Inbound", 1217 | "transparent": true, 1218 | "type": "singlestat", 1219 | "valueFontSize": "50%", 1220 | "valueMaps": [ 1221 | { 1222 | "op": "=", 1223 | "text": "N/A", 1224 | "value": "null" 1225 | } 1226 | ], 1227 | "valueName": "current" 1228 | }, 1229 | { 1230 | "cacheTimeout": null, 1231 | "colorBackground": false, 1232 | "colorValue": true, 1233 | "colors": [ 1234 | "#299c46", 1235 | "rgba(237, 129, 40, 0.89)", 1236 | "#d44a3a" 1237 | ], 1238 | "datasource": "accounting_es", 1239 | "format": "bytes", 1240 | "gauge": { 1241 | "maxValue": 100, 1242 | "minValue": 0, 1243 | "show": false, 1244 | "thresholdLabels": false, 1245 | "thresholdMarkers": true 1246 | }, 1247 | "gridPos": { 1248 | "h": 2, 1249 | "w": 4, 1250 | "x": 20, 1251 | "y": 12 1252 | }, 1253 | "id": 249, 1254 | "interval": null, 1255 | "links": [], 1256 | "mappingType": 1, 1257 | "mappingTypes": [ 1258 | { 1259 | "name": "value to text", 1260 | "value": 1 1261 | }, 1262 | { 1263 | "name": "range to text", 1264 | "value": 2 1265 | } 1266 | ], 1267 | "maxDataPoints": 100, 1268 | "nullPointMode": "connected", 1269 | "nullText": null, 1270 | "postfix": "", 1271 | "postfixFontSize": "50%", 1272 | "prefix": "", 1273 | "prefixFontSize": "50%", 1274 | "rangeMaps": [ 1275 | { 1276 | "from": "null", 1277 | "text": "N/A", 1278 | "to": "null" 1279 | } 1280 | ], 1281 | "repeatDirection": "h", 1282 | "sparkline": { 1283 | "fillColor": "rgba(31, 118, 189, 0.18)", 1284 | "full": false, 1285 | "lineColor": "rgb(31, 120, 193)", 1286 | "show": true 1287 | }, 1288 | "tableColumn": "", 1289 | "targets": [ 1290 | { 1291 | "alias": "{{field}}", 1292 | "bucketAggs": [ 1293 | { 1294 | "field": "@timestamp", 1295 | "id": "2", 1296 | "settings": { 1297 | "interval": "auto", 1298 | "min_doc_count": 0, 1299 | "trimEdges": 1 1300 | }, 1301 | "type": "date_histogram" 1302 | } 1303 | ], 1304 | "metrics": [ 1305 | { 1306 | "field": "out_bytes", 1307 | "id": "1", 1308 | "meta": {}, 1309 | "settings": {}, 1310 | "type": "sum" 1311 | } 1312 | ], 1313 | "query": "accounting_id: [[accounting_id]]", 1314 | "refId": "A", 1315 | "timeField": "@timestamp" 1316 | } 1317 | ], 1318 | "thresholds": "", 1319 | "title": "Outbound", 1320 | "transparent": true, 1321 | "type": "singlestat", 1322 | "valueFontSize": "50%", 1323 | "valueMaps": [ 1324 | { 1325 | "op": "=", 1326 | "text": "N/A", 1327 | "value": "null" 1328 | } 1329 | ], 1330 | "valueName": "current" 1331 | } 1332 | ], 1333 | "refresh": "30s", 1334 | "schemaVersion": 16, 1335 | "style": "dark", 1336 | "tags": [], 1337 | "templating": { 1338 | "list": [ 1339 | { 1340 | "allValue": null, 1341 | "current": {}, 1342 | "datasource": "accounting_es", 1343 | "hide": 0, 1344 | "includeAll": true, 1345 | "label": "category", 1346 | "multi": false, 1347 | "name": "entry_type", 1348 | "options": [], 1349 | "query": "{\"find\": \"terms\", \"field\": \"entry_type\"}", 1350 | "refresh": 1, 1351 | "regex": "", 1352 | "sort": 0, 1353 | "tagValuesQuery": "", 1354 | "tags": [], 1355 | "tagsQuery": "", 1356 | "type": "query", 1357 | "useTags": false 1358 | }, 1359 | { 1360 | "allValue": null, 1361 | "current": {}, 1362 | "datasource": "accounting_es", 1363 | "hide": 0, 1364 | "includeAll": true, 1365 | "label": null, 1366 | "multi": true, 1367 | "name": "accounting_id", 1368 | "options": [], 1369 | "query": "{\"find\": \"terms\", \"field\": \"accounting_id\", \"query\": \"entry_type:$entry_type\"}", 1370 | "refresh": 2, 1371 | "regex": "", 1372 | "sort": 0, 1373 | "tagValuesQuery": "", 1374 | "tags": [], 1375 | "tagsQuery": "", 1376 | "type": "query", 1377 | "useTags": false 1378 | }, 1379 | { 1380 | "allValue": null, 1381 | "current": {}, 1382 | "datasource": "accounting_es", 1383 | "hide": 2, 1384 | "includeAll": true, 1385 | "label": null, 1386 | "multi": false, 1387 | "name": "status", 1388 | "options": [], 1389 | "query": "{\"find\":\"fields\"}", 1390 | "refresh": 1, 1391 | "regex": "/^@nr_status.(.*)/", 1392 | "sort": 3, 1393 | "tagValuesQuery": "", 1394 | "tags": [], 1395 | "tagsQuery": "", 1396 | "type": "query", 1397 | "useTags": false 1398 | }, 1399 | { 1400 | "allValue": null, 1401 | "current": {}, 1402 | "datasource": "accounting_es", 1403 | "hide": 2, 1404 | "includeAll": true, 1405 | "label": null, 1406 | "multi": false, 1407 | "name": "agg_status", 1408 | "options": [], 1409 | "query": "{\"find\":\"fields\"}", 1410 | "refresh": 1, 1411 | "regex": "/^@agg_status.(.*)/", 1412 | "sort": 3, 1413 | "tagValuesQuery": "", 1414 | "tags": [], 1415 | "tagsQuery": "", 1416 | "type": "query", 1417 | "useTags": false 1418 | } 1419 | ] 1420 | }, 1421 | "time": { 1422 | "from": "now-1h", 1423 | "to": "now" 1424 | }, 1425 | "timepicker": { 1426 | "hidden": false, 1427 | "nowDelay": "", 1428 | "refresh_intervals": [ 1429 | "30s", 1430 | "1m", 1431 | "3m", 1432 | "5m", 1433 | "1h" 1434 | ], 1435 | "time_options": [ 1436 | "15m", 1437 | "1h", 1438 | "6h", 1439 | "12h", 1440 | "24h", 1441 | "2d", 1442 | "7d", 1443 | "30d" 1444 | ] 1445 | }, 1446 | "timezone": "", 1447 | "title": "Accounting", 1448 | "uid": "ekEH8Mnmk", 1449 | "version": 1 1450 | } 1451 | -------------------------------------------------------------------------------- /samples/grafana/dashboards/accounting.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | 3 | providers: 4 | - name: 'Accounting' 5 | orgId: 1 6 | folder: '' 7 | type: file 8 | disableDeletion: false 9 | options: 10 | path: /etc/grafana/provisioning/dashboards 11 | -------------------------------------------------------------------------------- /samples/grafana/datasources/datasource.yaml: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | 3 | datasources: 4 | - name: accounting_es 5 | type: elasticsearch 6 | access: proxy 7 | database: "ngx_accounting-*" 8 | url: http://elasticsearch:9200 9 | jsonData: 10 | esVersion: "7.10.1" 11 | timeInterval: 1m 12 | timeField: "@timestamp" 13 | -------------------------------------------------------------------------------- /samples/http.conf: -------------------------------------------------------------------------------- 1 | accounting on; 2 | accounting_interval 60; 3 | accounting_perturb on; 4 | accounting_id 'HTTP'; 5 | accounting_log logs/http-accounting.log; 6 | accounting_log syslog:server=logstash:29124,tag=http_accounting,nohostname notice; 7 | 8 | server { 9 | listen 8080; 10 | server_name localhost; 11 | 12 | #charset koi8-r; 13 | 14 | location / { 15 | return 200 ''; 16 | accounting_id $uri; 17 | } 18 | 19 | location /index { 20 | alias html; 21 | index index.html index.htm; 22 | accounting_id "INDEX"; 23 | } 24 | 25 | location /echo { 26 | proxy_pass http://localhost:8888; 27 | proxy_set_header Host "echo"; 28 | proxy_buffering off; 29 | accounting_id "HTTP_PROXY_ECHO"; 30 | } 31 | 32 | #error_page 404 /404.html; 33 | error_page 500 502 503 504 /50x.html; 34 | location = /50x.html { 35 | root html; 36 | } 37 | } 38 | 39 | server { 40 | listen 8888; 41 | server_name echo; 42 | accounting_id $host; 43 | 44 | location / { 45 | echo hello; 46 | echo_flush; 47 | echo_sleep 1.5; # in sec 48 | echo world; 49 | echo_flush; 50 | echo_sleep 2.5; # in sec 51 | echo world; 52 | echo_flush; 53 | echo_sleep 1.5; # in sec 54 | echo nginx; 55 | echo_flush; 56 | 57 | accounting_id 'HTTP_ECHO'; 58 | } 59 | 60 | location /echo/now { 61 | echo "Hello world!"; 62 | } 63 | } 64 | -------------------------------------------------------------------------------- /samples/logstash/es_template_accounting.json: -------------------------------------------------------------------------------- 1 | { 2 | "version": 1, 3 | "index_patterns": ["ngx_accounting-*"], 4 | "settings": { 5 | "index": { 6 | "refresh_interval": "5s" 7 | } 8 | }, 9 | "mappings": { 10 | "dynamic_templates": [ 11 | { 12 | "accounting_status_fields": { 13 | "path_match": "@nr_status.*", 14 | "mapping": { 15 | "type": "integer" 16 | } 17 | } 18 | }, 19 | { 20 | "accounting_aggregate_status_fields": { 21 | "path_match": "@agg_status.*", 22 | "mapping": { 23 | "type": "integer" 24 | } 25 | } 26 | } 27 | ], 28 | "properties": { 29 | "type": { 30 | "type": "keyword" 31 | }, 32 | "@timestamp": { 33 | "type": "date" 34 | }, 35 | "@from": { 36 | "type": "date" 37 | }, 38 | "@to": { 39 | "type": "date" 40 | }, 41 | "accounting_id": { 42 | "type": "keyword" 43 | }, 44 | "entry_type": { 45 | "type": "keyword" 46 | }, 47 | "nr_entries": { 48 | "type": "integer" 49 | }, 50 | "nr_open_entries": { 51 | "type": "integer" 52 | }, 53 | "nr_close_entries": { 54 | "type": "integer" 55 | }, 56 | "in_bytes": { 57 | "type": "integer" 58 | }, 59 | "out_bytes": { 60 | "type": "integer" 61 | }, 62 | "latency_ms": { 63 | "type": "integer" 64 | }, 65 | "upstream_latency_ms": { 66 | "type": "integer" 67 | }, 68 | "message": { 69 | "type": "text" 70 | } 71 | } 72 | }, 73 | "aliases": {} 74 | } 75 | -------------------------------------------------------------------------------- /samples/logstash/logstash.conf: -------------------------------------------------------------------------------- 1 | input { 2 | syslog { 3 | port => 29124 4 | } 5 | } 6 | filter { 7 | grok { 8 | pattern_definitions => { 9 | "TIMESTAMP_NGX" => "%{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{HOUR}:?%{MINUTE}(?::?%{SECOND})" 10 | } 11 | match => { 12 | "message" => 13 | "%{TIMESTAMP_NGX} \[%{LOGLEVEL}] %{NUMBER}\#%{NUMBER}: pid:%{NUMBER:pid:int}\|from:(?\d{10})\|to:(?\d{10})\|accounting_id:(?[^|]+)\|%{WORD:entry_type}:%{NUMBER:nr_entries:int}\|bytes_in:%{NUMBER:in_bytes:int}\|bytes_out:%{NUMBER:out_bytes:int}\|latency_ms:%{NUMBER:latency_ms:int}\|upstream_latency_ms:%{NUMBER:upstream_latency_ms:int}\|%{GREEDYDATA:statuses}" 14 | } 15 | remove_field => [ "host", "severity", "facility", "priority", "severity_label", "facility_label" ] 16 | remove_tag => [] 17 | } 18 | # date { 19 | # match => [ "timestamp" , "yyyy/MM/dd HH:mm:ss" ] 20 | # target => '@timestamp' 21 | # # timezone => 'Asia/Shanghai' 22 | # remove_field => [ "timestamp" ] 23 | # } 24 | date { 25 | match => [ "from" , "UNIX" ] 26 | target => '@from' 27 | remove_field => [ "from" ] 28 | } 29 | date { 30 | match => [ "to" , "UNIX" ] 31 | target => '@to' 32 | remove_field => [ "to" ] 33 | } 34 | kv { 35 | source => "statuses" 36 | target => "statuses_kv" 37 | field_split => "|" 38 | value_split => ":" 39 | remove_field => [ "statuses" ] 40 | } 41 | ruby { 42 | code => "s={};agg={};agg.default=0;(event.get('statuses_kv')||{}).each{|k,v|agg['%dxx'%k[0]]+=v.to_i;s[k]=v.to_i};event.set('@agg_status',agg);event.set('@nr_status',s)" 43 | remove_field => [ "statuses_kv" ] 44 | } 45 | } 46 | output { 47 | stdout { codec => rubydebug } 48 | elasticsearch { 49 | hosts => ["http://elasticsearch:9200"] 50 | manage_template => true 51 | template_overwrite => true 52 | template => "/etc/logstash/es_template_accounting.json" 53 | index => "ngx_accounting-%{+xxxx.ww}" 54 | } 55 | } 56 | -------------------------------------------------------------------------------- /samples/nginx.conf: -------------------------------------------------------------------------------- 1 | load_module modules/ngx_http_accounting_module.so; 2 | #load_module modules/ngx_stream_accounting_module.so; 3 | load_module modules/ngx_http_echo_module.so; 4 | 5 | worker_processes auto; 6 | error_log logs/error.log notice; 7 | 8 | events { 9 | worker_connections 1024; 10 | } 11 | 12 | http { 13 | include mime.types; 14 | default_type application/octet-stream; 15 | 16 | access_log off; 17 | sendfile on; 18 | keepalive_timeout 65; 19 | log_not_found off; 20 | 21 | include http.conf; 22 | } 23 | 24 | stream { 25 | include stream.conf; 26 | } 27 | -------------------------------------------------------------------------------- /samples/stream.conf: -------------------------------------------------------------------------------- 1 | accounting on; 2 | accounting_id 'STREAM'; 3 | accounting_interval 60; 4 | accounting_perturb on; 5 | accounting_log logs/stream-accounting.log; 6 | accounting_log syslog:server=logstash:29124,tag=stream_accounting,nohostname info; 7 | 8 | upstream echo { 9 | server 127.0.0.1:8888 max_fails=3 fail_timeout=30s; 10 | } 11 | 12 | server { 13 | listen 9999; 14 | proxy_connect_timeout 1s; 15 | proxy_timeout 30s; 16 | proxy_pass echo; 17 | resolver 127.0.0.1; 18 | accounting_id "TCP_PROXY_ECHO"; 19 | } 20 | 21 | 22 | upstream dns { 23 | server 8.8.8.8:53; 24 | } 25 | 26 | server { 27 | listen 53 udp; 28 | proxy_responses 1; 29 | proxy_timeout 20s; 30 | proxy_pass dns; 31 | accounting_id $remote_addr; 32 | } 33 | -------------------------------------------------------------------------------- /src/http/ngx_http_accounting_module.c: -------------------------------------------------------------------------------- 1 | 2 | /* 3 | * Copyright (C) Liu Lantao 4 | */ 5 | 6 | 7 | #include 8 | #include 9 | #include "ngx_http_accounting_module.h" 10 | 11 | 12 | static char entry_n[] = "requests"; 13 | static u_char *ngx_http_accounting_title = (u_char *)"NgxAccounting"; 14 | 15 | static ngx_int_t ngx_http_accounting_init(ngx_conf_t *cf); 16 | 17 | static ngx_int_t ngx_http_accounting_process_init(ngx_cycle_t *cycle); 18 | static void ngx_http_accounting_process_exit(ngx_cycle_t *cycle); 19 | 20 | static void worker_process_alarm_handler(ngx_event_t *ev); 21 | 22 | static char *ngx_http_accounting_set_accounting_id(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); 23 | 24 | static ngx_int_t ngx_http_accounting_request_handler(ngx_http_request_t *r); 25 | static ngx_str_t *ngx_http_accounting_get_accounting_id(ngx_http_request_t *r); 26 | 27 | 28 | static ngx_command_t ngx_http_accounting_commands[] = { 29 | { ngx_string("accounting"), 30 | NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1, 31 | ngx_conf_set_flag_slot, 32 | NGX_HTTP_MAIN_CONF_OFFSET, 33 | offsetof(ngx_http_accounting_main_conf_t, enable), 34 | NULL}, 35 | 36 | { ngx_string("accounting_interval"), 37 | NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1, 38 | ngx_conf_set_sec_slot, 39 | NGX_HTTP_MAIN_CONF_OFFSET, 40 | offsetof(ngx_http_accounting_main_conf_t, interval), 41 | NULL}, 42 | 43 | { ngx_string("accounting_perturb"), 44 | NGX_HTTP_MAIN_CONF|NGX_CONF_TAKE1, 45 | ngx_conf_set_flag_slot, 46 | NGX_HTTP_MAIN_CONF_OFFSET, 47 | offsetof(ngx_http_accounting_main_conf_t, perturb), 48 | NULL}, 49 | 50 | { ngx_string("accounting_log"), 51 | NGX_HTTP_MAIN_CONF|NGX_CONF_1MORE, 52 | ngx_traffic_accounting_set_log, 53 | NGX_HTTP_MAIN_CONF_OFFSET, 54 | 0, 55 | NULL}, 56 | 57 | { ngx_string("accounting_id"), 58 | NGX_HTTP_MAIN_CONF|NGX_HTTP_SRV_CONF|NGX_HTTP_LOC_CONF|NGX_HTTP_LIF_CONF 59 | |NGX_CONF_TAKE1, 60 | ngx_http_accounting_set_accounting_id, 61 | NGX_HTTP_LOC_CONF_OFFSET, 62 | 0, 63 | NULL}, 64 | 65 | ngx_null_command 66 | }; 67 | 68 | 69 | static ngx_http_module_t ngx_http_accounting_ctx = { 70 | NULL, /* preconfiguration */ 71 | ngx_http_accounting_init, /* postconfiguration */ 72 | ngx_traffic_accounting_create_main_conf,/* create main configuration */ 73 | ngx_traffic_accounting_init_main_conf, /* init main configuration */ 74 | NULL, /* create server configuration */ 75 | NULL, /* merge server configuration */ 76 | ngx_traffic_accounting_create_loc_conf, /* create location configuration */ 77 | ngx_traffic_accounting_merge_loc_conf /* merge location configuration */ 78 | }; 79 | 80 | 81 | ngx_module_t ngx_http_accounting_module = { 82 | NGX_MODULE_V1, 83 | &ngx_http_accounting_ctx, /* module context */ 84 | ngx_http_accounting_commands, /* module directives */ 85 | NGX_HTTP_MODULE, /* module type */ 86 | NULL, /* init master */ 87 | NULL, /* init module */ 88 | ngx_http_accounting_process_init, /* init process */ 89 | NULL, /* init thread */ 90 | NULL, /* exit thread */ 91 | ngx_http_accounting_process_exit, /* exit process */ 92 | NULL, /* exit master */ 93 | NGX_MODULE_V1_PADDING 94 | }; 95 | 96 | 97 | static ngx_int_t 98 | ngx_http_accounting_init(ngx_conf_t *cf) 99 | { 100 | ngx_http_handler_pt *h; 101 | ngx_http_core_main_conf_t *cmcf; 102 | ngx_http_accounting_main_conf_t *amcf; 103 | 104 | amcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_accounting_module); 105 | if (!amcf->enable) { 106 | return NGX_OK; 107 | } 108 | 109 | cmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_core_module); 110 | 111 | h = ngx_array_push(&cmcf->phases[NGX_HTTP_LOG_PHASE].handlers); 112 | if (h == NULL) { 113 | return NGX_ERROR; 114 | } 115 | 116 | *h = ngx_http_accounting_request_handler; 117 | 118 | return NGX_OK; 119 | } 120 | 121 | 122 | static ngx_int_t 123 | ngx_http_accounting_process_init(ngx_cycle_t *cycle) 124 | { 125 | ngx_http_accounting_main_conf_t *amcf; 126 | ngx_event_t *ev; 127 | time_t perturb_factor = 1000; 128 | 129 | amcf = ngx_http_cycle_get_module_main_conf(cycle, ngx_http_accounting_module); 130 | if (!amcf->enable) { 131 | return NGX_OK; 132 | } 133 | 134 | if (amcf->log != NULL) { 135 | ngx_log_error(NGXTA_LOG_LEVEL, amcf->log, 0, "pid:%i|start http traffic accounting", ngx_getpid()); 136 | } else { 137 | openlog((char *)ngx_http_accounting_title, LOG_NDELAY, LOG_SYSLOG); 138 | syslog(LOG_INFO, "pid:%i|start http traffic accounting", ngx_getpid()); 139 | } 140 | 141 | if (amcf->current == NULL) { 142 | if (ngx_traffic_accounting_period_create(amcf) != NGX_OK) 143 | return NGX_ERROR; 144 | } 145 | 146 | ev = ngx_pcalloc(cycle->pool, sizeof(ngx_event_t)); 147 | if (ev == NULL) 148 | return NGX_ERROR; 149 | 150 | ev->data = NULL; 151 | ev->log = cycle->log; 152 | ev->handler = worker_process_alarm_handler; 153 | ev->cancelable = 1; 154 | 155 | if (amcf->perturb) { 156 | srand(ngx_getpid() * ngx_max_module + ngx_http_accounting_module.ctx_index); 157 | perturb_factor = (1000 - rand() % 200); 158 | } 159 | 160 | ngx_add_timer(ev, amcf->interval * perturb_factor); 161 | 162 | return NGX_OK; 163 | } 164 | 165 | 166 | static void 167 | ngx_http_accounting_process_exit(ngx_cycle_t *cycle) 168 | { 169 | ngx_http_accounting_main_conf_t *amcf; 170 | 171 | amcf = ngx_http_cycle_get_module_main_conf(cycle, ngx_http_accounting_module); 172 | if (!amcf->enable) { 173 | return; 174 | } 175 | 176 | worker_process_alarm_handler(NULL); 177 | 178 | if (amcf->log != NULL) { 179 | ngx_log_error(NGXTA_LOG_LEVEL, amcf->log, 0, "pid:%i|stop http traffic accounting", ngx_getpid()); 180 | } else { 181 | syslog(LOG_INFO, "pid:%i|stop http traffic accounting", ngx_getpid()); 182 | } 183 | } 184 | 185 | static ngx_int_t 186 | worker_process_export_metrics(void *val, void *para1, void *para2) 187 | { 188 | ngx_http_accounting_main_conf_t *amcf; 189 | ngx_int_t rc; 190 | 191 | amcf = ngx_http_cycle_get_module_main_conf(ngx_cycle, ngx_http_accounting_module); 192 | 193 | rc = ngx_traffic_accounting_log_metrics(val, para1, para2, 194 | amcf->log, entry_n, 195 | ngx_http_statuses, 196 | ngx_http_statuses_len ); 197 | if (rc == NGX_OK) { return NGX_DONE; } /* NGX_DONE -> destroy node */ 198 | 199 | return rc; 200 | } 201 | 202 | static void 203 | worker_process_alarm_handler(ngx_event_t *ev) 204 | { 205 | ngx_http_accounting_main_conf_t *amcf; 206 | 207 | amcf = ngx_http_cycle_get_module_main_conf(ngx_cycle, ngx_http_accounting_module); 208 | 209 | ngx_traffic_accounting_period_rotate(amcf); 210 | ngx_traffic_accounting_period_rbtree_iterate(amcf->previous, 211 | worker_process_export_metrics, 212 | amcf->previous->created_at, 213 | amcf->previous->updated_at ); 214 | 215 | if (ngx_exiting || ev == NULL) 216 | return; 217 | 218 | ngx_add_timer(ev, (ngx_msec_t)amcf->interval * 1000); 219 | } 220 | 221 | 222 | static char * 223 | ngx_http_accounting_set_accounting_id(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) 224 | { 225 | return ngx_traffic_accounting_set_accounting_id(cf, cmd, conf, ngx_http_get_variable_index); 226 | } 227 | 228 | 229 | static ngx_int_t 230 | ngx_http_accounting_request_handler(ngx_http_request_t *r) 231 | { 232 | ngx_str_t *accounting_id; 233 | ngx_traffic_accounting_metrics_t *metrics; 234 | ngx_http_accounting_main_conf_t *amcf; 235 | 236 | ngx_uint_t status, i; 237 | ngx_time_t *tp = ngx_timeofday(); 238 | ngx_msec_int_t ms = 0; 239 | ngx_http_upstream_state_t *state; 240 | 241 | accounting_id = ngx_http_accounting_get_accounting_id(r); 242 | if (accounting_id == NULL) { return NGX_ERROR; } 243 | 244 | amcf = ngx_http_get_module_main_conf(r, ngx_http_accounting_module); 245 | 246 | metrics = ngx_traffic_accounting_period_fetch_metrics(amcf->current, accounting_id, amcf->log); 247 | if (metrics == NULL) { return NGX_ERROR; } 248 | 249 | if (ngx_traffic_accounting_metrics_init(metrics, ngx_http_statuses_len, amcf->log) == NGX_ERROR) 250 | return NGX_ERROR; 251 | 252 | amcf->current->updated_at = ngx_timeofday(); 253 | 254 | metrics->nr_entries += 1; 255 | metrics->bytes_in += r->request_length; 256 | metrics->bytes_out += r->connection->sent; 257 | 258 | if (r->err_status) { 259 | status = r->err_status; 260 | } else if (r->headers_out.status) { 261 | status = r->headers_out.status; 262 | } else { 263 | status = NGX_HTTP_STATUS_UNSET; 264 | } 265 | 266 | metrics->nr_status[ngx_status_bsearch(status, ngx_http_statuses, ngx_http_statuses_len)] += 1; 267 | 268 | ms = (ngx_msec_int_t)((tp->sec - r->start_sec) * 1000 + (tp->msec - r->start_msec)); 269 | ms = ngx_max(ms, 0); 270 | 271 | metrics->total_latency_ms += ms; 272 | 273 | if (r->upstream_states != NULL && r->upstream_states->nelts != 0) { 274 | ms = 0; 275 | state = r->upstream_states->elts; 276 | 277 | for (i = 0; i < r->upstream_states->nelts; i++) { 278 | if (state[i].status) { 279 | #if (nginx_version < 1009000) 280 | ms += (state[i].response_sec * 1000 + state[i].response_msec); 281 | #else 282 | ms += state[i].response_time; 283 | #endif 284 | } 285 | } 286 | 287 | metrics->total_upstream_latency_ms += ms; 288 | } 289 | 290 | return NGX_DECLINED; 291 | } 292 | 293 | static ngx_http_accounting_loc_conf_t * 294 | ngx_http_accounting_get_loc_conf(void *entry) 295 | { 296 | return ngx_http_get_module_loc_conf((ngx_http_request_t *)entry, ngx_http_accounting_module); 297 | } 298 | 299 | static ngx_http_variable_value_t * 300 | ngx_http_accounting_get_indexed_variable(void *entry, ngx_uint_t index) 301 | { 302 | return ngx_http_get_indexed_variable((ngx_http_request_t *)entry, index); 303 | } 304 | 305 | static ngx_str_t * 306 | ngx_http_accounting_get_accounting_id(ngx_http_request_t *r) 307 | { 308 | return ngx_traffic_accounting_get_accounting_id(r, ngx_http_accounting_get_loc_conf, ngx_http_accounting_get_indexed_variable); 309 | } 310 | -------------------------------------------------------------------------------- /src/http/ngx_http_accounting_module.h: -------------------------------------------------------------------------------- 1 | 2 | /* 3 | * Copyright (C) Liu Lantao 4 | */ 5 | 6 | 7 | #ifndef _NGX_HTTP_ACCOUNTING_MODULE_H_INCLUDED_ 8 | #define _NGX_HTTP_ACCOUNTING_MODULE_H_INCLUDED_ 9 | 10 | 11 | #include 12 | #include 13 | #include "../ngx_traffic_accounting_module.h" 14 | 15 | 16 | typedef ngx_traffic_accounting_main_conf_t ngx_http_accounting_main_conf_t; 17 | typedef ngx_traffic_accounting_loc_conf_t ngx_http_accounting_loc_conf_t; 18 | 19 | extern ngx_module_t ngx_http_accounting_module; 20 | 21 | 22 | // Status Code. Default 0 23 | #define NGX_HTTP_STATUS_UNSET 0 24 | #define NGX_HTTP_MAX_STATUS NGX_HTTP_INSUFFICIENT_STORAGE 25 | 26 | extern ngx_uint_t ngx_http_statuses[]; 27 | extern ngx_uint_t ngx_http_statuses_len; 28 | 29 | 30 | #endif /* _NGX_HTTP_ACCOUNTING_MODULE_H_INCLUDED_ */ 31 | -------------------------------------------------------------------------------- /src/http/ngx_http_accounting_statuses.c: -------------------------------------------------------------------------------- 1 | 2 | /* 3 | * Copyright (C) Liu Lantao 4 | */ 5 | 6 | 7 | #include "ngx_http_accounting_module.h" 8 | 9 | 10 | ngx_uint_t ngx_http_statuses[] = { 11 | NGX_HTTP_STATUS_UNSET, 12 | NGX_HTTP_CONTINUE, 13 | NGX_HTTP_SWITCHING_PROTOCOLS, 14 | NGX_HTTP_PROCESSING, 15 | NGX_HTTP_OK, 16 | NGX_HTTP_CREATED, 17 | NGX_HTTP_ACCEPTED, 18 | NGX_HTTP_NO_CONTENT, 19 | NGX_HTTP_PARTIAL_CONTENT, 20 | NGX_HTTP_SPECIAL_RESPONSE, 21 | NGX_HTTP_MOVED_PERMANENTLY, 22 | NGX_HTTP_MOVED_TEMPORARILY, 23 | NGX_HTTP_SEE_OTHER, 24 | NGX_HTTP_NOT_MODIFIED, 25 | NGX_HTTP_TEMPORARY_REDIRECT, 26 | NGX_HTTP_PERMANENT_REDIRECT, 27 | NGX_HTTP_BAD_REQUEST, 28 | NGX_HTTP_UNAUTHORIZED, 29 | NGX_HTTP_FORBIDDEN, 30 | NGX_HTTP_NOT_FOUND, 31 | NGX_HTTP_NOT_ALLOWED, 32 | NGX_HTTP_REQUEST_TIME_OUT, 33 | NGX_HTTP_CONFLICT, 34 | NGX_HTTP_LENGTH_REQUIRED, 35 | NGX_HTTP_PRECONDITION_FAILED, 36 | NGX_HTTP_REQUEST_ENTITY_TOO_LARGE, 37 | NGX_HTTP_REQUEST_URI_TOO_LARGE, 38 | NGX_HTTP_UNSUPPORTED_MEDIA_TYPE, 39 | NGX_HTTP_RANGE_NOT_SATISFIABLE, 40 | NGX_HTTP_MISDIRECTED_REQUEST, 41 | NGX_HTTP_TOO_MANY_REQUESTS, 42 | NGX_HTTP_CLOSE, 43 | NGX_HTTP_NGINX_CODES, 44 | NGX_HTTP_REQUEST_HEADER_TOO_LARGE, 45 | NGX_HTTPS_CERT_ERROR, 46 | NGX_HTTPS_NO_CERT, 47 | NGX_HTTP_TO_HTTPS, 48 | NGX_HTTP_CLIENT_CLOSED_REQUEST, 49 | NGX_HTTP_INTERNAL_SERVER_ERROR, 50 | NGX_HTTP_NOT_IMPLEMENTED, 51 | NGX_HTTP_BAD_GATEWAY, 52 | NGX_HTTP_SERVICE_UNAVAILABLE, 53 | NGX_HTTP_GATEWAY_TIME_OUT, 54 | NGX_HTTP_VERSION_NOT_SUPPORTED, 55 | NGX_HTTP_INSUFFICIENT_STORAGE 56 | }; 57 | 58 | ngx_uint_t ngx_http_statuses_len = sizeof(ngx_http_statuses)/sizeof(ngx_http_statuses[0]); 59 | -------------------------------------------------------------------------------- /src/ngx_traffic_accounting.h: -------------------------------------------------------------------------------- 1 | 2 | /* 3 | * Copyright (C) Liu Lantao 4 | */ 5 | 6 | 7 | #ifndef _NGX_TRAFFIC_ACCOUNTING_H_INCLUDED_ 8 | #define _NGX_TRAFFIC_ACCOUNTING_H_INCLUDED_ 9 | 10 | #include 11 | 12 | 13 | /* 14 | * Status Code 15 | */ 16 | 17 | ngx_uint_t ngx_status_bsearch(ngx_uint_t status, ngx_uint_t statuses[], ngx_uint_t size); 18 | 19 | /* 20 | * Period / Metrics 21 | */ 22 | 23 | typedef struct { 24 | ngx_rbtree_node_t rbnode; 25 | 26 | ngx_str_t name; 27 | 28 | ngx_uint_t nr_entries; 29 | ngx_uint_t bytes_in; 30 | ngx_uint_t bytes_out; 31 | ngx_uint_t total_latency_ms; 32 | ngx_uint_t total_upstream_latency_ms; 33 | ngx_uint_t *nr_status; 34 | ngx_uint_t *nr_upstream_status; 35 | } ngx_traffic_accounting_metrics_t; 36 | 37 | typedef struct { 38 | ngx_rbtree_t rbtree; 39 | ngx_rbtree_node_t sentinel; 40 | 41 | ngx_time_t *created_at; 42 | ngx_time_t *updated_at; 43 | } ngx_traffic_accounting_period_t; 44 | 45 | ngx_int_t ngx_traffic_accounting_metrics_init(ngx_traffic_accounting_metrics_t *metrics, size_t len, ngx_log_t *log); 46 | 47 | ngx_int_t ngx_traffic_accounting_period_init(ngx_traffic_accounting_period_t *period); 48 | void ngx_traffic_accounting_period_insert(ngx_traffic_accounting_period_t *period, ngx_str_t *name, ngx_log_t *log); 49 | void ngx_traffic_accounting_period_insert_metrics(ngx_traffic_accounting_period_t *period, ngx_traffic_accounting_metrics_t *metrics); 50 | void ngx_traffic_accounting_period_delete(ngx_traffic_accounting_period_t *period, ngx_str_t *name); 51 | void ngx_traffic_accounting_period_delete_metrics(ngx_traffic_accounting_period_t *period, ngx_traffic_accounting_metrics_t *metrics); 52 | ngx_traffic_accounting_metrics_t * ngx_traffic_accounting_period_lookup_metrics(ngx_traffic_accounting_period_t *period, ngx_str_t *name); 53 | ngx_traffic_accounting_metrics_t * ngx_traffic_accounting_period_fetch_metrics(ngx_traffic_accounting_period_t *period, ngx_str_t *name, ngx_log_t *log); 54 | 55 | typedef ngx_int_t (*ngx_traffic_accounting_period_iterate_func)(void *val, void *para1, void *para2); 56 | 57 | ngx_int_t ngx_traffic_accounting_period_rbtree_iterate(ngx_traffic_accounting_period_t *period, 58 | ngx_traffic_accounting_period_iterate_func func, void *para1, void *para2); 59 | 60 | 61 | /* 62 | * Log 63 | */ 64 | 65 | #define NGXTA_LOG_LEVEL NGX_LOG_NOTICE 66 | 67 | ngx_int_t ngx_traffic_accounting_log_metrics(void *val, void *para1, void *para2, 68 | ngx_log_t *log, char entry_n[], ngx_uint_t statuses[], ngx_uint_t statuses_len); 69 | 70 | 71 | #endif /* _NGX_TRAFFIC_ACCOUNTING_H_INCLUDED_ */ 72 | -------------------------------------------------------------------------------- /src/ngx_traffic_accounting_log.c: -------------------------------------------------------------------------------- 1 | 2 | /* 3 | * Copyright (C) Liu Lantao 4 | */ 5 | 6 | 7 | #include 8 | #include "ngx_traffic_accounting.h" 9 | 10 | 11 | ngx_int_t 12 | ngx_traffic_accounting_log_metrics(void *val, void *para1, void *para2, 13 | ngx_log_t *log, char entry_n[], ngx_uint_t statuses[], ngx_uint_t statuses_len) 14 | { 15 | ngx_traffic_accounting_metrics_t *metrics = (ngx_traffic_accounting_metrics_t *)val; 16 | ngx_time_t *created_at = para1; 17 | ngx_time_t *updated_at = para2; 18 | 19 | ngx_str_t accounting_msg; 20 | u_char msg_buf[NGX_MAX_ERROR_STR]; 21 | u_char *p, *last; 22 | ngx_uint_t i; 23 | 24 | if (metrics->nr_entries == 0) { 25 | return NGX_OK; 26 | } 27 | 28 | if (metrics->name.len > 255) { 29 | // Possible buffer overrun. Do not log it. 30 | return NGX_OK; 31 | } 32 | 33 | // Output message buffer 34 | p = msg_buf; 35 | last = msg_buf + NGX_MAX_ERROR_STR; 36 | 37 | p = ngx_slprintf(p, last, 38 | "pid:%i|from:%i|to:%i|accounting_id:%V|%s:%ui|bytes_in:%ui|bytes_out:%ui|latency_ms:%ui|upstream_latency_ms:%ui", 39 | ngx_getpid(), 40 | created_at->sec, updated_at->sec, 41 | &metrics->name, entry_n, 42 | metrics->nr_entries, 43 | metrics->bytes_in, metrics->bytes_out, 44 | metrics->total_latency_ms, 45 | metrics->total_upstream_latency_ms 46 | ); 47 | 48 | for (i = 0; i < statuses_len; i++) { 49 | if (metrics->nr_status[i] == 0) 50 | continue; 51 | 52 | p = ngx_slprintf(p, last, "|%i:%i", 53 | statuses[i], metrics->nr_status[i] ); 54 | } 55 | 56 | if (p > last - NGX_LINEFEED_SIZE) { 57 | p = last - NGX_LINEFEED_SIZE; 58 | } 59 | 60 | *p++ = '\0'; 61 | 62 | accounting_msg.len = p - msg_buf; 63 | accounting_msg.data = msg_buf; 64 | 65 | if (log != NULL) { 66 | ngx_log_error(NGXTA_LOG_LEVEL, log, 0, "%V", &accounting_msg); 67 | } else { 68 | syslog(LOG_INFO, "%s", msg_buf); 69 | } 70 | 71 | return NGX_OK; 72 | } 73 | -------------------------------------------------------------------------------- /src/ngx_traffic_accounting_module.c: -------------------------------------------------------------------------------- 1 | 2 | /* 3 | * Copyright (C) Liu Lantao 4 | */ 5 | 6 | 7 | #include "ngx_traffic_accounting.h" 8 | #include "ngx_traffic_accounting_module.h" 9 | 10 | 11 | ngx_int_t 12 | ngx_traffic_accounting_period_create(ngx_traffic_accounting_main_conf_t *amcf) 13 | { 14 | ngx_traffic_accounting_period_t *period; 15 | 16 | period = ngx_calloc(sizeof(ngx_traffic_accounting_period_t), amcf->log); 17 | if (period == NULL) 18 | return NGX_ERROR; 19 | 20 | ngx_traffic_accounting_period_init(period); 21 | 22 | period->created_at = ngx_timeofday(); 23 | 24 | amcf->current = period; 25 | 26 | return NGX_OK; 27 | } 28 | 29 | ngx_int_t 30 | ngx_traffic_accounting_period_rotate(ngx_traffic_accounting_main_conf_t *amcf) 31 | { 32 | ngx_free(amcf->previous); 33 | 34 | amcf->previous = amcf->current; 35 | 36 | return ngx_traffic_accounting_period_create(amcf); 37 | } 38 | -------------------------------------------------------------------------------- /src/ngx_traffic_accounting_module.h: -------------------------------------------------------------------------------- 1 | 2 | /* 3 | * Copyright (C) Liu Lantao 4 | */ 5 | 6 | 7 | #ifndef _NGX_TRAFFIC_ACCOUNTING_MODULE_H_INCLUDED_ 8 | #define _NGX_TRAFFIC_ACCOUNTING_MODULE_H_INCLUDED_ 9 | 10 | #include 11 | #include "ngx_traffic_accounting.h" 12 | 13 | 14 | #define NGX_CONF_INDEX_UNSET -128 15 | 16 | typedef struct { 17 | ngx_flag_t enable; 18 | ngx_log_t *log; 19 | time_t interval; 20 | ngx_flag_t perturb; 21 | 22 | ngx_traffic_accounting_period_t *current; 23 | ngx_traffic_accounting_period_t *previous; 24 | } ngx_traffic_accounting_main_conf_t; 25 | 26 | typedef struct { 27 | ngx_str_t accounting_id; 28 | ngx_int_t index; 29 | } ngx_traffic_accounting_loc_conf_t; 30 | 31 | void * ngx_traffic_accounting_create_main_conf(ngx_conf_t *cf); 32 | char * ngx_traffic_accounting_init_main_conf(ngx_conf_t *cf, void *conf); 33 | void * ngx_traffic_accounting_create_loc_conf(ngx_conf_t *cf); 34 | char * ngx_traffic_accounting_merge_loc_conf(ngx_conf_t *cf, void *parent, void *child); 35 | 36 | char * ngx_traffic_accounting_set_log(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); 37 | 38 | typedef ngx_int_t (*ngx_get_variable_index_pt) (ngx_conf_t *cf, ngx_str_t *name); 39 | char * ngx_traffic_accounting_set_accounting_id(ngx_conf_t *cf, ngx_command_t *cmd, void *conf, 40 | ngx_get_variable_index_pt get_variable_index); 41 | 42 | 43 | typedef ngx_traffic_accounting_loc_conf_t *(*ngx_get_loc_conf_pt) (void *entry); 44 | typedef ngx_variable_value_t *(*ngx_get_indexed_variable_pt) (void *entry, ngx_uint_t index); 45 | ngx_str_t * ngx_traffic_accounting_get_accounting_id(void *entry, ngx_get_loc_conf_pt get_loc_conf, 46 | ngx_get_indexed_variable_pt get_indexed_variable); 47 | 48 | 49 | ngx_int_t ngx_traffic_accounting_period_create(ngx_traffic_accounting_main_conf_t *amcf); 50 | ngx_int_t ngx_traffic_accounting_period_rotate(ngx_traffic_accounting_main_conf_t *amcf); 51 | 52 | 53 | #endif /* _NGX_TRAFFIC_ACCOUNTING_MODULE_H_INCLUDED_ */ 54 | -------------------------------------------------------------------------------- /src/ngx_traffic_accounting_module_conf.c: -------------------------------------------------------------------------------- 1 | 2 | /* 3 | * Copyright (C) Liu Lantao 4 | */ 5 | 6 | 7 | #include "ngx_traffic_accounting.h" 8 | #include "ngx_traffic_accounting_module.h" 9 | 10 | 11 | void * 12 | ngx_traffic_accounting_create_main_conf(ngx_conf_t *cf) 13 | { 14 | ngx_traffic_accounting_main_conf_t *amcf; 15 | 16 | amcf = ngx_pcalloc(cf->pool, sizeof(ngx_traffic_accounting_main_conf_t)); 17 | if (amcf == NULL) 18 | return NULL; 19 | 20 | amcf->enable = NGX_CONF_UNSET; 21 | amcf->interval = NGX_CONF_UNSET; 22 | amcf->perturb = NGX_CONF_UNSET; 23 | 24 | return amcf; 25 | } 26 | 27 | char * 28 | ngx_traffic_accounting_init_main_conf(ngx_conf_t *cf, void *conf) 29 | { 30 | ngx_traffic_accounting_main_conf_t *amcf = conf; 31 | 32 | if (amcf->enable == NGX_CONF_UNSET) { amcf->enable = 0; } 33 | if (amcf->interval == NGX_CONF_UNSET) { amcf->interval = 60; } 34 | if (amcf->perturb == NGX_CONF_UNSET) { amcf->perturb = 0; } 35 | 36 | return NGX_CONF_OK; 37 | } 38 | 39 | void * 40 | ngx_traffic_accounting_create_loc_conf(ngx_conf_t *cf) 41 | { 42 | ngx_traffic_accounting_loc_conf_t *conf; 43 | 44 | conf = ngx_pcalloc(cf->pool, sizeof(ngx_traffic_accounting_loc_conf_t)); 45 | if(conf == NULL) { return NULL; } 46 | 47 | conf->index = NGX_CONF_UNSET; 48 | 49 | return conf; 50 | } 51 | 52 | char * 53 | ngx_traffic_accounting_merge_loc_conf(ngx_conf_t *cf, void *parent, void *child) 54 | { 55 | ngx_traffic_accounting_loc_conf_t *prev = parent; 56 | ngx_traffic_accounting_loc_conf_t *conf = child; 57 | 58 | if (conf->index == NGX_CONF_UNSET) { // accounting_id is not set in current location 59 | ngx_conf_merge_str_value(conf->accounting_id, prev->accounting_id, "default"); 60 | conf->index = prev->index; 61 | } 62 | 63 | return NGX_CONF_OK; 64 | } 65 | 66 | 67 | char * 68 | ngx_traffic_accounting_set_log(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) 69 | { 70 | ngx_traffic_accounting_main_conf_t *amcf = conf; 71 | char *rc; 72 | ngx_log_t *log; 73 | 74 | rc = ngx_log_set_log(cf, &amcf->log); 75 | if (rc != NGX_CONF_OK) { return rc; } 76 | 77 | log = amcf->log; 78 | while (log) { 79 | if (log->log_level < NGXTA_LOG_LEVEL) { 80 | log->log_level = NGXTA_LOG_LEVEL; 81 | } 82 | 83 | log = log->next; 84 | } 85 | 86 | return NGX_CONF_OK; 87 | } 88 | 89 | char * 90 | ngx_traffic_accounting_set_accounting_id(ngx_conf_t *cf, ngx_command_t *cmd, void *conf, 91 | ngx_get_variable_index_pt get_variable_index) 92 | { 93 | ngx_traffic_accounting_loc_conf_t *alcf = conf; 94 | ngx_str_t *value; 95 | 96 | value = cf->args->elts; 97 | 98 | if (value[1].data[0] == '$') { 99 | value[1].len--; 100 | value[1].data++; 101 | 102 | alcf->index = get_variable_index(cf, &value[1]); 103 | if (alcf->index == NGX_ERROR) { 104 | return NGX_CONF_ERROR; 105 | } 106 | 107 | alcf->accounting_id = value[1]; 108 | 109 | return NGX_CONF_OK; 110 | } 111 | 112 | alcf->accounting_id = value[1]; 113 | alcf->index = NGX_CONF_INDEX_UNSET; 114 | 115 | return NGX_CONF_OK; 116 | } 117 | 118 | 119 | 120 | ngx_str_t * 121 | ngx_traffic_accounting_get_accounting_id(void *entry, ngx_get_loc_conf_pt get_loc_conf, 122 | ngx_get_indexed_variable_pt get_indexed_variable) 123 | { 124 | ngx_traffic_accounting_loc_conf_t *alcf; 125 | ngx_variable_value_t *vv; 126 | static ngx_str_t accounting_id; 127 | 128 | alcf = get_loc_conf(entry); 129 | if (alcf == NULL) 130 | return NULL; 131 | 132 | if (alcf->index != NGX_CONF_UNSET && alcf->index != NGX_CONF_INDEX_UNSET) { 133 | vv = get_indexed_variable(entry, alcf->index); 134 | 135 | if (vv != NULL) { 136 | if (vv->not_found) { 137 | vv->no_cacheable = 1; 138 | return NULL; 139 | } 140 | 141 | if (!vv->not_found) { 142 | // vv->data[vv->len] = '\0'; 143 | 144 | accounting_id.len = vv->len; 145 | accounting_id.data = vv->data; 146 | 147 | return &accounting_id; 148 | } 149 | } 150 | } 151 | 152 | return &alcf->accounting_id; 153 | } 154 | -------------------------------------------------------------------------------- /src/ngx_traffic_accounting_period_metrics.c: -------------------------------------------------------------------------------- 1 | 2 | /* 3 | * Copyright (C) Liu Lantao 4 | */ 5 | 6 | 7 | #include "ngx_traffic_accounting.h" 8 | 9 | 10 | static void ngx_traffic_accounting_period_insert_value(ngx_rbtree_node_t *temp, ngx_rbtree_node_t *node, ngx_rbtree_node_t *sentinel); 11 | 12 | ngx_int_t 13 | ngx_traffic_accounting_metrics_init(ngx_traffic_accounting_metrics_t *metrics, size_t len, ngx_log_t *log) 14 | { 15 | if (metrics->nr_status == NULL) { 16 | metrics->nr_status = ngx_calloc(sizeof(ngx_uint_t) * len, log); 17 | 18 | if (metrics->nr_status == NULL) 19 | return NGX_ERROR; 20 | } 21 | 22 | if (metrics->nr_upstream_status == NULL) { 23 | metrics->nr_upstream_status = ngx_calloc(sizeof(ngx_uint_t) * len, log); 24 | 25 | if (metrics->nr_upstream_status == NULL) 26 | return NGX_ERROR; 27 | } 28 | 29 | return NGX_OK; 30 | } 31 | 32 | ngx_int_t 33 | ngx_traffic_accounting_period_init(ngx_traffic_accounting_period_t *period) 34 | { 35 | ngx_rbtree_init(&period->rbtree, &period->sentinel, 36 | ngx_traffic_accounting_period_insert_value); 37 | 38 | return NGX_OK; 39 | } 40 | 41 | void 42 | ngx_traffic_accounting_period_insert(ngx_traffic_accounting_period_t *period, ngx_str_t *name, ngx_log_t *log) 43 | { 44 | ngx_traffic_accounting_metrics_t *metrics; 45 | 46 | metrics = ngx_calloc(sizeof(ngx_traffic_accounting_metrics_t), log); 47 | 48 | void *data; 49 | data = ngx_calloc(name->len+1, log); 50 | ngx_memcpy(data, name->data, name->len); 51 | 52 | metrics->name.data = data; 53 | metrics->name.len = name->len; 54 | 55 | ngx_traffic_accounting_period_insert_metrics(period, metrics); 56 | } 57 | 58 | void 59 | ngx_traffic_accounting_period_insert_metrics(ngx_traffic_accounting_period_t *period, ngx_traffic_accounting_metrics_t *metrics) 60 | { 61 | ngx_rbtree_t *rbtree = &period->rbtree; 62 | ngx_rbtree_node_t *node = &metrics->rbnode; 63 | 64 | node->key = ngx_hash_key_lc(metrics->name.data, metrics->name.len); 65 | 66 | ngx_rbtree_insert(rbtree, node); 67 | } 68 | 69 | void 70 | ngx_traffic_accounting_period_delete(ngx_traffic_accounting_period_t *period, ngx_str_t *name) 71 | { 72 | ngx_traffic_accounting_metrics_t *metrics; 73 | 74 | metrics = ngx_traffic_accounting_period_lookup_metrics(period, name); 75 | if (metrics == NULL) 76 | return; 77 | 78 | ngx_traffic_accounting_period_delete_metrics(period, metrics); 79 | } 80 | 81 | void 82 | ngx_traffic_accounting_period_delete_metrics(ngx_traffic_accounting_period_t *period, ngx_traffic_accounting_metrics_t *metrics) 83 | { 84 | ngx_rbtree_delete(&period->rbtree, &metrics->rbnode); 85 | ngx_free(metrics); 86 | } 87 | 88 | ngx_traffic_accounting_metrics_t * 89 | ngx_traffic_accounting_period_lookup_metrics(ngx_traffic_accounting_period_t *period, ngx_str_t *name) 90 | { 91 | ngx_int_t rc; 92 | ngx_traffic_accounting_metrics_t *n; 93 | ngx_rbtree_node_t *node, *sentinel; 94 | 95 | ngx_rbtree_key_t hash = ngx_hash_key_lc(name->data, name->len); 96 | 97 | ngx_rbtree_t *rbtree = &period->rbtree; 98 | node = rbtree->root; 99 | sentinel = rbtree->sentinel; 100 | 101 | while (node != sentinel) { 102 | if (hash != node->key) { 103 | node = (hash < node->key) ? node->left : node->right; 104 | continue; 105 | } 106 | 107 | n = (ngx_traffic_accounting_metrics_t *) node; 108 | rc = ngx_rstrncmp(name->data, n->name.data, name->len); 109 | 110 | if (rc < 0) { 111 | node = node->left; 112 | continue; 113 | } 114 | 115 | if (rc > 0) { 116 | node = node->right; 117 | continue; 118 | } 119 | 120 | return n; 121 | } 122 | 123 | return NULL; 124 | } 125 | 126 | ngx_traffic_accounting_metrics_t * 127 | ngx_traffic_accounting_period_fetch_metrics(ngx_traffic_accounting_period_t *period, ngx_str_t *name, ngx_log_t *log) 128 | { 129 | ngx_traffic_accounting_metrics_t *n; 130 | 131 | n = ngx_traffic_accounting_period_lookup_metrics(period, name); 132 | if (n != NULL) 133 | return n; 134 | 135 | ngx_traffic_accounting_period_insert(period, name, log); 136 | 137 | return ngx_traffic_accounting_period_lookup_metrics(period, name); 138 | } 139 | 140 | ngx_int_t 141 | ngx_traffic_accounting_period_rbtree_iterate(ngx_traffic_accounting_period_t *period, 142 | ngx_traffic_accounting_period_iterate_func func, void *para1, void *para2) 143 | { 144 | ngx_rbtree_t *rbtree; 145 | ngx_rbtree_node_t *node, *sentinel; 146 | ngx_traffic_accounting_metrics_t *n; 147 | ngx_int_t rc; 148 | 149 | rbtree = &period->rbtree; 150 | node = rbtree->root; 151 | sentinel = rbtree->sentinel; 152 | 153 | while (node != sentinel) { 154 | n = (ngx_traffic_accounting_metrics_t *) node; 155 | 156 | rc = func(n, para1, para2); 157 | 158 | if (rc == NGX_DONE) { 159 | /* NGX_DONE -> destroy node */ 160 | ngx_rbtree_delete(rbtree, node); 161 | ngx_free(n->nr_status); 162 | ngx_free(n->nr_upstream_status); 163 | ngx_free(n->name.data); 164 | ngx_free(n); 165 | 166 | goto done; 167 | } 168 | 169 | if (rc != NGX_OK) { return rc; } 170 | 171 | done: 172 | node = rbtree->root; 173 | } 174 | 175 | return NGX_OK; 176 | } 177 | 178 | static void 179 | ngx_traffic_accounting_period_insert_value(ngx_rbtree_node_t *temp, 180 | ngx_rbtree_node_t *node, ngx_rbtree_node_t *sentinel) 181 | { 182 | ngx_traffic_accounting_metrics_t *n, *t; // node 183 | ngx_rbtree_node_t **p; 184 | 185 | for ( ;; ) { 186 | n = (ngx_traffic_accounting_metrics_t *) node; 187 | t = (ngx_traffic_accounting_metrics_t *) temp; 188 | 189 | if (node->key != temp->key) { 190 | p = (node->key < temp->key) ? &temp->left : &temp->right; 191 | } else if (n->name.len != t->name.len) { 192 | p = (n->name.len < t->name.len) ? &temp->left : &temp->right; 193 | } else { 194 | p = (ngx_memcmp(n->name.data, t->name.data, n->name.len) < 0) 195 | ? &temp->left : &temp->right; 196 | } 197 | 198 | if (*p == sentinel) { 199 | break; 200 | } 201 | 202 | temp = *p; 203 | } 204 | 205 | *p = node; 206 | node->parent = temp; 207 | node->left = sentinel; 208 | node->right = sentinel; 209 | ngx_rbt_red(node); 210 | } 211 | -------------------------------------------------------------------------------- /src/ngx_traffic_accounting_statuses.c: -------------------------------------------------------------------------------- 1 | 2 | /* 3 | * Copyright (C) Liu Lantao 4 | */ 5 | 6 | 7 | #include "ngx_traffic_accounting.h" 8 | 9 | 10 | static int statuses_cmp(const void * a, const void * b); 11 | 12 | ngx_uint_t 13 | ngx_status_bsearch(ngx_uint_t status, ngx_uint_t statuses[], ngx_uint_t size) 14 | { 15 | ngx_uint_t *match; 16 | 17 | match = bsearch(&status, statuses, size, sizeof(ngx_uint_t), statuses_cmp); 18 | if ( match == NULL ) { 19 | return 0; 20 | } 21 | 22 | return ((ngx_uint_t *)match - statuses); 23 | } 24 | 25 | static int 26 | statuses_cmp(const void * a, const void * b) { 27 | return ( *(int*)a - *(int*)b ); 28 | } 29 | -------------------------------------------------------------------------------- /src/stream/ngx_stream_accounting_module.c: -------------------------------------------------------------------------------- 1 | 2 | /* 3 | * Copyright (C) Liu Lantao 4 | */ 5 | 6 | 7 | #include 8 | #include 9 | #include "ngx_stream_accounting_module.h" 10 | 11 | 12 | static char entry_n[] = "sessions"; 13 | static u_char *ngx_stream_accounting_title = (u_char *)"NgxAccounting"; 14 | 15 | static ngx_int_t ngx_stream_accounting_init(ngx_conf_t *cf); 16 | 17 | static ngx_int_t ngx_stream_accounting_process_init(ngx_cycle_t *cycle); 18 | static void ngx_stream_accounting_process_exit(ngx_cycle_t *cycle); 19 | 20 | static void worker_process_alarm_handler(ngx_event_t *ev); 21 | 22 | static char *ngx_stream_accounting_set_accounting_id(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); 23 | 24 | static ngx_int_t ngx_stream_accounting_session_handler(ngx_stream_session_t *r); 25 | static ngx_str_t *ngx_stream_accounting_get_accounting_id(ngx_stream_session_t *r); 26 | 27 | 28 | static ngx_command_t ngx_stream_accounting_commands[] = { 29 | { ngx_string("accounting"), 30 | NGX_STREAM_MAIN_CONF|NGX_CONF_TAKE1, 31 | ngx_conf_set_flag_slot, 32 | NGX_STREAM_MAIN_CONF_OFFSET, 33 | offsetof(ngx_stream_accounting_main_conf_t, enable), 34 | NULL}, 35 | 36 | { ngx_string("accounting_interval"), 37 | NGX_STREAM_MAIN_CONF|NGX_CONF_TAKE1, 38 | ngx_conf_set_sec_slot, 39 | NGX_STREAM_MAIN_CONF_OFFSET, 40 | offsetof(ngx_stream_accounting_main_conf_t, interval), 41 | NULL}, 42 | 43 | { ngx_string("accounting_perturb"), 44 | NGX_STREAM_MAIN_CONF|NGX_CONF_TAKE1, 45 | ngx_conf_set_flag_slot, 46 | NGX_STREAM_MAIN_CONF_OFFSET, 47 | offsetof(ngx_stream_accounting_main_conf_t, perturb), 48 | NULL}, 49 | 50 | { ngx_string("accounting_log"), 51 | NGX_STREAM_MAIN_CONF|NGX_CONF_1MORE, 52 | ngx_traffic_accounting_set_log, 53 | NGX_STREAM_MAIN_CONF_OFFSET, 54 | 0, 55 | NULL}, 56 | 57 | { ngx_string("accounting_id"), 58 | NGX_STREAM_MAIN_CONF|NGX_STREAM_SRV_CONF|NGX_CONF_TAKE1, 59 | ngx_stream_accounting_set_accounting_id, 60 | NGX_STREAM_SRV_CONF_OFFSET, 61 | 0, 62 | NULL}, 63 | 64 | ngx_null_command 65 | }; 66 | 67 | 68 | static ngx_stream_module_t ngx_stream_accounting_ctx = { 69 | NULL, /* preconfiguration */ 70 | ngx_stream_accounting_init, /* postconfiguration */ 71 | ngx_traffic_accounting_create_main_conf,/* create main configuration */ 72 | ngx_traffic_accounting_init_main_conf, /* init main configuration */ 73 | ngx_traffic_accounting_create_loc_conf, /* create server configuration */ 74 | ngx_traffic_accounting_merge_loc_conf /* merge server configuration */ 75 | }; 76 | 77 | 78 | ngx_module_t ngx_stream_accounting_module = { 79 | NGX_MODULE_V1, 80 | &ngx_stream_accounting_ctx, /* module context */ 81 | ngx_stream_accounting_commands, /* module directives */ 82 | NGX_STREAM_MODULE, /* module type */ 83 | NULL, /* init master */ 84 | NULL, /* init module */ 85 | ngx_stream_accounting_process_init, /* init process */ 86 | NULL, /* init thread */ 87 | NULL, /* exit thread */ 88 | ngx_stream_accounting_process_exit, /* exit process */ 89 | NULL, /* exit master */ 90 | NGX_MODULE_V1_PADDING 91 | }; 92 | 93 | 94 | static ngx_int_t 95 | ngx_stream_accounting_init(ngx_conf_t *cf) 96 | { 97 | ngx_stream_handler_pt *h; 98 | ngx_stream_core_main_conf_t *cmcf; 99 | ngx_stream_accounting_main_conf_t *amcf; 100 | 101 | amcf = ngx_stream_conf_get_module_main_conf(cf, ngx_stream_accounting_module); 102 | if (!amcf->enable) { 103 | return NGX_OK; 104 | } 105 | 106 | cmcf = ngx_stream_conf_get_module_main_conf(cf, ngx_stream_core_module); 107 | 108 | h = ngx_array_push(&cmcf->phases[NGX_STREAM_LOG_PHASE].handlers); 109 | if (h == NULL) { 110 | return NGX_ERROR; 111 | } 112 | 113 | *h = ngx_stream_accounting_session_handler; 114 | 115 | return NGX_OK; 116 | } 117 | 118 | 119 | static ngx_int_t 120 | ngx_stream_accounting_process_init(ngx_cycle_t *cycle) 121 | { 122 | ngx_stream_accounting_main_conf_t *amcf; 123 | ngx_event_t *ev; 124 | time_t perturb_factor = 1000; 125 | 126 | amcf = ngx_stream_cycle_get_module_main_conf(cycle, ngx_stream_accounting_module); 127 | if (!amcf->enable) { 128 | return NGX_OK; 129 | } 130 | 131 | if (amcf->log != NULL) { 132 | ngx_log_error(NGXTA_LOG_LEVEL, amcf->log, 0, "pid:%i|start stream traffic accounting", ngx_getpid()); 133 | } else { 134 | openlog((char *)ngx_stream_accounting_title, LOG_NDELAY, LOG_SYSLOG); 135 | syslog(LOG_INFO, "pid:%i|start stream traffic accounting", ngx_getpid()); 136 | } 137 | 138 | if (amcf->current == NULL) { 139 | if (ngx_traffic_accounting_period_create(amcf) != NGX_OK) 140 | return NGX_ERROR; 141 | } 142 | 143 | ev = ngx_pcalloc(cycle->pool, sizeof(ngx_event_t)); 144 | if (ev == NULL) 145 | return NGX_ERROR; 146 | 147 | ev->data = NULL; 148 | ev->log = cycle->log; 149 | ev->handler = worker_process_alarm_handler; 150 | ev->cancelable = 1; 151 | 152 | if (amcf->perturb) { 153 | srand(ngx_getpid() * ngx_max_module + ngx_stream_accounting_module.ctx_index); 154 | perturb_factor = (1000 - rand() % 200); 155 | } 156 | 157 | ngx_add_timer(ev, amcf->interval * perturb_factor); 158 | 159 | return NGX_OK; 160 | } 161 | 162 | 163 | static void 164 | ngx_stream_accounting_process_exit(ngx_cycle_t *cycle) 165 | { 166 | ngx_stream_accounting_main_conf_t *amcf; 167 | 168 | amcf = ngx_stream_cycle_get_module_main_conf(cycle, ngx_stream_accounting_module); 169 | if (!amcf->enable) { 170 | return; 171 | } 172 | 173 | worker_process_alarm_handler(NULL); 174 | 175 | if (amcf->log != NULL) { 176 | ngx_log_error(NGXTA_LOG_LEVEL, amcf->log, 0, "pid:%i|stop stream traffic accounting", ngx_getpid()); 177 | } else { 178 | syslog(LOG_INFO, "pid:%i|stop stream traffic accounting", ngx_getpid()); 179 | } 180 | } 181 | 182 | static ngx_int_t 183 | worker_process_export_metrics(void *val, void *para1, void *para2) 184 | { 185 | ngx_stream_accounting_main_conf_t *amcf; 186 | ngx_int_t rc; 187 | 188 | amcf = ngx_stream_cycle_get_module_main_conf(ngx_cycle, ngx_stream_accounting_module); 189 | 190 | rc = ngx_traffic_accounting_log_metrics(val, para1, para2, 191 | amcf->log, entry_n, 192 | ngx_stream_statuses, 193 | ngx_stream_statuses_len ); 194 | if (rc == NGX_OK) { return NGX_DONE; } /* NGX_DONE -> destroy node */ 195 | 196 | return rc; 197 | } 198 | 199 | static void 200 | worker_process_alarm_handler(ngx_event_t *ev) 201 | { 202 | ngx_stream_accounting_main_conf_t *amcf; 203 | 204 | amcf = ngx_stream_cycle_get_module_main_conf(ngx_cycle, ngx_stream_accounting_module); 205 | 206 | ngx_traffic_accounting_period_rotate(amcf); 207 | ngx_traffic_accounting_period_rbtree_iterate(amcf->previous, 208 | worker_process_export_metrics, 209 | amcf->previous->created_at, 210 | amcf->previous->updated_at ); 211 | 212 | if (ngx_exiting || ev == NULL) 213 | return; 214 | 215 | ngx_add_timer(ev, (ngx_msec_t)amcf->interval * 1000); 216 | } 217 | 218 | 219 | static char * 220 | ngx_stream_accounting_set_accounting_id(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) 221 | { 222 | return ngx_traffic_accounting_set_accounting_id(cf, cmd, conf, ngx_stream_get_variable_index); 223 | } 224 | 225 | 226 | static ngx_int_t 227 | ngx_stream_accounting_session_handler(ngx_stream_session_t *s) 228 | { 229 | ngx_str_t *accounting_id; 230 | ngx_traffic_accounting_metrics_t *metrics; 231 | ngx_stream_accounting_main_conf_t *amcf; 232 | 233 | ngx_uint_t status, i; 234 | ngx_time_t *tp = ngx_timeofday(); 235 | ngx_msec_int_t ms = 0; 236 | ngx_stream_upstream_state_t *state; 237 | 238 | accounting_id = ngx_stream_accounting_get_accounting_id(s); 239 | if (accounting_id == NULL) { return NGX_ERROR; } 240 | 241 | amcf = ngx_stream_get_module_main_conf(s, ngx_stream_accounting_module); 242 | 243 | metrics = ngx_traffic_accounting_period_fetch_metrics(amcf->current, accounting_id, amcf->log); 244 | if (metrics == NULL) { return NGX_ERROR; } 245 | 246 | if (ngx_traffic_accounting_metrics_init(metrics, ngx_stream_statuses_len, amcf->log) == NGX_ERROR) 247 | return NGX_ERROR; 248 | 249 | amcf->current->updated_at = ngx_timeofday(); 250 | 251 | metrics->nr_entries += 1; 252 | metrics->bytes_in += s->received; 253 | metrics->bytes_out += s->connection->sent; 254 | 255 | if (s->status) { 256 | status = s->status; 257 | } else { 258 | status = NGX_STREAM_STATUS_UNSET; 259 | } 260 | 261 | metrics->nr_status[ngx_status_bsearch(status, ngx_stream_statuses, ngx_stream_statuses_len)] += 1; 262 | 263 | ms = (ngx_msec_int_t)((tp->sec - s->start_sec) * 1000 + (tp->msec - s->start_msec)); 264 | ms = ngx_max(ms, 0); 265 | 266 | metrics->total_latency_ms += ms; 267 | 268 | if (s->upstream_states != NULL && s->upstream_states->nelts != 0) { 269 | ms = 0; 270 | state = s->upstream_states->elts; 271 | 272 | for (i = 0; i < s->upstream_states->nelts; i++) { 273 | ms += state[i].response_time; 274 | } 275 | ms = ngx_max(ms, 0); 276 | 277 | metrics->total_upstream_latency_ms += ms; 278 | } 279 | 280 | return NGX_DECLINED; 281 | } 282 | 283 | static ngx_stream_accounting_loc_conf_t * 284 | ngx_stream_accounting_get_srv_conf(void *entry) 285 | { 286 | return ngx_stream_get_module_srv_conf((ngx_stream_session_t *)entry, ngx_stream_accounting_module); 287 | } 288 | 289 | static ngx_stream_variable_value_t * 290 | ngx_stream_accounting_get_indexed_variable(void *entry, ngx_uint_t index) 291 | { 292 | return ngx_stream_get_indexed_variable((ngx_stream_session_t *)entry, index); 293 | } 294 | 295 | static ngx_str_t * 296 | ngx_stream_accounting_get_accounting_id(ngx_stream_session_t *r) 297 | { 298 | return ngx_traffic_accounting_get_accounting_id(r, ngx_stream_accounting_get_srv_conf, ngx_stream_accounting_get_indexed_variable); 299 | } 300 | -------------------------------------------------------------------------------- /src/stream/ngx_stream_accounting_module.h: -------------------------------------------------------------------------------- 1 | 2 | /* 3 | * Copyright (C) Liu Lantao 4 | */ 5 | 6 | 7 | #ifndef _NGX_STREAM_ACCOUNTING_MODULE_H_INCLUDED_ 8 | #define _NGX_STREAM_ACCOUNTING_MODULE_H_INCLUDED_ 9 | 10 | 11 | #include 12 | #include 13 | #include "../ngx_traffic_accounting_module.h" 14 | 15 | 16 | typedef ngx_traffic_accounting_main_conf_t ngx_stream_accounting_main_conf_t; 17 | typedef ngx_traffic_accounting_loc_conf_t ngx_stream_accounting_loc_conf_t; 18 | 19 | extern ngx_module_t ngx_stream_accounting_module; 20 | 21 | 22 | // Status Code. Default 0 23 | #define NGX_STREAM_STATUS_UNSET 0 24 | #define NGX_STREAM_MAX_STATUS NGX_STREAM_SERVICE_UNAVAILABLE 25 | 26 | extern ngx_uint_t ngx_stream_statuses[]; 27 | extern ngx_uint_t ngx_stream_statuses_len; 28 | 29 | 30 | #endif /* _NGX_STREAM_ACCOUNTING_MODULE_H_INCLUDED_ */ 31 | -------------------------------------------------------------------------------- /src/stream/ngx_stream_accounting_statuses.c: -------------------------------------------------------------------------------- 1 | 2 | /* 3 | * Copyright (C) Liu Lantao 4 | */ 5 | 6 | 7 | #include "ngx_stream_accounting_module.h" 8 | 9 | 10 | ngx_uint_t ngx_stream_statuses[] = { 11 | NGX_STREAM_STATUS_UNSET, 12 | 13 | NGX_STREAM_OK, 14 | NGX_STREAM_BAD_REQUEST, 15 | NGX_STREAM_FORBIDDEN, 16 | NGX_STREAM_INTERNAL_SERVER_ERROR, 17 | NGX_STREAM_BAD_GATEWAY, 18 | NGX_STREAM_SERVICE_UNAVAILABLE 19 | }; 20 | 21 | ngx_uint_t ngx_stream_statuses_len = sizeof(ngx_stream_statuses)/sizeof(ngx_stream_statuses[0]); 22 | --------------------------------------------------------------------------------