├── README.md ├── config ├── graphite_module_v1_15_4.patch ├── graphite_module_v1_2_5.patch ├── graphite_module_v1_7_7.patch ├── lua_module_v0_10_10.patch ├── lua_module_v0_10_12.patch ├── lua_module_v0_10_2.patch ├── lua_module_v0_10_6.patch ├── nginx_error_log_limiting_v1_15.4.patch └── src ├── ngx_http_graphite_allocator.c ├── ngx_http_graphite_allocator.h ├── ngx_http_graphite_array.c ├── ngx_http_graphite_array.h ├── ngx_http_graphite_bsearch.h ├── ngx_http_graphite_module.c ├── ngx_http_graphite_module.h ├── ngx_http_graphite_net.c └── ngx_http_graphite_net.h /README.md: -------------------------------------------------------------------------------- 1 | graphite-nginx-module 2 | ===================== 3 | 4 | An nginx module for collecting location stats into Graphite. 5 | 6 | *This module is not distributed with the Nginx source.* See [the installation instructions](#installation). 7 | 8 | Features 9 | ======== 10 | 11 | * Aggregation of location, server or http metrics 12 | * Calculation of percentiles 13 | * Sending data to Graphite over UDP or TCP in non-blocking way 14 | * Sending custom metrics from lua 15 | 16 | Version 17 | ======= 18 | 19 | This document describes graphite-nginx-module [v2.3.0](https://github.com/mailru/graphite-nginx-module/tags) released on 21 August 2018. 20 | 21 | Synopsis 22 | ======== 23 | 24 | ```nginx 25 | 26 | http { 27 | graphite_config prefix=playground server=127.0.0.1; 28 | server { 29 | location /foo/ { 30 | graphite_data nginx.foo; 31 | } 32 | } 33 | } 34 | ``` 35 | 36 | Description 37 | =========== 38 | 39 | This module use shared memory segment to collect aggregated stats from all workers and send calculated values for last minute to Graphite every 60s (default) over UDP or TCP in non-blocking way. 40 | Stats aggegation made on the fly in fixed size buffer allocated on server start and does't affect server performance. 41 | 42 | This module is in active use on [Mail.Ru Sites](http://mail.ru/) (one of largest web-services in Russia) for about a year and considered stable and well-tested. 43 | 44 | To collect metrics from nginx core modules (ssl, gzip, upstream) little patch must be applied on nginx source tree. See [the installation instructions](#installation). 45 | You can build this module as a dynamic one, but then you won't be able to collect metrics from nginx core modules (ssl, gzip, upstream) and lua functions. 46 | 47 | 48 | Directives 49 | ========== 50 | 51 | ### graphite_config 52 | 53 | **syntax:** *graphite_config key1=<value1> key2=<value2> ... keyN=<valueN>* 54 | 55 | **context:** *http* 56 | 57 | Specify global settings for a whole server instance. 58 | 59 | Param | Required | Default | Description 60 | --------- | -------- | ------------- | ----------- 61 | prefix | | | path prefix for all graphs 62 | host | | gethostname() | host name for all graphs 63 | server | Yes | | carbon-cache server IP address 64 | protocol | | udp | carbon-cache server protocol (udp or tcp) 65 | port | | 2003 | carbon-cache server port 66 | frequency | | 60 | how often send values to Graphite (seconds) 67 | intervals | | 1m | aggregation intervals, time interval list, vertical bar separator (`m` - minutes) 68 | params | | * | limit metrics list to track, vertical bar separator 69 | shared | | 2m | shared memory size, increase in case of `too small shared memory` error 70 | buffer | | 64k | network buffer size, increase in case of `too small buffer size` error 71 | package | | 1400 | maximum UDP packet size 72 | template | | | template for graph name (default is $prefix.$host.$split.$param_$interval) 73 | error\_log| | | path suffix for error logs graphs (\*) 74 | 75 | (\*): works only when nginx_error\_log\_limiting\*.patch is applied to the nginx source code 76 | 77 | Example (standard): 78 | 79 | ```nginx 80 | http { 81 | graphite_config prefix=playground server=127.0.0.1; 82 | } 83 | ``` 84 | 85 | Example (custom): 86 | 87 | ```nginx 88 | http { 89 | graphite_config prefix=playground server=127.0.0.1 intervals=1m|5m|15m params=rps|request_time|upstream_time template=$prefix.$host.$split.$param_$interval; 90 | } 91 | ``` 92 | 93 | Example (error_log): 94 | 95 | ```nginx 96 | http { 97 | graphite_config prefix=playground server=127.0.0.1 error_log=log; 98 | } 99 | ``` 100 | 101 | ### graphite_default_data 102 | 103 | **syntax:** *graphite_default_data <path prefix> [params=<params>] [if=<condition>]* 104 | 105 | **context:** *http, server* 106 | 107 | Create measurement point in all nested locations. 108 | You can use "$location" or "$server" variables which represent the name of the current location and the name of current server with all non-alphanumeric characters replaced with "\_." Leading and trailing "\_" are deleted. 109 | 110 | Example: 111 | 112 | ```nginx 113 | 114 | graphite_default_data nginx.$location; 115 | 116 | location /foo/ { 117 | } 118 | 119 | location /bar/ { 120 | } 121 | ``` 122 | 123 | Data for `/foo/` will be sent to `nginx.foo`, data for `/bar/` - to `nginx.bar`. 124 | The `` parameter (1.3.0) specifies list of params to be collected for all nested locations. To add all default params, use \*. 125 | The `` parameter (1.1.0) enables conditional logging. A request will not be logged if the condition evaluates to "0" or an empty string. 126 | 127 | Example(with $server): 128 | ```nginx 129 | 130 | graphite_default_data nginx.$server.$location 131 | 132 | server { 133 | server_name foo_host; 134 | 135 | location /foo/ { 136 | } 137 | } 138 | 139 | server { 140 | server_name bar_host; 141 | 142 | location /bar/ { 143 | } 144 | } 145 | ``` 146 | 147 | Data for `/foo/` will be sent to `nginx.foo_host.foo`, data for `/bar/` - to `nginx.bar_host.bar`. 148 | 149 | ### graphite_data 150 | 151 | **syntax:** *graphite_data <path prefix> [params=<params>] [if=<condition>]* 152 | 153 | **context:** *http, server, location, if* 154 | 155 | Create measurement point in specific location. 156 | 157 | Example: 158 | 159 | ```nginx 160 | 161 | location /foo/ { 162 | graphite_data nginx.foo; 163 | } 164 | ``` 165 | 166 | The `` parameter (1.3.0) specifies list of params to be collected for this location. To add all default params, use \*. 167 | The `` parameter (1.1.0) enables conditional logging. A request will not be logged if the condition evaluates to "0" or an empty string. 168 | 169 | Example: 170 | 171 | ```nginx 172 | 173 | map $scheme $is_http { http 1; } 174 | map $scheme $is_https { https 1; } 175 | 176 | ... 177 | 178 | location /bar/ { 179 | graphite_data nginx.all.bar; 180 | graphite_data nginx.http.bar if=$is_http; 181 | graphite_data nginx.https.bar if=$is_https; 182 | graphite_data nginx.arg params=rps|request_time; 183 | graphite_data nginx.ext params=*|rps|request_time; 184 | } 185 | ``` 186 | 187 | ### graphite_param 188 | 189 | **syntax:** *graphite_param name=<path> interval=<time value> aggregate=<func>* 190 | 191 | **context:** *http, server, location, if* 192 | 193 | Param | Required | Description 194 | ---------- | -------- | ----------- 195 | name | Yes | path prefix for all graphs 196 | interval | Yes\* | aggregation interval, time intrval value format (`m` - minutes) 197 | aggregate | Yes\* | aggregation function on values 198 | percentile | Yes\* | percentile level 199 | 200 | #### aggregate functions 201 | func | Description 202 | ------ | ----------- 203 | sum | sum of values per interval 204 | persec | sum of values per second (`sum` divided on seconds in `interval`) 205 | avg | average value on interval 206 | gauge | gauge value 207 | 208 | Example: see below. 209 | 210 | Nginx API for Lua 211 | ================= 212 | 213 | **syntax:** *ngx.graphite.param(<name>)* 214 | 215 | Get a link on a graphite parameter name, to use it in place of the name for the functions below. 216 | The link is valid up to nginx reload. After getting the link of a parameter, you can still pass 217 | the parameter name to the functions below. You can get the link of a parameter multiple times, 218 | you'll always get the same object by the same name (a lightuserdata). The function returns false 219 | if the parameter specified by name doesn't exist. The function returns nil on link getting errors. 220 | Functions access parameters information by link faster than by name. 221 | 222 | *Available after applying patch to lua-nginx-module.* The feature is present in the patch for lua 223 | module v0.10.12. See [the installation instructions](#build-nginx-with-lua-and-graphite-modules). 224 | 225 | **syntax:** *ngx.graphite(<name_or_link>,<value>[,<config>])* 226 | 227 | Write stat value into aggregator function. Floating point numbers accepted in `value`. 228 | 229 | *Available after applying patch to lua-nginx-module.* See [the installation instructions](#build-nginx-with-lua-and-graphite-modules). 230 | 231 | ```lua 232 | ngx.graphite(name, value, config) 233 | ``` 234 | 235 | Example: 236 | 237 | ```nginx 238 | 239 | location /foo/ { 240 | graphite_param name=lua.foo_sum aggregate=sum interval=1m; 241 | graphite_param name=lua.foo_rps aggregate=persec interval=1m; 242 | graphite_param name=lua.foo_avg aggregate=avg interval=1m; 243 | graphite_param name=lua.foo_gauge aggregate=gauge; 244 | 245 | content_by_lua ' 246 | ngx.graphite("lua.foo_sum", 0.01) 247 | ngx.graphite("lua.foo_rps", 1) 248 | ngx.graphite("lua.foo_avg", ngx.var.request_uri:len()) 249 | local foo_gauge_link = ngx.graphite.param("lua.foo_gauge") 250 | ngx.graphite(foo_gauge_link, 10) 251 | ngx.graphite(foo_gauge_link, -2) 252 | ngx.graphite("lua.auto_rps", 1, "aggregate=persec interval=1m percentile=50|90|99") 253 | ngx.say("hello") 254 | '; 255 | } 256 | ``` 257 | 258 | You must either specify the `graphite_param` command or pass the `config` argument. 259 | If you choose the second option, the data for this graph will not be sent until the first call to ngx.graphite. 260 | 261 | **Warning:** 262 | If you do not declare graph using `graphite_param` command then memory for the graph will be allocated dynamically in module's shared memory. 263 | If module's shared memory is exhausted while nginx is running, no new graphs will be created and an error message will be logged. 264 | 265 | **syntax:** *ngx.graphite.get(<name_or_link>)* 266 | 267 | Get value of the gauge param with specified `name_or_link`. 268 | 269 | **syntax:** *ngx.graphite.set(<name>,<value>)* 270 | 271 | Set `value` to the gauge param with specified `name_or_link`. 272 | 273 | Params 274 | ====== 275 | 276 | Param | Units | Func | Description 277 | ----------------------- | ----- | ---- | ------------------------------------------ 278 | request\_time | ms | avg | total time spent on serving request 279 | bytes\_sent | bytes | avg | http response length 280 | body\_bytes\_sent | bytes | avg | http response body length 281 | request\_length | bytes | avg | http request length 282 | ssl\_handshake\_time | ms | avg | time spent on ssl handsake 283 | ssl\_cache\_usage | % | last | how much SSL cache used 284 | content\_time | ms | avg | time spent generating content inside nginx 285 | gzip\_time | ms | avg | time spent gzipping content ob-the-fly 286 | lua\_time | ms | avg | time spent on lua code 287 | upstream\_time | ms | avg | time spent tailking with upstream 288 | upstream\_connect\_time | ms | avg | time spent on upstream connect (nginx >= 1.9.1) 289 | upstream\_header\_time | ms | avg | time spent on upstream header (nginx >= 1.9.1) 290 | upstream\_response\_2xx\_rps | rps | sum | total upstream responses number with 2xx code (nginx >= 1.9.1) 291 | upstream\_response\_3xx\_rps | rps | sum | total upstream responses number with 3xx code (nginx >= 1.9.1) 292 | upstream\_response\_4xx\_rps | rps | sum | total upstream responses number with 4xx code (nginx >= 1.9.1) 293 | upstream\_response\_5xx\_rps | rps | sum | total upstream responses number with 5xx code (nginx >= 1.9.1) 294 | upstream\_response\_[0-9]{3}\_rps | rps | sum | total upstream responses number with given code (nginx >= 1.9.1) 295 | rps | rps | sum | total requests number per second 296 | keepalive\_rps | rps | sum | requests number sent over previously opened keepalive connection 297 | response\_2xx\_rps | rps | sum | total responses number with 2xx code 298 | response\_3xx\_rps | rps | sum | total responses number with 3xx code 299 | response\_4xx\_rps | rps | sum | total responses number with 4xx code 300 | response\_5xx\_rps | rps | sum | total responses number with 5xx code 301 | response\_[0-9]{3}\_rps | rps | sum | total responses number with given code 302 | upstream\_cache\_(miss\|bypass\|expired\|stale\|updating\|revalidated\|hit)\_rps | rps | sum | totar responses with a given upstream cache status 303 | 304 | Percentiles 305 | =========== 306 | 307 | To calculate percentile value for any parameter, set percentile level via `/`. E.g. `request_time/50|request_time/90|request_time/99`. 308 | 309 | Installation 310 | ============ 311 | 312 | #### Requirements 313 | * nginx: 1.2.0 - 1.14.x 314 | * lua-nginx-module: 0.8.6 - 0.10.13 (optional) 315 | 316 | #### Build nginx with graphite module 317 | ```bash 318 | 319 | wget 'http://nginx.org/download/nginx-1.9.2.tar.gz' 320 | tar -xzf nginx-1.9.2.tar.gz 321 | cd nginx-1.9.2/ 322 | 323 | # patch to collect ssl_cache_usage, ssl_handshake_time content_time, gzip_time, upstream_time, upstream_connect_time, upstream_header_time graphs (optional) 324 | patch -p1 < /path/to/graphite-nginx-module/graphite_module_v1_7_7.patch 325 | 326 | ./configure --add-module=/path/to/graphite-nginx-module 327 | 328 | make 329 | make install 330 | ``` 331 | 332 | #### Build nginx with graphite dynamic module 333 | ```bash 334 | 335 | wget 'http://nginx.org/download/nginx-1.9.2.tar.gz' 336 | tar -xzf nginx-1.9.2.tar.gz 337 | cd nginx-1.9.2/ 338 | 339 | ./configure --add-dynamic-module=/path/to/graphite-nginx-module 340 | 341 | make 342 | make install 343 | ``` 344 | 345 | #### Build nginx with lua and graphite modules 346 | ```bash 347 | 348 | wget 'https://github.com/chaoslawful/lua-nginx-module/archive/v0.9.16.tar.gz' 349 | tar -xzf v0.9.16.tar.gz 350 | cd lua-nginx-module-0.9.16/ 351 | # patch to add api for sending metrics from lua code (optional) 352 | patch -p1 < /path/to/graphite-nginx-module/lua_module_v0_9_11.patch 353 | cd .. 354 | 355 | wget 'http://nginx.org/download/nginx-1.9.2.tar.gz' 356 | tar -xzf nginx-1.9.2.tar.gz 357 | cd nginx-1.9.2/ 358 | 359 | # patch to collect ssl_cache_usage, ssl_handshake_time content_time, gzip_time, upstream_time, upstream_connect_time, upstream_header_time graphs (optional) 360 | patch -p1 < /path/to/graphite-nginx-module/graphite_module_v1_7_7.patch 361 | 362 | ./configure \ 363 | --add-module=/path/to/ngx_devel_kit \ 364 | --add-module=/path/to/lua-nginx-module \ 365 | --add-module=/path/to/graphite-nginx-module 366 | 367 | make 368 | make install 369 | ``` 370 | 371 | Instructions on installing lua-nginx-module can be found in [documentation on lua-nginx-module](https://github.com/chaoslawful/lua-nginx-module#installation). 372 | 373 | License 374 | ======= 375 | 376 | Copyright (c) 2013-2018, Mail.Ru Ltd. 377 | 378 | This module is licensed under the terms of the BSD license. 379 | 380 | Redistribution and use in source and binary forms, with or without 381 | modification, are permitted provided that the following conditions 382 | are met: 383 | 384 | * Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. 385 | * Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. 386 | 387 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 388 | "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 389 | LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR 390 | A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT 391 | HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, 392 | SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED 393 | TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR 394 | PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF 395 | LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 396 | NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 397 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 398 | -------------------------------------------------------------------------------- /config: -------------------------------------------------------------------------------- 1 | ngx_feature="Graphite module" 2 | have=NGX_GRAPHITE . auto/have 3 | 4 | ngx_addon_name=ngx_http_graphite_module 5 | 6 | if test -n "$ngx_module_link"; then 7 | ngx_module_type=HTTP 8 | ngx_module_name=ngx_http_graphite_module 9 | ngx_module_incs="$ngx_addon_dir/src" 10 | ngx_module_srcs="\ 11 | $ngx_addon_dir/src/ngx_http_graphite_allocator.c\ 12 | $ngx_addon_dir/src/ngx_http_graphite_array.c\ 13 | $ngx_addon_dir/src/ngx_http_graphite_module.c\ 14 | $ngx_addon_dir/src/ngx_http_graphite_net.c\ 15 | " 16 | . auto/module 17 | else 18 | HTTP_MODULES="$HTTP_MODULES ngx_http_graphite_module" 19 | HTTP_INCS="$HTTP_INCS $ngx_addon_dir/src" 20 | NGX_ADDON_SRCS="$NGX_ADDON_SRCS \ 21 | $ngx_addon_dir/src/ngx_http_graphite_allocator.c \ 22 | $ngx_addon_dir/src/ngx_http_graphite_array.c \ 23 | $ngx_addon_dir/src/ngx_http_graphite_module.c \ 24 | $ngx_addon_dir/src/ngx_http_graphite_net.c \ 25 | " 26 | fi 27 | -------------------------------------------------------------------------------- /graphite_module_v1_15_4.patch: -------------------------------------------------------------------------------- 1 | diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c 2 | index 3a0e150..0eb06aa 100644 3 | --- a/src/event/ngx_event_openssl.c 4 | +++ b/src/event/ngx_event_openssl.c 5 | @@ -1363,6 +1363,16 @@ ngx_ssl_handshake(ngx_connection_t *c) 6 | 7 | ngx_ssl_clear_error(c->log); 8 | 9 | +#if (NGX_GRAPHITE) 10 | + struct timeval tp; 11 | + if (c->ssl->handshake_process == 0) { 12 | + c->ssl->handshake_process = 1; 13 | + ngx_gettimeofday(&tp); 14 | + c->ssl->handshake_start_sec = tp.tv_sec; 15 | + c->ssl->handshake_start_msec = tp.tv_usec / 1000; 16 | + } 17 | +#endif 18 | + 19 | n = SSL_do_handshake(c->ssl->connection); 20 | 21 | ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_do_handshake: %d", n); 22 | @@ -1401,6 +1411,13 @@ ngx_ssl_handshake(ngx_connection_t *c) 23 | #endif 24 | #endif 25 | 26 | +#if (NGX_GRAPHITE) 27 | + c->ssl->handshake_process = 0; 28 | + ngx_gettimeofday(&tp); 29 | + c->ssl->handshake_end_sec = tp.tv_sec; 30 | + c->ssl->handshake_end_msec = tp.tv_usec / 1000; 31 | +#endif 32 | + 33 | return NGX_OK; 34 | } 35 | 36 | diff --git a/src/event/ngx_event_openssl.h b/src/event/ngx_event_openssl.h 37 | index abd84cc..6735aa9 100644 38 | --- a/src/event/ngx_event_openssl.h 39 | +++ b/src/event/ngx_event_openssl.h 40 | @@ -89,6 +89,14 @@ struct ngx_ssl_connection_s { 41 | 42 | u_char early_buf; 43 | 44 | +#if (NGX_GRAPHITE) 45 | + ngx_uint_t handshake_process; 46 | + time_t handshake_start_sec; 47 | + ngx_msec_t handshake_start_msec; 48 | + time_t handshake_end_sec; 49 | + ngx_msec_t handshake_end_msec; 50 | +#endif 51 | + 52 | unsigned handshaked:1; 53 | unsigned renegotiation:1; 54 | unsigned buffer:1; 55 | diff --git a/src/http/modules/ngx_http_gzip_filter_module.c b/src/http/modules/ngx_http_gzip_filter_module.c 56 | index e4c343c..1f1bd43 100644 57 | --- a/src/http/modules/ngx_http_gzip_filter_module.c 58 | +++ b/src/http/modules/ngx_http_gzip_filter_module.c 59 | @@ -427,7 +427,16 @@ ngx_http_gzip_body_filter(ngx_http_request_t *r, ngx_chain_t *in) 60 | } 61 | 62 | 63 | +#if (NGX_GRAPHITE) 64 | + struct timeval start_tp; 65 | + ngx_gettimeofday(&start_tp); 66 | +#endif 67 | rc = ngx_http_gzip_filter_deflate(r, ctx); 68 | +#if (NGX_GRAPHITE) 69 | + struct timeval stop_tp; 70 | + ngx_gettimeofday(&stop_tp); 71 | + r->gzip_time += (stop_tp.tv_sec - start_tp.tv_sec) * 1000 + (stop_tp.tv_usec - start_tp.tv_usec) / 1000.0; 72 | +#endif 73 | 74 | if (rc == NGX_OK) { 75 | break; 76 | diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c 77 | index c57ec00..8e97fa3 100644 78 | --- a/src/http/ngx_http_core_module.c 79 | +++ b/src/http/ngx_http_core_module.c 80 | @@ -1166,7 +1166,17 @@ ngx_http_core_content_phase(ngx_http_request_t *r, 81 | 82 | if (r->content_handler) { 83 | r->write_event_handler = ngx_http_request_empty_handler; 84 | - ngx_http_finalize_request(r, r->content_handler(r)); 85 | +#if (NGX_GRAPHITE) 86 | + struct timeval start_tp; 87 | + ngx_gettimeofday(&start_tp); 88 | +#endif 89 | + ngx_int_t rc = r->content_handler(r); 90 | +#if (NGX_GRAPHITE) 91 | + struct timeval stop_tp; 92 | + ngx_gettimeofday(&stop_tp); 93 | + r->content_time += (stop_tp.tv_sec - start_tp.tv_sec) * 1000 + (stop_tp.tv_usec - start_tp.tv_usec) / 1000.0; 94 | +#endif 95 | + ngx_http_finalize_request(r, rc); 96 | return NGX_OK; 97 | } 98 | 99 | diff --git a/src/http/ngx_http_request.h b/src/http/ngx_http_request.h 100 | index 6bfff96..c67378d 100644 101 | --- a/src/http/ngx_http_request.h 102 | +++ b/src/http/ngx_http_request.h 103 | @@ -586,6 +586,20 @@ struct ngx_http_request_s { 104 | 105 | unsigned http_minor:16; 106 | unsigned http_major:16; 107 | + 108 | +#define NGX_GRAPHITE_PATCH 109 | + 110 | +#if (NGX_GRAPHITE) 111 | + double content_time; 112 | +#endif 113 | + 114 | +#if ((NGX_HTTP_GZIP) && (NGX_GRAPHITE)) 115 | + double gzip_time; 116 | +#endif 117 | + 118 | +#if (NGX_GRAPHITE) 119 | + double lua_time; 120 | +#endif 121 | }; 122 | 123 | 124 | -------------------------------------------------------------------------------- /graphite_module_v1_2_5.patch: -------------------------------------------------------------------------------- 1 | diff -u3 -p -r a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c 2 | --- a/src/event/ngx_event_openssl.c 2012-11-12 22:00:32.000000000 +0400 3 | +++ b/src/event/ngx_event_openssl.c 2012-12-17 16:30:47.639146541 +0400 4 | @@ -604,6 +604,16 @@ ngx_ssl_handshake(ngx_connection_t *c) 5 | 6 | ngx_ssl_clear_error(c->log); 7 | 8 | +#if (NGX_GRAPHITE) 9 | + struct timeval tp; 10 | + if (c->ssl->handshake_process == 0) { 11 | + c->ssl->handshake_process = 1; 12 | + ngx_gettimeofday(&tp); 13 | + c->ssl->handshake_start_sec = tp.tv_sec; 14 | + c->ssl->handshake_start_msec = tp.tv_usec / 1000; 15 | + } 16 | +#endif 17 | + 18 | n = SSL_do_handshake(c->ssl->connection); 19 | 20 | ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_do_handshake: %d", n); 21 | @@ -667,6 +677,12 @@ ngx_ssl_handshake(ngx_connection_t *c) 22 | 23 | c->ssl->handshaked = 1; 24 | 25 | +#if (NGX_GRAPHITE) 26 | + c->ssl->handshake_process = 0; 27 | + ngx_gettimeofday(&tp); 28 | + c->ssl->handshake_end_sec = tp.tv_sec; 29 | + c->ssl->handshake_end_msec = tp.tv_usec / 1000; 30 | +#endif 31 | c->recv = ngx_ssl_recv; 32 | c->send = ngx_ssl_write; 33 | c->recv_chain = ngx_ssl_recv_chain; 34 | diff -u3 -p -r a/src/event/ngx_event_openssl.h b/src/event/ngx_event_openssl.h 35 | --- a/src/event/ngx_event_openssl.h 2012-11-13 14:42:16.000000000 +0400 36 | +++ b/src/event/ngx_event_openssl.h 2012-12-17 16:30:47.639146541 +0400 37 | @@ -42,6 +42,14 @@ typedef struct { 38 | ngx_event_handler_pt saved_read_handler; 39 | ngx_event_handler_pt saved_write_handler; 40 | 41 | +#if (NGX_GRAPHITE) 42 | + ngx_uint_t handshake_process; 43 | + time_t handshake_start_sec; 44 | + ngx_msec_t handshake_start_msec; 45 | + time_t handshake_end_sec; 46 | + ngx_msec_t handshake_end_msec; 47 | +#endif 48 | + 49 | unsigned handshaked:1; 50 | unsigned renegotiation:1; 51 | unsigned buffer:1; 52 | diff -u3 -p -r a/src/http/modules/ngx_http_gzip_filter_module.c b/src/http/modules/ngx_http_gzip_filter_module.c 53 | --- a/src/http/modules/ngx_http_gzip_filter_module.c 2012-02-13 19:23:43.000000000 +0400 54 | +++ b/src/http/modules/ngx_http_gzip_filter_module.c 2012-12-17 16:30:52.299146347 +0400 55 | @@ -363,6 +363,16 @@ ngx_http_gzip_body_filter(ngx_http_reque 56 | } 57 | } 58 | 59 | +#if (NGX_GRAPHITE) 60 | + struct timeval tp; 61 | + if (r->gzip_process == 0) { 62 | + r->gzip_process = 1; 63 | + ngx_gettimeofday(&tp); 64 | + r->gzip_start_sec = tp.tv_sec; 65 | + r->gzip_start_msec = tp.tv_usec / 1000; 66 | + } 67 | +#endif 68 | + 69 | if (in) { 70 | if (ngx_chain_add_copy(r->pool, &ctx->in, in) != NGX_OK) { 71 | goto failed; 72 | @@ -432,6 +442,11 @@ ngx_http_gzip_body_filter(ngx_http_reque 73 | if (ctx->out == NULL) { 74 | ngx_http_gzip_filter_free_copy_buf(r, ctx); 75 | 76 | +#if (NGX_GRAPHITE) 77 | + ngx_gettimeofday(&tp); 78 | + r->gzip_end_sec = tp.tv_sec; 79 | + r->gzip_end_msec = tp.tv_usec / 1000; 80 | +#endif 81 | return ctx->busy ? NGX_AGAIN : NGX_OK; 82 | } 83 | 84 | @@ -456,6 +471,12 @@ ngx_http_gzip_body_filter(ngx_http_reque 85 | ctx->nomem = 0; 86 | 87 | if (ctx->done) { 88 | + 89 | +#if (NGX_GRAPHITE) 90 | + ngx_gettimeofday(&tp); 91 | + r->gzip_end_sec = tp.tv_sec; 92 | + r->gzip_end_msec = tp.tv_usec / 1000; 93 | +#endif 94 | return rc; 95 | } 96 | } 97 | @@ -474,6 +495,12 @@ failed: 98 | 99 | ngx_http_gzip_filter_free_copy_buf(r, ctx); 100 | 101 | +#if (NGX_GRAPHITE) 102 | + ngx_gettimeofday(&tp); 103 | + r->gzip_end_sec = tp.tv_sec; 104 | + r->gzip_end_msec = tp.tv_usec / 1000; 105 | +#endif 106 | + 107 | return NGX_ERROR; 108 | } 109 | 110 | diff -u3 -p -r a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c 111 | --- a/src/http/ngx_http_core_module.c 2012-09-24 23:06:48.000000000 +0400 112 | +++ b/src/http/ngx_http_core_module.c 2012-12-17 17:29:35.654998991 +0400 113 | @@ -1393,7 +1393,19 @@ ngx_http_core_content_phase(ngx_http_req 114 | 115 | if (r->content_handler) { 116 | r->write_event_handler = ngx_http_request_empty_handler; 117 | - ngx_http_finalize_request(r, r->content_handler(r)); 118 | +#if (NGX_GRAPHITE) 119 | + struct timeval tp; 120 | + ngx_gettimeofday(&tp); 121 | + r->content_start_sec = tp.tv_sec; 122 | + r->content_start_msec = tp.tv_usec / 1000; 123 | +#endif 124 | + ngx_int_t rc = r->content_handler(r); 125 | +#if (NGX_GRAPHITE) 126 | + ngx_gettimeofday(&tp); 127 | + r->content_end_sec = tp.tv_sec; 128 | + r->content_end_msec = tp.tv_usec / 1000; 129 | +#endif 130 | + ngx_http_finalize_request(r, rc); 131 | return NGX_OK; 132 | } 133 | 134 | diff -u3 -p -r a/src/http/ngx_http_request.h b/src/http/ngx_http_request.h 135 | --- a/src/http/ngx_http_request.h 2012-07-02 21:41:52.000000000 +0400 136 | +++ b/src/http/ngx_http_request.h 2012-12-17 17:05:39.755059043 +0400 137 | @@ -562,6 +562,21 @@ struct ngx_http_request_s { 138 | 139 | unsigned http_minor:16; 140 | unsigned http_major:16; 141 | + 142 | +#if (NGX_GRAPHITE) 143 | + time_t content_start_sec; 144 | + ngx_msec_t content_start_msec; 145 | + time_t content_end_sec; 146 | + ngx_msec_t content_end_msec; 147 | +#endif 148 | + 149 | +#if ((NGX_HTTP_GZIP) && (NGX_GRAPHITE)) 150 | + ngx_uint_t gzip_process; 151 | + time_t gzip_start_sec; 152 | + ngx_msec_t gzip_start_msec; 153 | + time_t gzip_end_sec; 154 | + ngx_msec_t gzip_end_msec; 155 | +#endif 156 | }; 157 | 158 | 159 | -------------------------------------------------------------------------------- /graphite_module_v1_7_7.patch: -------------------------------------------------------------------------------- 1 | diff --git a/src/event/ngx_event_openssl.c b/src/event/ngx_event_openssl.c 2 | index 2c4e114..98ae6ba 100644 3 | --- a/src/event/ngx_event_openssl.c 4 | +++ b/src/event/ngx_event_openssl.c 5 | @@ -1228,6 +1228,16 @@ ngx_ssl_handshake(ngx_connection_t *c) 6 | 7 | ngx_ssl_clear_error(c->log); 8 | 9 | +#if (NGX_GRAPHITE) 10 | + struct timeval tp; 11 | + if (c->ssl->handshake_process == 0) { 12 | + c->ssl->handshake_process = 1; 13 | + ngx_gettimeofday(&tp); 14 | + c->ssl->handshake_start_sec = tp.tv_sec; 15 | + c->ssl->handshake_start_msec = tp.tv_usec / 1000; 16 | + } 17 | +#endif 18 | + 19 | n = SSL_do_handshake(c->ssl->connection); 20 | 21 | ngx_log_debug1(NGX_LOG_DEBUG_EVENT, c->log, 0, "SSL_do_handshake: %d", n); 22 | @@ -1307,6 +1317,13 @@ ngx_ssl_handshake(ngx_connection_t *c) 23 | #endif 24 | #endif 25 | 26 | +#if (NGX_GRAPHITE) 27 | + c->ssl->handshake_process = 0; 28 | + ngx_gettimeofday(&tp); 29 | + c->ssl->handshake_end_sec = tp.tv_sec; 30 | + c->ssl->handshake_end_msec = tp.tv_usec / 1000; 31 | +#endif 32 | + 33 | return NGX_OK; 34 | } 35 | 36 | diff --git a/src/event/ngx_event_openssl.h b/src/event/ngx_event_openssl.h 37 | index 2a14980..daac038 100644 38 | --- a/src/event/ngx_event_openssl.h 39 | +++ b/src/event/ngx_event_openssl.h 40 | @@ -79,6 +79,14 @@ struct ngx_ssl_connection_s { 41 | ngx_event_handler_pt saved_read_handler; 42 | ngx_event_handler_pt saved_write_handler; 43 | 44 | +#if (NGX_GRAPHITE) 45 | + ngx_uint_t handshake_process; 46 | + time_t handshake_start_sec; 47 | + ngx_msec_t handshake_start_msec; 48 | + time_t handshake_end_sec; 49 | + ngx_msec_t handshake_end_msec; 50 | +#endif 51 | + 52 | unsigned handshaked:1; 53 | unsigned renegotiation:1; 54 | unsigned buffer:1; 55 | diff --git a/src/http/modules/ngx_http_gzip_filter_module.c b/src/http/modules/ngx_http_gzip_filter_module.c 56 | index 287fd36..656dc75 100644 57 | --- a/src/http/modules/ngx_http_gzip_filter_module.c 58 | +++ b/src/http/modules/ngx_http_gzip_filter_module.c 59 | @@ -424,7 +424,16 @@ ngx_http_gzip_body_filter(ngx_http_request_t *r, ngx_chain_t *in) 60 | } 61 | 62 | 63 | +#if (NGX_GRAPHITE) 64 | + struct timeval start_tp; 65 | + ngx_gettimeofday(&start_tp); 66 | +#endif 67 | rc = ngx_http_gzip_filter_deflate(r, ctx); 68 | +#if (NGX_GRAPHITE) 69 | + struct timeval stop_tp; 70 | + ngx_gettimeofday(&stop_tp); 71 | + r->gzip_time += (stop_tp.tv_sec - start_tp.tv_sec) * 1000 + (stop_tp.tv_usec - start_tp.tv_usec) / 1000.0; 72 | +#endif 73 | 74 | if (rc == NGX_OK) { 75 | break; 76 | diff --git a/src/http/ngx_http_core_module.c b/src/http/ngx_http_core_module.c 77 | index 7e40e78..3d7336d 100644 78 | --- a/src/http/ngx_http_core_module.c 79 | +++ b/src/http/ngx_http_core_module.c 80 | @@ -1384,7 +1395,17 @@ ngx_http_core_content_phase(ngx_http_request_t *r, 81 | 82 | if (r->content_handler) { 83 | r->write_event_handler = ngx_http_request_empty_handler; 84 | - ngx_http_finalize_request(r, r->content_handler(r)); 85 | +#if (NGX_GRAPHITE) 86 | + struct timeval start_tp; 87 | + ngx_gettimeofday(&start_tp); 88 | +#endif 89 | + ngx_int_t rc = r->content_handler(r); 90 | +#if (NGX_GRAPHITE) 91 | + struct timeval stop_tp; 92 | + ngx_gettimeofday(&stop_tp); 93 | + r->content_time += (stop_tp.tv_sec - start_tp.tv_sec) * 1000 + (stop_tp.tv_usec - start_tp.tv_usec) / 1000.0; 94 | +#endif 95 | + ngx_http_finalize_request(r, rc); 96 | return NGX_OK; 97 | } 98 | 99 | diff --git a/src/http/ngx_http_request.h b/src/http/ngx_http_request.h 100 | index 283c582..68690ec 100644 101 | --- a/src/http/ngx_http_request.h 102 | +++ b/src/http/ngx_http_request.h 103 | @@ -580,6 +580,20 @@ struct ngx_http_request_s { 104 | 105 | unsigned http_minor:16; 106 | unsigned http_major:16; 107 | + 108 | +#define NGX_GRAPHITE_PATCH 109 | + 110 | +#if (NGX_GRAPHITE) 111 | + double content_time; 112 | +#endif 113 | + 114 | +#if ((NGX_HTTP_GZIP) && (NGX_GRAPHITE)) 115 | + double gzip_time; 116 | +#endif 117 | + 118 | +#if (NGX_GRAPHITE) 119 | + double lua_time; 120 | +#endif 121 | }; 122 | 123 | 124 | -------------------------------------------------------------------------------- /lua_module_v0_10_10.patch: -------------------------------------------------------------------------------- 1 | diff --git a/config b/config 2 | index 044deb97..46d8f23b 100644 3 | --- a/config 4 | +++ b/config 5 | @@ -360,6 +360,7 @@ HTTP_LUA_SRCS=" \ 6 | $ngx_addon_dir/src/ngx_http_lua_ssl_session_storeby.c \ 7 | $ngx_addon_dir/src/ngx_http_lua_ssl_session_fetchby.c \ 8 | $ngx_addon_dir/src/ngx_http_lua_ssl.c \ 9 | + $ngx_addon_dir/src/ngx_http_lua_graphite.c \ 10 | $ngx_addon_dir/src/ngx_http_lua_log_ringbuf.c \ 11 | " 12 | 13 | @@ -421,6 +422,7 @@ HTTP_LUA_DEPS=" \ 14 | $ngx_addon_dir/src/ngx_http_lua_ssl_session_storeby.h \ 15 | $ngx_addon_dir/src/ngx_http_lua_ssl_session_fetchby.h \ 16 | $ngx_addon_dir/src/ngx_http_lua_ssl.h \ 17 | + $ngx_addon_dir/src/ngx_http_lua_graphite.h \ 18 | $ngx_addon_dir/src/ngx_http_lua_log_ringbuf.h \ 19 | " 20 | 21 | diff --git a/src/ngx_http_lua_graphite.c b/src/ngx_http_lua_graphite.c 22 | new file mode 100644 23 | index 0000000..20812f9 24 | --- /dev/null 25 | +++ b/src/ngx_http_lua_graphite.c 26 | @@ -0,0 +1,117 @@ 27 | +#ifndef DDEBUG 28 | +#define DDEBUG 0 29 | +#endif 30 | +#include "ddebug.h" 31 | + 32 | + 33 | +#include "ngx_http_graphite_module.h" 34 | +#include "ngx_http_lua_graphite.h" 35 | +#include "ngx_http_lua_util.h" 36 | + 37 | + 38 | +static int ngx_http_lua_graphite(lua_State *L); 39 | +static int ngx_http_lua_graphite_get(lua_State *L); 40 | +static int ngx_http_lua_graphite_set(lua_State *L); 41 | + 42 | + 43 | +void 44 | +ngx_http_lua_inject_graphite_api(lua_State *L) 45 | +{ 46 | + lua_createtable(L, 0, 2); 47 | + lua_newtable(L); 48 | + lua_pushcfunction(L, ngx_http_lua_graphite); 49 | + lua_setfield(L, -2, "__call"); 50 | + lua_setmetatable(L, -2); 51 | + 52 | + lua_pushcfunction(L, ngx_http_lua_graphite_get); 53 | + lua_setfield(L, -2, "get"); 54 | + 55 | + lua_pushcfunction(L, ngx_http_lua_graphite_set); 56 | + lua_setfield(L, -2, "set"); 57 | + 58 | + lua_setfield(L, -2, "graphite"); 59 | +} 60 | + 61 | + 62 | +static int 63 | +ngx_http_lua_graphite(lua_State *L) { 64 | + 65 | + size_t n = lua_gettop(L) - 1; 66 | + if (n != 2 && n != 3) { 67 | + return luaL_error(L, "ngx.graphite expecting 2 or 3 arguments got %d", n); 68 | + } 69 | + 70 | + ngx_http_request_t *r; 71 | + r = ngx_http_lua_get_req(L); 72 | + 73 | + if (r == NULL) { 74 | + return luaL_error(L, "no request object found"); 75 | + } 76 | + 77 | + ngx_str_t name; 78 | + name.data = (u_char*)lua_tolstring(L, 2, &name.len); 79 | + if (name.data == NULL) 80 | + return 0; 81 | + 82 | + double value = lua_tonumber(L, 3); 83 | + 84 | + ngx_http_graphite(r, &name, value); 85 | + 86 | + return 0; 87 | +} 88 | + 89 | + 90 | +static int 91 | +ngx_http_lua_graphite_get(lua_State *L) { 92 | + 93 | + size_t n = lua_gettop(L); 94 | + if (n != 1) { 95 | + return luaL_error(L, "ngx.graphite.get expecting 1 argument got %d", n); 96 | + } 97 | + 98 | + ngx_http_request_t *r; 99 | + r = ngx_http_lua_get_req(L); 100 | + 101 | + if (r == NULL) { 102 | + return luaL_error(L, "no request object found"); 103 | + } 104 | + 105 | + ngx_str_t name; 106 | + name.data = (u_char*)lua_tolstring(L, 1, &name.len); 107 | + if (name.data == NULL) 108 | + return 0; 109 | + 110 | + double value = ngx_http_graphite_get(r, &name); 111 | + 112 | + lua_pushnumber(L, value); 113 | + 114 | + return 1; 115 | +} 116 | + 117 | + 118 | +static int 119 | +ngx_http_lua_graphite_set(lua_State *L) { 120 | + 121 | + size_t n = lua_gettop(L); 122 | + if (n != 2) { 123 | + return luaL_error(L, "ngx.graphite.get expecting 2 arguments got %d", n); 124 | + } 125 | + 126 | + ngx_http_request_t *r; 127 | + r = ngx_http_lua_get_req(L); 128 | + 129 | + if (r == NULL) { 130 | + return luaL_error(L, "no request object found"); 131 | + } 132 | + 133 | + ngx_str_t name; 134 | + name.data = (u_char*)lua_tolstring(L, 1, &name.len); 135 | + if (name.data == NULL) 136 | + return 0; 137 | + 138 | + double value = lua_tonumber(L, 2); 139 | + 140 | + ngx_http_graphite_set(r, &name, value); 141 | + 142 | + return 0; 143 | +} 144 | diff --git a/src/ngx_http_lua_util.c b/src/ngx_http_lua_util.c 145 | index c7bee3e8..9379ccee 100644 146 | --- a/src/ngx_http_lua_util.c 147 | +++ b/src/ngx_http_lua_util.c 148 | @@ -53,6 +53,7 @@ 149 | #include "ngx_http_lua_ssl.h" 150 | #include "ngx_http_lua_log_ringbuf.h" 151 | 152 | +#include "ngx_http_lua_graphite.h" 153 | 154 | #if 1 155 | #undef ngx_http_lua_probe_info 156 | @@ -710,7 +711,7 @@ static void 157 | ngx_http_lua_inject_ngx_api(lua_State *L, ngx_http_lua_main_conf_t *lmcf, 158 | ngx_log_t *log) 159 | { 160 | - lua_createtable(L, 0 /* narr */, 116 /* nrec */); /* ngx.* */ 161 | + lua_createtable(L, 0 /* narr */, 117 /* nrec */); /* ngx.* */ 162 | 163 | lua_pushcfunction(L, ngx_http_lua_get_raw_phase_context); 164 | lua_setfield(L, -2, "_phase_ctx"); 165 | @@ -744,6 +745,7 @@ ngx_http_lua_inject_ngx_api(lua_State *L, ngx_http_lua_main_conf_t *lmcf, 166 | ngx_http_lua_inject_timer_api(L); 167 | ngx_http_lua_inject_config_api(L); 168 | ngx_http_lua_inject_worker_api(L); 169 | + ngx_http_lua_inject_graphite_api(L); 170 | 171 | ngx_http_lua_inject_misc_api(L); 172 | 173 | @@ -1003,7 +1005,16 @@ ngx_http_lua_run_thread(lua_State *L, ngx_http_request_t *r, 174 | ngx_http_lua_assert(orig_coctx->co_top + nrets 175 | == lua_gettop(orig_coctx->co)); 176 | 177 | +#ifdef NGX_GRAPHITE_PATCH 178 | + struct timeval start_tp; 179 | + ngx_gettimeofday(&start_tp); 180 | +#endif 181 | rv = lua_resume(orig_coctx->co, nrets); 182 | +#ifdef NGX_GRAPHITE_PATCH 183 | + struct timeval stop_tp; 184 | + ngx_gettimeofday(&stop_tp); 185 | + r->lua_time += (stop_tp.tv_sec - start_tp.tv_sec) * 1000 + (stop_tp.tv_usec - start_tp.tv_usec) / 1000.0; 186 | +#endif 187 | 188 | #if (NGX_PCRE) 189 | /* XXX: work-around to nginx regex subsystem */ 190 | -------------------------------------------------------------------------------- /lua_module_v0_10_12.patch: -------------------------------------------------------------------------------- 1 | From 9a1e0a0ec78c8281e1ccd006554b1c3dca2742a8 Mon Sep 17 00:00:00 2001 2 | From: Mikhail Kirichenko 3 | Date: Mon, 19 Apr 2021 20:21:23 +0300 4 | Subject: [PATCH 1/3] initial patch 5 | 6 | --- 7 | config | 2 + 8 | src/ngx_http_lua_graphite.c | 117 ++++++++++++++++++++++++++++++++++++ 9 | src/ngx_http_lua_graphite.h | 11 ++++ 10 | src/ngx_http_lua_util.c | 13 +++- 11 | 4 files changed, 142 insertions(+), 1 deletion(-) 12 | create mode 100644 src/ngx_http_lua_graphite.c 13 | create mode 100644 src/ngx_http_lua_graphite.h 14 | 15 | diff --git a/config b/config 16 | index 044deb97..46d8f23b 100644 17 | --- a/config 18 | +++ b/config 19 | @@ -360,6 +360,7 @@ HTTP_LUA_SRCS=" \ 20 | $ngx_addon_dir/src/ngx_http_lua_ssl_session_storeby.c \ 21 | $ngx_addon_dir/src/ngx_http_lua_ssl_session_fetchby.c \ 22 | $ngx_addon_dir/src/ngx_http_lua_ssl.c \ 23 | + $ngx_addon_dir/src/ngx_http_lua_graphite.c \ 24 | $ngx_addon_dir/src/ngx_http_lua_log_ringbuf.c \ 25 | " 26 | 27 | @@ -421,6 +422,7 @@ HTTP_LUA_DEPS=" \ 28 | $ngx_addon_dir/src/ngx_http_lua_ssl_session_storeby.h \ 29 | $ngx_addon_dir/src/ngx_http_lua_ssl_session_fetchby.h \ 30 | $ngx_addon_dir/src/ngx_http_lua_ssl.h \ 31 | + $ngx_addon_dir/src/ngx_http_lua_graphite.h \ 32 | $ngx_addon_dir/src/ngx_http_lua_log_ringbuf.h \ 33 | " 34 | 35 | diff --git a/src/ngx_http_lua_graphite.c b/src/ngx_http_lua_graphite.c 36 | new file mode 100644 37 | index 00000000..9178553d 38 | --- /dev/null 39 | +++ b/src/ngx_http_lua_graphite.c 40 | @@ -0,0 +1,117 @@ 41 | +#ifndef DDEBUG 42 | +#define DDEBUG 0 43 | +#endif 44 | +#include "ddebug.h" 45 | + 46 | + 47 | +#include "ngx_http_graphite_module.h" 48 | +#include "ngx_http_lua_graphite.h" 49 | +#include "ngx_http_lua_util.h" 50 | + 51 | + 52 | +static int ngx_http_lua_graphite(lua_State *L); 53 | +static int ngx_http_lua_graphite_get(lua_State *L); 54 | +static int ngx_http_lua_graphite_set(lua_State *L); 55 | + 56 | + 57 | +void 58 | +ngx_http_lua_inject_graphite_api(lua_State *L) 59 | +{ 60 | + lua_createtable(L, 0, 2); 61 | + lua_newtable(L); 62 | + lua_pushcfunction(L, ngx_http_lua_graphite); 63 | + lua_setfield(L, -2, "__call"); 64 | + lua_setmetatable(L, -2); 65 | + 66 | + lua_pushcfunction(L, ngx_http_lua_graphite_get); 67 | + lua_setfield(L, -2, "get"); 68 | + 69 | + lua_pushcfunction(L, ngx_http_lua_graphite_set); 70 | + lua_setfield(L, -2, "set"); 71 | + 72 | + lua_setfield(L, -2, "graphite"); 73 | +} 74 | + 75 | + 76 | +static int 77 | +ngx_http_lua_graphite(lua_State *L) { 78 | + 79 | + size_t n = lua_gettop(L) - 1; 80 | + if (n != 2 && n != 3) { 81 | + return luaL_error(L, "ngx.graphite expecting 2 or 3 arguments got %d", n); 82 | + } 83 | + 84 | + ngx_http_request_t *r; 85 | + r = ngx_http_lua_get_req(L); 86 | + 87 | + if (r == NULL) { 88 | + return luaL_error(L, "no request object found"); 89 | + } 90 | + 91 | + ngx_str_t name; 92 | + name.data = (u_char*)lua_tolstring(L, 2, &name.len); 93 | + if (name.data == NULL) 94 | + return 0; 95 | + 96 | + double value = lua_tonumber(L, 3); 97 | + 98 | + ngx_http_graphite(r, &name, value); 99 | + 100 | + return 0; 101 | +} 102 | + 103 | + 104 | +static int 105 | +ngx_http_lua_graphite_get(lua_State *L) { 106 | + 107 | + size_t n = lua_gettop(L); 108 | + if (n != 1) { 109 | + return luaL_error(L, "ngx.graphite.get expecting 1 argument got %d", n); 110 | + } 111 | + 112 | + ngx_http_request_t *r; 113 | + r = ngx_http_lua_get_req(L); 114 | + 115 | + if (r == NULL) { 116 | + return luaL_error(L, "no request object found"); 117 | + } 118 | + 119 | + ngx_str_t name; 120 | + name.data = (u_char*)lua_tolstring(L, 1, &name.len); 121 | + if (name.data == NULL) 122 | + return 0; 123 | + 124 | + double value = ngx_http_graphite_get(r, &name); 125 | + 126 | + lua_pushnumber(L, value); 127 | + 128 | + return 1; 129 | +} 130 | + 131 | + 132 | +static int 133 | +ngx_http_lua_graphite_set(lua_State *L) { 134 | + 135 | + size_t n = lua_gettop(L); 136 | + if (n != 2) { 137 | + return luaL_error(L, "ngx.graphite.get expecting 2 arguments got %d", n); 138 | + } 139 | + 140 | + ngx_http_request_t *r; 141 | + r = ngx_http_lua_get_req(L); 142 | + 143 | + if (r == NULL) { 144 | + return luaL_error(L, "no request object found"); 145 | + } 146 | + 147 | + ngx_str_t name; 148 | + name.data = (u_char*)lua_tolstring(L, 1, &name.len); 149 | + if (name.data == NULL) 150 | + return 0; 151 | + 152 | + double value = lua_tonumber(L, 2); 153 | + 154 | + ngx_http_graphite_set(r, &name, value); 155 | + 156 | + return 0; 157 | +} 158 | diff --git a/src/ngx_http_lua_graphite.h b/src/ngx_http_lua_graphite.h 159 | new file mode 100644 160 | index 00000000..cf76efa3 161 | --- /dev/null 162 | +++ b/src/ngx_http_lua_graphite.h 163 | @@ -0,0 +1,11 @@ 164 | +#ifndef NGX_HTTP_LUA_GRAPHITE_H 165 | +#define NGX_HTTP_LUA_GRAPHITE_H 166 | + 167 | +#include "ngx_http_lua_common.h" 168 | + 169 | + 170 | +void ngx_http_lua_inject_graphite_api(lua_State *L); 171 | + 172 | + 173 | +#endif /* NGX_HTTP_LUA_GRAPHITE_H */ 174 | + 175 | diff --git a/src/ngx_http_lua_util.c b/src/ngx_http_lua_util.c 176 | index f7a537ee..6292d8b5 100644 177 | --- a/src/ngx_http_lua_util.c 178 | +++ b/src/ngx_http_lua_util.c 179 | @@ -53,6 +53,7 @@ 180 | #include "ngx_http_lua_ssl.h" 181 | #include "ngx_http_lua_log_ringbuf.h" 182 | 183 | +#include "ngx_http_lua_graphite.h" 184 | 185 | #if 1 186 | #undef ngx_http_lua_probe_info 187 | @@ -710,7 +711,7 @@ static void 188 | ngx_http_lua_inject_ngx_api(lua_State *L, ngx_http_lua_main_conf_t *lmcf, 189 | ngx_log_t *log) 190 | { 191 | - lua_createtable(L, 0 /* narr */, 117 /* nrec */); /* ngx.* */ 192 | + lua_createtable(L, 0 /* narr */, 118 /* nrec */); /* ngx.* */ 193 | 194 | lua_pushcfunction(L, ngx_http_lua_get_raw_phase_context); 195 | lua_setfield(L, -2, "_phase_ctx"); 196 | @@ -744,6 +745,7 @@ ngx_http_lua_inject_ngx_api(lua_State *L, ngx_http_lua_main_conf_t *lmcf, 197 | ngx_http_lua_inject_timer_api(L); 198 | ngx_http_lua_inject_config_api(L); 199 | ngx_http_lua_inject_worker_api(L); 200 | + ngx_http_lua_inject_graphite_api(L); 201 | 202 | ngx_http_lua_inject_misc_api(L); 203 | 204 | @@ -1007,7 +1009,16 @@ ngx_http_lua_run_thread(lua_State *L, ngx_http_request_t *r, 205 | ngx_http_lua_assert(orig_coctx->co_top + nrets 206 | == lua_gettop(orig_coctx->co)); 207 | 208 | +#ifdef NGX_GRAPHITE_PATCH 209 | + struct timeval start_tp; 210 | + ngx_gettimeofday(&start_tp); 211 | +#endif 212 | rv = lua_resume(orig_coctx->co, nrets); 213 | +#ifdef NGX_GRAPHITE_PATCH 214 | + struct timeval stop_tp; 215 | + ngx_gettimeofday(&stop_tp); 216 | + r->lua_time += (stop_tp.tv_sec - start_tp.tv_sec) * 1000 + (stop_tp.tv_usec - start_tp.tv_usec) / 1000.0; 217 | +#endif 218 | 219 | #if (NGX_PCRE) 220 | /* XXX: work-around to nginx regex subsystem */ 221 | -- 222 | 2.25.1 223 | 224 | 225 | From 49a877588691ea7136a7472ab95df2665d546c64 Mon Sep 17 00:00:00 2001 226 | From: Alexander Drozdov 227 | Date: Mon, 19 Apr 2021 20:09:05 +0300 228 | Subject: [PATCH 2/3] add ngx.graphite.param feature 229 | 230 | Add ngx.graphite.param() function that returns an object by graphite 231 | parameter name. We name it 'parameter link'. The link then 232 | can be passed to ngx.graphite(), ngx.graphite.get() & 233 | ngx.graphite.set() in place of the name. 234 | --- 235 | src/ngx_http_lua_graphite.c | 90 +++++++++++++++++++++++++++++-------- 236 | 1 file changed, 72 insertions(+), 18 deletions(-) 237 | 238 | diff --git a/src/ngx_http_lua_graphite.c b/src/ngx_http_lua_graphite.c 239 | index 9178553d..142c6e9f 100644 240 | --- a/src/ngx_http_lua_graphite.c 241 | +++ b/src/ngx_http_lua_graphite.c 242 | @@ -10,6 +10,7 @@ 243 | 244 | 245 | static int ngx_http_lua_graphite(lua_State *L); 246 | +static int ngx_http_lua_graphite_param(lua_State *L); 247 | static int ngx_http_lua_graphite_get(lua_State *L); 248 | static int ngx_http_lua_graphite_set(lua_State *L); 249 | 250 | @@ -23,6 +24,9 @@ ngx_http_lua_inject_graphite_api(lua_State *L) 251 | lua_setfield(L, -2, "__call"); 252 | lua_setmetatable(L, -2); 253 | 254 | + lua_pushcfunction(L, ngx_http_lua_graphite_param); 255 | + lua_setfield(L, -2, "param"); 256 | + 257 | lua_pushcfunction(L, ngx_http_lua_graphite_get); 258 | lua_setfield(L, -2, "get"); 259 | 260 | @@ -48,18 +52,50 @@ ngx_http_lua_graphite(lua_State *L) { 261 | return luaL_error(L, "no request object found"); 262 | } 263 | 264 | - ngx_str_t name; 265 | - name.data = (u_char*)lua_tolstring(L, 2, &name.len); 266 | - if (name.data == NULL) 267 | - return 0; 268 | + int type = lua_type(L, 2); 269 | + if (type == LUA_TLIGHTUSERDATA) { 270 | + const ngx_http_graphite_link_t *link = lua_touserdata(L, 2); 271 | 272 | - double value = lua_tonumber(L, 3); 273 | - 274 | - ngx_http_graphite(r, &name, value); 275 | + ngx_http_graphite_by_link(r, link, lua_tonumber(L, 3)); 276 | + } 277 | + else if (type == LUA_TBOOLEAN) 278 | + return 0; 279 | + else { 280 | + ngx_str_t name; 281 | + name.data = (u_char*)lua_tolstring(L, 2, &name.len); 282 | + if (name.data == NULL) 283 | + return 0; 284 | + ngx_http_graphite(r, &name, lua_tonumber(L, 3)); 285 | + } 286 | 287 | return 0; 288 | } 289 | 290 | +static int 291 | +ngx_http_lua_graphite_param(lua_State *L) { 292 | + 293 | + int n = lua_gettop(L); 294 | + if (n != 1) { 295 | + return luaL_error(L, "ngx.graphite.param expecting 1 argument got %d", n); 296 | + } 297 | + 298 | + ngx_http_request_t *r; 299 | + r = ngx_http_lua_get_req(L); 300 | + 301 | + if (r == NULL) { 302 | + return luaL_error(L, "no request object found"); 303 | + } 304 | + ngx_str_t name; 305 | + const ngx_http_graphite_link_t *link; 306 | + name.data = (u_char*)lua_tolstring(L, 1, &name.len); 307 | + if (name.data == NULL || (link = ngx_http_graphite_link(r, &name)) == NULL) { 308 | + lua_pushboolean(L, 0); 309 | + return 1; 310 | + } 311 | + 312 | + lua_pushlightuserdata(L, (void *)link); 313 | + return 1; 314 | +} 315 | 316 | static int 317 | ngx_http_lua_graphite_get(lua_State *L) { 318 | @@ -76,12 +112,22 @@ ngx_http_lua_graphite_get(lua_State *L) { 319 | return luaL_error(L, "no request object found"); 320 | } 321 | 322 | - ngx_str_t name; 323 | - name.data = (u_char*)lua_tolstring(L, 1, &name.len); 324 | - if (name.data == NULL) 325 | - return 0; 326 | + double value; 327 | + int type = lua_type(L, 1); 328 | + if (type == LUA_TLIGHTUSERDATA) { 329 | + const ngx_http_graphite_link_t *link = lua_touserdata(L, 1); 330 | 331 | - double value = ngx_http_graphite_get(r, &name); 332 | + value = ngx_http_graphite_get_by_link(r, link); 333 | + } 334 | + else if (type == LUA_TBOOLEAN) 335 | + return 0; 336 | + else { 337 | + ngx_str_t name; 338 | + name.data = (u_char*)lua_tolstring(L, 1, &name.len); 339 | + if (name.data == NULL) 340 | + return 0; 341 | + value = ngx_http_graphite_get(r, &name); 342 | + } 343 | 344 | lua_pushnumber(L, value); 345 | 346 | @@ -104,14 +150,22 @@ ngx_http_lua_graphite_set(lua_State *L) { 347 | return luaL_error(L, "no request object found"); 348 | } 349 | 350 | - ngx_str_t name; 351 | - name.data = (u_char*)lua_tolstring(L, 1, &name.len); 352 | - if (name.data == NULL) 353 | - return 0; 354 | - 355 | double value = lua_tonumber(L, 2); 356 | + int type = lua_type(L, 1); 357 | + if (type == LUA_TLIGHTUSERDATA) { 358 | + const ngx_http_graphite_link_t *link = lua_touserdata(L, 1); 359 | + ngx_http_graphite_set_by_link(r, link, value); 360 | + } 361 | + else if (type == LUA_TBOOLEAN) 362 | + return 0; 363 | + else { 364 | + ngx_str_t name; 365 | + name.data = (u_char*)lua_tolstring(L, 1, &name.len); 366 | + if (name.data == NULL) 367 | + return 0; 368 | 369 | - ngx_http_graphite_set(r, &name, value); 370 | + ngx_http_graphite_set(r, &name, value); 371 | + } 372 | 373 | return 0; 374 | } 375 | -- 376 | 2.25.1 377 | 378 | 379 | From c40cf28bbf35da8c18668f5b8f165aa89a6202dc Mon Sep 17 00:00:00 2001 380 | From: Alexander Drozdov 381 | Date: Sat, 19 Jun 2021 12:39:39 +0300 382 | Subject: [PATCH 3/3] add new graphite parameters on-the-fly 383 | 384 | --- 385 | src/ngx_http_lua_graphite.c | 19 +++++++++++++++---- 386 | 1 file changed, 15 insertions(+), 4 deletions(-) 387 | 388 | diff --git a/src/ngx_http_lua_graphite.c b/src/ngx_http_lua_graphite.c 389 | index 142c6e9f..9c660ad1 100644 390 | --- a/src/ngx_http_lua_graphite.c 391 | +++ b/src/ngx_http_lua_graphite.c 392 | @@ -62,10 +62,16 @@ ngx_http_lua_graphite(lua_State *L) { 393 | return 0; 394 | else { 395 | ngx_str_t name; 396 | + ngx_str_t config, *pconfig = NULL; 397 | + 398 | name.data = (u_char*)lua_tolstring(L, 2, &name.len); 399 | if (name.data == NULL) 400 | return 0; 401 | - ngx_http_graphite(r, &name, lua_tonumber(L, 3)); 402 | + if (n >= 3) { 403 | + config.data = (u_char*)lua_tolstring(L, 4, &config.len); 404 | + pconfig = &config; 405 | + } 406 | + ngx_http_graphite(r, &name, lua_tonumber(L, 3), pconfig); 407 | } 408 | 409 | return 0; 410 | @@ -75,8 +81,8 @@ static int 411 | ngx_http_lua_graphite_param(lua_State *L) { 412 | 413 | int n = lua_gettop(L); 414 | - if (n != 1) { 415 | - return luaL_error(L, "ngx.graphite.param expecting 1 argument got %d", n); 416 | + if (n != 1 && n != 2) { 417 | + return luaL_error(L, "ngx.graphite.param expecting 1 or 2 arguments, got %d", n); 418 | } 419 | 420 | ngx_http_request_t *r; 421 | @@ -86,9 +92,14 @@ ngx_http_lua_graphite_param(lua_State *L) { 422 | return luaL_error(L, "no request object found"); 423 | } 424 | ngx_str_t name; 425 | + ngx_str_t config, *pconfig = NULL; 426 | const ngx_http_graphite_link_t *link; 427 | name.data = (u_char*)lua_tolstring(L, 1, &name.len); 428 | - if (name.data == NULL || (link = ngx_http_graphite_link(r, &name)) == NULL) { 429 | + if (n >= 2) { 430 | + config.data = (u_char*)lua_tolstring(L, 2, &config.len); 431 | + pconfig = &config; 432 | + } 433 | + if (name.data == NULL || (link = ngx_http_graphite_link(r, &name, pconfig)) == NULL) { 434 | lua_pushboolean(L, 0); 435 | return 1; 436 | } 437 | -- 438 | 2.25.1 439 | 440 | -------------------------------------------------------------------------------- /lua_module_v0_10_2.patch: -------------------------------------------------------------------------------- 1 | diff --git a/config b/config 2 | index c91a07e..9f89c8e 100644 3 | --- a/config 4 | +++ b/config 5 | @@ -357,6 +357,7 @@ HTTP_LUA_SRCS=" \ 6 | $ngx_addon_dir/src/ngx_http_lua_ssl_ocsp.c \ 7 | $ngx_addon_dir/src/ngx_http_lua_lex.c \ 8 | $ngx_addon_dir/src/ngx_http_lua_balancer.c \ 9 | + $ngx_addon_dir/src/ngx_http_lua_graphite.c \ 10 | " 11 | 12 | HTTP_LUA_DEPS=" \ 13 | @@ -414,6 +415,7 @@ HTTP_LUA_DEPS=" \ 14 | $ngx_addon_dir/src/ngx_http_lua_ssl_certby.h \ 15 | $ngx_addon_dir/src/ngx_http_lua_lex.h \ 16 | $ngx_addon_dir/src/ngx_http_lua_balancer.h \ 17 | + $ngx_addon_dir/src/ngx_http_lua_graphite.h \ 18 | " 19 | 20 | CFLAGS="$CFLAGS -DNDK_SET_VAR" 21 | diff --git a/src/ngx_http_lua_graphite.c b/src/ngx_http_lua_graphite.c 22 | new file mode 100644 23 | index 0000000..20812f9 24 | --- /dev/null 25 | +++ b/src/ngx_http_lua_graphite.c 26 | @@ -0,0 +1,117 @@ 27 | +#ifndef DDEBUG 28 | +#define DDEBUG 0 29 | +#endif 30 | +#include "ddebug.h" 31 | + 32 | + 33 | +#include "ngx_http_graphite_module.h" 34 | +#include "ngx_http_lua_graphite.h" 35 | +#include "ngx_http_lua_util.h" 36 | + 37 | + 38 | +static int ngx_http_lua_graphite(lua_State *L); 39 | +static int ngx_http_lua_graphite_get(lua_State *L); 40 | +static int ngx_http_lua_graphite_set(lua_State *L); 41 | + 42 | + 43 | +void 44 | +ngx_http_lua_inject_graphite_api(lua_State *L) 45 | +{ 46 | + lua_createtable(L, 0, 2); 47 | + lua_newtable(L); 48 | + lua_pushcfunction(L, ngx_http_lua_graphite); 49 | + lua_setfield(L, -2, "__call"); 50 | + lua_setmetatable(L, -2); 51 | + 52 | + lua_pushcfunction(L, ngx_http_lua_graphite_get); 53 | + lua_setfield(L, -2, "get"); 54 | + 55 | + lua_pushcfunction(L, ngx_http_lua_graphite_set); 56 | + lua_setfield(L, -2, "set"); 57 | + 58 | + lua_setfield(L, -2, "graphite"); 59 | +} 60 | + 61 | + 62 | +static int 63 | +ngx_http_lua_graphite(lua_State *L) { 64 | + 65 | + size_t n = lua_gettop(L) - 1; 66 | + if (n != 2 && n != 3) { 67 | + return luaL_error(L, "ngx.graphite expecting 2 or 3 arguments got %d", n); 68 | + } 69 | + 70 | + ngx_http_request_t *r; 71 | + r = ngx_http_lua_get_req(L); 72 | + 73 | + if (r == NULL) { 74 | + return luaL_error(L, "no request object found"); 75 | + } 76 | + 77 | + ngx_str_t name; 78 | + name.data = (u_char*)lua_tolstring(L, 2, &name.len); 79 | + if (name.data == NULL) 80 | + return 0; 81 | + 82 | + double value = lua_tonumber(L, 3); 83 | + 84 | + ngx_http_graphite(r, &name, value); 85 | + 86 | + return 0; 87 | +} 88 | + 89 | + 90 | +static int 91 | +ngx_http_lua_graphite_get(lua_State *L) { 92 | + 93 | + size_t n = lua_gettop(L); 94 | + if (n != 1) { 95 | + return luaL_error(L, "ngx.graphite.get expecting 1 argument got %d", n); 96 | + } 97 | + 98 | + ngx_http_request_t *r; 99 | + r = ngx_http_lua_get_req(L); 100 | + 101 | + if (r == NULL) { 102 | + return luaL_error(L, "no request object found"); 103 | + } 104 | + 105 | + ngx_str_t name; 106 | + name.data = (u_char*)lua_tolstring(L, 1, &name.len); 107 | + if (name.data == NULL) 108 | + return 0; 109 | + 110 | + double value = ngx_http_graphite_get(r, &name); 111 | + 112 | + lua_pushnumber(L, value); 113 | + 114 | + return 1; 115 | +} 116 | + 117 | + 118 | +static int 119 | +ngx_http_lua_graphite_set(lua_State *L) { 120 | + 121 | + size_t n = lua_gettop(L); 122 | + if (n != 2) { 123 | + return luaL_error(L, "ngx.graphite.get expecting 2 arguments got %d", n); 124 | + } 125 | + 126 | + ngx_http_request_t *r; 127 | + r = ngx_http_lua_get_req(L); 128 | + 129 | + if (r == NULL) { 130 | + return luaL_error(L, "no request object found"); 131 | + } 132 | + 133 | + ngx_str_t name; 134 | + name.data = (u_char*)lua_tolstring(L, 1, &name.len); 135 | + if (name.data == NULL) 136 | + return 0; 137 | + 138 | + double value = lua_tonumber(L, 2); 139 | + 140 | + ngx_http_graphite_set(r, &name, value); 141 | + 142 | + return 0; 143 | +} 144 | diff --git a/src/ngx_http_lua_util.c b/src/ngx_http_lua_util.c 145 | index 1bd1388..0bebff3 100644 146 | --- a/src/ngx_http_lua_util.c 147 | +++ b/src/ngx_http_lua_util.c 148 | @@ -51,6 +51,7 @@ 149 | #include "ngx_http_lua_socket_tcp.h" 150 | #include "ngx_http_lua_ssl_certby.h" 151 | 152 | +#include "ngx_http_lua_graphite.h" 153 | 154 | #if 1 155 | #undef ngx_http_lua_probe_info 156 | @@ -719,7 +720,7 @@ static void 157 | ngx_http_lua_inject_ngx_api(lua_State *L, ngx_http_lua_main_conf_t *lmcf, 158 | ngx_log_t *log) 159 | { 160 | - lua_createtable(L, 0 /* narr */, 116 /* nrec */); /* ngx.* */ 161 | + lua_createtable(L, 0 /* narr */, 117 /* nrec */); /* ngx.* */ 162 | 163 | lua_pushcfunction(L, ngx_http_lua_get_raw_phase_context); 164 | lua_setfield(L, -2, "_phase_ctx"); 165 | @@ -753,6 +754,7 @@ ngx_http_lua_inject_ngx_api(lua_State *L, ngx_http_lua_main_conf_t *lmcf, 166 | ngx_http_lua_inject_timer_api(L); 167 | ngx_http_lua_inject_config_api(L); 168 | ngx_http_lua_inject_worker_api(L); 169 | + ngx_http_lua_inject_graphite_api(L); 170 | 171 | ngx_http_lua_inject_misc_api(L); 172 | 173 | @@ -1003,7 +1005,16 @@ ngx_http_lua_run_thread(lua_State *L, ngx_http_request_t *r, 174 | ngx_http_lua_assert(orig_coctx->co_top + nrets 175 | == lua_gettop(orig_coctx->co)); 176 | 177 | +#ifdef NGX_GRAPHITE_PATCH 178 | + struct timeval start_tp; 179 | + ngx_gettimeofday(&start_tp); 180 | +#endif 181 | rv = lua_resume(orig_coctx->co, nrets); 182 | +#ifdef NGX_GRAPHITE_PATCH 183 | + struct timeval stop_tp; 184 | + ngx_gettimeofday(&stop_tp); 185 | + r->lua_time += (stop_tp.tv_sec - start_tp.tv_sec) * 1000 + (stop_tp.tv_usec - start_tp.tv_usec) / 1000.0; 186 | +#endif 187 | 188 | #if (NGX_PCRE) 189 | /* XXX: work-around to nginx regex subsystem */ 190 | -------------------------------------------------------------------------------- /lua_module_v0_10_6.patch: -------------------------------------------------------------------------------- 1 | diff --git a/config b/config 2 | index 0f2749d..7b8d1eb 100644 3 | --- a/config 4 | +++ b/config 5 | @@ -360,6 +360,7 @@ HTTP_LUA_SRCS=" \ 6 | $ngx_addon_dir/src/ngx_http_lua_ssl_session_storeby.c \ 7 | $ngx_addon_dir/src/ngx_http_lua_ssl_session_fetchby.c \ 8 | $ngx_addon_dir/src/ngx_http_lua_ssl.c \ 9 | + $ngx_addon_dir/src/ngx_http_lua_graphite.c \ 10 | " 11 | 12 | HTTP_LUA_DEPS=" \ 13 | @@ -420,6 +421,7 @@ HTTP_LUA_DEPS=" \ 14 | $ngx_addon_dir/src/ngx_http_lua_ssl_session_storeby.h \ 15 | $ngx_addon_dir/src/ngx_http_lua_ssl_session_fetchby.h \ 16 | $ngx_addon_dir/src/ngx_http_lua_ssl.h \ 17 | + $ngx_addon_dir/src/ngx_http_lua_graphite.h \ 18 | " 19 | 20 | CFLAGS="$CFLAGS -DNDK_SET_VAR" 21 | diff --git a/src/ngx_http_lua_graphite.c b/src/ngx_http_lua_graphite.c 22 | new file mode 100644 23 | index 0000000..20812f9 24 | --- /dev/null 25 | +++ b/src/ngx_http_lua_graphite.c 26 | @@ -0,0 +1,117 @@ 27 | +#ifndef DDEBUG 28 | +#define DDEBUG 0 29 | +#endif 30 | +#include "ddebug.h" 31 | + 32 | + 33 | +#include "ngx_http_graphite_module.h" 34 | +#include "ngx_http_lua_graphite.h" 35 | +#include "ngx_http_lua_util.h" 36 | + 37 | + 38 | +static int ngx_http_lua_graphite(lua_State *L); 39 | +static int ngx_http_lua_graphite_get(lua_State *L); 40 | +static int ngx_http_lua_graphite_set(lua_State *L); 41 | + 42 | + 43 | +void 44 | +ngx_http_lua_inject_graphite_api(lua_State *L) 45 | +{ 46 | + lua_createtable(L, 0, 2); 47 | + lua_newtable(L); 48 | + lua_pushcfunction(L, ngx_http_lua_graphite); 49 | + lua_setfield(L, -2, "__call"); 50 | + lua_setmetatable(L, -2); 51 | + 52 | + lua_pushcfunction(L, ngx_http_lua_graphite_get); 53 | + lua_setfield(L, -2, "get"); 54 | + 55 | + lua_pushcfunction(L, ngx_http_lua_graphite_set); 56 | + lua_setfield(L, -2, "set"); 57 | + 58 | + lua_setfield(L, -2, "graphite"); 59 | +} 60 | + 61 | + 62 | +static int 63 | +ngx_http_lua_graphite(lua_State *L) { 64 | + 65 | + size_t n = lua_gettop(L) - 1; 66 | + if (n != 2 && n != 3) { 67 | + return luaL_error(L, "ngx.graphite expecting 2 or 3 arguments got %d", n); 68 | + } 69 | + 70 | + ngx_http_request_t *r; 71 | + r = ngx_http_lua_get_req(L); 72 | + 73 | + if (r == NULL) { 74 | + return luaL_error(L, "no request object found"); 75 | + } 76 | + 77 | + ngx_str_t name; 78 | + name.data = (u_char*)lua_tolstring(L, 2, &name.len); 79 | + if (name.data == NULL) 80 | + return 0; 81 | + 82 | + double value = lua_tonumber(L, 3); 83 | + 84 | + ngx_http_graphite(r, &name, value); 85 | + 86 | + return 0; 87 | +} 88 | + 89 | + 90 | +static int 91 | +ngx_http_lua_graphite_get(lua_State *L) { 92 | + 93 | + size_t n = lua_gettop(L); 94 | + if (n != 1) { 95 | + return luaL_error(L, "ngx.graphite.get expecting 1 argument got %d", n); 96 | + } 97 | + 98 | + ngx_http_request_t *r; 99 | + r = ngx_http_lua_get_req(L); 100 | + 101 | + if (r == NULL) { 102 | + return luaL_error(L, "no request object found"); 103 | + } 104 | + 105 | + ngx_str_t name; 106 | + name.data = (u_char*)lua_tolstring(L, 1, &name.len); 107 | + if (name.data == NULL) 108 | + return 0; 109 | + 110 | + double value = ngx_http_graphite_get(r, &name); 111 | + 112 | + lua_pushnumber(L, value); 113 | + 114 | + return 1; 115 | +} 116 | + 117 | + 118 | +static int 119 | +ngx_http_lua_graphite_set(lua_State *L) { 120 | + 121 | + size_t n = lua_gettop(L); 122 | + if (n != 2) { 123 | + return luaL_error(L, "ngx.graphite.get expecting 2 arguments got %d", n); 124 | + } 125 | + 126 | + ngx_http_request_t *r; 127 | + r = ngx_http_lua_get_req(L); 128 | + 129 | + if (r == NULL) { 130 | + return luaL_error(L, "no request object found"); 131 | + } 132 | + 133 | + ngx_str_t name; 134 | + name.data = (u_char*)lua_tolstring(L, 1, &name.len); 135 | + if (name.data == NULL) 136 | + return 0; 137 | + 138 | + double value = lua_tonumber(L, 2); 139 | + 140 | + ngx_http_graphite_set(r, &name, value); 141 | + 142 | + return 0; 143 | +} 144 | diff --git a/src/ngx_http_lua_util.c b/src/ngx_http_lua_util.c 145 | index 7f59833..43ca208 100644 146 | --- a/src/ngx_http_lua_util.c 147 | +++ b/src/ngx_http_lua_util.c 148 | @@ -52,6 +52,7 @@ 149 | #include "ngx_http_lua_ssl_certby.h" 150 | #include "ngx_http_lua_ssl.h" 151 | 152 | +#include "ngx_http_lua_graphite.h" 153 | 154 | #if 1 155 | #undef ngx_http_lua_probe_info 156 | @@ -709,7 +710,7 @@ static void 157 | ngx_http_lua_inject_ngx_api(lua_State *L, ngx_http_lua_main_conf_t *lmcf, 158 | ngx_log_t *log) 159 | { 160 | - lua_createtable(L, 0 /* narr */, 116 /* nrec */); /* ngx.* */ 161 | + lua_createtable(L, 0 /* narr */, 117 /* nrec */); /* ngx.* */ 162 | 163 | lua_pushcfunction(L, ngx_http_lua_get_raw_phase_context); 164 | lua_setfield(L, -2, "_phase_ctx"); 165 | @@ -743,6 +744,7 @@ ngx_http_lua_inject_ngx_api(lua_State *L, ngx_http_lua_main_conf_t *lmcf, 166 | ngx_http_lua_inject_timer_api(L); 167 | ngx_http_lua_inject_config_api(L); 168 | ngx_http_lua_inject_worker_api(L); 169 | + ngx_http_lua_inject_graphite_api(L); 170 | 171 | ngx_http_lua_inject_misc_api(L); 172 | 173 | @@ -1003,7 +1005,16 @@ ngx_http_lua_run_thread(lua_State *L, ngx_http_request_t *r, 174 | ngx_http_lua_assert(orig_coctx->co_top + nrets 175 | == lua_gettop(orig_coctx->co)); 176 | 177 | +#ifdef NGX_GRAPHITE_PATCH 178 | + struct timeval start_tp; 179 | + ngx_gettimeofday(&start_tp); 180 | +#endif 181 | rv = lua_resume(orig_coctx->co, nrets); 182 | +#ifdef NGX_GRAPHITE_PATCH 183 | + struct timeval stop_tp; 184 | + ngx_gettimeofday(&stop_tp); 185 | + r->lua_time += (stop_tp.tv_sec - start_tp.tv_sec) * 1000 + (stop_tp.tv_usec - start_tp.tv_usec) / 1000.0; 186 | +#endif 187 | 188 | #if (NGX_PCRE) 189 | /* XXX: work-around to nginx regex subsystem */ 190 | -------------------------------------------------------------------------------- /nginx_error_log_limiting_v1_15.4.patch: -------------------------------------------------------------------------------- 1 | From 5b4c58739a91cbe1b0d200b5df448fd29b4b8ce6 Mon Sep 17 00:00:00 2001 2 | From: Alexander Drozdov 3 | Date: Thu, 25 Jun 2020 09:20:58 +0300 4 | Subject: [PATCH 1/6] RB-27104: log: refactor writing to error_log, part 1 5 | 6 | ngx_log_error_core(): 7 | We split code which checks if a message should be written to 8 | each (ngx_log_t *) from the one which really writes. 9 | The corresponding functions are: 10 | - ngx_log_error_core_check() 11 | - ngx_log_error_core_write() 12 | --- 13 | src/core/ngx_log.c | 62 +++++++++++++++++++++++++++++----------------- 14 | 1 file changed, 39 insertions(+), 23 deletions(-) 15 | 16 | diff --git a/src/core/ngx_log.c b/src/core/ngx_log.c 17 | index 8e9408df..c8a55581 100644 18 | --- a/src/core/ngx_log.c 19 | +++ b/src/core/ngx_log.c 20 | @@ -90,6 +90,42 @@ static const char *debug_levels[] = { 21 | }; 22 | 23 | 24 | +static void 25 | +ngx_log_error_core_write(ngx_log_t *log, ngx_uint_t level, 26 | + void *buf, size_t size, ngx_uint_t *wrote_stderr) 27 | +{ 28 | + ssize_t n; 29 | + 30 | + if (log->writer) { 31 | + log->writer(log, level, buf, size); 32 | + return; 33 | + } 34 | + n = ngx_write_fd(log->file->fd, buf, size); 35 | + 36 | + if (n == -1 && ngx_errno == NGX_ENOSPC) { 37 | + log->disk_full_time = ngx_time(); 38 | + } 39 | + 40 | + if (log->file->fd == ngx_stderr) { 41 | + *wrote_stderr = 1; 42 | + } 43 | +} 44 | + 45 | +static ngx_int_t 46 | +ngx_log_error_core_check(ngx_log_t *log, size_t size, ngx_uint_t *wrote_stderr) 47 | +{ 48 | + if (log->writer == NULL && ngx_time() == log->disk_full_time) { 49 | + /* 50 | + * on FreeBSD writing to a full filesystem with enabled softupdates 51 | + * may block process for much longer time than writing to non-full 52 | + * filesystem, so we skip writing to a log for one second 53 | + */ 54 | + 55 | + return NGX_DECLINED; 56 | + } 57 | + return NGX_OK; 58 | +} 59 | + 60 | #if (NGX_HAVE_VARIADIC_MACROS) 61 | 62 | void 63 | @@ -108,7 +144,6 @@ ngx_log_error_core(ngx_uint_t level, ngx_log_t *log, ngx_err_t err, 64 | va_list args; 65 | #endif 66 | u_char *p, *last, *msg; 67 | - ssize_t n; 68 | ngx_uint_t wrote_stderr, debug_connection; 69 | u_char errstr[NGX_MAX_ERROR_STR]; 70 | 71 | @@ -164,31 +199,12 @@ ngx_log_error_core(ngx_uint_t level, ngx_log_t *log, ngx_err_t err, 72 | break; 73 | } 74 | 75 | - if (log->writer) { 76 | - log->writer(log, level, errstr, p - errstr); 77 | + if (ngx_log_error_core_check( 78 | + log, p - errstr, &wrote_stderr) != NGX_OK) { 79 | goto next; 80 | } 81 | 82 | - if (ngx_time() == log->disk_full_time) { 83 | - 84 | - /* 85 | - * on FreeBSD writing to a full filesystem with enabled softupdates 86 | - * may block process for much longer time than writing to non-full 87 | - * filesystem, so we skip writing to a log for one second 88 | - */ 89 | - 90 | - goto next; 91 | - } 92 | - 93 | - n = ngx_write_fd(log->file->fd, errstr, p - errstr); 94 | - 95 | - if (n == -1 && ngx_errno == NGX_ENOSPC) { 96 | - log->disk_full_time = ngx_time(); 97 | - } 98 | - 99 | - if (log->file->fd == ngx_stderr) { 100 | - wrote_stderr = 1; 101 | - } 102 | + ngx_log_error_core_write(log, level, errstr, p - errstr, &wrote_stderr); 103 | 104 | next: 105 | 106 | -- 107 | 2.25.1 108 | 109 | 110 | From 18e5f6752fa27e28b15d4bf442b33b378950d116 Mon Sep 17 00:00:00 2001 111 | From: Alexander Drozdov 112 | Date: Tue, 30 Jun 2020 12:20:56 +0300 113 | Subject: [PATCH 2/6] RB-27104: log: refactor writing to error_log, part 2 114 | 115 | ngx_log_error_core_write(): 116 | return NGX_DECLINED on a log message writing failure. 117 | We'll use the feature later, see subsequent changesets. 118 | --- 119 | src/core/ngx_log.c | 15 +++++++++++---- 120 | 1 file changed, 11 insertions(+), 4 deletions(-) 121 | 122 | diff --git a/src/core/ngx_log.c b/src/core/ngx_log.c 123 | index c8a55581..858210ec 100644 124 | --- a/src/core/ngx_log.c 125 | +++ b/src/core/ngx_log.c 126 | @@ -90,25 +90,32 @@ static const char *debug_levels[] = { 127 | }; 128 | 129 | 130 | -static void 131 | +static ngx_int_t 132 | ngx_log_error_core_write(ngx_log_t *log, ngx_uint_t level, 133 | void *buf, size_t size, ngx_uint_t *wrote_stderr) 134 | { 135 | ssize_t n; 136 | + ngx_int_t rc = NGX_OK; 137 | 138 | if (log->writer) { 139 | log->writer(log, level, buf, size); 140 | - return; 141 | + goto done; 142 | } 143 | n = ngx_write_fd(log->file->fd, buf, size); 144 | 145 | - if (n == -1 && ngx_errno == NGX_ENOSPC) { 146 | - log->disk_full_time = ngx_time(); 147 | + if (n == -1) { 148 | + if (ngx_errno == NGX_ENOSPC) { 149 | + log->disk_full_time = ngx_time(); 150 | + } 151 | + rc = NGX_DECLINED; 152 | } 153 | 154 | if (log->file->fd == ngx_stderr) { 155 | *wrote_stderr = 1; 156 | } 157 | + 158 | +done: 159 | + return rc; 160 | } 161 | 162 | static ngx_int_t 163 | -- 164 | 2.25.1 165 | 166 | 167 | From aa5838f0334c01e56fca0b75032468cdfbe89637 Mon Sep 17 00:00:00 2001 168 | From: Alexander Drozdov 169 | Date: Thu, 25 Jun 2020 14:10:46 +0300 170 | Subject: [PATCH 3/6] RB-27104: log: allow to load extra error_log parameters 171 | 172 | We renamed ngx_log_set_levels() to ngx_log_set_params() to allow 173 | the function to read not only log levels but also a list of extra 174 | parameters. The parameters list now is empty. We'll add one soon, 175 | see subsequent changesets. 176 | --- 177 | src/core/ngx_log.c | 37 ++++++++++++++++++++++++++++++++++--- 178 | 1 file changed, 34 insertions(+), 3 deletions(-) 179 | 180 | diff --git a/src/core/ngx_log.c b/src/core/ngx_log.c 181 | index 858210ec..53ad8d82 100644 182 | --- a/src/core/ngx_log.c 183 | +++ b/src/core/ngx_log.c 184 | @@ -10,7 +10,7 @@ 185 | 186 | 187 | static char *ngx_error_log(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); 188 | -static char *ngx_log_set_levels(ngx_conf_t *cf, ngx_log_t *log); 189 | +static char *ngx_log_set_params(ngx_conf_t *cf, ngx_log_t *log); 190 | static void ngx_log_insert(ngx_log_t *log, ngx_log_t *new_log); 191 | 192 | 193 | @@ -89,6 +89,11 @@ static const char *debug_levels[] = { 194 | "debug_http", "debug_mail", "debug_stream" 195 | }; 196 | 197 | +static const struct ngx_log_parameter { 198 | + const char *name; 199 | + char *(*load)(ngx_conf_t *cf, ngx_log_t *log, u_char *name, u_char *value); 200 | +} log_parameters[] = { 201 | +}; 202 | 203 | static ngx_int_t 204 | ngx_log_error_core_write(ngx_log_t *log, ngx_uint_t level, 205 | @@ -502,7 +507,7 @@ ngx_log_get_file_log(ngx_log_t *head) 206 | 207 | 208 | static char * 209 | -ngx_log_set_levels(ngx_conf_t *cf, ngx_log_t *log) 210 | +ngx_log_set_params(ngx_conf_t *cf, ngx_log_t *log) 211 | { 212 | ngx_uint_t i, n, d, found; 213 | ngx_str_t *value; 214 | @@ -515,8 +520,34 @@ ngx_log_set_levels(ngx_conf_t *cf, ngx_log_t *log) 215 | value = cf->args->elts; 216 | 217 | for (i = 2; i < cf->args->nelts; i++) { 218 | + u_char *param_value; 219 | + 220 | found = 0; 221 | 222 | + if ((param_value = (u_char *)ngx_strchr(value[i].data, '=')) != NULL) { 223 | + *param_value++ = '\0'; 224 | + 225 | + for (n = 0; n != sizeof(log_parameters) / sizeof(log_parameters[0]); 226 | + n++) { 227 | + char *load_rv; 228 | + 229 | + if (ngx_strcmp(value[i].data, log_parameters[n].name) != 0) 230 | + continue; 231 | + load_rv = log_parameters[n].load( 232 | + cf, log, value[i].data, param_value); 233 | + if (load_rv != NGX_CONF_OK) 234 | + return load_rv; 235 | + found = 1; 236 | + break; 237 | + } 238 | + if (!found) { 239 | + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, 240 | + "unknown parameter \"%s\"", 241 | + value[i].data); 242 | + return NGX_CONF_ERROR; 243 | + } 244 | + } 245 | + 246 | for (n = 1; n <= NGX_LOG_DEBUG; n++) { 247 | if (ngx_strcmp(value[i].data, err_levels[n].data) == 0) { 248 | 249 | @@ -687,7 +718,7 @@ ngx_log_set_log(ngx_conf_t *cf, ngx_log_t **head) 250 | } 251 | } 252 | 253 | - if (ngx_log_set_levels(cf, new_log) != NGX_CONF_OK) { 254 | + if (ngx_log_set_params(cf, new_log) != NGX_CONF_OK) { 255 | return NGX_CONF_ERROR; 256 | } 257 | 258 | -- 259 | 2.25.1 260 | 261 | 262 | From 275f5f66fdf1dfd0b0788f0115c6ac0936ce00ee Mon Sep 17 00:00:00 2001 263 | From: Alexander Drozdov 264 | Date: Thu, 25 Jun 2020 20:38:36 +0300 265 | Subject: [PATCH 4/6] RB-27104: log: add error_log grow limit support 266 | 267 | Add 'grow_limit' parameter to error_log directive. 268 | The parameter sets an error_log grow limit, in bytes per second. 269 | When the limit is reached then no more messages are printed 270 | this second, and "[N messages skipped]" is printed after the 271 | second. 272 | 273 | Example: 274 | error_log error.log error grow_limit=1k; 275 | --- 276 | src/core/ngx_connection.h | 2 + 277 | src/core/ngx_cycle.c | 22 ++- 278 | src/core/ngx_log.c | 282 +++++++++++++++++++++++++++++++++++++- 279 | src/core/ngx_log.h | 24 ++++ 280 | 4 files changed, 326 insertions(+), 4 deletions(-) 281 | 282 | diff --git a/src/core/ngx_connection.h b/src/core/ngx_connection.h 283 | index 54059629..7ac9fa0d 100644 284 | --- a/src/core/ngx_connection.h 285 | +++ b/src/core/ngx_connection.h 286 | @@ -203,6 +203,8 @@ struct ngx_connection_s { 287 | c->log->next = l->next; \ 288 | c->log->writer = l->writer; \ 289 | c->log->wdata = l->wdata; \ 290 | + c->log->shared = l->shared; \ 291 | + c->log->grow_limit = l->grow_limit; \ 292 | if (!(c->log->log_level & NGX_LOG_DEBUG_CONNECTION)) { \ 293 | c->log->log_level = l->log_level; \ 294 | } 295 | diff --git a/src/core/ngx_cycle.c b/src/core/ngx_cycle.c 296 | index f3ac24d7..914155fe 100644 297 | --- a/src/core/ngx_cycle.c 298 | +++ b/src/core/ngx_cycle.c 299 | @@ -630,6 +630,16 @@ ngx_init_cycle(ngx_cycle_t *old_cycle) 300 | 301 | /* close and delete stuff that lefts from an old cycle */ 302 | 303 | + /* 304 | + * log the remaining events of the old cycle with the new log as the old one 305 | + * may become unusable while cleaning the data 306 | + */ 307 | + 308 | + log = cycle->log; 309 | + old_cycle->log = log; 310 | + old_cycle->pool->log = log; 311 | + conf.temp_pool->log = log; 312 | + 313 | /* free the unnecessary shared memory */ 314 | 315 | opart = &old_cycle->shared_memory.part; 316 | @@ -785,12 +795,12 @@ old_shm_zone_done: 317 | ngx_memzero(ngx_old_cycles.elts, n * sizeof(ngx_cycle_t *)); 318 | 319 | ngx_cleaner_event.handler = ngx_clean_old_cycles; 320 | - ngx_cleaner_event.log = cycle->log; 321 | ngx_cleaner_event.data = &dumb; 322 | dumb.fd = (ngx_socket_t) -1; 323 | } 324 | 325 | ngx_temp_pool->log = cycle->log; 326 | + ngx_cleaner_event.log = cycle->log; 327 | 328 | old = ngx_array_push(&ngx_old_cycles); 329 | if (old == NULL) { 330 | @@ -808,6 +818,16 @@ old_shm_zone_done: 331 | 332 | failed: 333 | 334 | + /* 335 | + * log the remaining events with the previous log as the current one 336 | + * may become unusable while cleaning the data 337 | + */ 338 | + 339 | + cycle->log = log; 340 | + pool->log = log; 341 | + conf.temp_pool->log = log; 342 | + conf.log = log; 343 | + 344 | if (!ngx_is_init_cycle(old_cycle)) { 345 | old_ccf = (ngx_core_conf_t *) ngx_get_conf(old_cycle->conf_ctx, 346 | ngx_core_module); 347 | diff --git a/src/core/ngx_log.c b/src/core/ngx_log.c 348 | index 53ad8d82..a3f3ed29 100644 349 | --- a/src/core/ngx_log.c 350 | +++ b/src/core/ngx_log.c 351 | @@ -8,10 +8,18 @@ 352 | #include 353 | #include 354 | 355 | +#define NGX_LOG_SHM_PREFIX "_ngx_log_" 356 | +#define NGX_LOG_SHM_PART_SIZE (32 * 1024) 357 | +#define NGX_LOG_SHM_DATAS_PER_PART \ 358 | + (NGX_LOG_SHM_PART_SIZE / 2 / sizeof(ngx_log_shared_t)) 359 | 360 | +static void *ngx_log_module_create_conf(ngx_cycle_t *cycle); 361 | +static char *ngx_log_module_init_conf(ngx_cycle_t *cycle, void *conf); 362 | static char *ngx_error_log(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); 363 | static char *ngx_log_set_params(ngx_conf_t *cf, ngx_log_t *log); 364 | static void ngx_log_insert(ngx_log_t *log, ngx_log_t *new_log); 365 | +static char *ngx_log_param_grow_limit( 366 | + ngx_conf_t *cf, ngx_log_t *log, u_char *name, u_char *value); 367 | 368 | 369 | #if (NGX_DEBUG) 370 | @@ -46,8 +54,8 @@ static ngx_command_t ngx_errlog_commands[] = { 371 | 372 | static ngx_core_module_t ngx_errlog_module_ctx = { 373 | ngx_string("errlog"), 374 | - NULL, 375 | - NULL 376 | + ngx_log_module_create_conf, 377 | + ngx_log_module_init_conf 378 | }; 379 | 380 | 381 | @@ -93,8 +101,23 @@ static const struct ngx_log_parameter { 382 | const char *name; 383 | char *(*load)(ngx_conf_t *cf, ngx_log_t *log, u_char *name, u_char *value); 384 | } log_parameters[] = { 385 | + {"grow_limit", ngx_log_param_grow_limit}, 386 | }; 387 | 388 | + 389 | +static void 390 | +ngx_log_error_core_skip(ngx_log_t *log, size_t size) 391 | +{ 392 | + ngx_log_grow_t *grow = &log->shared->grow; 393 | + 394 | + if (grow == NULL) { 395 | + return; 396 | + } 397 | + 398 | + ngx_atomic_fetch_add(&grow->total_bytes_skipped, size); 399 | + ngx_atomic_fetch_add(&grow->last_msgs_skipped, 1); 400 | +} 401 | + 402 | static ngx_int_t 403 | ngx_log_error_core_write(ngx_log_t *log, ngx_uint_t level, 404 | void *buf, size_t size, ngx_uint_t *wrote_stderr) 405 | @@ -123,6 +146,37 @@ done: 406 | return rc; 407 | } 408 | 409 | +static ngx_int_t 410 | +ngx_log_error_core_write_skipped( 411 | + ngx_log_t *log, size_t last_msgs_skipped, ngx_uint_t *wrote_stderr) 412 | +{ 413 | + u_char *p, *last; 414 | + u_char errstr[NGX_MAX_ERROR_STR]; 415 | + ngx_uint_t level = NGX_LOG_ERR; 416 | + 417 | + last = errstr + NGX_MAX_ERROR_STR; 418 | + 419 | + p = ngx_cpymem(errstr, ngx_cached_err_log_time.data, 420 | + ngx_cached_err_log_time.len); 421 | + 422 | + p = ngx_slprintf(p, last, " [%V] ", &err_levels[level]); 423 | + 424 | + /* pid#tid */ 425 | + p = ngx_slprintf(p, last, "%P#" NGX_TID_T_FMT ": ", 426 | + ngx_log_pid, ngx_log_tid); 427 | + 428 | + p = ngx_slprintf(p, last, "%uz messages skipped", last_msgs_skipped); 429 | + 430 | + if (p > last - NGX_LINEFEED_SIZE) { 431 | + p = last - NGX_LINEFEED_SIZE; 432 | + } 433 | + 434 | + ngx_linefeed(p); 435 | + 436 | + return ngx_log_error_core_write( 437 | + log, level, errstr, p - errstr, wrote_stderr); 438 | +} 439 | + 440 | static ngx_int_t 441 | ngx_log_error_core_check(ngx_log_t *log, size_t size, ngx_uint_t *wrote_stderr) 442 | { 443 | @@ -135,6 +189,66 @@ ngx_log_error_core_check(ngx_log_t *log, size_t size, ngx_uint_t *wrote_stderr) 444 | 445 | return NGX_DECLINED; 446 | } 447 | + 448 | + size_t grow_limit = log->grow_limit; 449 | + ngx_log_grow_t *grow = &log->shared->grow; 450 | + 451 | + if (grow == NULL || (!grow_limit && !grow->last_msgs_skipped)) { 452 | + return NGX_OK; 453 | + } 454 | + 455 | + time_t last_time, now_time; 456 | + size_t total_bytes, prev_total_bytes; 457 | + 458 | + do { 459 | + ngx_msec_t now_time_msec = ngx_current_msec; 460 | + 461 | + last_time = grow->last_time; 462 | + ngx_memory_barrier(); 463 | + prev_total_bytes = grow->prev_total_bytes; 464 | + total_bytes = grow->total_bytes; 465 | + 466 | + if (now_time_msec < (ngx_msec_t)(last_time + 1) * 1000) { 467 | + if (!grow_limit 468 | + || total_bytes < prev_total_bytes 469 | + || total_bytes - prev_total_bytes < grow_limit) { 470 | + return NGX_OK; 471 | + } 472 | + return NGX_DECLINED; 473 | + } 474 | + now_time = now_time_msec / 1000; 475 | + } while (!ngx_atomic_cmp_set( 476 | + &grow->last_time, last_time, now_time) 477 | + /* 478 | + * prev_total_bytes should now be actual 479 | + * until now_time is obsoleted 480 | + */ 481 | + || !ngx_atomic_cmp_set( 482 | + &grow->prev_total_bytes, prev_total_bytes, total_bytes)); 483 | + 484 | + /* we've updated grow->last_time & grow->prev_total_bytes */ 485 | + 486 | + size_t last_msgs_skipped; 487 | + 488 | + /* swap (&grow->last_msgs_skipped, 0) */ 489 | + do { 490 | + last_msgs_skipped = grow->last_msgs_skipped; 491 | + if (!last_msgs_skipped) 492 | + break; 493 | + } while (!ngx_atomic_cmp_set( 494 | + &grow->last_msgs_skipped, last_msgs_skipped, 0)); 495 | + 496 | + if (last_msgs_skipped) { 497 | + if (ngx_log_error_core_write_skipped( 498 | + log, last_msgs_skipped, wrote_stderr) != NGX_OK) { 499 | + 500 | + /* push last_msgs_skipped back to the counder to retry later */ 501 | + 502 | + ngx_atomic_fetch_add(&grow->last_msgs_skipped, last_msgs_skipped); 503 | + return NGX_DECLINED; 504 | + } 505 | + } 506 | + 507 | return NGX_OK; 508 | } 509 | 510 | @@ -211,12 +325,20 @@ ngx_log_error_core(ngx_uint_t level, ngx_log_t *log, ngx_err_t err, 511 | break; 512 | } 513 | 514 | + if (log->shared != NULL) { 515 | + ngx_atomic_fetch_add(&log->shared->grow.total_bytes, p - errstr); 516 | + } 517 | + 518 | if (ngx_log_error_core_check( 519 | log, p - errstr, &wrote_stderr) != NGX_OK) { 520 | + ngx_log_error_core_skip(log, p - errstr); 521 | goto next; 522 | } 523 | 524 | - ngx_log_error_core_write(log, level, errstr, p - errstr, &wrote_stderr); 525 | + if (ngx_log_error_core_write( 526 | + log, level, errstr, p - errstr, &wrote_stderr) != NGX_OK) { 527 | + ngx_log_error_core_skip(log, p - errstr); 528 | + } 529 | 530 | next: 531 | 532 | @@ -505,6 +627,23 @@ ngx_log_get_file_log(ngx_log_t *head) 533 | return NULL; 534 | } 535 | 536 | +static char * 537 | +ngx_log_param_grow_limit( 538 | + ngx_conf_t *cf, ngx_log_t *log, u_char *name, u_char *value_) 539 | +{ 540 | + ngx_str_t str_value = {ngx_strlen(value_), value_}; 541 | + ssize_t value; 542 | + 543 | + value = ngx_parse_size(&str_value); 544 | + if (value == NGX_ERROR) { 545 | + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, 546 | + "cannot parse \"%s\" parameter value: \"%s\"", 547 | + name, value_); 548 | + return NGX_CONF_ERROR; 549 | + } 550 | + log->grow_limit = value; 551 | + return NGX_CONF_OK; 552 | +} 553 | 554 | static char * 555 | ngx_log_set_params(ngx_conf_t *cf, ngx_log_t *log) 556 | @@ -594,6 +733,32 @@ ngx_log_set_params(ngx_conf_t *cf, ngx_log_t *log) 557 | return NGX_CONF_OK; 558 | } 559 | 560 | +static void * 561 | +ngx_log_module_create_conf(ngx_cycle_t *cycle) 562 | +{ 563 | + ngx_log_conf_t *lcf; 564 | + 565 | + lcf = ngx_pcalloc(cycle->pool, sizeof(*lcf)); 566 | + if (lcf == NULL) { 567 | + return NULL; 568 | + } 569 | + lcf->init = ngx_pcalloc(cycle->pool, sizeof(*lcf->init)); 570 | + if (lcf->init == NULL) { 571 | + return NULL; 572 | + } 573 | + 574 | + return lcf; 575 | +} 576 | + 577 | +static char * 578 | +ngx_log_module_init_conf(ngx_cycle_t *cycle, void *conf) 579 | +{ 580 | + ngx_log_conf_t *lcf = conf; 581 | + 582 | + lcf->init = NULL; 583 | + 584 | + return NGX_CONF_OK; 585 | +} 586 | 587 | static char * 588 | ngx_error_log(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) 589 | @@ -605,6 +770,113 @@ ngx_error_log(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) 590 | return ngx_log_set_log(cf, &dummy); 591 | } 592 | 593 | +static ngx_int_t 594 | +ngx_log_shm_init(ngx_shm_zone_t *shm_zone, void *data) 595 | +{ 596 | + ngx_slab_pool_t *shpool; 597 | + ngx_list_t *l; 598 | + ngx_log_t **plog; 599 | + ngx_list_part_t *part; 600 | + ngx_uint_t i; 601 | + 602 | + if (data != NULL) { 603 | + ngx_log_error(NGX_LOG_ERR, shm_zone->shm.log, 0, 604 | + "shared memory is reused"); 605 | + return NGX_ERROR; 606 | + } 607 | + 608 | + shpool = (ngx_slab_pool_t *)shm_zone->shm.addr; 609 | + 610 | + l = shm_zone->data; 611 | + part = &l->part; 612 | + plog = part->elts; 613 | + 614 | + for (i = 0; /* void */ ; i++) { 615 | + ngx_log_shared_t *shared; 616 | + ngx_log_t *log; 617 | + 618 | + if (i >= part->nelts) { 619 | + if (part->next == NULL) { 620 | + break; 621 | + } 622 | + part = part->next; 623 | + plog = part->elts; 624 | + i = 0; 625 | + } 626 | + 627 | + log = plog[i]; 628 | + if ((shared = ngx_slab_calloc(shpool, sizeof(*shared))) == NULL) { 629 | + ngx_log_error(NGX_LOG_ERR, shm_zone->shm.log, 0, 630 | + "cannot allocate %uz shared memory bytes", 631 | + sizeof(*shared)); 632 | + return NGX_ERROR; 633 | + } 634 | + log->shared = shared; 635 | + } 636 | + 637 | + return (NGX_OK); 638 | +} 639 | + 640 | +char * 641 | +ngx_log_shm_add(ngx_conf_t *cf, ngx_log_t *log) 642 | +{ 643 | + ngx_log_conf_t *lcf; 644 | + ngx_shm_zone_t *shm_zone; 645 | + u_char *ptr; 646 | + ngx_str_t name; 647 | + size_t name_size; 648 | + ngx_log_t **plog; 649 | + 650 | + lcf = (ngx_log_conf_t *)ngx_get_conf(cf->cycle->conf_ctx, ngx_errlog_module); 651 | + 652 | + if (lcf->init->shm_zone_nlogs < NGX_LOG_SHM_DATAS_PER_PART) { 653 | + lcf->init->shm_zone_nlogs++; 654 | + } else { 655 | + lcf->init->shm_zone_idx++; 656 | + lcf->init->shm_zone_nlogs = 0; 657 | + } 658 | + 659 | + name_size = sizeof(NGX_LOG_SHM_PREFIX) + 19; 660 | + name.data = ngx_palloc(cf->pool, name_size); 661 | + if (name.data == NULL) { 662 | + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, 663 | + "cannot allocate memory for shared memory name"); 664 | + return NGX_CONF_ERROR; 665 | + } 666 | + ptr = ngx_snprintf( 667 | + name.data, name_size, "%s%ui", 668 | + NGX_LOG_SHM_PREFIX, lcf->init->shm_zone_idx); 669 | + name.len = ptr - name.data; 670 | + shm_zone = ngx_shared_memory_add( 671 | + cf, &name, NGX_LOG_SHM_PART_SIZE, &ngx_errlog_module); 672 | + if (shm_zone == NULL) { 673 | + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, 674 | + "cannot add shared memory for %uz bytes", 675 | + (size_t)NGX_LOG_SHM_PART_SIZE); 676 | + return NGX_CONF_ERROR; 677 | + } 678 | + if (shm_zone->init == NULL) { 679 | + ngx_list_t *l; 680 | + 681 | + l = ngx_list_create(cf->temp_pool, 8, sizeof(ngx_log_t *)); 682 | + if (l == NULL) { 683 | + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, 684 | + "cannot allocate memory for a list"); 685 | + return NGX_CONF_ERROR; 686 | + } 687 | + 688 | + shm_zone->init = ngx_log_shm_init; 689 | + shm_zone->data = l; 690 | + shm_zone->noreuse = 1; 691 | + } 692 | + if ((plog = ngx_list_push(shm_zone->data)) == NULL) { 693 | + ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "cannot insert to a list"); 694 | + return NGX_CONF_ERROR; 695 | + } 696 | + *plog = log; 697 | + 698 | + return NGX_CONF_OK; 699 | +} 700 | 701 | char * 702 | ngx_log_set_log(ngx_conf_t *cf, ngx_log_t **head) 703 | @@ -628,6 +900,10 @@ ngx_log_set_log(ngx_conf_t *cf, ngx_log_t **head) 704 | } 705 | } 706 | 707 | + if (ngx_log_shm_add(cf, new_log) != NGX_CONF_OK) { 708 | + return NGX_CONF_ERROR; 709 | + } 710 | + 711 | value = cf->args->elts; 712 | 713 | if (ngx_strcmp(value[1].data, "stderr") == 0) { 714 | diff --git a/src/core/ngx_log.h b/src/core/ngx_log.h 715 | index afb73bf7..905b9bd1 100644 716 | --- a/src/core/ngx_log.h 717 | +++ b/src/core/ngx_log.h 718 | @@ -46,6 +46,17 @@ typedef u_char *(*ngx_log_handler_pt) (ngx_log_t *log, u_char *buf, size_t len); 719 | typedef void (*ngx_log_writer_pt) (ngx_log_t *log, ngx_uint_t level, 720 | u_char *buf, size_t len); 721 | 722 | +typedef struct { 723 | + ngx_atomic_t last_time; 724 | + ngx_atomic_t last_msgs_skipped; 725 | + ngx_atomic_t total_bytes; 726 | + ngx_atomic_t prev_total_bytes; 727 | + ngx_atomic_t total_bytes_skipped; 728 | +} ngx_log_grow_t; 729 | + 730 | +typedef struct { 731 | + ngx_log_grow_t grow; 732 | +} ngx_log_shared_t; 733 | 734 | struct ngx_log_s { 735 | ngx_uint_t log_level; 736 | @@ -61,6 +72,9 @@ struct ngx_log_s { 737 | ngx_log_writer_pt writer; 738 | void *wdata; 739 | 740 | + size_t grow_limit; 741 | + ngx_log_shared_t *shared; 742 | + 743 | /* 744 | * we declare "action" as "char *" because the actions are usually 745 | * the static strings and in the "u_char *" case we have to override 746 | @@ -72,6 +86,16 @@ struct ngx_log_s { 747 | ngx_log_t *next; 748 | }; 749 | 750 | +typedef struct { 751 | + ngx_uint_t shm_zone_idx; 752 | + ngx_uint_t shm_zone_nlogs; 753 | +} ngx_log_conf_init_t; 754 | + 755 | +typedef struct { 756 | + /* init pointer is valid only at the config loading stage */ 757 | + ngx_log_conf_init_t *init; 758 | +} ngx_log_conf_t; 759 | + 760 | 761 | #define NGX_MAX_ERROR_STR 2048 762 | 763 | -- 764 | 2.25.1 765 | 766 | 767 | From 6b6409d58b99e50e3a6f08bf4b9942b0d0ac4950 Mon Sep 17 00:00:00 2001 768 | From: Alexander Drozdov 769 | Date: Tue, 7 Jul 2020 14:29:26 +0300 770 | Subject: [PATCH 5/6] RB-27104: log: create a global list of all the logs 771 | 772 | ngx_log_set_log(): 773 | Link all created ngx_log_t object to a list to allow other 774 | modules to iterate them all. 775 | --- 776 | src/core/ngx_log.c | 7 +++++++ 777 | src/core/ngx_log.h | 3 +++ 778 | 2 files changed, 10 insertions(+) 779 | 780 | diff --git a/src/core/ngx_log.c b/src/core/ngx_log.c 781 | index a3f3ed29..e1307553 100644 782 | --- a/src/core/ngx_log.c 783 | +++ b/src/core/ngx_log.c 784 | @@ -1002,6 +1002,13 @@ ngx_log_set_log(ngx_conf_t *cf, ngx_log_t **head) 785 | ngx_log_insert(*head, new_log); 786 | } 787 | 788 | + ngx_log_conf_t *lcf; 789 | + 790 | + lcf = (ngx_log_conf_t *)ngx_get_conf(cf->cycle->conf_ctx, ngx_errlog_module); 791 | + 792 | + new_log->global_next = lcf->global_logs; 793 | + lcf->global_logs = new_log; 794 | + 795 | return NGX_CONF_OK; 796 | } 797 | 798 | diff --git a/src/core/ngx_log.h b/src/core/ngx_log.h 799 | index 905b9bd1..aa568e75 100644 800 | --- a/src/core/ngx_log.h 801 | +++ b/src/core/ngx_log.h 802 | @@ -84,6 +84,7 @@ struct ngx_log_s { 803 | char *action; 804 | 805 | ngx_log_t *next; 806 | + ngx_log_t *global_next; 807 | }; 808 | 809 | typedef struct { 810 | @@ -94,6 +95,8 @@ typedef struct { 811 | typedef struct { 812 | /* init pointer is valid only at the config loading stage */ 813 | ngx_log_conf_init_t *init; 814 | + 815 | + ngx_log_t *global_logs; 816 | } ngx_log_conf_t; 817 | 818 | 819 | -- 820 | 2.25.1 821 | 822 | 823 | From ed25de6dea013abff673c866e31bc949fbd26f81 Mon Sep 17 00:00:00 2001 824 | From: Alexander Drozdov 825 | Date: Fri, 10 Jul 2020 12:14:47 +0300 826 | Subject: [PATCH 6/6] RB-27104: log: mark logging limiting changes with a macro 827 | 828 | Define NGX_LOG_LIMIT_ENABLED macro to ease feature detection 829 | for other modules. 830 | --- 831 | src/core/ngx_log.h | 1 + 832 | 1 file changed, 1 insertion(+) 833 | 834 | diff --git a/src/core/ngx_log.h b/src/core/ngx_log.h 835 | index aa568e75..4dacfe75 100644 836 | --- a/src/core/ngx_log.h 837 | +++ b/src/core/ngx_log.h 838 | @@ -12,6 +12,7 @@ 839 | #include 840 | #include 841 | 842 | +#define NGX_LOG_LIMIT_ENABLED 843 | 844 | #define NGX_LOG_STDERR 0 845 | #define NGX_LOG_EMERG 1 846 | -- 847 | 2.25.1 848 | 849 | -------------------------------------------------------------------------------- /src/ngx_http_graphite_allocator.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | 6 | #include "ngx_http_graphite_allocator.h" 7 | 8 | void ngx_http_graphite_allocator_init(ngx_http_graphite_allocator_t *allocator, void *pool, void* (*alloc)(void *, size_t), void (*free)(void *, void *)) { 9 | allocator->pool = pool; 10 | allocator->alloc = alloc; 11 | allocator->free = free; 12 | allocator->nomemory = 0; 13 | } 14 | 15 | void *ngx_http_graphite_allocator_pool_alloc(void *pool, size_t size) { 16 | return ngx_palloc((ngx_pool_t*)pool, size); 17 | } 18 | 19 | void ngx_http_graphite_allocator_pool_free(void *pool, void *p) { 20 | ngx_pfree((ngx_pool_t*)pool, p); 21 | } 22 | 23 | void *ngx_http_graphite_allocator_slab_alloc(void *pool, size_t size) { 24 | return ngx_slab_alloc_locked((ngx_slab_pool_t*)pool, size); 25 | } 26 | 27 | void ngx_http_graphite_allocator_slab_free(void *pool, void *p) { 28 | ngx_slab_free_locked((ngx_slab_pool_t*)pool, p); 29 | } 30 | 31 | void *ngx_http_graphite_allocator_alloc(ngx_http_graphite_allocator_t *allocator, size_t size) { 32 | void *p = allocator->alloc(allocator->pool, size); 33 | if (p == NULL) 34 | allocator->nomemory = 1; 35 | return p; 36 | } 37 | 38 | void ngx_http_graphite_allocator_free(ngx_http_graphite_allocator_t *allocator, void *p) { 39 | allocator->free(allocator->pool, p); 40 | } 41 | -------------------------------------------------------------------------------- /src/ngx_http_graphite_allocator.h: -------------------------------------------------------------------------------- 1 | #ifndef _NGX_HTTP_GRAPHITE_ALLOCATOR_H_INCLUDED_ 2 | #define _NGX_HTTP_GRAPHITE_ALLOCATOR_H_INCLUDED_ 3 | 4 | #include 5 | #include 6 | #include 7 | #include 8 | 9 | typedef struct ngx_http_graphite_allocator_s { 10 | void *pool; 11 | void *(*alloc)(void *pool, size_t size); 12 | void (*free)(void *pool, void *p); 13 | ngx_int_t nomemory; 14 | } ngx_http_graphite_allocator_t; 15 | 16 | void ngx_http_graphite_allocator_init(ngx_http_graphite_allocator_t *allocator, void *pool, void* (*alloc)(void *, size_t), void (*free)(void *, void *)); 17 | void *ngx_http_graphite_allocator_pool_alloc(void *pool, size_t size); 18 | void ngx_http_graphite_allocator_pool_free(void *pool, void *p); 19 | void *ngx_http_graphite_allocator_slab_alloc(void *pool, size_t size); 20 | void ngx_http_graphite_allocator_slab_free(void *pool, void *p); 21 | 22 | void *ngx_http_graphite_allocator_alloc(ngx_http_graphite_allocator_t *allocator, size_t size); 23 | void ngx_http_graphite_allocator_free(ngx_http_graphite_allocator_t *allocator, void *p); 24 | 25 | #endif 26 | -------------------------------------------------------------------------------- /src/ngx_http_graphite_array.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | 6 | #include "ngx_http_graphite_array.h" 7 | 8 | ngx_http_graphite_array_t * 9 | ngx_http_graphite_array_create(ngx_http_graphite_allocator_t *allocator, ngx_uint_t n, size_t size) { 10 | 11 | ngx_http_graphite_array_t *array = ngx_http_graphite_allocator_alloc(allocator, sizeof(ngx_http_graphite_array_t)); 12 | if (array == NULL) 13 | return NULL; 14 | 15 | if (ngx_http_graphite_array_init(array, allocator, n, size) != NGX_OK) 16 | return NULL; 17 | 18 | return array; 19 | } 20 | 21 | ngx_int_t 22 | ngx_http_graphite_array_init(ngx_http_graphite_array_t *array, ngx_http_graphite_allocator_t *allocator, ngx_uint_t n, size_t size) { 23 | 24 | array->nelts = 0; 25 | array->size = size; 26 | array->nalloc = n; 27 | array->allocator = allocator; 28 | 29 | array->elts = ngx_http_graphite_allocator_alloc(allocator, n * size); 30 | if (array->elts == NULL) { 31 | return NGX_ERROR; 32 | } 33 | 34 | return NGX_OK; 35 | } 36 | 37 | void 38 | ngx_http_graphite_array_destroy(ngx_http_graphite_array_t *array) { 39 | ngx_http_graphite_allocator_t *allocator = array->allocator; 40 | ngx_http_graphite_allocator_free(allocator, array->elts); 41 | ngx_http_graphite_allocator_free(allocator, array); 42 | } 43 | 44 | void * 45 | ngx_http_graphite_array_push(ngx_http_graphite_array_t *a) { 46 | return ngx_http_graphite_array_push_n(a, 1); 47 | } 48 | 49 | void * 50 | ngx_http_graphite_array_push_n(ngx_http_graphite_array_t *array, ngx_uint_t n) { 51 | 52 | ngx_uint_t new_nelts = array->nelts + n; 53 | if (new_nelts > array->nalloc) { 54 | 55 | ngx_http_graphite_allocator_t *allocator = array->allocator; 56 | ngx_uint_t nalloc = 2 * ((n >= array->nalloc) ? n : array->nalloc); 57 | 58 | void *new = ngx_http_graphite_allocator_alloc(allocator, nalloc * array->size); 59 | if (new == NULL) 60 | return NULL; 61 | 62 | ngx_memcpy(new, array->elts, array->nelts * array->size); 63 | ngx_http_graphite_allocator_free(allocator, array->elts); 64 | array->elts = new; 65 | array->nalloc = nalloc; 66 | } 67 | 68 | void *elt = (u_char*)array->elts + array->size * array->nelts; 69 | array->nelts = new_nelts; 70 | 71 | return elt; 72 | } 73 | 74 | ngx_http_graphite_array_t * 75 | ngx_http_graphite_array_copy(ngx_http_graphite_allocator_t *allocator, ngx_http_graphite_array_t *array) { 76 | 77 | ngx_http_graphite_array_t *copy = ngx_http_graphite_array_create(allocator, array->nalloc, array->size); 78 | if (copy == NULL) 79 | return NULL; 80 | if (ngx_http_graphite_array_push_n(copy, array->nelts) == NULL) 81 | return NULL; 82 | ngx_memcpy(copy->elts, array->elts, array->nelts * array->size); 83 | 84 | return copy; 85 | } 86 | -------------------------------------------------------------------------------- /src/ngx_http_graphite_array.h: -------------------------------------------------------------------------------- 1 | #ifndef _NGX_HTTP_GRAPHITE_ARRAY_H_INCLUDED_ 2 | #define _NGX_HTTP_GRAPHITE_ARRAY_H_INCLUDED_ 3 | 4 | #include 5 | #include 6 | #include 7 | #include 8 | 9 | #include "ngx_http_graphite_allocator.h" 10 | 11 | typedef struct ngx_http_graphite_array_s { 12 | void *elts; 13 | ngx_uint_t nelts; 14 | size_t size; 15 | ngx_uint_t nalloc; 16 | ngx_http_graphite_allocator_t *allocator; 17 | } ngx_http_graphite_array_t; 18 | 19 | ngx_http_graphite_array_t *ngx_http_graphite_array_create(ngx_http_graphite_allocator_t *allocator, ngx_uint_t n, size_t size); 20 | ngx_int_t ngx_http_graphite_array_init(ngx_http_graphite_array_t *array, ngx_http_graphite_allocator_t *allocator, ngx_uint_t n, size_t size); 21 | void ngx_http_graphite_array_destroy(ngx_http_graphite_array_t *array); 22 | void *ngx_http_graphite_array_push(ngx_http_graphite_array_t *a); 23 | void *ngx_http_graphite_array_push_n(ngx_http_graphite_array_t *a, ngx_uint_t n); 24 | ngx_http_graphite_array_t *ngx_http_graphite_array_copy(ngx_http_graphite_allocator_t *allocator, ngx_http_graphite_array_t *array); 25 | 26 | #endif 27 | -------------------------------------------------------------------------------- /src/ngx_http_graphite_bsearch.h: -------------------------------------------------------------------------------- 1 | #ifndef _NGX_HTTP_GRAPHITE_BSEARCH_H_INCLUDED_ 2 | #define _NGX_HTTP_GRAPHITE_BSEARCH_H_INCLUDED_ 3 | 4 | #include 5 | 6 | static inline void * 7 | ngx_graphite_bsearch( 8 | const void *key, const void *base, size_t nmemb, size_t size, int (*compar)(const void *, const void *), int *found) { 9 | 10 | size_t l = 0, r = nmemb; 11 | 12 | while (l < r) { 13 | size_t m = l + (r - l) / 2; 14 | 15 | int c = compar(key, (char*)base + m * size); 16 | if (c < 0) 17 | r = m; 18 | else if (c > 0) 19 | l = m + 1; 20 | else { 21 | *found = 1; 22 | return (void*)((char*)base + m * size); 23 | } 24 | } 25 | 26 | *found = 0; 27 | return (void*)((char*)base + l * size); 28 | } 29 | 30 | #endif 31 | -------------------------------------------------------------------------------- /src/ngx_http_graphite_module.h: -------------------------------------------------------------------------------- 1 | #ifndef _NGX_HTTP_GRAPHITE_MODULE_H_INCLUDED_ 2 | #define _NGX_HTTP_GRAPHITE_MODULE_H_INCLUDED_ 3 | 4 | #include 5 | #include 6 | #include 7 | #include 8 | 9 | #include "ngx_http_graphite_allocator.h" 10 | #include "ngx_http_graphite_array.h" 11 | 12 | typedef struct ngx_http_graphite_storage_s { 13 | time_t start_time; 14 | time_t last_time; 15 | time_t event_time; 16 | 17 | ngx_atomic_t rwlock; 18 | 19 | ngx_uint_t max_interval; 20 | 21 | ngx_http_graphite_allocator_t *allocator; 22 | 23 | ngx_http_graphite_array_t *metrics; 24 | ngx_http_graphite_array_t *gauges; 25 | ngx_http_graphite_array_t *statistics; 26 | 27 | ngx_http_graphite_array_t *params; 28 | ngx_http_graphite_array_t *internals; 29 | 30 | } ngx_http_graphite_storage_t; 31 | 32 | typedef struct { 33 | struct sockaddr *sockaddr; 34 | socklen_t socklen; 35 | ngx_str_t name; 36 | } ngx_http_graphite_server_t; 37 | 38 | typedef struct { 39 | 40 | ngx_uint_t enable; 41 | 42 | ngx_str_t host; 43 | ngx_str_t protocol; 44 | #ifdef NGX_LOG_LIMIT_ENABLED 45 | ngx_str_t error_log; 46 | #endif 47 | ngx_shm_zone_t *shared; 48 | ngx_buf_t buffer; 49 | 50 | ngx_str_t prefix; 51 | 52 | ngx_http_graphite_server_t server; 53 | int port; 54 | ngx_uint_t frequency; 55 | 56 | ngx_array_t *sources; 57 | ngx_array_t *intervals; 58 | ngx_array_t *splits; 59 | 60 | ngx_uint_t timeout; 61 | 62 | ngx_array_t *default_params; 63 | 64 | size_t shared_size; 65 | size_t buffer_size; 66 | size_t package_size; 67 | 68 | ngx_array_t *template; 69 | 70 | ngx_array_t *default_data_template; 71 | ngx_array_t *default_data_params; 72 | ngx_http_complex_value_t *default_data_filter; 73 | 74 | ngx_array_t *datas; 75 | ngx_array_t *internal_values; 76 | #ifdef NGX_LOG_LIMIT_ENABLED 77 | ngx_array_t *logs; 78 | #endif 79 | 80 | ngx_http_graphite_storage_t *storage; 81 | 82 | ngx_connection_t *connection; 83 | 84 | } ngx_http_graphite_main_conf_t; 85 | 86 | typedef struct ngx_http_graphite_link ngx_http_graphite_link_t; 87 | 88 | const ngx_http_graphite_link_t *ngx_http_graphite_link(ngx_http_request_t *r, const ngx_str_t *name, const ngx_str_t *config); 89 | 90 | ngx_int_t ngx_http_graphite(ngx_http_request_t *r, const ngx_str_t *name, double value, const ngx_str_t *config); 91 | ngx_int_t ngx_http_graphite_by_link(ngx_http_request_t *r, const ngx_http_graphite_link_t *link, double value); 92 | 93 | double ngx_http_graphite_get(ngx_http_request_t *r, const ngx_str_t *name); 94 | double ngx_http_graphite_get_by_link(ngx_http_request_t *r, const ngx_http_graphite_link_t *link); 95 | ngx_int_t ngx_http_graphite_set(ngx_http_request_t *r, const ngx_str_t *name, double value); 96 | ngx_int_t ngx_http_graphite_set_by_link(ngx_http_request_t *r, const ngx_http_graphite_link_t *link, double value); 97 | 98 | #endif 99 | -------------------------------------------------------------------------------- /src/ngx_http_graphite_net.c: -------------------------------------------------------------------------------- 1 | #include 2 | #include 3 | #include 4 | #include 5 | 6 | #include "ngx_http_graphite_module.h" 7 | 8 | static ngx_int_t ngx_http_graphite_net_send_tcp(ngx_http_graphite_main_conf_t *gmcf, ngx_log_t *log); 9 | static ngx_int_t ngx_http_graphite_net_send_udp(ngx_http_graphite_main_conf_t *gmcf, ngx_log_t *log); 10 | static ngx_int_t ngx_http_graphite_net_connect_tcp(ngx_http_graphite_main_conf_t *gmcf, ngx_log_t *log); 11 | static ngx_int_t ngx_http_graphite_net_connect_udp(ngx_http_graphite_main_conf_t *gmcf, ngx_log_t *log); 12 | static void ngx_http_graphite_tcp_read(ngx_event_t *rev); 13 | static void ngx_http_graphite_tcp_write(ngx_event_t *wev); 14 | 15 | ngx_int_t 16 | ngx_http_graphite_net_send_buffer(ngx_http_graphite_main_conf_t *gmcf, ngx_log_t *log) { 17 | 18 | ngx_int_t rc = NGX_ERROR; 19 | 20 | if (ngx_strncmp(gmcf->protocol.data, "tcp", 3) == 0) 21 | rc = ngx_http_graphite_net_send_tcp(gmcf, log); 22 | else if (ngx_strncmp(gmcf->protocol.data, "udp", 3) == 0) 23 | rc = ngx_http_graphite_net_send_udp(gmcf, log); 24 | 25 | return rc; 26 | } 27 | 28 | static ngx_int_t 29 | ngx_http_graphite_net_send_tcp(ngx_http_graphite_main_conf_t *gmcf, ngx_log_t *log) 30 | { 31 | 32 | if (gmcf->connection) 33 | return NGX_ERROR; 34 | 35 | ngx_int_t rc = ngx_http_graphite_net_connect_tcp(gmcf, log); 36 | if (rc == NGX_ERROR) 37 | return NGX_ERROR; 38 | 39 | gmcf->connection->data = gmcf; 40 | gmcf->connection->read->handler = ngx_http_graphite_tcp_read; 41 | gmcf->connection->write->handler = ngx_http_graphite_tcp_write; 42 | 43 | ngx_add_timer(gmcf->connection->write, (ngx_msec_t)(gmcf->timeout)); 44 | 45 | if (rc == NGX_OK) 46 | ngx_http_graphite_tcp_write(gmcf->connection->write); 47 | 48 | return NGX_OK; 49 | } 50 | 51 | static ngx_int_t 52 | ngx_http_graphite_net_send_udp(ngx_http_graphite_main_conf_t *gmcf, ngx_log_t *log) 53 | { 54 | 55 | if (gmcf->connection) 56 | return NGX_ERROR; 57 | 58 | if (ngx_http_graphite_net_connect_udp(gmcf, log) != NGX_OK) { 59 | ngx_log_error(NGX_LOG_ERR, log, 0, "graphite connect to %V failed", &gmcf->server.name); 60 | gmcf->connection = NULL; 61 | return NGX_ERROR; 62 | } 63 | 64 | gmcf->connection->data = gmcf; 65 | 66 | ngx_buf_t *b = &gmcf->buffer; 67 | 68 | u_char *part = b->start; 69 | u_char *next = NULL; 70 | u_char *nl = NULL; 71 | 72 | while (*part) { 73 | next = part; 74 | nl = part; 75 | 76 | while ((next = (u_char*)ngx_strchr(next, '\n')) && ((size_t)(next - part) <= gmcf->package_size)) { 77 | nl = next; 78 | next++; 79 | } 80 | if (nl > part) { 81 | ssize_t n = ngx_send(gmcf->connection, part, nl - part + 1); 82 | 83 | if (n == -1) { 84 | ngx_log_error(NGX_LOG_ERR, log, n, "graphite udp send error"); 85 | goto failed; 86 | } 87 | 88 | if (n != nl - part + 1) { 89 | ngx_log_error(NGX_LOG_ERR, log, 0, "graphite udp send incomplete"); 90 | goto failed; 91 | } 92 | } 93 | else { 94 | ngx_log_error(NGX_LOG_ERR, log, 0, "graphite package size too small, need send %z", (size_t)(next - part)); 95 | } 96 | 97 | part = nl + 1; 98 | } 99 | 100 | ngx_close_connection(gmcf->connection); 101 | gmcf->connection = NULL; 102 | 103 | return NGX_OK; 104 | 105 | failed: 106 | 107 | ngx_close_connection(gmcf->connection); 108 | gmcf->connection = NULL; 109 | 110 | return NGX_ERROR; 111 | } 112 | 113 | static ngx_int_t 114 | ngx_http_graphite_net_connect_tcp(ngx_http_graphite_main_conf_t *gmcf, ngx_log_t *log) 115 | { 116 | ngx_socket_t s = ngx_socket(gmcf->server.sockaddr->sa_family, SOCK_STREAM, 0); 117 | 118 | if (s == (ngx_socket_t) -1) { 119 | ngx_log_error(NGX_LOG_ERR, log, ngx_socket_errno, ngx_socket_n " failed"); 120 | return NGX_ERROR; 121 | } 122 | 123 | ngx_connection_t *c = ngx_get_connection(s, log); 124 | 125 | if (c == NULL) { 126 | if (ngx_close_socket(s) == -1) 127 | ngx_log_error(NGX_LOG_ERR, log, ngx_socket_errno, ngx_close_socket_n "failed"); 128 | 129 | return NGX_ERROR; 130 | } 131 | 132 | if (ngx_nonblocking(s) == -1) { 133 | ngx_log_error(NGX_LOG_ERR, log, ngx_socket_errno, ngx_nonblocking_n " failed"); 134 | 135 | goto failed; 136 | } 137 | 138 | ngx_event_t *rev = c->read; 139 | ngx_event_t *wev = c->write; 140 | 141 | rev->log = log; 142 | wev->log = log; 143 | 144 | gmcf->connection = c; 145 | 146 | c->number = ngx_atomic_fetch_add(ngx_connection_counter, 1); 147 | 148 | if (ngx_add_conn) { 149 | if (ngx_add_conn(c) == NGX_ERROR) 150 | goto failed; 151 | } 152 | 153 | int rc = connect(s, gmcf->server.sockaddr, gmcf->server.socklen); 154 | 155 | if (rc == -1) { 156 | ngx_err_t err = ngx_socket_errno; 157 | 158 | if (err != NGX_EINPROGRESS 159 | #if (NGX_WIN32) 160 | && err != NGX_EAGAIN 161 | #endif 162 | ) 163 | { 164 | ngx_log_error(NGX_LOG_ERR, c->log, err, "graphite connect to %V failed", &gmcf->server.name); 165 | 166 | goto failed; 167 | } 168 | } 169 | 170 | if (ngx_add_conn) { 171 | if (rc == -1) 172 | return NGX_AGAIN; 173 | 174 | wev->ready = 1; 175 | return NGX_OK; 176 | } 177 | 178 | if (ngx_event_flags & NGX_USE_IOCP_EVENT) { 179 | 180 | if (ngx_blocking(s) == -1) { 181 | ngx_log_error(NGX_LOG_ERR, log, ngx_socket_errno, ngx_blocking_n " failed"); 182 | goto failed; 183 | } 184 | 185 | rev->ready = 1; 186 | wev->ready = 1; 187 | 188 | return NGX_OK; 189 | } 190 | 191 | ngx_int_t event = (ngx_event_flags & NGX_USE_CLEAR_EVENT) ? NGX_CLEAR_EVENT: NGX_LEVEL_EVENT; 192 | 193 | if (ngx_add_event(rev, NGX_READ_EVENT, event) != NGX_OK) { 194 | goto failed; 195 | } 196 | 197 | if (rc == -1) { 198 | if (ngx_add_event(wev, NGX_WRITE_EVENT, event) != NGX_OK) 199 | goto failed; 200 | return NGX_AGAIN; 201 | } 202 | 203 | wev->ready = 1; 204 | 205 | return NGX_OK; 206 | 207 | failed: 208 | 209 | ngx_close_connection(c); 210 | gmcf->connection = NULL; 211 | 212 | return NGX_ERROR; 213 | } 214 | 215 | static ngx_int_t 216 | ngx_http_graphite_net_connect_udp(ngx_http_graphite_main_conf_t *gmcf, ngx_log_t *log) 217 | { 218 | 219 | ngx_socket_t s = ngx_socket(gmcf->server.sockaddr->sa_family, SOCK_DGRAM, 0); 220 | 221 | if (s == (ngx_socket_t) -1) { 222 | ngx_log_error(NGX_LOG_ERR, log, ngx_socket_errno, ngx_socket_n " failed"); 223 | return NGX_ERROR; 224 | } 225 | 226 | ngx_connection_t *c = ngx_get_connection(s, log); 227 | 228 | if (c == NULL) { 229 | if (ngx_close_socket(s) == -1) { 230 | ngx_log_error(NGX_LOG_ERR, log, ngx_socket_errno, ngx_close_socket_n " failed"); 231 | } 232 | 233 | return NGX_ERROR; 234 | } 235 | 236 | ngx_event_t *rev = c->read; 237 | ngx_event_t *wev = c->write; 238 | 239 | rev->log = log; 240 | wev->log = log; 241 | 242 | gmcf->connection = c; 243 | 244 | c->number = ngx_atomic_fetch_add(ngx_connection_counter, 1); 245 | 246 | int rc = connect(s, gmcf->server.sockaddr, gmcf->server.socklen); 247 | 248 | if (rc == -1) { 249 | ngx_log_error(NGX_LOG_ERR, log, ngx_socket_errno, "connect failed"); 250 | goto failed; 251 | } 252 | 253 | wev->ready = 1; 254 | 255 | ngx_int_t event = (ngx_event_flags & NGX_USE_CLEAR_EVENT) ? NGX_CLEAR_EVENT: NGX_LEVEL_EVENT; 256 | 257 | if (ngx_add_event(rev, NGX_READ_EVENT, event) != NGX_OK) 258 | goto failed; 259 | 260 | return NGX_OK; 261 | 262 | failed: 263 | 264 | ngx_close_connection(c); 265 | 266 | return NGX_ERROR; 267 | } 268 | 269 | static void 270 | ngx_http_graphite_tcp_read(ngx_event_t *rev) 271 | { 272 | } 273 | 274 | static void 275 | ngx_http_graphite_tcp_write(ngx_event_t *wev) 276 | { 277 | ngx_connection_t *c = wev->data; 278 | ngx_http_graphite_main_conf_t *gmcf = (ngx_http_graphite_main_conf_t*)(c->data); 279 | ngx_buf_t *b = &gmcf->buffer; 280 | 281 | if (wev->timedout) { 282 | ngx_log_error(NGX_LOG_ERR, wev->log, 0, "graphite tcp connection timeout"); 283 | goto failed; 284 | } 285 | 286 | off_t sent = c->sent; 287 | 288 | while (wev->ready && b->pos < b->last) { 289 | ssize_t n = ngx_send(c, b->pos, b->last - b->pos); 290 | 291 | if (n == NGX_AGAIN) 292 | break; 293 | 294 | if (n == NGX_ERROR) { 295 | ngx_log_error(NGX_LOG_ERR, wev->log, 0, "graphite tcp send error"); 296 | goto failed; 297 | } 298 | 299 | b->pos += n; 300 | } 301 | 302 | if (b->pos != b->start) { 303 | b->last = ngx_movemem(b->start, b->pos, b->last - b->pos); 304 | b->pos = b->start; 305 | } 306 | 307 | if (b->last - b->pos == 0) { 308 | ngx_close_connection(c); 309 | gmcf->connection = NULL; 310 | return; 311 | } 312 | 313 | if (c->sent != sent) 314 | ngx_add_timer(wev, (ngx_msec_t)(gmcf->timeout)); 315 | 316 | if (ngx_handle_write_event(wev, 0) != NGX_OK) 317 | goto failed; 318 | 319 | return; 320 | 321 | failed: 322 | ngx_close_connection(c); 323 | gmcf->connection = NULL; 324 | } 325 | -------------------------------------------------------------------------------- /src/ngx_http_graphite_net.h: -------------------------------------------------------------------------------- 1 | #ifndef _NGX_HTTP_GRAPHITE_NET_H_INCLUDED_ 2 | #define _NGX_HTTP_GRAPHITE_NET_H_INCLUDED_ 3 | 4 | #include 5 | #include 6 | #include 7 | #include 8 | 9 | #include "ngx_http_graphite_module.h" 10 | 11 | ngx_int_t ngx_http_graphite_net_send_buffer(ngx_http_graphite_main_conf_t *gmcf, ngx_log_t *log); 12 | 13 | #endif 14 | --------------------------------------------------------------------------------