├── LICENSE ├── README.markdown ├── conf └── nginx.conf ├── logging.lua └── logs └── .gitignore /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright 2016 Matthieu Tourne 2 | 3 | Licensed under the Apache License, Version 2.0 (the "License"); 4 | you may not use this file except in compliance with the License. 5 | You may obtain a copy of the License at 6 | 7 | http://www.apache.org/licenses/LICENSE-2.0 8 | 9 | Unless required by applicable law or agreed to in writing, software 10 | distributed under the License is distributed on an "AS IS" BASIS, 11 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 12 | See the License for the specific language governing permissions and 13 | limitations under the License. 14 | -------------------------------------------------------------------------------- /README.markdown: -------------------------------------------------------------------------------- 1 | This is an example for embedding Lua into the logging phase of Nginx and building a simple log aggregation service. 2 | 3 | Installation 4 | ============ 5 | 6 | One of the only dependencies not directly included in OpenResty is libpcre. 7 | You can either install it from source, or use your distribution. 8 | A modern version (>8.21) is recommended to leverage PCRE_JIT (faster regex), but it will still work with an older version. 9 | 10 | In my case on Mac OS using [Homebrew](http://mxcl.github.com/homebrew/) : 11 | 12 | $ brew install pcre 13 | 14 | 15 | Then download the latest OpenResty from this [page](http://openresty.org/#Download) 16 | and compile it. 17 | More up do date installation intruction can be found on the [Openresty install page](http://openresty.org/#Installation) 18 | 19 | $ wget http://agentzh.org/misc/nginx/ngx_openresty-1.2.4.11.tar.gz 20 | $ tar xvzf ngx_openresty-1.2.4.11.tar.gz 21 | $ cd ngx_openresty-1.2.4.11 22 | 23 | Now let's finally compile it. 24 | I found a slight issue that only happens on Mac OS while using hombrew, and I had to the configure line where to find my pcre library (`--with-ld-opt="-L/usr/local/lib"`) 25 | This should get fixed in the next releases 26 | 27 | $ ./configure --with-luajit --with-ld-opt="-L/usr/local/lib" 28 | $ make 29 | 30 | This will create a /usr/local/openresty/ directory that contains everything you need 31 | 32 | $ make install 33 | 34 | 35 | Launching the demo server 36 | ========================= 37 | 38 | Let's check that the Nginx we just compiled has all the necessary to run this example 39 | 40 | $ /usr/local/openresty/nginx/sbin/nginx -V 41 | 42 | This will display all the module that Nginx has been built with, including ngx_lua-0.7.5 here 43 | (the dev version of ngx_lua can be found [here](https://github.com/chaoslawful/lua-nginx-module)) 44 | 45 | configure arguments: --prefix=/usr/local/openresty/nginx --add-module=../ngx_devel_kit-0.2.17 --add-module=../echo-nginx-module-0.41 --add-module=../xss-nginx-module-0.03rc9 --add-module=../ngx_coolkit-0.2rc1 --add-module=../set-misc-nginx-module-0.22rc8 --add-module=../form-input-nginx-module-0.07rc5 --add-module=../encrypted-session-nginx-module-0.02 --add-module=../srcache-nginx-module-0.16 --add-module=../ngx_lua-0.7.5 --add-module=../headers-more-nginx-module-0.19 --add-module=../array-var-nginx-module-0.03rc1 --add-module=../memc-nginx-module-0.13rc3 --add-module=../redis2-nginx-module-0.09 --add-module=../redis-nginx-module-0.3.6 --add-module=../auth-request-nginx-module-0.2 --add-module=../rds-json-nginx-module-0.12rc10 --add-module=../rds-csv-nginx-module-0.05rc2 --with-http_ssl_module 46 | 47 | Now let's run Nginx with the configuration example. 48 | (You don't need to run it as root, as the example doesn't listen on port 80) 49 | 50 | $ git clone https://github.com/mtourne/nginx_log_by_lua.git 51 | 52 | The server will run until you send you hit Control+C 53 | 54 | $ /usr/local/openresty/nginx/sbin/nginx -p nginx_log_by_lua 55 | 56 | 57 | Testing it 58 | ========== 59 | 60 | $ curl localhost:8080 61 | 62 | If everything went right, you should see the sever response : 63 | 64 | Hello 65 | 66 | 67 | Now, let's query our log aggregation port : 68 | 69 | $ curl localhost:6080 70 | 71 | And here is the result : 72 | 73 | Since last measure: 1.0729999542236 secs 74 | Request Count: 1 75 | Average req time: 0 secs 76 | Requests per Secs: 0.93196648896742 77 | 78 | 79 | As a final test you can try hammering your web application and see the log aggregation in action : 80 | 81 | ab -n 1000 -c 100 http://localhost:8080/ > /dev/null && echo && curl http://localhost:6080/ 82 | 83 | Completed 100 requests 84 | Completed 200 requests 85 | Completed 300 requests 86 | Completed 400 requests 87 | Completed 500 requests 88 | Completed 600 requests 89 | Completed 700 requests 90 | Completed 800 requests 91 | Completed 900 requests 92 | Completed 1000 requests 93 | Finished 1000 requests 94 | 95 | Since last measure: 0.10899996757507 secs 96 | Request Count: 1000 97 | Average req time: 0.0012160038948059 secs 98 | Requests per Secs: 9174.3146557475 99 | 100 | 101 | Beyond the example 102 | ================== 103 | 104 | For the purpose of the example the code is very simple, but with very simple additions a json printer could be added to make the output easier to consume by other services (OpenResty does include the package [lua-cjson](http://www.kyne.com.au/~mark/software/lua-cjson.php)) 105 | 106 | We could also push the example further and aggregate logs to calculate timings similarly to `ab`, which displays what is the average request_time for the 90, 95, 99 percentile. 107 | 108 | Happy Hacking! 109 | 110 | -------------------------------------------------------------------------------- /conf/nginx.conf: -------------------------------------------------------------------------------- 1 | daemon off; 2 | 3 | worker_processes 2; 4 | 5 | error_log /tmp/nginx_error.log warn; 6 | 7 | events { 8 | } 9 | 10 | http { 11 | # Lua configuration 12 | lua_package_path "$prefix/?.lua;;"; 13 | lua_shared_dict log_dict 1M; 14 | 15 | server { 16 | listen 8080; 17 | 18 | location / { 19 | # Replace with your web application 20 | content_by_lua " 21 | ngx.say('Hello'); 22 | "; 23 | 24 | 25 | # this logging code can be added to any existing nginx.conf 26 | log_by_lua ' 27 | local logging = require("logging") 28 | 29 | local request_time = ngx.now() - ngx.req.start_time() 30 | logging.add_plot(ngx.shared.log_dict, "request_time", request_time) 31 | '; 32 | } 33 | } 34 | 35 | # log server - print the log generated by our web application 36 | server { 37 | 38 | listen 127.0.0.1:6080; 39 | 40 | location / { 41 | content_by_lua ' 42 | local logging = require("logging") 43 | 44 | local count, avg, elapsed_time = 45 | logging.get_plot(ngx.shared.log_dict, "request_time") 46 | 47 | local qps = 0 48 | 49 | ngx.say("Since last measure:\t", elapsed_time, " secs") 50 | ngx.say("Request Count:\t\t", count) 51 | ngx.say("Average req time:\t", avg, " secs") 52 | if elapsed_time > 0 then 53 | qps = count / elapsed_time 54 | end 55 | ngx.say("Requests per Secs:\t", qps) 56 | '; 57 | } 58 | } 59 | } -------------------------------------------------------------------------------- /logging.lua: -------------------------------------------------------------------------------- 1 | -- Copyright (C) 2012 Matthieu Tourne 2 | -- @author Matthieu Tourne 3 | 4 | -- Simple helper functions for logging 5 | 6 | local logging = {} 7 | 8 | local module = logging 9 | 10 | local function incr(dict, key, increment) 11 | increment = increment or 1 12 | local newval, err = dict:incr(key, increment) 13 | if err then 14 | dict:set(key, increment) 15 | newval = increment 16 | end 17 | return newval 18 | end 19 | 20 | function logging.add_plot(dict, key, value) 21 | local sum_key = key .. "-sum" 22 | local count_key = key .. "-count" 23 | local start_time_key = key .. "-start_time" 24 | 25 | local start_time = dict:get(start_time_key) 26 | if not start_time then 27 | ngx.log(ngx.ERR, 'now: ', ngx.now()) 28 | dict:set(start_time_key, ngx.now()) 29 | end 30 | 31 | local sum = incr(dict, sum_key, value) 32 | incr(dict, count_key) 33 | end 34 | 35 | function logging.get_plot(dict, key) 36 | local sum_key = key .. "-sum" 37 | local count_key = key .. "-count" 38 | local start_time_key = key .. "-start_time" 39 | 40 | local elapsed_time = 0 41 | local avg = 0 42 | 43 | local start_time = dict:get(start_time_key) 44 | if start_time then 45 | elapsed_time = ngx.now() - start_time 46 | end 47 | dict:delete(start_time_key) 48 | 49 | local count = dict:get(count_key) or 0 50 | dict:delete(count_key) 51 | 52 | local sum = dict:get(sum_key) or 0 53 | dict:delete(sum_key) 54 | 55 | if count > 0 then 56 | avg = sum / count 57 | end 58 | 59 | return count, avg, elapsed_time 60 | end 61 | 62 | 63 | -- safety net 64 | local module_mt = { 65 | __newindex = ( 66 | function (table, key, val) 67 | error('Attempt to write to undeclared variable "' .. key .. '"') 68 | end), 69 | } 70 | 71 | setmetatable(module, module_mt) 72 | 73 | -- expose the module 74 | return logging 75 | -------------------------------------------------------------------------------- /logs/.gitignore: -------------------------------------------------------------------------------- 1 | # Nginx needs a logs/ directory to start 2 | # Ignore everything in this directory 3 | * 4 | # Except this file 5 | !.gitignore --------------------------------------------------------------------------------