├── .gitignore ├── BuildOpenResty.md ├── README.md ├── Snapshots.md ├── docker-compose.yml ├── docker ├── backend │ ├── Dockerfile │ ├── build.sh │ └── push.sh ├── grafana │ ├── Dockerfile │ ├── build.sh │ ├── datasource.yml │ ├── grafana.ini │ ├── nginx-api-dashboard.json │ ├── nginx-api-host-endpoint-url.json │ ├── nginx-api-host-endpoint.json │ ├── nginx-api-host.json │ ├── provisioning.yml │ └── push.sh ├── nginx │ ├── Dockerfile │ ├── buid.sh │ └── push.sh └── prometheus │ ├── Dockerfile │ ├── build.sh │ ├── nginx.alert.rule │ ├── nginx.counter.rule │ ├── nginx.errrate.rule │ ├── nginx.latency.rule │ ├── nginx.qps.rule │ ├── prometheus.yml │ └── push.sh ├── gitpush.sh ├── pictures ├── resty-lua-prometheus-snapshot01.png ├── resty-lua-prometheus-snapshot02.png ├── resty-lua-prometheus-snapshot03.png └── resty-lua-prometheus-snapshot04.png └── workdir ├── conf.d ├── backend.conf └── counter.conf ├── html ├── 50x.html └── index.html ├── lua ├── counter.lua └── prometheus.lua └── nginx.conf /.gitignore: -------------------------------------------------------------------------------- 1 | *.jar 2 | workdir/client_body_temp 3 | workdir/fastcgi_temp 4 | workdir/logs 5 | workdir/proxy_temp 6 | workdir/scgi_temp 7 | workdir/uwsgi_temp 8 | -------------------------------------------------------------------------------- /BuildOpenResty.md: -------------------------------------------------------------------------------- 1 | ### Build OpenResty from source 2 | ```shell 3 | su root 4 | apt install libpcre3-dev libssl-dev perl make build-essential curl -y 5 | 6 | export RESTY_HOME=/opt/openresty 7 | mkdir -p $RESTY_HOME/build 8 | cd $RESTY_HOME 9 | 10 | # build openresty 11 | wget https://openresty.org/download/openresty-1.13.6.2.tar.gz 12 | tar -xvf openresty-1.13.6.2.tar.gz 13 | cd openresty-1.13.6.2 14 | ./configure --prefix=$RESTY_HOME/build -j8 15 | make -j8 16 | make install 17 | $RESTY_HOME/build/nginx/sbin/nginx -V 18 | nginx version: openresty/1.13.6.2 19 | built by gcc 8.3.0 (Ubuntu 8.3.0-6ubuntu1) 20 | built with OpenSSL 1.1.1b 26 Feb 2019 21 | TLS SNI support enabled 22 | configure arguments: --prefix=/opt/openresty/build/nginx --with-cc-opt=-O2 ... --with-http_ssl_module 23 | $RESTY_HOME/build/nginx/sbin/nginx 24 | curl localhost 25 | 26 | ... 27 |

Thank you for flying OpenResty.

28 | ... 29 | 30 | pkill nginx 31 | ``` 32 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # prometheus-lua-nginx 2 | ![GitHub stars](https://img.shields.io/github/stars/zrbcool/prometheus-lua-nginx.svg?style=social) 3 | ![GitHub followers](https://img.shields.io/github/followers/zrbcool.svg?style=social) 4 | ## Overview 5 | Project **prometheus-lua-nginx** is a complete solution for openresty based api gateway monitoring, **prometheus-lua-nginx** use prometheus lua lib and openresty http request life cycle to inject monitoring code, it is async, high performance, and business awareness, it also have built-in grafana dashboard out of box, please see [Snapshots](https://github.com/zrbcool/prometheus-lua-nginx/blob/master/Snapshots.md) 6 | 7 | ## Concept 8 | How to measure your API performance? how stable your service are? 9 | 10 | The answer is metrics monitoring: 11 | 12 | From last one years, our best practise is the following metrics and dimensions 13 | - P99(latency percent 99%), P999, P90 this most important metrics, it will impact your users expirence about your apis. 14 | - Error rate 15 | - QPS 16 | 17 | Dimensions: 18 | - host, every web site use virtual host to isolate sub business systems 19 | - endpoint, endpoint is the root access point of your api like /myapi 20 | - fullurl, this is the detail of one specific api, not aggrated data, it will help you find out the service quality trends 21 | - SLA， correct response/all request * 100% during a timewindow 22 | - statuscode, method, scheme 23 | 24 | **prometheus-lua-nginx** use Prometheus to collect request latency metrics, below is the data model 25 | 26 | `{=, ...}` 27 | 28 | for example: 29 | ```yaml 30 | nginx_http_request_duration_seconds_bucket{host="demo1.coohua.com",status="200",scheme="http",method="GET",endpoint="/service",fullurl="/service/latency",le="00.010"} 42 31 | nginx_http_request_duration_seconds_bucket{host="demo1.coohua.com",status="200",scheme="http",method="GET",endpoint="/service",fullurl="/service/latency",le="00.020"} 249 32 | nginx_http_request_duration_seconds_bucket{host="demo1.coohua.com",status="200",scheme="http",method="GET",endpoint="/service",fullurl="/service/latency",le="00.030"} 249 33 | nginx_http_request_duration_seconds_bucket{host="demo1.coohua.com",status="200",scheme="http",method="GET",endpoint="/service",fullurl="/service/latency",le="00.050"} 249 34 | nginx_http_request_duration_seconds_bucket{host="demo1.coohua.com",status="200",scheme="http",method="GET",endpoint="/service",fullurl="/service/latency",le="00.075"} 249 35 | nginx_http_request_duration_seconds_bucket{host="demo1.coohua.com",status="200",scheme="http",method="GET",endpoint="/service",fullurl="/service/latency",le="00.100"} 249 36 | nginx_http_request_duration_seconds_bucket{host="demo1.coohua.com",status="200",scheme="http",method="GET",endpoint="/service",fullurl="/service/latency",le="00.200"} 249 37 | nginx_http_request_duration_seconds_bucket{host="demo1.coohua.com",status="200",scheme="http",method="GET",endpoint="/service",fullurl="/service/latency",le="00.300"} 249 38 | nginx_http_request_duration_seconds_bucket{host="demo1.coohua.com",status="200",scheme="http",method="GET",endpoint="/service",fullurl="/service/latency",le="00.400"} 249 39 | nginx_http_request_duration_seconds_bucket{host="demo1.coohua.com",status="200",scheme="http",method="GET",endpoint="/service",fullurl="/service/latency",le="00.500"} 249 40 | nginx_http_request_duration_seconds_bucket{host="demo1.coohua.com",status="200",scheme="http",method="GET",endpoint="/service",fullurl="/service/latency",le="00.750"} 249 41 | ... 42 | nginx_http_request_duration_seconds_bucket{host="demo1.coohua.com",status="499",scheme="http",method="GET",endpoint="/service",fullurl="/service/latency",le="+Inf"} 1 43 | nginx_http_request_duration_seconds_count{host="demo1.coohua.com",status="200",scheme="http",method="GET",endpoint="/service",fullurl="/service/latency"} 249 44 | nginx_http_request_duration_seconds_count{host="demo1.coohua.com",status="499",scheme="http",method="GET",endpoint="/service",fullurl="/service/latency"} 1 45 | nginx_http_request_duration_seconds_sum{host="demo1.coohua.com",status="200",scheme="http",method="GET",endpoint="/service",fullurl="/service/latency"} 2.82 46 | nginx_http_request_duration_seconds_sum{host="demo1.coohua.com",status="499",scheme="http",method="GET",endpoint="/service",fullurl="/service/latency"} 0.008 47 | ``` 48 | and then we use grafana to setup a dashboard, and add a prometheus datasource, then use prometheus functions to get the calculate metrics like P99, P999, SLA, ERR rate, etc. 49 | ## Architecture 50 | 51 | ## Features 52 | - Base on promethues pull mode to collect metrics 53 | - Multiple dimension latency metrics collection(host,status code,schema,etc.) 54 | - Out-of-box grafana dashboards 55 | - LUA based latency metrics collection, high performance, low memory use, async 56 | ## Getting started 57 | ### Quickly start up a demo by docker-compose 58 | ```shell 59 | # please instal docker-ce, docker-compose correctly 60 | docker-compose up 61 | ``` 62 | this compose will install a set of containers for a demo use: 63 | - a grafana has built in dashboards 64 | - a backend services for mock latency and the real workload 65 | - a openresty as proxy with counter.lua and promethues.lua integrated, can collect latency metrics, and expose them by :9145/metrics endpoint 66 | - a promethues has pre configured previous oprenresty as scrape target 67 | 68 | how to access: 69 | - access grafana by [hostip]:3000 70 | - access promethues by [hostip]:9090 71 | - access openresty exposed metrics by [hostip]:9145/metrics 72 | - access [hostip]:80/service/latency/[latency]/bytes/[bytes] to mock a api call, 73 | > for instance -> curl -H "host:anyhost.com" [hostip]:80/service/latency/10/bytes/10 74 | > or use ab to simulate real workload like: ab -n 10 -H "host:anyhost.com" officepc:80/service/latency/10/bytes/10 75 | - check the dashboard 76 | ![](http://oss.zrbcool.top/picgo/20190706151826.png) 77 | ## Building 78 | ### About Demo build 79 | You should aware that the demo is only for demo use, if you want to use it in your production environment, you need to completely know how the demo and the solution runs. If you are ready, please see ./docker/xxx for more details 80 | ### Build OpenResty from source and integrate LUA metrics collection code 81 | - build OpenResty from source, please follow [BuildOpenResty](https://github.com/zrbcool/prometheus-lua-nginx/blob/master/BuildOpenResty.md) 82 | - copy workdir/conf.d/counter.conf to your conf.d dir 83 | - copy workdir/lua/prometheus.lua to your openresty lib dir 84 | - change lua_package_path in counter.conf to define lib location 85 | - copy /workdir/lua/counter.lua to your openresty lua dir 86 | here is our workdir structure 87 | ```shell 88 | workdir 89 | ├── conf.d 90 | │ ├── backend.conf # backend service config, replace it by your real workload 91 | │ └── counter.conf # define lua-prometheus related vars and import deps 92 | ├── html 93 | │ ├── 50x.html # demo use static html 94 | │ └── index.html # demo use static html 95 | ├── lua 96 | │ ├── counter.lua # metric collection code 97 | │ └── prometheus.lua # prometheus.lua is offical Prometheus LUA Library you can upgrade it from https://github.com/knyar/nginx-lua-prometheus 98 | └── nginx.conf # main config 99 | ``` 100 | - if you just want run current project, please follow: 101 | ```java 102 | export RESTY_HOME=/opt/openresty 103 | cd $RESTY_HOME 104 | git clone https://github.com/zrbcool/prometheus-lua-nginx.git 105 | ln -s $RESTY_HOME/prometheus-lua-nginx/workdir $RESTY_HOME/workdir 106 | mkdir $RESTY_HOME/workdir/logs 107 | cd $RESTY_HOME 108 | git clone https://github.com/knyar/nginx-lua-prometheus.git 109 | cp nginx-lua-prometheus/prometheus.lua workdir/lua 110 | $RESTY_HOME/build/nginx/sbin/nginx \ 111 | -p $RESTY_HOME/workdir \ 112 | -c $RESTY_HOME/workdir/nginx.conf 113 | 114 | curl localhost:8080 115 |

hello, world

116 | curl localhost:9145/metrics 117 | # HELP nginx_metric_errors_total Number of nginx-lua-prometheus errors 118 | # TYPE nginx_metric_errors_total counter 119 | nginx_metric_errors_total 0 120 | pkill nginx 121 | ``` 122 | ## Contact 123 | ### E-Mail 124 | - zhangrongbincool@163.com 125 | - zhangrongbincool@gmail.com 126 | - zhangrongbin@coohua.com 127 | ### Blog 128 | [blog.zrbcool.top](http://blog.zrbcool.top) 129 | ### Other Topics 130 | [github](https://github.com/zrbcool/blog-public) 131 | -------------------------------------------------------------------------------- /Snapshots.md: -------------------------------------------------------------------------------- 1 | ## Snapshots 2 | ### all hosts overall 3 | ![](https://github.com/zrbcool/prometheus-lua-nginx/blob/master/pictures/resty-lua-prometheus-snapshot01.png) 4 | ### drill down to one specific host dashboard 5 | ![](https://github.com/zrbcool/prometheus-lua-nginx/blob/master/pictures/resty-lua-prometheus-snapshot02.png) 6 | ### drill down to one specific endpoint dashboard 7 | ![](https://github.com/zrbcool/prometheus-lua-nginx/blob/master/pictures/resty-lua-prometheus-snapshot03.png) 8 | ### drill down to one specific URL(http api) dashboard 9 | ![](https://github.com/zrbcool/prometheus-lua-nginx/blob/master/pictures/resty-lua-prometheus-snapshot04.png) 10 | 11 | -------------------------------------------------------------------------------- /docker-compose.yml: -------------------------------------------------------------------------------- 1 | version: '3' 2 | services: 3 | 4 | prom: 5 | image: "zrbcool/prometheus-lua-prom:v1.0" 6 | ports: 7 | - "9090:9090" 8 | 9 | grafana: 10 | image: "zrbcool/prometheus-lua-grafana:v1.0" 11 | ports: 12 | - "3000:3000" 13 | 14 | resty: 15 | image: "zrbcool/prometheus-lua-nginx:v1.0" 16 | ports: 17 | - "80:80" 18 | - "9145:9145" 19 | 20 | backend: 21 | image: "zrbcool/prometheus-lua-backend:latest" 22 | ports: 23 | - "8081:8081" 24 | -------------------------------------------------------------------------------- /docker/backend/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM openjdk:8-jdk-alpine 2 | VOLUME /tmp 3 | ARG JAR_FILE=caf-sample-web-tomcat-1.7-SNAPSHOT.jar 4 | COPY ${JAR_FILE} app.jar 5 | ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"] 6 | -------------------------------------------------------------------------------- /docker/backend/build.sh: -------------------------------------------------------------------------------- 1 | docker build -t zrbcool/prometheus-lua-backend:latest -f Dockerfile . 2 | -------------------------------------------------------------------------------- /docker/backend/push.sh: -------------------------------------------------------------------------------- 1 | docker tag prometheus-lua-grafana:latest zrbcool/prometheus-lua-grafana:latest 2 | docker push zrbcool/prometheus-lua-grafana:latest 3 | 4 | docker tag prometheus-lua-nginx:latest zrbcool/prometheus-lua-nginx:latest 5 | docker push zrbcool/prometheus-lua-nginx:latest 6 | 7 | docker tag prometheus-lua-backend:latest zrbcool/prometheus-lua-backend:latest 8 | docker push zrbcool/prometheus-lua-backend:latest 9 | -------------------------------------------------------------------------------- /docker/grafana/Dockerfile: -------------------------------------------------------------------------------- 1 | FROM grafana/grafana 2 | 3 | COPY grafana.ini /etc/grafana/grafana.ini 4 | COPY provisioning.yml /etc/grafana/provisioning/dashboards/provisioning.yml 5 | COPY datasource.yml /etc/grafana/provisioning/datasources/datasource.yml 6 | 7 | RUN mkdir /var/lib/grafana/dashboards 8 | 9 | COPY nginx-api-dashboard.json /var/lib/grafana/dashboards/nginx-api-dashboard.json 10 | COPY nginx-api-host-endpoint.json /var/lib/grafana/dashboards/nginx-api-host-endpoint.json 11 | COPY nginx-api-host-endpoint-url.json /var/lib/grafana/dashboards/nginx-api-host-endpoint-url.json 12 | COPY nginx-api-host.json /var/lib/grafana/dashboards/nginx-api-host.json 13 | 14 | -------------------------------------------------------------------------------- /docker/grafana/build.sh: -------------------------------------------------------------------------------- 1 | docker build -t zrbcool/prometheus-lua-grafana:latest -f Dockerfile . 2 | -------------------------------------------------------------------------------- /docker/grafana/datasource.yml: -------------------------------------------------------------------------------- 1 | apiVersion: 1 2 | 3 | deleteDatasources: 4 | - name: Prometheus 5 | orgId: 1 6 | 7 | datasources: 8 | - name: Prometheus-Aggregator 9 | type: prometheus 10 | access: proxy 11 | orgId: 1 12 | url: http://prom:9090 13 | isDefault: true 14 | jsonData: 15 | tlsAuth: false 16 | tlsAuthWithCACert: false 17 | # json object of data that will be encrypted. 18 | secureJsonData: 19 | version: 1 20 | editable: false 21 | 22 | -------------------------------------------------------------------------------- /docker/grafana/grafana.ini: -------------------------------------------------------------------------------- 1 | ##################### Grafana Configuration Defaults ##################### 2 | # 3 | # Do not modify this file in grafana installs 4 | # 5 | 6 | # possible values : production, development 7 | app_mode = production 8 | 9 | # instance name, defaults to HOSTNAME environment variable value or hostname if HOSTNAME var is empty 10 | instance_name = ${HOSTNAME} 11 | 12 | #################################### Paths ############################### 13 | [paths] 14 | # Path to where grafana can store temp files, sessions, and the sqlite3 db (if that is used) 15 | data = data 16 | 17 | # Temporary files in `data` directory older than given duration will be removed 18 | temp_data_lifetime = 24h 19 | 20 | # Directory where grafana can store logs 21 | logs = data/log 22 | 23 | # Directory where grafana will automatically scan and look for plugins 24 | plugins = data/plugins 25 | 26 | # folder that contains provisioning config files that grafana will apply on startup and while running. 27 | provisioning = conf/provisioning 28 | 29 | #################################### Server ############################## 30 | [server] 31 | # Protocol (http, https, socket) 32 | protocol = http 33 | 34 | # The ip address to bind to, empty will bind to all interfaces 35 | http_addr = 36 | 37 | # The http port to use 38 | http_port = 3000 39 | 40 | # The public facing domain name used to access grafana from a browser 41 | domain = localhost 42 | 43 | # Redirect to correct domain if host header does not match domain 44 | # Prevents DNS rebinding attacks 45 | enforce_domain = false 46 | 47 | # The full public facing url 48 | root_url = %(protocol)s://%(domain)s:%(http_port)s/ 49 | 50 | # Log web requests 51 | router_logging = false 52 | 53 | # the path relative working path 54 | static_root_path = public 55 | 56 | # enable gzip 57 | enable_gzip = false 58 | 59 | # https certs & key file 60 | cert_file = 61 | cert_key = 62 | 63 | # Unix socket path 64 | socket = /tmp/grafana.sock 65 | 66 | #################################### Database ############################ 67 | [database] 68 | # You can configure the database connection by specifying type, host, name, user and password 69 | # as separate properties or as on string using the url property. 70 | 71 | # Either "mysql", "postgres" or "sqlite3", it's your choice 72 | type = sqlite3 73 | host = 127.0.0.1:3306 74 | name = grafana 75 | user = root 76 | # If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;""" 77 | password = 78 | # Use either URL or the previous fields to configure the database 79 | # Example: mysql://user:secret@host:port/database 80 | url = 81 | 82 | # Max idle conn setting default is 2 83 | max_idle_conn = 2 84 | 85 | # Max conn setting default is 0 (mean not set) 86 | max_open_conn = 87 | 88 | # Connection Max Lifetime default is 14400 (means 14400 seconds or 4 hours) 89 | conn_max_lifetime = 14400 90 | 91 | # Set to true to log the sql calls and execution times. 92 | log_queries = 93 | 94 | # For "postgres", use either "disable", "require" or "verify-full" 95 | # For "mysql", use either "true", "false", or "skip-verify". 96 | ssl_mode = disable 97 | 98 | ca_cert_path = 99 | client_key_path = 100 | client_cert_path = 101 | server_cert_name = 102 | 103 | # For "sqlite3" only, path relative to data_path setting 104 | path = grafana.db 105 | 106 | # For "sqlite3" only. cache mode setting used for connecting to the database 107 | cache_mode = private 108 | 109 | #################################### Cache server ############################# 110 | [remote_cache] 111 | # Either "redis", "memcached" or "database" default is "database" 112 | type = database 113 | 114 | # cache connectionstring options 115 | # database: will use Grafana primary database. 116 | # redis: config like redis server e.g. `addr=127.0.0.1:6379,pool_size=100,db=0`. Only addr is required. 117 | # memcache: 127.0.0.1:11211 118 | ;connstr = 119 | 120 | #################################### Data proxy ########################### 121 | [dataproxy] 122 | 123 | # This enables data proxy logging, default is false 124 | logging = false 125 | 126 | # How long the data proxy should wait before timing out default is 30 (seconds) 127 | timeout = 30 128 | 129 | # If enabled and user is not anonymous, data proxy will add X-Grafana-User header with username into the request, default is false. 130 | send_user_header = false 131 | 132 | #################################### Analytics ########################### 133 | [analytics] 134 | # Server reporting, sends usage counters to stats.grafana.org every 24 hours. 135 | # No ip addresses are being tracked, only simple counters to track 136 | # running instances, dashboard and error counts. It is very helpful to us. 137 | # Change this option to false to disable reporting. 138 | reporting_enabled = true 139 | 140 | # Set to false to disable all checks to https://grafana.com 141 | # for new versions (grafana itself and plugins), check is used 142 | # in some UI views to notify that grafana or plugin update exists 143 | # This option does not cause any auto updates, nor send any information 144 | # only a GET request to https://grafana.com to get latest versions 145 | check_for_updates = true 146 | 147 | # Google Analytics universal tracking code, only enabled if you specify an id here 148 | google_analytics_ua_id = 149 | 150 | # Google Tag Manager ID, only enabled if you specify an id here 151 | google_tag_manager_id = 152 | 153 | #################################### Security ############################ 154 | [security] 155 | # default admin user, created on startup 156 | admin_user = admin 157 | 158 | # default admin password, can be changed before first start of grafana, or in profile settings 159 | admin_password = admin 160 | 161 | # used for signing 162 | secret_key = SW2YcwTIb9zpOOhoPsMm 163 | 164 | # disable gravatar profile images 165 | disable_gravatar = false 166 | 167 | # data source proxy whitelist (ip_or_domain:port separated by spaces) 168 | data_source_proxy_whitelist = 169 | 170 | # disable protection against brute force login attempts 171 | disable_brute_force_login_protection = false 172 | 173 | # set to true if you host Grafana behind HTTPS. default is false. 174 | cookie_secure = false 175 | 176 | # set cookie SameSite attribute. defaults to `lax`. can be set to "lax", "strict" and "none" 177 | cookie_samesite = lax 178 | 179 | # set to true if you want to allow browsers to render Grafana in a ,