├── LICENSE ├── README.md ├── config ├── ngx_http_elastic_client_module.c ├── runtest.sh └── t ├── 001-elastic-client-conn-test.t ├── 002-elastic-client-create-test.t ├── 003-elastic-client-search-test.t └── 004-elastic-client-delete-test.t /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 2-Clause License 2 | 3 | Copyright (c) 2018, Taymindis Woon 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | * Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | * Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 17 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 18 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 20 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 21 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 22 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 23 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 24 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 25 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | nginx-elastic-client 2 | ==================== 3 | 4 | To structure your elastic client command in your nginx proxy for multiple elasticsearch server. 5 | 6 | Table of Contents 7 | ================= 8 | 9 | * [Introduction](#introduction) 10 | * [Usage](#usage) 11 | * [Installation](#installation) 12 | * [Test](#test) 13 | * [Why need this](#why-need-this) 14 | * [Support](#support) 15 | * [Copyright & License](#copyright--license) 16 | 17 | Introduction 18 | ============ 19 | 20 | nginx-elastic-client is a nginx module which allow to proxy to elastic server with given pre-defined/dynamic command. 21 | 22 | 23 | Usage 24 | ======= 25 | ### 0. Setup your elastic upstream 26 | ```nginx 27 | # nginx.conf 28 | upstream elastic_upstream { 29 | server 127.0.0.1:9200; 30 | } 31 | ``` 32 | 33 | 34 | ### 1. Simple create index command, please noted that http method is defined inside command, it regardless what you called to nginx. 35 | ```nginx 36 | # nginx.conf 37 | 38 | server { 39 | .... 40 | location /test { 41 | elastic_pass http://elastic_upstream; 42 | elastic_send PUT /testindex/testdoc/100; 43 | elastic_query '{ 44 | "testname": "mytest" 45 | }'; 46 | } 47 | } 48 | ``` 49 | 50 | 51 | ### 2. If you don't want any command inside config, just pass ngixn variables to the elastic host, please setup the config below, you may refer the [nginx variables](https://gist.github.com/esfand/8262283) for more details 52 | ```nginx 53 | # nginx.conf 54 | 55 | server { 56 | .... 57 | 58 | location / { 59 | elastic_pass http://elastic_upstream; 60 | elastic_send $request_method $uri?$query_string; 61 | elastic_query $request_body; 62 | } 63 | } 64 | ``` 65 | 66 | 67 | ### 3. Create index based on your args, for example /test?new_id=Staff1232 68 | ```nginx 69 | # nginx.conf 70 | 71 | server { 72 | .... 73 | location /test { 74 | elastic_pass http://elastic_upstream; 75 | elastic_send PUT /testindex/testdoc/$arg_new_id; 76 | elastic_query '{ 77 | "testname": "mytest-$arg_new_id" 78 | }'; 79 | } 80 | } 81 | ``` 82 | 83 | 84 | ### 4. search all 85 | ```nginx 86 | # nginx.conf 87 | 88 | server { 89 | .... 90 | location /test { 91 | elastic_pass http://elastic_upstream; 92 | elastic_send POST /testindex/testdoc/_search/; 93 | elastic_query '{"query": 94 | { 95 | "match_all": {} 96 | } 97 | }'; 98 | } 99 | } 100 | ``` 101 | 102 | 103 | ### 5. search all but group the docs by index. 104 | ```nginx 105 | # nginx.conf 106 | 107 | server { 108 | .... 109 | location /test { 110 | elastic_pass http://elastic_upstream; 111 | elastic_send POST /testindex/testdoc/_search?size=100 index_docs; 112 | 113 | elastic_query '{"query": 114 | { 115 | "match_all": {} 116 | } 117 | }'; 118 | } 119 | } 120 | ``` 121 | 122 | 123 | ### 6. search all but skipmeta data, keep index, type, and id. 124 | ```nginx 125 | # nginx.conf 126 | 127 | server { 128 | .... 129 | location /test { 130 | elastic_pass http://elastic_upstream; 131 | elastic_send POST /testindex/testdoc/_search skipmeta; 132 | 133 | elastic_query '{"query": 134 | { 135 | "match_all": {} 136 | } 137 | }'; 138 | } 139 | } 140 | ``` 141 | 142 | 143 | ### 7. search all but source only data. 144 | ```nginx 145 | # nginx.conf 146 | 147 | server { 148 | .... 149 | location /test { 150 | elastic_pass http://elastic_upstream; 151 | elastic_send POST /testindex/testdoc/_search source; 152 | 153 | elastic_query '{"query": 154 | { 155 | "match_all": {} 156 | } 157 | }'; 158 | } 159 | } 160 | ``` 161 | 162 | 163 | ### 8. search with dynamic input. 164 | ```nginx 165 | # nginx.conf 166 | 167 | server { 168 | .... 169 | location /test { 170 | elastic_pass http://elastic_upstream; 171 | elastic_send POST /testindex/testdoc/_search; 172 | elastic_query $request_body; 173 | } 174 | 175 | location /test2 { 176 | elastic_pass http://elastic_upstream; 177 | elastic_send POST /testindex/testdoc/_search; 178 | elastic_query '{"query": {"match" : {"$arg_field" : "$arg_value"}}}'; 179 | } 180 | } 181 | ``` 182 | 183 | 184 | ### 9. Delete index. 185 | ```nginx 186 | # nginx.conf 187 | 188 | server { 189 | .... 190 | location /test { 191 | elastic_pass http://elastic_upstream; 192 | elastic_send DELETE /testindex/; 193 | } 194 | } 195 | ``` 196 | 197 | 198 | 199 | Installation 200 | ============ 201 | 202 | ```bash 203 | wget 'http://nginx.org/download/nginx-1.13.7.tar.gz' 204 | tar -xzvf nginx-1.13.7.tar.gz 205 | cd nginx-1.13.7/ 206 | 207 | ./configure --add-module=/path/to/nginx-elastic-client 208 | 209 | make -j2 210 | sudo make install 211 | ``` 212 | 213 | [Back to TOC](#table-of-contents) 214 | 215 | 216 | Test 217 | ===== 218 | 219 | It depends on nginx test suite libs, please refer [test-nginx](https://github.com/openresty/test-nginx) for installation. 220 | 221 | 222 | ```bash 223 | cd /path/to/nginx-elastic-client 224 | export PATH=/path/to/nginx-dirname:$PATH 225 | sudo TEST_NGINX_SLEEP=0.3 prove t 226 | ``` 227 | 228 | [Back to TOC](#table-of-contents) 229 | 230 | 231 | Why need this 232 | ============== 233 | 234 | Not everyone knows how to query elasticsearch, some client might prefer pass the argument rather than query themselves. 235 | 236 | The query inside the nginx.conf has version controlled, maintained, tracked and it's transparent to every of your team member. Central based and pass to multiple elastic server. 237 | 238 | Known issue faster, make change 1 to every client, if many client has their on client query logic, the issue will be hard to maintained. 239 | 240 | Client can be Zero Down time! nginx -s reload make the changed. 241 | 242 | Nginx is reliable proxy server, the client is auto load-balance upstream to elasticsearch server. 243 | 244 | [Back to TOC](#table-of-contents) 245 | 246 | Support 247 | ======= 248 | 249 | Please do not hesitate to contact minikawoon2017@gmail.com for any queries or development improvement. 250 | 251 | 252 | [Back to TOC](#table-of-contents) 253 | 254 | Copyright & License 255 | =================== 256 | 257 | Copyright (c) 2018, Taymindis 258 | 259 | This module is licensed under the terms of the BSD license. 260 | 261 | All rights reserved. 262 | 263 | Redistribution and use in source and binary forms, with or without 264 | modification, are permitted provided that the following conditions are met: 265 | 266 | 1. Redistributions of source code must retain the above copyright notice, this 267 | list of conditions and the following disclaimer. 268 | 2. Redistributions in binary form must reproduce the above copyright notice, 269 | this list of conditions and the following disclaimer in the documentation 270 | and/or other materials provided with the distribution. 271 | 272 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 273 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 274 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 275 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR 276 | ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 277 | (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 278 | LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 279 | ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 280 | (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 281 | SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 282 | 283 | [Back to TOC](#table-of-contents) 284 | -------------------------------------------------------------------------------- /config: -------------------------------------------------------------------------------- 1 | ngx_addon_name=ngx_http_elastic_client_module 2 | 3 | 4 | HTTP_HELLOW_WORLD_SRCS="$ngx_addon_dir/ngx_http_elastic_client_module.c \ 5 | " 6 | 7 | HTTP_HELLOW_WORLD_DEPS="$NGX_ADDON_DEPS \ 8 | " 9 | 10 | if test -n "$ngx_module_link"; then 11 | ngx_module_type=HTTP_FILTER 12 | ngx_module_name=$ngx_addon_name 13 | ngx_module_deps="$HTTP_HELLOW_WORLD_DEPS" 14 | ngx_module_srcs="$HTTP_HELLOW_WORLD_SRCS" 15 | . auto/module 16 | else 17 | #If we want add a "string filter", which chain should be registered 18 | HTTP_FILTER_MODULES="$HTTP_FILTER_MODULES ngx_http_elastic_client_module" 19 | NGX_ADDON_SRCS="$NGX_ADDON_SRCS $HTTP_HELLOW_WORLD_SRCS" 20 | NGX_ADDON_DEPS="$NGX_ADDON_DEPS $HTTP_HELLOW_WORLD_DEPS" 21 | fi 22 | -------------------------------------------------------------------------------- /ngx_http_elastic_client_module.c: -------------------------------------------------------------------------------- 1 | /** 2 | * @file ngx_http_elastic_client_module.c 3 | * @author taymindis 4 | * @date Sun JAN 28 12:06:52 2018 5 | * 6 | * @brief A nginx-elastic-client module for Nginx. 7 | * 8 | * @section LICENSE 9 | * 10 | * Copyright (c) 2018, Taymindis 11 | * 12 | * This module is licensed under the terms of the BSD license. 13 | * 14 | * Redistribution and use in source and binary forms, with or without 15 | * modification, are permitted provided that the following conditions are met: 16 | * 17 | * 1. Redistributions of source code must retain the above copyright notice, this 18 | * list of conditions and the following disclaimer. 19 | * 2. Redistributions in binary form must reproduce the above copyright notice, 20 | * this list of conditions and the following disclaimer in the documentation 21 | * and/or other materials provided with the distribution. 22 | * 23 | * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 24 | * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 25 | * WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 26 | * DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR 27 | * ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES 28 | * (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 29 | * LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 30 | * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT 31 | * (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS 32 | * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 33 | */ 34 | 35 | #include 36 | #include 37 | #include 38 | 39 | /*** this module depends on ngx_http_proxy_module ****/ 40 | extern ngx_module_t ngx_http_proxy_module; 41 | 42 | #define A_SKIP_METADATA "&filter_path=hits.total,hits.hits._index,hits.hits._type,hits.hits._source" 43 | #define Q_SKIP_METADATA "?filter_path=hits.total,hits.hits._index,hits.hits._type,hits.hits._source" 44 | #define SKIP_METADATA_LEN 74 45 | 46 | #define A_SOURCE_ONLY "&filter_path=hits.total,hits.hits._source" 47 | #define Q_SOURCE_ONLY "?filter_path=hits.total,hits.hits._source" 48 | #define SOURCE_ONLY_LEN 41 49 | 50 | #define A_INDEX_DOCLIST "&filter_path=hits.hits._index,hits.hits._source&sort=_index:asc" 51 | #define Q_INDEX_DOCLIST "?filter_path=hits.hits._index,hits.hits._source&sort=_index:asc" 52 | #define INDEX_DOCLIST_LEN 63 53 | 54 | #define CLEAN_BUF(b) \ 55 | cl->buf->pos = cl->buf->start; \ 56 | cl->buf->last = cl->buf->start; \ 57 | cl->buf->flush = 1; \ 58 | cl->buf->temporary = 0; \ 59 | cl->buf->memory = 0; 60 | 61 | #define CLEAR_BUF(b) \ 62 | cl->buf->pos = cl->buf->last; 63 | 64 | typedef struct { 65 | ngx_http_complex_value_t *req_method; 66 | ngx_http_complex_value_t *index_n_path; 67 | ngx_uint_t resp_opt; 68 | } ngx_http_elastic_client_loc_conf_t; 69 | 70 | typedef struct { 71 | ngx_chain_t *free; 72 | ngx_chain_t *busy; 73 | off_t length; 74 | ngx_buf_t *saved_buf; 75 | } ngx_http_elastic_client_ctx_t; 76 | 77 | static ngx_int_t ngx_http_elastic_client_init(ngx_conf_t *cf); 78 | static void *ngx_http_elastic_client_create_loc_conf(ngx_conf_t *cf); 79 | static char *ngx_http_elastic_client_merge_loc_conf(ngx_conf_t *cf, void *parent, void *child); 80 | 81 | static char *ngx_conf_elastic_client_set_path_slot(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); 82 | static char *ngx_conf_elastic_client_set_query_slot(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); 83 | static char *ngx_conf_elastic_client_set_enum_slot(ngx_conf_t *cf, ngx_command_t *cmd, void *conf, ngx_uint_t varargs); 84 | static char *ngx_conf_elastic_client_proxy_pass_command(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); 85 | static ngx_int_t ngx_http_elastic_client_access_handler(ngx_http_request_t *r); 86 | static ngx_int_t ngx_http_elastic_client_header_filter(ngx_http_request_t *r); 87 | static ngx_int_t ngx_http_elastic_client_body_filter(ngx_http_request_t *r, ngx_chain_t *in); 88 | 89 | static ngx_http_output_header_filter_pt ngx_http_next_header_filter; 90 | static ngx_http_output_body_filter_pt ngx_http_next_body_filter; 91 | 92 | #define NGX_HTTP_ELASTIC_CLIENT_SKIPMETA_RESP 905 93 | #define NGX_HTTP_ELASTIC_CLIENT_SOURCEONLY_RESP 906 94 | #define NGX_HTTP_ELASTIC_CLIENT_INDEXDOCLIST_RESP 907 95 | #define NGX_HTTP_ELASTIC_CLIENT_RESP_OPT_OFF 0 96 | 97 | static ngx_flag_t index_docs_enabled = 0; 98 | #define FIRST_INDEX "[{\"_index\":\"" 99 | #define SIZEOF_FIRSTINDEX 12 100 | #define FOLLOW_INDEX ",{\"_index\":\"" 101 | #define SIZEOF_FOLLOWINDEX 12 102 | #define _SOURCE ",\"_source\":" 103 | #define SIZEOF_SOURCE 11 104 | 105 | #define ARRAY_BRACKET_OPEN "[" 106 | #define ARRAY_BRACKET_CLOSED "]" 107 | #define OBJ_BRACKET_OPEN "{" 108 | #define OBJ_BRACKET_CLOSED "}" 109 | #define COMMA "," 110 | #define DOUBLE_QUOTE "\"" 111 | #define DOUBLE_QUOTE_COLON "\":" 112 | #define COLON ":" 113 | 114 | 115 | void *ngx_elasic_client_index_realloc(ngx_http_request_t *r, void *ptr, size_t old_length, size_t new_length) 116 | { 117 | void *ptrNew = ngx_palloc(r->pool, new_length); 118 | if (ptrNew) { 119 | memcpy(ptrNew, ptr, old_length); 120 | ngx_pfree(r->pool, ptr); // free with pool 121 | } 122 | return ptrNew; 123 | } 124 | 125 | #define is_index_existed(__index_buckets__, __index_size__, __index__, __index_len__) ({\ 126 | size_t z, existed=0;\ 127 | ngx_str_t *index_val = &__index_buckets__[0];\ 128 | for(z=0;z<__index_size__;z++,index_val++){\ 129 | if(index_val->len == __index_len__ && ngx_strncmp(__index__, index_val->data, __index_len__)==0){\ 130 | existed=1; break;\ 131 | }}\ 132 | existed;}) 133 | 134 | static ngx_conf_enum_t ngx_http_elastic_client_resp_opt[] = { 135 | { ngx_string("skipmeta"), NGX_HTTP_ELASTIC_CLIENT_SKIPMETA_RESP }, 136 | { ngx_string("source"), NGX_HTTP_ELASTIC_CLIENT_SOURCEONLY_RESP }, 137 | { ngx_string("index_docs"), NGX_HTTP_ELASTIC_CLIENT_INDEXDOCLIST_RESP }, 138 | { ngx_null_string, 0 } 139 | }; 140 | 141 | 142 | static ngx_command_t ngx_http_elastic_client_commands[] = { 143 | 144 | { ngx_string("elastic_pass"), 145 | NGX_HTTP_LOC_CONF | NGX_HTTP_LIF_CONF | NGX_HTTP_LMT_CONF | NGX_CONF_TAKE1, 146 | ngx_conf_elastic_client_proxy_pass_command, 147 | NGX_HTTP_LOC_CONF_OFFSET, 148 | 0, 149 | NULL 150 | }, 151 | 152 | { ngx_string("elastic_send"), 153 | NGX_HTTP_LOC_CONF | NGX_HTTP_LIF_CONF | NGX_CONF_TAKE23, 154 | ngx_conf_elastic_client_set_path_slot, 155 | NGX_HTTP_LOC_CONF_OFFSET, 156 | offsetof(ngx_http_elastic_client_loc_conf_t, resp_opt), 157 | ngx_http_elastic_client_resp_opt 158 | }, 159 | 160 | { ngx_string("elastic_query"), 161 | NGX_HTTP_LOC_CONF | NGX_HTTP_LIF_CONF | NGX_CONF_TAKE1, 162 | ngx_conf_elastic_client_set_query_slot, 163 | NGX_HTTP_LOC_CONF_OFFSET, 164 | 0, 165 | NULL 166 | }, 167 | 168 | ngx_null_command 169 | }; 170 | 171 | static ngx_http_module_t ngx_http_elastic_client_module_ctx = { 172 | NULL, /* preconfiguration */ 173 | ngx_http_elastic_client_init, /* postconfiguration */ 174 | 175 | NULL, /* create main configuration */ 176 | NULL, /* init main configuration */ 177 | 178 | NULL, /* create server configuration */ 179 | NULL, /* merge server configuration */ 180 | 181 | ngx_http_elastic_client_create_loc_conf, /* create location configuration */ 182 | ngx_http_elastic_client_merge_loc_conf /* merge location configuration */ 183 | }; 184 | 185 | ngx_module_t ngx_http_elastic_client_module = { 186 | NGX_MODULE_V1, 187 | &ngx_http_elastic_client_module_ctx, /* module context */ 188 | ngx_http_elastic_client_commands, /* module directives */ 189 | NGX_HTTP_MODULE, /* module type */ 190 | NULL, /* init master */ 191 | NULL, /* init module */ 192 | NULL, /* init process */ 193 | NULL, /* init thread */ 194 | NULL, /* exit thread */ 195 | NULL, /* exit process */ 196 | NULL, /* exit master */ 197 | NGX_MODULE_V1_PADDING 198 | }; 199 | 200 | static void * 201 | ngx_http_elastic_client_create_loc_conf(ngx_conf_t *cf) 202 | { 203 | ngx_http_elastic_client_loc_conf_t *conf; 204 | 205 | conf = ngx_pcalloc(cf->pool, sizeof(ngx_http_elastic_client_loc_conf_t)); 206 | if (conf == NULL) { 207 | return NULL; 208 | } 209 | 210 | conf->req_method = NGX_CONF_UNSET_PTR; 211 | conf->index_n_path = NGX_CONF_UNSET_PTR; 212 | conf->resp_opt = NGX_CONF_UNSET_UINT; 213 | 214 | return conf; 215 | } 216 | 217 | static char * 218 | ngx_http_elastic_client_merge_loc_conf(ngx_conf_t *cf, void *parent, void *child) { 219 | ngx_http_elastic_client_loc_conf_t *prev = parent; 220 | ngx_http_elastic_client_loc_conf_t *conf = child; 221 | 222 | ngx_conf_merge_ptr_value(conf->index_n_path, prev->index_n_path, NULL); 223 | ngx_conf_merge_ptr_value(conf->req_method, prev->req_method, NULL); 224 | ngx_conf_merge_uint_value(conf->resp_opt, prev->resp_opt, NGX_HTTP_ELASTIC_CLIENT_RESP_OPT_OFF); 225 | 226 | 227 | return NGX_CONF_OK; 228 | } 229 | 230 | static ngx_int_t 231 | ngx_http_elastic_client_access_handler(ngx_http_request_t *r) 232 | { 233 | ngx_http_elastic_client_loc_conf_t *eclcf; 234 | ngx_str_t http_method; 235 | ngx_str_t index_path; 236 | u_char *p/*, *target_opt*/; 237 | size_t len/*, target_opt_len*/; 238 | 239 | 240 | eclcf = ngx_http_get_module_loc_conf(r, ngx_http_elastic_client_module); 241 | 242 | if (eclcf->index_n_path) { 243 | if (ngx_http_complex_value(r, eclcf->req_method, &http_method) != NGX_OK || 244 | ngx_http_complex_value(r, eclcf->index_n_path, &index_path) != NGX_OK) { 245 | return NGX_ERROR; 246 | } 247 | 248 | // if (eclcf->resp_opt == NGX_HTTP_ELASTIC_CLIENT_SKIPMETA_RESP) { 249 | // if (ngx_strchr(index_path.data, '?')) { 250 | // len = index_path.len + SKIP_METADATA_LEN; 251 | // p = r->unparsed_uri.data = ngx_palloc(r->pool, len); 252 | // p = ngx_copy(p, index_path.data, index_path.len); 253 | // p = ngx_copy(p, A_SKIP_METADATA, SKIP_METADATA_LEN); 254 | // r->unparsed_uri.len = len; 255 | // } else { 256 | // len = index_path.len + SKIP_METADATA_LEN; 257 | // p = r->unparsed_uri.data = ngx_palloc(r->pool, len); 258 | // p = ngx_copy(p, index_path.data, index_path.len); 259 | // p = ngx_copy(p, Q_SKIP_METADATA, SKIP_METADATA_LEN); 260 | // r->unparsed_uri.len = len; 261 | // } 262 | // } else if (eclcf->resp_opt == NGX_HTTP_ELASTIC_CLIENT_SOURCEONLY_RESP) { 263 | // if (ngx_strchr(index_path.data, '?')) { 264 | // len = index_path.len + SOURCE_ONLY_LEN; 265 | // p = r->unparsed_uri.data = ngx_palloc(r->pool, len); 266 | // p = ngx_copy(p, index_path.data, index_path.len); 267 | // p = ngx_copy(p, A_SOURCE_ONLY, SOURCE_ONLY_LEN); 268 | // r->unparsed_uri.len = len; 269 | // } else { 270 | // len = index_path.len + SOURCE_ONLY_LEN; 271 | // p = r->unparsed_uri.data = ngx_palloc(r->pool, len); 272 | // p = ngx_copy(p, index_path.data, index_path.len); 273 | // p = ngx_copy(p, Q_SOURCE_ONLY, SOURCE_ONLY_LEN); 274 | // r->unparsed_uri.len = len; 275 | // } 276 | // } else { 277 | // r->unparsed_uri = index_path; 278 | // } 279 | 280 | switch (eclcf->resp_opt) { 281 | case NGX_HTTP_ELASTIC_CLIENT_SKIPMETA_RESP: 282 | if (ngx_strchr(index_path.data, '?')) { 283 | len = index_path.len + SKIP_METADATA_LEN; 284 | p = r->unparsed_uri.data = ngx_palloc(r->pool, len); 285 | p = ngx_copy(p, index_path.data, index_path.len); 286 | p = ngx_copy(p, A_SKIP_METADATA, SKIP_METADATA_LEN); 287 | r->unparsed_uri.len = len; 288 | } else { 289 | len = index_path.len + SKIP_METADATA_LEN; 290 | p = r->unparsed_uri.data = ngx_palloc(r->pool, len); 291 | p = ngx_copy(p, index_path.data, index_path.len); 292 | p = ngx_copy(p, Q_SKIP_METADATA, SKIP_METADATA_LEN); 293 | r->unparsed_uri.len = len; 294 | } 295 | break; 296 | case NGX_HTTP_ELASTIC_CLIENT_SOURCEONLY_RESP: 297 | if (ngx_strchr(index_path.data, '?')) { 298 | len = index_path.len + SOURCE_ONLY_LEN; 299 | p = r->unparsed_uri.data = ngx_palloc(r->pool, len); 300 | p = ngx_copy(p, index_path.data, index_path.len); 301 | p = ngx_copy(p, A_SOURCE_ONLY, SOURCE_ONLY_LEN); 302 | r->unparsed_uri.len = len; 303 | } else { 304 | len = index_path.len + SOURCE_ONLY_LEN; 305 | p = r->unparsed_uri.data = ngx_palloc(r->pool, len); 306 | p = ngx_copy(p, index_path.data, index_path.len); 307 | p = ngx_copy(p, Q_SOURCE_ONLY, SOURCE_ONLY_LEN); 308 | r->unparsed_uri.len = len; 309 | } 310 | break; 311 | case NGX_HTTP_ELASTIC_CLIENT_INDEXDOCLIST_RESP: 312 | if (ngx_strchr(index_path.data, '?')) { 313 | len = index_path.len + INDEX_DOCLIST_LEN; 314 | p = r->unparsed_uri.data = ngx_palloc(r->pool, len); 315 | p = ngx_copy(p, index_path.data, index_path.len); 316 | p = ngx_copy(p, A_INDEX_DOCLIST, INDEX_DOCLIST_LEN); 317 | r->unparsed_uri.len = len; 318 | } else { 319 | len = index_path.len + INDEX_DOCLIST_LEN; 320 | p = r->unparsed_uri.data = ngx_palloc(r->pool, len); 321 | p = ngx_copy(p, index_path.data, index_path.len); 322 | p = ngx_copy(p, Q_INDEX_DOCLIST, INDEX_DOCLIST_LEN); 323 | r->unparsed_uri.len = len; 324 | } 325 | break; 326 | default: 327 | r->unparsed_uri = index_path; 328 | } 329 | r->method_name = http_method; 330 | 331 | // ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, "uri is =%V ", &r->unparsed_uri); 332 | } 333 | 334 | return NGX_DECLINED; 335 | } 336 | 337 | static ngx_int_t 338 | ngx_http_elastic_client_init(ngx_conf_t *cf) { 339 | if (index_docs_enabled) { 340 | ngx_http_next_header_filter = ngx_http_top_header_filter; 341 | ngx_http_top_header_filter = ngx_http_elastic_client_header_filter; 342 | 343 | ngx_http_next_body_filter = ngx_http_top_body_filter; 344 | ngx_http_top_body_filter = ngx_http_elastic_client_body_filter; 345 | } 346 | 347 | ngx_http_handler_pt *h; 348 | ngx_http_core_main_conf_t *cmcf; 349 | 350 | cmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_core_module); 351 | 352 | h = ngx_array_push(&cmcf->phases[NGX_HTTP_ACCESS_PHASE].handlers); 353 | if (h == NULL) { 354 | return NGX_ERROR; 355 | } 356 | 357 | *h = ngx_http_elastic_client_access_handler; 358 | 359 | return NGX_OK; 360 | } 361 | 362 | static char * 363 | ngx_conf_elastic_client_proxy_pass_command(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) { 364 | u_char *p; 365 | ngx_str_t *upstream_name, *value = cf->args->elts; 366 | upstream_name = &value[1]; 367 | 368 | if (upstream_name->len) { 369 | /*** check only allow domain name ***/ 370 | if ((p = (u_char*)ngx_strstr(upstream_name->data, "//"))) { 371 | if (!(*(p += 2))) 372 | goto ELASTIC_ERROR_CONF; 373 | 374 | if ((p = (u_char*)ngx_strchr(p, '/'))) { 375 | upstream_name->len--; 376 | *p = '\0'; 377 | p++; 378 | if (*p) { 379 | goto ELASTIC_ERROR_CONF; 380 | } 381 | } 382 | } else 383 | goto ELASTIC_ERROR_CONF; 384 | 385 | ngx_command_t *proxy_cmd = ngx_http_proxy_module.commands; 386 | if (proxy_cmd != NULL) { 387 | for ( /* void */ ; proxy_cmd->name.len; proxy_cmd++) { 388 | if ( ngx_strcmp(proxy_cmd->name.data, "proxy_pass"/*, (sizeof("proxy_pass") - 1)*/) == 0 ) { 389 | // printf("upstream=%.*s\n", (int)upstream_name->len, upstream_name->data); 390 | ngx_str_t *values = cf->args->elts; 391 | cf->args->nelts = 2; 392 | values[1].data = upstream_name->data; 393 | values[1].len = upstream_name->len; 394 | proxy_cmd->set(cf, proxy_cmd, ngx_http_conf_get_module_loc_conf(cf, ngx_http_proxy_module) ); 395 | } 396 | 397 | /* set request header content type application/json */ 398 | if ( ngx_strcmp(proxy_cmd->name.data, "proxy_set_header") == 0 ) { 399 | char *conf = ngx_http_conf_get_module_loc_conf(cf, ngx_http_proxy_module); 400 | ngx_array_t **a; 401 | ngx_keyval_t *kv; 402 | 403 | a = (ngx_array_t **) (conf + proxy_cmd->offset); 404 | 405 | if (*a == NULL) { 406 | *a = ngx_array_create(cf->pool, 4, sizeof(ngx_keyval_t)); 407 | if (*a == NULL) { 408 | return NGX_CONF_ERROR; 409 | } 410 | } 411 | 412 | kv = ngx_array_push(*a); 413 | if (kv == NULL) { 414 | return NGX_CONF_ERROR; 415 | } 416 | 417 | ngx_str_set(&kv->key, "Content-Type"); 418 | ngx_str_set(&kv->value, "application/json"); 419 | } 420 | } 421 | } 422 | } 423 | 424 | return NGX_CONF_OK; 425 | ELASTIC_ERROR_CONF: 426 | ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "%s", "invalid upstream url while elastic pass"); 427 | return NGX_CONF_ERROR; 428 | } /* ngx_http_elastic_client_by_pass_post_command */ 429 | 430 | 431 | static char * 432 | ngx_conf_elastic_client_set_path_slot(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) { 433 | ngx_http_elastic_client_loc_conf_t *eclcf = conf; 434 | 435 | ngx_str_t *value; 436 | ngx_http_complex_value_t **cv; 437 | ngx_http_compile_complex_value_t ccv; 438 | char *rv; 439 | 440 | cv = &eclcf->req_method; 441 | 442 | if (*cv != NGX_CONF_UNSET_PTR) { 443 | return "is duplicate"; 444 | } 445 | 446 | *cv = ngx_palloc(cf->pool, sizeof(ngx_http_complex_value_t)); 447 | if (*cv == NULL) { 448 | return NGX_CONF_ERROR; 449 | } 450 | 451 | value = cf->args->elts; 452 | 453 | if ( cf->args->nelts == 4) { 454 | rv = ngx_conf_elastic_client_set_enum_slot(cf, cmd, conf, 3); 455 | 456 | if (rv != NGX_CONF_OK) { 457 | return rv; 458 | } 459 | } 460 | 461 | if (value[1].len == 0 || value[2].len == 0) { 462 | ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, "no method given, e.g. GET/PUT/POST? "); 463 | return NGX_CONF_ERROR; 464 | } 465 | 466 | // ngx_strlow(value[1].data, value[1].data, value[1].len); 467 | ngx_memzero(&ccv, sizeof(ngx_http_compile_complex_value_t)); 468 | 469 | ccv.cf = cf; 470 | ccv.value = &value[1]; 471 | ccv.complex_value = *cv; 472 | 473 | if (ngx_http_compile_complex_value(&ccv) != NGX_OK) { 474 | return NGX_CONF_ERROR; 475 | } 476 | 477 | // index path complex value compilation 478 | cv = &eclcf->index_n_path; 479 | 480 | if (*cv != NGX_CONF_UNSET_PTR) { 481 | return "is duplicate"; 482 | } 483 | 484 | *cv = ngx_palloc(cf->pool, sizeof(ngx_http_complex_value_t)); 485 | if (*cv == NULL) { 486 | return NGX_CONF_ERROR; 487 | } 488 | 489 | ngx_memzero(&ccv, sizeof(ngx_http_compile_complex_value_t)); 490 | 491 | ccv.cf = cf; 492 | ccv.value = &value[2]; 493 | ccv.complex_value = *cv; 494 | 495 | if (ngx_http_compile_complex_value(&ccv) != NGX_OK) { 496 | return NGX_CONF_ERROR; 497 | } 498 | 499 | return NGX_CONF_OK; 500 | } 501 | 502 | static char * 503 | ngx_conf_elastic_client_set_query_slot(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) { 504 | ngx_command_t *proxy_cmd = ngx_http_proxy_module.commands; 505 | if (proxy_cmd != NULL) { 506 | for ( /* void */ ; proxy_cmd->name.len; proxy_cmd++) { 507 | /* set elastic query to proxy_body */ 508 | if ( ngx_strcmp(proxy_cmd->name.data, "proxy_set_body") == 0 ) { 509 | proxy_cmd->set(cf, proxy_cmd, ngx_http_conf_get_module_loc_conf(cf, ngx_http_proxy_module) ); 510 | break; 511 | } 512 | } 513 | } 514 | return NGX_CONF_OK; 515 | } 516 | 517 | static char * 518 | ngx_conf_elastic_client_set_enum_slot(ngx_conf_t *cf, ngx_command_t *cmd, void *conf, ngx_uint_t varargs) { 519 | char *p = conf; 520 | 521 | ngx_uint_t *np, i; 522 | ngx_str_t *value; 523 | ngx_conf_enum_t *e; 524 | 525 | np = (ngx_uint_t *) (p + cmd->offset); 526 | 527 | if (*np != NGX_CONF_UNSET_UINT) { 528 | return "is duplicate"; 529 | } 530 | 531 | value = cf->args->elts; 532 | e = cmd->post; 533 | 534 | for (i = 0; e[i].name.len != 0; i++) { 535 | if (e[i].name.len != value[varargs].len 536 | || ngx_strcasecmp(e[i].name.data, value[varargs].data) != 0) 537 | { 538 | continue; 539 | } 540 | 541 | *np = e[i].value; 542 | 543 | if ( ngx_strcmp(value[varargs].data, "index_docs" ) == 0 ) { 544 | index_docs_enabled = 1; 545 | } 546 | 547 | return NGX_CONF_OK; 548 | } 549 | 550 | ngx_conf_log_error(NGX_LOG_EMERG, cf, 0, 551 | "invalid value \"%s\", opts: index_docs, skipmeta, source", value[varargs].data); 552 | 553 | return NGX_CONF_ERROR; 554 | } 555 | 556 | static ngx_int_t 557 | ngx_http_elastic_client_header_filter(ngx_http_request_t *r) { 558 | ngx_http_elastic_client_loc_conf_t *eclcf; 559 | ngx_http_elastic_client_ctx_t *ctx; 560 | 561 | eclcf = ngx_http_get_module_loc_conf(r, ngx_http_elastic_client_module); 562 | 563 | if (r->headers_out.status == NGX_HTTP_NOT_MODIFIED 564 | || r->headers_out.status == NGX_HTTP_NO_CONTENT 565 | || r->headers_out.status < NGX_HTTP_OK 566 | || r != r->main 567 | || r->method == NGX_HTTP_HEAD) 568 | { 569 | ngx_log_error(NGX_LOG_DEBUG, r->connection->log, 0, "======== head response only ============="); 570 | } 571 | 572 | if (eclcf->resp_opt != NGX_HTTP_ELASTIC_CLIENT_INDEXDOCLIST_RESP/* 573 | || r->headers_out.content_type.len != sizeof("application/json") - 1 574 | || ngx_strncmp(r->headers_out.content_type.data, "application/json", 575 | r->headers_out.content_type.len) != 0*/) { 576 | goto SKIP_DOCLIST_FILTER; 577 | } 578 | 579 | // Response Json content type only 580 | // r->headers_out.content_type_len = sizeof("application/json") - 1; 581 | // ngx_str_set(&r->headers_out.content_type, "application/json"); 582 | 583 | 584 | 585 | 586 | if (r->header_only) { 587 | return ngx_http_next_header_filter(r); 588 | } 589 | 590 | ctx = ngx_pcalloc(r->pool, sizeof(ngx_http_elastic_client_ctx_t)); 591 | if (ctx == NULL) { 592 | return NGX_ERROR; 593 | } 594 | 595 | ctx->length = r->headers_out.content_length_n; 596 | if (!ctx->saved_buf) { 597 | ctx->saved_buf = ngx_calloc_buf(r->pool); 598 | ctx->saved_buf->pos = ngx_palloc(r->pool, ctx->length); 599 | ctx->saved_buf->start = ctx->saved_buf->last = ctx->saved_buf->pos; 600 | ctx->saved_buf->end = ctx->saved_buf->pos + ctx->length; 601 | } 602 | 603 | ngx_http_set_ctx(r, ctx, ngx_http_elastic_client_module); 604 | 605 | SKIP_DOCLIST_FILTER: 606 | r->filter_need_in_memory = 1; 607 | 608 | // if (r == r->main) { 609 | 610 | ngx_http_clear_content_length(r); 611 | ngx_http_clear_accept_ranges(r); 612 | ngx_http_weak_etag(r); 613 | // } 614 | 615 | 616 | return ngx_http_next_header_filter(r); 617 | } 618 | 619 | static ngx_int_t 620 | ngx_index_docs_parser(ngx_http_request_t *r, ngx_buf_t *b, u_char *p, size_t buf_size ) { 621 | 622 | u_char *out; 623 | if (!b) { 624 | b = ngx_calloc_buf(r->pool); 625 | } 626 | b->start = b->pos = out = ngx_palloc(r->pool, buf_size); 627 | b->end = b->start + buf_size; 628 | ngx_str_t *index_str; 629 | u_char *_index; 630 | u_char *_source; 631 | size_t curr_size, index_len, i = 0, index_size = 10; 632 | 633 | ngx_str_t* index_buckets = ngx_pcalloc( r->pool, index_size * sizeof(ngx_str_t)); 634 | 635 | out = ngx_copy(out, OBJ_BRACKET_OPEN, 1); 636 | 637 | if ((_index = (u_char*)ngx_strstr(p, FIRST_INDEX))) { 638 | _index = _index + SIZEOF_FIRSTINDEX; 639 | if ((_source = (u_char*)ngx_strstr(_index, _SOURCE))) { 640 | index_len = _source - _index - 1; 641 | out = ngx_copy(out, DOUBLE_QUOTE, 1); 642 | out = ngx_copy(out, _index, index_len); 643 | out = ngx_copy(out, DOUBLE_QUOTE_COLON, 2); 644 | out = ngx_copy(out, ARRAY_BRACKET_OPEN, 1); 645 | index_str = &index_buckets[i++]; 646 | index_str->data = _index; 647 | index_str->len = index_len; 648 | _source = _source + SIZEOF_SOURCE; 649 | 650 | for (; (_index = (u_char*)ngx_strstr(_source, FOLLOW_INDEX)); i++ ) { 651 | out = ngx_copy(out, _source, _index - _source - 1); 652 | 653 | _index = _index + SIZEOF_FOLLOWINDEX; 654 | if (i >= index_size) { 655 | curr_size = index_size * sizeof(ngx_str_t); 656 | index_buckets = ngx_elasic_client_index_realloc(r, index_buckets, curr_size, curr_size * 2); 657 | if (!index_buckets) 658 | return NGX_ERROR; 659 | index_size *= 2; 660 | } 661 | 662 | if ((_source = (u_char*)ngx_strstr(_index, _SOURCE))) { 663 | index_len = _source - _index - 1; 664 | if ( !(is_index_existed(index_buckets, i, _index, index_len)) ) { 665 | /** new index **/ 666 | out = ngx_copy(out, ARRAY_BRACKET_CLOSED, 1); 667 | out = ngx_copy(out, COMMA, 1); 668 | out = ngx_copy(out, DOUBLE_QUOTE, 1); 669 | out = ngx_copy(out, _index, index_len); 670 | out = ngx_copy(out, DOUBLE_QUOTE_COLON, 2); 671 | out = ngx_copy(out, ARRAY_BRACKET_OPEN, 1); 672 | } else { 673 | out = ngx_copy(out, COMMA, 1); 674 | } 675 | _source = _source + SIZEOF_SOURCE; 676 | } 677 | index_str = &index_buckets[i]; 678 | index_str->data = _index; 679 | index_str->len = index_len; 680 | } 681 | 682 | // LAST Source 683 | int source_open_close = 0; 684 | size_t n = b->end - _source; 685 | register const u_char *__p = _source; 686 | for (; n != 0; n--) { 687 | if (*__p == '{') 688 | source_open_close++; 689 | else if (*__p == '}') 690 | source_open_close--; 691 | 692 | __p++; 693 | if (source_open_close == 0) 694 | break; 695 | } 696 | out = ngx_copy(out, _source, __p - _source ); 697 | out = ngx_copy(out, ARRAY_BRACKET_CLOSED, 1); 698 | } 699 | } 700 | 701 | 702 | out = ngx_copy(out, OBJ_BRACKET_CLOSED, 1); 703 | b->last = out; 704 | 705 | return NGX_OK; 706 | } 707 | 708 | static ngx_int_t 709 | ngx_http_elastic_client_body_filter(ngx_http_request_t *r, ngx_chain_t *in) { 710 | ngx_http_elastic_client_loc_conf_t *eclcf; 711 | 712 | eclcf = ngx_http_get_module_loc_conf(r, ngx_http_elastic_client_module); 713 | 714 | if (eclcf->resp_opt != NGX_HTTP_ELASTIC_CLIENT_INDEXDOCLIST_RESP) { 715 | return ngx_http_next_body_filter(r, in); 716 | } 717 | 718 | ngx_int_t rc; 719 | ngx_buf_t *b; 720 | ngx_chain_t *cl, *tl, *out, **ll; 721 | ngx_http_elastic_client_ctx_t *ctx; 722 | 723 | ctx = ngx_http_get_module_ctx(r, ngx_http_elastic_client_module); 724 | if (ctx == NULL) { 725 | return NGX_ERROR; 726 | } 727 | 728 | /* create a new chain "out" from "in" with all the changes */ 729 | 730 | ll = &out; 731 | 732 | for (cl = in; cl; cl = cl->next) { 733 | 734 | ctx->saved_buf->last = ngx_copy(ctx->saved_buf->last, cl->buf->pos, ngx_buf_size(cl->buf)); 735 | /* loop until last buf then we do the json parsing */ 736 | if (cl->buf->last_buf) { 737 | 738 | tl = ngx_chain_get_free_buf(r->pool, &ctx->free); 739 | if (tl == NULL) { 740 | return NGX_ERROR; 741 | } 742 | 743 | // *ctx->saved_buf->last = '\0'; // set NULL terminator 744 | b = tl->buf; 745 | 746 | ngx_index_docs_parser(r, b, ctx->saved_buf->pos, ctx->saved_buf->last - ctx->saved_buf->pos); 747 | 748 | b->tag = (ngx_buf_tag_t) &ngx_http_elastic_client_module; 749 | b->temporary = 1; 750 | b->last_buf = 1; 751 | // ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, "Buff= %O", b->last - b->pos ); 752 | // ngx_log_error(NGX_LOG_INFO, r->connection->log, 0, "cxt->length= %O", ctx->length ); 753 | 754 | 755 | *ll = tl; 756 | ll = &tl->next; 757 | CLEAR_BUF(cl->buf); 758 | goto out_result; 759 | } 760 | 761 | CLEAR_BUF(cl->buf); // clear current buffer 762 | 763 | } 764 | out_result: 765 | *ll = NULL; 766 | 767 | /* send the new chain */ 768 | 769 | rc = ngx_http_next_body_filter(r, out); 770 | 771 | /* update "busy" and "free" chains for reuse */ 772 | 773 | ngx_chain_update_chains(r->pool, &ctx->free, &ctx->busy, &out, 774 | (ngx_buf_tag_t) &ngx_http_elastic_client_module); 775 | 776 | return rc; 777 | } -------------------------------------------------------------------------------- /runtest.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | TEST_NGINX_SLEEP=0.3 prove t 4 | -------------------------------------------------------------------------------- /t/001-elastic-client-conn-test.t: -------------------------------------------------------------------------------- 1 | # vi:filetype=perl 2 | 3 | use lib 'inc'; 4 | use lib 'lib'; 5 | use Test::Nginx::Socket 'no_plan'; 6 | 7 | 8 | $ENV{TEST_NGINX_ELASTIC_HOST} ||= '10.2.140.88'; 9 | $ENV{TEST_NGINX_ELASTIC_PORT} ||= 9200; 10 | 11 | our $http_config = <<'_EOC_'; 12 | upstream elastic_upstream { 13 | server $TEST_NGINX_ELASTIC_HOST:$TEST_NGINX_ELASTIC_PORT; 14 | keepalive 1; 15 | } 16 | _EOC_ 17 | 18 | no_shuffle(); 19 | run_tests(); 20 | 21 | #no_diff(); 22 | 23 | __DATA__ 24 | 25 | === TEST 1: test elastic connection 26 | --- http_config eval: $::http_config 27 | --- config 28 | location /test { 29 | elastic_pass http://elastic_upstream; 30 | elastic_send GET /; 31 | } 32 | --- request 33 | GET /test 34 | --- error_code: 200 35 | --- timeout: 10 36 | --- response_headers 37 | Content-Type: application/json; charset=UTF-8 -------------------------------------------------------------------------------- /t/002-elastic-client-create-test.t: -------------------------------------------------------------------------------- 1 | # vi:filetype=perl 2 | 3 | use lib 'inc'; 4 | use lib 'lib'; 5 | use Test::Nginx::Socket 'no_plan'; 6 | 7 | 8 | $ENV{TEST_NGINX_ELASTIC_HOST} ||= '10.2.140.88'; 9 | $ENV{TEST_NGINX_ELASTIC_PORT} ||= 9200; 10 | 11 | our $http_config = <<'_EOC_'; 12 | upstream elastic_upstream { 13 | server $TEST_NGINX_ELASTIC_HOST:$TEST_NGINX_ELASTIC_PORT; 14 | keepalive 1; 15 | } 16 | _EOC_ 17 | 18 | no_shuffle(); 19 | run_tests(); 20 | 21 | #no_diff(); 22 | 23 | __DATA__ 24 | 25 | 26 | === TEST 2: create index by id 100 27 | --- http_config eval: $::http_config 28 | --- config 29 | location /test { 30 | elastic_pass http://elastic_upstream; 31 | elastic_send PUT /testindex/testdoc/100; 32 | elastic_query '{ 33 | "testname": "mytest" 34 | }'; 35 | } 36 | --- request 37 | GET /test 38 | --- error_code: 201 39 | --- timeout: 10 40 | --- response_headers 41 | Content-Type: application/json; charset=UTF-8 42 | 43 | 44 | 45 | === TEST 3: create index by arg id 46 | --- http_config eval: $::http_config 47 | --- config 48 | location /test { 49 | elastic_pass http://elastic_upstream; 50 | elastic_send PUT /testindex/testdoc/$arg_new_id; 51 | elastic_query '{ 52 | "testname": "mytest-$arg_new_id" 53 | }'; 54 | } 55 | --- request 56 | GET /test?new_id=300 57 | --- error_code: 201 58 | --- timeout: 10 59 | --- response_headers 60 | Content-Type: application/json; charset=UTF-8 -------------------------------------------------------------------------------- /t/003-elastic-client-search-test.t: -------------------------------------------------------------------------------- 1 | # vi:filetype=perl 2 | 3 | use lib 'inc'; 4 | use lib 'lib'; 5 | use Test::Nginx::Socket 'no_plan'; 6 | 7 | 8 | $ENV{TEST_NGINX_ELASTIC_HOST} ||= '10.2.140.88'; 9 | $ENV{TEST_NGINX_ELASTIC_PORT} ||= 9200; 10 | 11 | our $http_config = <<'_EOC_'; 12 | upstream elastic_upstream { 13 | server $TEST_NGINX_ELASTIC_HOST:$TEST_NGINX_ELASTIC_PORT; 14 | keepalive 1; 15 | } 16 | _EOC_ 17 | 18 | no_shuffle(); 19 | run_tests(); 20 | 21 | #no_diff(); 22 | 23 | __DATA__ 24 | 25 | 26 | 27 | === TEST 4: search all 28 | --- http_config eval: $::http_config 29 | --- config 30 | location /test { 31 | elastic_pass http://elastic_upstream; 32 | elastic_send POST /testindex/testdoc/_search/; 33 | elastic_query '{"query": 34 | { 35 | "match_all": {} 36 | } 37 | }'; 38 | } 39 | --- request 40 | GET /test 41 | --- error_code: 200 42 | --- timeout: 10 43 | --- response_headers 44 | Content-Type: application/json; charset=UTF-8 45 | --- response_body_like eval chomp 46 | qr/.*?\"_source\":.*/ 47 | 48 | 49 | 50 | === TEST 5: search all but skipmeta data 51 | --- http_config eval: $::http_config 52 | --- config 53 | location /test { 54 | elastic_pass http://elastic_upstream; 55 | elastic_send POST /testindex/testdoc/_search skipmeta; 56 | 57 | elastic_query '{"query": 58 | { 59 | "match_all": {} 60 | } 61 | }'; 62 | } 63 | --- request 64 | GET /test 65 | --- error_code: 200 66 | --- timeout: 10 67 | --- response_headers 68 | Content-Type: application/json; charset=UTF-8 69 | --- response_body_like eval chomp 70 | qr/.*?\"_source\":.*/ 71 | 72 | 73 | 74 | 75 | === TEST 6: search all but source only data 76 | --- http_config eval: $::http_config 77 | --- config 78 | location /test { 79 | elastic_pass http://elastic_upstream; 80 | elastic_send POST /testindex/testdoc/_search source; 81 | 82 | elastic_query '{"query": 83 | { 84 | "match_all": {} 85 | } 86 | }'; 87 | } 88 | --- request 89 | GET /test 90 | --- error_code: 200 91 | --- timeout: 10 92 | --- response_headers 93 | Content-Type: application/json; charset=UTF-8 94 | --- response_body_like eval chomp 95 | qr/.*?\"_source\":.*/ 96 | 97 | 98 | 99 | === TEST 7: search with dynamic input 100 | --- http_config eval: $::http_config 101 | --- config 102 | location /test { 103 | elastic_pass http://elastic_upstream; 104 | elastic_send POST /testindex/testdoc/_search; 105 | elastic_query $request_body; 106 | } 107 | --- request 108 | POST /test 109 | {"query":{"match_all": {}}} 110 | --- error_code: 200 111 | --- timeout: 10 112 | --- response_headers 113 | Content-Type: application/json; charset=UTF-8 114 | --- response_body_like eval chomp 115 | qr/.*?\"_source\":.*/ 116 | 117 | 118 | 119 | === TEST 8: search with dynamic input 2 120 | --- http_config eval: $::http_config 121 | --- config 122 | location /test { 123 | elastic_pass http://elastic_upstream; 124 | elastic_send POST /testindex/testdoc/_search; 125 | elastic_query '{"query": {"match" : {"$arg_field" : "$arg_value"}}}'; 126 | } 127 | --- request 128 | GET /test?field=testname&value=mytest 129 | --- error_code: 200 130 | --- timeout: 10 131 | --- response_headers 132 | Content-Type: application/json; charset=UTF-8 133 | --- response_body_like eval chomp 134 | qr/.*?\"_source\":.*/ -------------------------------------------------------------------------------- /t/004-elastic-client-delete-test.t: -------------------------------------------------------------------------------- 1 | # vi:filetype=perl 2 | 3 | use lib 'inc'; 4 | use lib 'lib'; 5 | use Test::Nginx::Socket 'no_plan'; 6 | 7 | 8 | $ENV{TEST_NGINX_ELASTIC_HOST} ||= '10.2.140.88'; 9 | $ENV{TEST_NGINX_ELASTIC_PORT} ||= 9200; 10 | 11 | our $http_config = <<'_EOC_'; 12 | upstream elastic_upstream { 13 | server $TEST_NGINX_ELASTIC_HOST:$TEST_NGINX_ELASTIC_PORT; 14 | keepalive 1; 15 | } 16 | _EOC_ 17 | 18 | no_shuffle(); 19 | run_tests(); 20 | 21 | #no_diff(); 22 | 23 | __DATA__ 24 | 25 | 26 | 27 | === TEST 9: Delete test index 28 | --- http_config eval: $::http_config 29 | --- config 30 | location /test { 31 | elastic_pass http://elastic_upstream; 32 | elastic_send DELETE /testindex/; 33 | } 34 | --- request 35 | GET /test 36 | --- error_code: 200 37 | --- timeout: 10 38 | --- response_headers 39 | Content-Type: application/json; charset=UTF-8 40 | 41 | 42 | --------------------------------------------------------------------------------