└── README.rst /README.rst: -------------------------------------------------------------------------------- 1 | =============================================== 2 | NGINX proxy pitfalls related with DNS resolving 3 | =============================================== 4 | 5 | If you're using ``proxy_pass`` and your endpoint's IPs can vary in time, please read this article to avoid misunderstandings about how nginx works. 6 | 7 | .. contents:: 8 | 9 | TL;DR 10 | ===== 11 | 12 | If you want to force nginx resolve your endpoints, you should: 13 | 14 | * Use variables within ``proxy_pass`` directive, e.g. ``proxy_pass https://$endpoint/;``, where ``$endpoint`` can be manually setted or extracted from location regexp. `Read more `_. 15 | * Make sure that your endpoint isn't used in the another locations w/o variables, because in this case resolving won't work. To fix this move endpoint domain to the ``upstream`` or use variables in the ``proxy_pass`` in all locations to make resolving works. `Read more `_. 16 | * `You can have both resolve and non-resolve locations for same domain `_. 17 | * When a variable is used in proxy_pass directive, the location header is not longer adjusted. To get around this, simply set ``proxy_redirect``. `Read more `_. 18 | 19 | But I recommend to read full article, because it's interesting. 20 | 21 | Explanatory example 22 | =================== 23 | 24 | .. code:: nginx 25 | 26 | location /api/ { 27 | proxy_pass http://api.com/; 28 | } 29 | 30 | In this case nginx will resolve api.com only once at startup (or reload). But there are some cases when your endpoint can be resolved to any IP, e.g. if you're using load balancer which doing magic failover via DNS mapping. If api.com will point to another IP your proxying will fail. 31 | 32 | Finding the solution 33 | ==================== 34 | 35 | Add a resolver directive 36 | ------------------------ 37 | 38 | You can check `official nginx documentation `_ and find `resolver `_ directive: 39 | 40 | .. code:: nginx 41 | 42 | location /api/ { 43 | resolver 8.8.8.8; 44 | proxy_pass https://api.com/; 45 | } 46 | 47 | No, it will not work. Even this will not work: 48 | 49 | .. code:: nginx 50 | 51 | location /api/ { 52 | resolver 8.8.8.8 valid=1s; 53 | proxy_pass https://api.com/; 54 | } 55 | 56 | It's because of nginx doesn't respect ``resolver`` directive in this case. It will resolve api.com only at startup (or reload) by system resolver (/etc/resolv.conf), even if real TTL of A/AAAA record api.com is 1s. 57 | 58 | Add variables 59 | ------------- 60 | 61 | You can google a bit and find that `nginx try to resolve proxy endpoint with variables `_. Also `official documentation for proxy_pass directive notices this too `_. Hmmm, I think this should be noticed in the ``resolver`` description, but let's try anyway: 62 | 63 | .. code:: nginx 64 | 65 | location = /proxy/ { 66 | set $endpoint proxy.com; 67 | resolver 8.8.8.8 valid=10s; 68 | proxy_pass https://$endpoint/; 69 | } 70 | 71 | Works as expected, nginx will query proxy.com every 10s on particular requests. These configurations works too: 72 | 73 | .. code:: nginx 74 | 75 | set $endpoint api.com; 76 | location ~ ^/api/(.*)$ { 77 | resolver 8.8.8.8 valid=60s; 78 | proxy_pass https://$endpoint/$1$is_args$args; 79 | } 80 | 81 | .. code:: nginx 82 | 83 | location ~ ^/(?[\w-]+)(?:/(?.*))? { 84 | resolver 8.8.8.8 ipv6=off valid=60s; 85 | proxy_pass https://${dest_proxy}.example.com/${path_proxy}$is_args$args; 86 | } 87 | 88 | Notice that nginx will start even without ``resolver`` directive, but will fail with 502 at runtime, because "no resolver defined to resolve". 89 | 90 | Caveats 91 | ------- 92 | 93 | .. code:: nginx 94 | 95 | location = /api_version/ { 96 | proxy_pass https://api.com/version/; 97 | } 98 | 99 | location ~ ^/api/(.*)$ { 100 | set $endpoint api.com; 101 | resolver 8.8.8.8 valid=60s; 102 | proxy_pass https://$endpoint/$1$is_args$args; 103 | } 104 | 105 | In this case nginx will resolve api.com once at startup with system resolver and then will never do re-resolve even for /api/ requests. *Example with /api_version/ is just synthetic example, you can use more complex scenarios with headers set, etc.* 106 | 107 | Use variables everywhere to make it work as expected: 108 | 109 | .. code:: nginx 110 | 111 | location = /api_version/ { 112 | set $endpoint api.com; 113 | resolver 8.8.8.8 valid=60s; 114 | proxy_pass https://$endpoint/version/; 115 | } 116 | 117 | location ~ ^/api/(.*)$ { 118 | set $endpoint api.com; 119 | resolver 8.8.8.8 valid=60s; 120 | proxy_pass https://$endpoint/$1$is_args$args; 121 | } 122 | 123 | You can move ``set`` and ``resolver`` to the ``server`` or ``http`` (or use ``include``) directives to avoid copy-paste (also I assume that it will increase perfomance a bit, but I haven't tested it). 124 | 125 | If response from proxy contains ``Location`` header, as in the case of a redirect, nginx will automatically replace these values as needed. However, if variables are used in ``proxy_pass``, this must be done explicitly via ``proxy_redirect``: 126 | 127 | .. code:: nginx 128 | 129 | location = /api_version/ { 130 | set $endpoint api.com; 131 | resolver 8.8.8.8 valid=60s; 132 | proxy_pass https://$endpoint/version/; 133 | proxy_redirect https://$endpoint/ /; 134 | } 135 | 136 | location ~ ^/api/(.*)$ { 137 | set $endpoint api.com; 138 | resolver 8.8.8.8 valid=60s; 139 | proxy_pass https://$endpoint/$1$is_args$args; 140 | proxy_redirect https://$endpoint/ /; 141 | } 142 | 143 | Single line in `nginx docs `_ that mention it: 144 | 145 | The default parameter is not permitted if proxy_pass is specified using variables. 146 | 147 | Upstreams 148 | ========= 149 | 150 | If you're using nginx plus, you can use ``resolve`` parameter, `check out documentation `_. I assume that it will be efficient, because documentation says "monitors changes of the IP addresses that correspond to a domain name of the server", while solutions listed above will query DNS on the particular requests. But if you're using open source nginx, no honey is available for you. No money — no honey. 151 | 152 | You can have both resolve and non-resolve locations for same domain 153 | ------------------------------------------------------------------- 154 | 155 | .. code:: nginx 156 | 157 | upstream proxy { 158 | server proxy.com:443; 159 | } 160 | 161 | server { 162 | listen 80; 163 | server_name fillo.me; 164 | 165 | location = /proxy-with-resolve/ { 166 | set $endpoint proxy.com; 167 | resolver 8.8.8.8 valid=1s; 168 | proxy_pass https://$endpoint/; 169 | } 170 | 171 | location = /proxy-without-resolve/ { 172 | proxy_pass https://proxy/; 173 | proxy_set_header Host proxy.com; 174 | } 175 | } 176 | 177 | Yes, http://fillo.me/proxy-with-resolve/ will resolve proxy.com every 1s on particular requests, while http://fillo.me/proxy-without-resolve/ will not resolve proxy.com (nginx will resolve proxy.com at startup/reload once). This magic works because ``upstream`` directive is used. 178 | 179 | Another example: 180 | 181 | .. code:: nginx 182 | 183 | upstream api_version { 184 | server version.api.com:443; 185 | } 186 | 187 | server { 188 | listen 80; 189 | server_name fillo.me; 190 | 191 | location = /api_version/ { 192 | proxy_pass https://api_version/version/; 193 | proxy_set_header Host version.api.com; 194 | } 195 | 196 | location ~ ^/api/(?[\w-]+)(?:/(?.*))? { 197 | resolver 8.8.8.8 valid=60s; 198 | proxy_pass https://${dest_proxy}.api.com/${path_proxy}$is_args$args; 199 | } 200 | } 201 | 202 | * If you will open http://fillo.me/api_version/ then no resolve will be done, because of nginx resolved version.api.com at startup. 203 | * If you will open http://fillo.me/api/version/version/ then it will work as expected, nginx will resolve version.api.com every 60s on particular request. 204 | * If you will open http://fillo.me/api/checkout/items/ then it will work as expected, nginx will resolve checkout.api.com every 60s on particular request. 205 | 206 | Tested on 207 | ========= 208 | 209 | * 1.9.6 210 | * 1.10.1 211 | 212 | Although I think it works for many other versions. 213 | 214 | Further research 215 | ================ 216 | 217 | * `This issue `_ says that changing HTTPS to the HTTP helps. Check how protocol changes affects examples above. 218 | * Compare perfomance with and without resolving. 219 | * Compare perfomance with different variables scopes. 220 | * How to force upstream resolving. 221 | --------------------------------------------------------------------------------