├── README.rst
├── build_dynamic_haproxy.sh
├── build_rpm.sh
├── build_static_haproxy.sh
├── get_endpoints.lst
├── haproxy-aws-arch-diagram.png
├── haproxy.cfg
├── haproxy.spec
├── log_parse_ddos_protection.sh
└── post_endpoints.lst
/README.rst:
--------------------------------------------------------------------------------
1 | =========================================
2 | Guidelines for HAProxy termination in AWS
3 | =========================================
4 |
5 | .. title:: Guidelines for HAProxy termination in AWS
6 | .. |gentime| date:: %F %H:%M %Z
7 | .. role:: lvl_low
8 | :class: st_gray
9 | .. role:: lvl_medium
10 | :class: st_blue
11 | .. role:: lvl_high
12 | :class: st_yellow
13 | .. role:: lvl_max
14 | :class: st_red
15 | .. role:: ready
16 | :class: st_green
17 | .. role:: not_ready
18 | :class: st_red
19 | .. |br| raw:: html
20 |
21 |
22 |
23 | .. sidebar:: Document status
24 |
25 | +-----------------------+------------------------------------------------------+
26 | |:not_ready:`NOT READY` | $Revision: $ @ |gentime| |
27 | +===============+=======+=============+=======================+================+
28 | |**Author** |Julien Vehent |**Review** | CloudOps |
29 | +---------------+---------------------+-----------------------+----------------+
30 |
31 | .. sectnum::
32 |
33 | .. contents:: **Table of contents**
34 | :depth: 2
35 |
36 | Summary & Scope
37 | ---------------
38 |
39 | This document explains how HAProxy and Elastic Load Balancer can be used in
40 | Amazon Web Services to provide performant and secure termination of traffic
41 | to an API service. The goal is to provide the following features:
42 |
43 | - **DDoS Protection**: we use HAProxy to mitigate low to medium DDoS attacks, with
44 | sane limits and custom blacklist.
45 |
46 | - **Application firewall**: we perform a first level of filtering in HAProxy, that
47 | protects NodeJS against all sorts of attack, known and to come. This will be done
48 | by inserting a set of regexes in HAProxy ACLs, that get updated when the
49 | application routes are updated. Note that managing these ACLs will not impact
50 | uptime, or require redeployment.
51 |
52 | - **SSL/TLS**: ELBs support the PROXY protocol, and so does HAProxy, which allows us
53 | to proxy the tcp connection to HAProxy. It gives us better TLS, backed by
54 | OpenSSL, at the cost of managing the TLS keys on the HAProxy instances.
55 |
56 | - **Logging**: ELBs have limited support for logging. HAProxy, however, has excellent
57 | logging for TCP, SSL and HTTPS. We leverage the flexibility of HAProxy's logging
58 | to improve our DDoS detection capabilities. We also want to uniquely identify
59 | requests in HAProxy and NodeJS, and correlate events, using a `unique-id`.
60 |
61 | Architecture
62 | ------------
63 |
64 | Below is our target setup:
65 |
66 | .. image:: haproxy-aws-arch-diagram.png
67 | :alt: architecture diagram
68 |
69 | PROXY protocol between ELB and HAProxy
70 | --------------------------------------
71 |
72 | This configuration uses an Elastic Load Balancer in TCP mode, with PROXY
73 | protocol enabled. The PROXY protocol adds a string at the beginning of the TCP
74 | payload that is passed to the backend. This string contains the IP of the client
75 | that connected to the ELB, which allows HAProxy to feed its internal state with
76 | this information, and act as if it had a direct TCP connection to the client.
77 |
78 | For more information on the PROXY protocol, see
79 | http://haproxy.1wt.eu/download/1.5/doc/proxy-protocol.txt
80 |
81 | First, we need to create an ELB, and enable a TCP listener on port 443 that
82 | supports the PROXY protocol. The ELB will not decipher the SSL, but instead pass
83 | the entire TCP payload down to Haproxy.
84 |
85 | ELB Configuration
86 | ~~~~~~~~~~~~~~~~~
87 | PROXY protocol support must be enabled on the ELB.
88 |
89 | .. code:: bash
90 |
91 | $ ./elb-describe-lb-policy-types -I AKIA... -S Ww1... --region us-east-1
92 | POLICY_TYPE ProxyProtocolPolicyType Policy that controls whether to include the
93 | IP address and port of the originating request
94 | for TCP messages. This policy operates on
95 | TCP/SSL listeners only
96 |
97 | The policy name we want to enable is `ProxyProtocolPolicyType`. We need the load
98 | balancer name for that, and the following command:
99 |
100 | .. code:: bash
101 |
102 | $ ./elb-create-lb-policy elb123-testproxyprotocol \
103 | --policy-name EnableProxyProtocol \
104 | --policy-type ProxyProtocolPolicyType \
105 | --attribute "name=ProxyProtocol, value=true" \
106 | -I AKIA... -S Ww1... --region us-east-1
107 |
108 | OK-Creating LoadBalancer Policy
109 |
110 |
111 | $ ./elb-set-lb-policies-for-backend-server elb123-testproxyprotocol \
112 | --policy-names EnableProxyProtocol \
113 | --instance-port 443 \
114 | -I AKIA... -S Ww1... --region us-east-1
115 |
116 | OK-Setting Policies
117 |
118 | Now configure a listener on TCP/443 on that ELB, that points to TCP/443 on the
119 | HAProxy instance. On the instance side, make sure that your security group
120 | accepts traffic from the ELB security group on port 443.
121 |
122 | HAProxy frontend
123 | ~~~~~~~~~~~~~~~~
124 |
125 | The HAProxy frontend listens on port 443 with a SSL configuration, as follow:
126 |
127 | .. code::
128 |
129 | frontend https
130 | bind 0.0.0.0:443 accept-proxy ssl ......
131 |
132 | Note the `accept-proxy` parameter of the bind command. This option tells HAProxy
133 | that whatever sits in front of it will append the PROXY header to TCP payloads.
134 |
135 | SSL/TLS Configuration
136 | ~~~~~~~~~~~~~~~~~~~~~
137 |
138 | HAProxy takes a SSL configuration on the `bind` line directly. The configuration
139 | requires a set of certificates and private key, and a ciphersuite.
140 |
141 | .. code::
142 |
143 | bind 0.0.0.0:443 accept-proxy ssl crt /etc/haproxy/bundle.pem ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:AES128:AES256:RC4-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
144 |
145 | Unlike most servers (Apache, Nginx, ...), HAProxy takes certificates and keys
146 | into a single file, here named `bundle.pem`. In this file are concatenated the
147 | server private key, the server public certificate, the CA intermediate
148 | certificate (if any) and a DH parameter (if any). For more information on DH
149 | parameters, see https://wiki.mozilla.org/Security/Server_Side_TLS .
150 |
151 | In the sample below, components of `bundle.pem` are concatenated as follow:
152 |
153 | - client certificate signed by CA XYZ
154 | - client private key
155 | - public DH parameter (2048 bits)
156 | - intermediate certificate of CA XYZ
157 |
158 | .. code::
159 |
160 | -----BEGIN CERTIFICATE-----
161 | MIIGYjCCBUqgAwIBAgIDDD5PMA0GCSqGSIb3DQEBBQUAMIGMMQswCQYDVQQGEwJJ
162 | ...
163 | ej2w/mPv
164 | -----END CERTIFICATE-----
165 | -----BEGIN RSA PRIVATE KEY-----
166 | MIIEpAIBAAKCAQEAvJQqCjE4I63S3kR9KV0EG9e/lX/bZxa/2QVvZGi9/Suj65nD
167 | ...
168 | RMSEpg+wuIVnKUi6KThiMKyXfZaTX7BDuR/ezE/JHs1TN5Hkw43TCQ==
169 | -----END RSA PRIVATE KEY-----
170 | -----BEGIN DH PARAMETERS-----
171 | MIICCAKCAgEA51RNlgY6j9MhmDURTpzydlJOsjk/TpU1BiY028SXAppuKJeFcx9S
172 | ...
173 | HgHeuQQRjuv+h+Wf4dBe2f/fU5w9Osvq299vtcCjvQ7EtZTKT8RfvIMCAQI=
174 | -----END DH PARAMETERS-----
175 | -----BEGIN CERTIFICATE-----
176 | MIIGNDCCBBygAwIBAgIBGDANBgkqhkiG9w0BAQUFADB9MQswCQYDVQQGEwJJTDEW
177 | ...
178 | 0q6Dp6jOW6c=
179 | -----END CERTIFICATE-----
180 |
181 | The rest of the `bind` line is a ciphersuite, taken from
182 | https://wiki.mozilla.org/Security/Server_Side_TLS .
183 |
184 | We can verify the configuration using `cipherscan`. Below is the expected output
185 | for our configuration:
186 |
187 | .. code:: bash
188 |
189 | $ ./cipherscan haproxytest1234.elb.amazonaws.com
190 | .........................
191 | prio ciphersuite protocols pfs_keysize
192 | 1 ECDHE-RSA-AES128-GCM-SHA256 SSLv3,TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits
193 | 2 ECDHE-RSA-AES256-GCM-SHA384 SSLv3,TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits
194 | 3 DHE-RSA-AES128-GCM-SHA256 SSLv3,TLSv1,TLSv1.1,TLSv1.2 DH,2048bits
195 | 4 DHE-RSA-AES256-GCM-SHA384 SSLv3,TLSv1,TLSv1.1,TLSv1.2 DH,2048bits
196 | 5 ECDHE-RSA-AES128-SHA256 SSLv3,TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits
197 | 6 ECDHE-RSA-AES128-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits
198 | 7 ECDHE-RSA-AES256-SHA384 SSLv3,TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits
199 | 8 ECDHE-RSA-AES256-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits
200 | 9 DHE-RSA-AES128-SHA256 SSLv3,TLSv1,TLSv1.1,TLSv1.2 DH,2048bits
201 | 10 DHE-RSA-AES128-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 DH,2048bits
202 | 11 DHE-RSA-AES256-SHA256 SSLv3,TLSv1,TLSv1.1,TLSv1.2 DH,2048bits
203 | 12 DHE-RSA-AES256-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 DH,2048bits
204 | 13 AES128-GCM-SHA256 SSLv3,TLSv1,TLSv1.1,TLSv1.2
205 | 14 AES256-GCM-SHA384 SSLv3,TLSv1,TLSv1.1,TLSv1.2
206 | 15 ECDHE-RSA-RC4-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 ECDH,P-256,256bits
207 | 16 AES128-SHA256 SSLv3,TLSv1,TLSv1.1,TLSv1.2
208 | 17 AES128-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2
209 | 18 AES256-SHA256 SSLv3,TLSv1,TLSv1.1,TLSv1.2
210 | 19 AES256-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2
211 | 20 RC4-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2
212 | 21 DHE-RSA-CAMELLIA256-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 DH,2048bits
213 | 22 CAMELLIA256-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2
214 | 23 DHE-RSA-CAMELLIA128-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2 DH,2048bits
215 | 24 CAMELLIA128-SHA SSLv3,TLSv1,TLSv1.1,TLSv1.2
216 |
217 | Healthchecks between ELB and HAProxy
218 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
219 |
220 | As of writing of this document, it appears that ELBs do not use the proxy
221 | protocol when running healthchecks against an instance. As a result, these
222 | healthchecks cannot be handled by the `https frontend`, because HAProxy will
223 | fail when looking for a PROXY header that isn't there.
224 |
225 | The workaround is to create a secondary `frontend` in HAProxy that is entirely
226 | dedicated to answering healthchecks from the ELB.
227 |
228 | The configuration below uses the `monitor` option to check the health of the
229 | nodejs backend. If more than one server is alive in that backend, then our
230 | `health` frontend will return `200 OK`. If no server is alive, a `503` will be
231 | returned. All the ELB has to do is to query the URL at
232 | http://haproxy_host:34180/haproxy_status . To reduce the overhead, we also
233 | disable SSL on the health frontend.
234 |
235 | .. code::
236 |
237 | # frontend used to return health status without requiring SSL
238 | frontend health
239 | bind 0.0.0.0:34180 # 34180 means EALTH ;)
240 | # create a status URI in /haproxy_status that will return
241 | # a 200 is backend is healthy, and 503 if it isn't. This
242 | # URI is queried by the ELB.
243 | acl backend_dead nbsrv(nodejs) lt 1
244 | monitor-uri /haproxy_status
245 | monitor fail if backend_dead
246 |
247 | *(note: we could also use ACLs in HAProxy to only expect the PROXY header on
248 | certain source IPs, but the approach of a dedicated health frontend seems
249 | cleaner)*
250 |
251 | ELB Logging
252 | -----------
253 | TODO
254 |
255 | HAProxy Logging
256 | ---------------
257 |
258 | HAProxy supports custom log format, which we want here, as opposed to default
259 | log format, in order to capture TCP, SSL and HTTP information on a single line.
260 |
261 | For our logging, we want the following:
262 |
263 | 1. TCP/IP logs first, such that these are always present, even if HAProxy cuts
264 | the connection before processing the SSL or HTTP traffic
265 | 2. SSL information
266 | 3. HTTP information
267 |
268 | .. code::
269 |
270 | log-format [%pid]\ [%Ts.%ms]\ %ac/%fc/%bc/%bq/%sc/%sq/%rc\ %Tq/%Tw/%Tc/%Tr/%Tt\ %tsc\ %ci:%cp\ %fi:%fp\ %si:%sp\ %ft\ %sslc\ %sslv\ %{+Q}r\ %ST\ %b:%s\ "%CC"\ "%hr"\ "%CS"\ "%hs"\ req_size=%U\ resp_size=%B
271 |
272 | The format above will generate:
273 |
274 | .. code::
275 |
276 | Mar 14 17:14:51 localhost haproxy[14887]: [14887] [1394817291.250] 10/5/2/0/3/0/0 48/0/0/624/672 ---- 1.10.2.10:35701 10.151.122.228:443 127.0.0.1:8000 logger - - "GET /v1/ HTTP/1.0" 404 fxa-nodejs:nodejs1 "-" "{||ApacheBench/2.3|over-100-active-connections,over-100-connections-in-10-seconds,high-error-rate,high-request-rate,|47B4176E:8B75_0A977AE4:01BB_5323390B_31E0:3A27}" "-" "" ireq_size=592 resp_size=787
277 |
278 | The log-format contains very detailed information on the connection itself, but
279 | also on the state of haproxy itself. Below is a description of the fields we
280 | used in our custom log format.
281 |
282 | * `%pid`: process ID of HAProxy
283 | * `%Ts.%ms`: unix timestamp + milliseconds
284 | * `%ac`: total number of concurrent connections
285 | * `%fc`: total number of concurrent connections on the frontend
286 | * `%bc`: total number of concurrent connections on the backend
287 | * `%bq`: queue size of the backend
288 | * `%sc`: total number of concurrent connections on the server
289 | * `%sq`: queue size of the server
290 | * `%rc`: connection retries to the server
291 | * `%Tq`: total time to get the client request (HTTP mode only)
292 | * `%Tw`: total time spent in the queues waiting for a connection slot
293 | * `%Tc`: total time to establish the TCP connection to the server
294 | * `%Tr`: server response time (HTTP mode only)
295 | * `%Tt`: total session duration time, between the moment the proxy accepted it
296 | and the moment both ends were closed.
297 | * `%tsc`: termination state (see `8.5. Session state at disconnection`)
298 | * `%ci:%cp`: client IP and Port
299 | * `%fi:%fp`: frontend IP and Port
300 | * `%si:%sp`: server IP and Port
301 | * `%ft`: transport type of the frontend (with a ~ suffix for SSL)
302 | * `%sslc %sslv`: SSL cipher and version
303 | * `%{+Q}r`: HTTP request, between double quotes
304 | * `%ST`: HTTP status code
305 | * `%b:%s`: backend name and server name
306 | * `%CC`: captured request cookies
307 | * `%hr`: captured request headers
308 | * `%CS`: captured response cookies
309 | * `%hs`: captured response headers
310 | * `%U`: bytes read from the client (request size)
311 | * `%B`: bytes read from server to client (response size)
312 |
313 | For more details on the available logging variables, see the HAProxy
314 | configuration, under `8.2.4. Custom log format`.
315 | http://haproxy.1wt.eu/download/1.5/doc/configuration.txt
316 |
317 | Unique request ID
318 | ~~~~~~~~~~~~~~~~~
319 |
320 | Tracking requests across multiple servers can be problematic, because the chain
321 | of events triggered by a request on the frontend are not tied to each other.
322 | HAProxy has a simple mechanism to insert a unique identifier to incoming
323 | requests, in the form of an ID inserted in the request headers, and passed to
324 | the backend server. This ID can then be logged by the backend server, and passed
325 | on to the next step. In a largely distributed environment, the unique ID makes
326 | tracking requests propagation a lot easier.
327 |
328 | The unique ID is declared on the HTTPS frontend as follow:
329 |
330 | .. code::
331 |
332 | # Insert a unique request identifier is the headers of the request
333 | # passed to the backend
334 | unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
335 | unique-id-header X-Unique-ID
336 |
337 | This will add an ID that is composed of hexadecimal variables, taken from the
338 | client IP and port, frontend IP and port, timestamp, request counter and PID.
339 | An example of generated ID is **485B7525:CB2F_0A977AE4:01BB_5319CB0C_000D:27C0**.
340 |
341 | The Unique ID is added to the request headers passed to the backend in the
342 | `X-Unique-ID` header. We will also capture it in the logs, as a request header.
343 |
344 | ::
345 |
346 | GET / HTTP/1.1
347 | Host: backendserver123.example.net
348 | User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0
349 | Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
350 | Accept-Language: en-US,en;q=0.5
351 | Accept-Encoding: gzip, deflate
352 | DNT: 1
353 | Cache-Control: max-age=0
354 | X-Unique-ID: 485B7525:CB70_0A977AE4:01BB_5319CD3F_0163:27C0
355 | X-Forwarded-For: 2.12.17.87
356 |
357 | Capturing headers and cookies
358 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
359 |
360 | In the log format, we defined fields for the request and response headers and
361 | cookies. But, by default, this fields will show empty in the logs. In order to
362 | log headers and cookies, the `capture` parameters must be set in the
363 | frontend.
364 |
365 | Here is how we can capture headers sent by the client in the HTTP request.
366 |
367 | .. code::
368 |
369 | capture request header Referrer len 64
370 | capture request header Content-Length len 10
371 | capture request header User-Agent len 64
372 |
373 | Cookies can be captures the same way:
374 |
375 | .. code::
376 |
377 | capture cookie mycookie123= len 32
378 |
379 | HAProxy will also add custom headers to the request, before passing it to the
380 | backend. However, added headers don't get logged, because the addition happens
381 | after the capture operation. To fix this issue, we are going to create a new
382 | frontend dedicated to logging.
383 |
384 | Logging in a separate frontend
385 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
386 |
387 | During processing of the request, we added custom headers, and we want these
388 | headers to appear in the logs. One solution is to route all the request to a
389 | secondary frontend that only does logging, and blocking or forwarding.
390 |
391 | Classic setup:
392 |
393 | ::
394 |
395 | {logging}
396 | request +--------------+ +---------------+
397 | +-------------->|frontend |+----->|backend | +---------+
398 | | fxa-https | | fxa-nodejs |+---->| |
399 | +--------------+ +---------------+ | NodeJS |
400 | | |
401 | +---------+
402 |
403 | Setup with separate logging frontend:
404 |
405 | ::
406 |
407 | {no logging}
408 | request +--------------+ +---------------+
409 | +-------------->|frontend | |backend | +---------+
410 | | fxa-https | | fxa-nodejs |+---->| |
411 | +--------------+ +---------------+ | NodeJS |
412 | + ^ | |
413 | | | +---------+
414 | | |
415 | +------v-------+ +-----+--------+
416 | |backend |+----->|frontend |
417 | | logger | | logger |
418 | +--------------+ +--------------+
419 | {logging}
420 |
421 |
422 | At the end of the configuration of frontend `fxa-https`, instead of sending
423 | requests to backend `fxa-nodejs`, we send them to backend `logger`.
424 |
425 | .. code::
426 |
427 | frontend fxa-https
428 | ...
429 | # Don't log here, log into logger frontend
430 | no log
431 | default_backend logger
432 |
433 | Then we declare a backend and a frontend for `logger`:
434 |
435 | .. code::
436 |
437 | backend logger
438 | server localhost localhost:55555 send-proxy
439 |
440 | # frontend use to log acl activity
441 | frontend logger
442 | bind localhost:55555 accept-proxy
443 |
444 | ...
445 |
446 | capture request header Referrer len 64
447 | capture request header Content-Length len 10
448 | capture request header User-Agent len 64
449 | capture request header X-Haproxy-ACL len 256
450 | capture request header X-Unique-ID len 64
451 |
452 | # if previous ACL didn't pass and aren't whitelisted
453 | acl whitelisted req.fhdr(X-Haproxy-ACL) -m beg whitelisted,
454 | acl fail-validation req.fhdr(X-Haproxy-ACL) -m found
455 | http-request deny if !whitelisted fail-validation
456 |
457 | default_backend fxa-nodejs
458 |
459 | Note the use of `send-proxy` and `accept-proxy` between the logger backend and
460 | frontend, allowing to keep the information about the client IP.
461 |
462 | **Isn't this slow and inefficient?**
463 |
464 | Well, obviously, routing request through HAProxy twice isn't the most elegant
465 | way of proxying. But in practice, this approach adds minimal overhead. Linux and
466 | HAProxy support TCP splicing, which provides zero-copy transfer of data between
467 | TCP sockets. When HAProxy forward the request to the logger socket, there is, in
468 | fact, no transfer of data at the kernel level. Benchmark it, it's fast!
469 |
470 | Rate limiting & DDoS protection
471 | -------------------------------
472 |
473 | One of the particularity of operating an infrastructure in AWS, is that control
474 | over the network is very limited. Techniques such as BGP blackholing are not
475 | available. And visibility over the layer 3 (IP) and 4 (TCP) is reduced. Building
476 | protection against DDoS means that we need to block traffic further down the
477 | stack, which consumes more resources. This is the main motivation for using ELBs
478 | in TCP mode with the PROXY protocol: it gives HAProxy low-level access to the
479 | TCP connection, and visibility of the client IP before parsing HTTP headers
480 | (like you would traditionally do with X-Forwarded-For).
481 |
482 | ELBs have limited resources, but simplify the management of public IPs in AWS.
483 | By offloading the SSL & HTTP processing to HAProxy, we reduce the pressure on
484 | ELB, while conserving the ability to manage the public endpoints through it.
485 |
486 | HAProxy maintains tons of detailed information on connections. One can use this
487 | information to accept, block or route connections. In the following section, we
488 | will discuss the use of ACLs and stick-tables to block clients that do not
489 | respect sane limits.
490 |
491 | Automated rate limiting
492 | ~~~~~~~~~~~~~~~~~~~~~~~
493 |
494 | The configuration below enable counters to track connections in a table where
495 | the key is the source IP of the client:
496 |
497 | .. code::
498 |
499 | # Define a table that will store IPs associated with counters
500 | stick-table type ip size 500k expire 30s store conn_cur,conn_rate(10s),http_req_rate(10s),http_err_rate(10s)
501 |
502 | # Enable tracking of src IP in the stick-table
503 | tcp-request content track-sc0 src
504 |
505 | Let's decompose this configuration. First, we define a `stick-table` that
506 | stores IP addresses as keys. We define a maximum size for this table
507 | of 500,000 IPs, and we tell HAProxy to expire the records after 30 seconds. If
508 | the table gets filled, HAProxy will delete records following the LRU logic.
509 |
510 | The `stick-table` will store a number of information associated with the IP
511 | address:
512 |
513 | - `conn_cur` is a counter of the concurrent connection count for this IP.
514 |
515 | - `conn_rate(10s)` is a sliding window that counts new TCP connections over a 10
516 | seconds period
517 |
518 | - `http_req_rate(10s)` is a sliding window that counts HTTP requests over a 10
519 | seconds period
520 |
521 | - `http_err_rate(10s)` is a sliding window that counts HTTP errors triggered by
522 | requests from that IP over a 10 seconds period
523 |
524 | By default, the stick table declaration doesn't do anything, we need to send
525 | data to it. This is what the `tcp-request content track-sc0 src` parameter does.
526 |
527 | Now that we have tracking in place, we can write ACLs that run tests against the
528 | content of the table. The examples below evaluate several of these counters
529 | against arbitrary limits. Tune these to your needs.
530 |
531 | .. code::
532 |
533 | # Reject the new connection if the client already has 100 opened
534 | http-request add-header X-Haproxy-ACL %[req.fhdr(X-Haproxy-ACL,-1)]over-100-active-connections, if { src_conn_cur ge 100 }
535 |
536 | # Reject the new connection if the client has opened more than 100 connections in 10 seconds
537 | http-request add-header X-Haproxy-ACL %[req.fhdr(X-Haproxy-ACL,-1)]over-100-connections-in-10-seconds, if { src_conn_rate ge 100 }
538 |
539 | # Reject the connection if the client has passed the HTTP error rate
540 | http-request add-header X-Haproxy-ACL %[req.fhdr(X-Haproxy-ACL,-1)]high-error-rate, if { sc0_http_err_rate() gt 100 }
541 |
542 | # Reject the connection if the client has passed the HTTP request rate
543 | http-request add-header X-Haproxy-ACL %[req.fhdr(X-Haproxy-ACL,-1)]high-request-rate, if { sc0_http_req_rate() gt 500 }
544 |
545 | HAProxy provides a lot of flexibility on what can be tracked in a `stick-table`.
546 | Take a look at section `7.3.2. Fetching samples at Layer 4` from the doc to get
547 | a better idea.
548 |
549 | Querying tables state in real time
550 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
551 |
552 | Tables are named after the name of the frontend or backend they live in. Our
553 | frontend called `fxa-https` will have a table called `fxa-https`, that can be
554 | queried through the stat socket:
555 |
556 | .. code::
557 |
558 | # echo "show table fxa-https" | socat unix:/var/lib/haproxy/stats -
559 | # table: fxa-https, type: ip, size:512000, used:1
560 | 0x1aa3358: key=1.10.2.10 use=1 exp=29957 conn_rate(10000)=43 conn_cur=1 http_req_rate(10000)=42 http_err_rate(10000)=42
561 |
562 | The line above shows a table entry for key `1.10.2.10`, which is a tracked IP
563 | address. The other entries on the line show the status of various counters that
564 | we defined in the configuration.
565 |
566 | Blacklists & Whitelists
567 | ~~~~~~~~~~~~~~~~~~~~~~~
568 |
569 | Blacklist and whitelists are simple lists of IP addresses that are checked by
570 | HAProxy as early on as possible. Blacklist are checked at the beginning of the
571 | TCP connection, which allows for early connection drops, and also means that
572 | blacklisting an IP always takes precedence over any other rule, including the
573 | whitelist.
574 |
575 | Whitelists are checked at the HTTP level, and allow to bypass ACLs and rate
576 | limiting.
577 |
578 | .. code::
579 |
580 | # Blacklist: Deny access to some IPs before anything else is checked
581 | tcp-request content reject if { src -f /etc/haproxy/blacklist.lst }
582 |
583 | # Whitelist: Allow IPs to bypass the filters
584 | http-request add-header X-Haproxy-ACL %[req.fhdr(X-Haproxy-ACL,-1)]whitelisted, if { src -f /etc/haproxy/whitelist.lst }
585 | http-request allow if { src -f /etc/haproxy/whitelist.lst }
586 |
587 | List files can contain IP addresses or networks in CIDR format.
588 |
589 | .. code::
590 |
591 | 10.0.0.0/8
592 | 172.16.0.0/12
593 | 192.168.0.0/16
594 | 8.8.8.8
595 |
596 | List files are loaded into HAProxy at startup. If you add or remove IPs from a
597 | list, make sure to perform a soft reload.
598 |
599 | .. code:: bash
600 |
601 | haproxy -f /etc/haproxy/haproxy.cfg -c && sudo haproxy -f /etc/haproxy/haproxy.cfg -sf $(pidof haproxy)
602 |
603 | Protect against slow clients (Slowloris attack)
604 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
605 |
606 | Slowloris is an attack where a client very slowly sends requests to the server,
607 | forcing it to allocate resources to that client that are only not used. This
608 | attack is commonly used in DDoS, by clients that send their requests characters
609 | by characters. HAProxy can block these clients, by allocating a maximum amount
610 | of time a client can take to send a full request. This is done with the `timeout
611 | http-request` parameter.
612 |
613 | .. code::
614 |
615 | # disconnect slow handshake clients early, protect from
616 | # resources exhaustion attacks
617 | timeout http-request 5s
618 |
619 | URL filtering with ACLs
620 | -----------------------
621 |
622 | HAProxy has the ability to inspect requests before passing them to the backend.
623 | This is limited to query strings, and doesn't support inspecting the body of a
624 | POST request. But we can already leverage this to filter out unwanted traffic.
625 |
626 | The first thing we need, is a list of endpoints sorted by HTTP method. This can
627 | be obtained from the web application directly. Note that some endpoints, such as
628 | `__heartbeat__` should be limited to HAProxy, and thus blocked from clients.
629 |
630 | For now, let's ignore GET URL parameters, and only build a list of request
631 | paths, that we store in two files: one for GET requests, and one for POST
632 | requests.
633 |
634 | `get_endpoints.lst`
635 |
636 | .. include :: get_endpoints.lst
637 | :code: bash
638 |
639 | `post_endpoints.lst`
640 |
641 | .. include :: post_endpoints.lst
642 | :code: bash
643 |
644 | In the HAProxy configuration, we can build ACLs around these files. The `http-request deny`
645 | method takes a condition, as described in the Haproxy documentation, section
646 | `7.2. Using ACLs to form conditions`.
647 |
648 | .. code::
649 |
650 | # Requests validation using ACLs ---
651 | acl valid-get path -f /etc/haproxy/get_endpoints.lst
652 | acl valid-post path -f /etc/haproxy/post_endpoints.lst
653 |
654 | # block requests that don't match the predefined endpoints
655 | http-request deny unless METH_GET valid-get or METH_POST valid-post
656 |
657 | `http-request deny` does the job, and return a 403 to the client. But if you want
658 | more visibility on ACL activity, you may want to use a custom header as describe
659 | later in this section.
660 |
661 | Filtering URL parameters on GET requests
662 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
663 |
664 | While HAProxy supports regexes on URLs, writing regexes that can validate URL
665 | parameters is a path that leads to frustration and insanity. A much simpler
666 | approach consists of using the `url_param` ACL provided by HAProxy.
667 |
668 | For example, take the NodeJS endpoint below:
669 |
670 | .. code:: javascript
671 |
672 | {
673 | method: 'GET',
674 | path: '/verify_email',
675 | config: {
676 | validate: {
677 | query: {
678 | code: isA.string().max(32).regex(HEX_STRING).required(),
679 | uid: isA.string().max(32).regex(HEX_STRING).required(),
680 | service: isA.string().max(16).alphanum().optional(),
681 | redirectTo: isA.string()
682 | .max(512)
683 | .regex(validators.domainRegex(redirectDomain))
684 | .optional()
685 | }
686 | }
687 | },
688 | handler: function (request, reply) {
689 | return reply().redirect(config.contentServer.url + request.raw.req.url)
690 | }
691 | },
692 |
693 | This endpoints receives requests on `/verify_email` with the parameters `code`,
694 | a 32 character hexadecimal, `uid`, a 32 character hexadecimal, `service`, a 16
695 | character string, and `redirectTo`, a FQDN. However, only `code` and `uid` are
696 | required.
697 |
698 | In the previous section, we validated that requests on `/verify_email` must use
699 | the method GET. Now we are taking the validation one step further, and blocking
700 | requests on this endpoint that do not match our prerequisite.
701 |
702 | .. code::
703 |
704 | acl endpoint-verify_email path /verify_email
705 | acl param-code urlp_reg(code) [0-9a-fA-F]{1,32}
706 | acl param-uid urlp_reg(uid) [0-9a-fA-F]{1,32}
707 | http-request deny if endpoint-verify_email !param-code or endpoint-verify_email !param-uid
708 |
709 | The follow request will be accepted, everything else will be rejected with a
710 | HTTP error 403.
711 |
712 | .. code::
713 |
714 | https://haproxy_server/verify_email?code=d64f53326cec3a1af60166a929ca52bd&uid=d64f53326cec3a1af60166a929c3d7b2131561792b4837377ed2e0cde3295df2
715 |
716 | Using regexes to validate URL parameters is a powerful feature. Below is another
717 | example that matches an email addresses using case-insensitive regex:
718 |
719 | .. code::
720 |
721 | acl endpoint-complete_reset_password path /complete_reset_password
722 | acl param-email urlp_reg(email) -i ^[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}$
723 | acl param-token urlp_reg(token) [0-9a-fA-F]{1,64}
724 | http-request deny if endpoint-complete_reset_password !param-email or endpoint-complete_reset_password !param-token or endpoint-complete_reset_password !param-code
725 |
726 | Note that we didn't redefine `param-code` when we reused it in the `http-request deny`
727 | command. This is because ACL are defined globally for a frontend, and can
728 | be reused multiple times.
729 |
730 | Filtering payloads on POST requests
731 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
732 |
733 | POST requests are harder to validate, because they do not follow a predefined
734 | format, but also because the client could be sending the body over a long period
735 | of time, split over dozens of packets.
736 |
737 | However, in the case of an API that only handles small POST payloads, we can at
738 | least verify the size of the payload sent by the client, and make sure that
739 | clients do not overload the backend with random data. This can be done using an
740 | ACL on the content-length header of the request. The ACL below discard requests
741 | that have a content-length larger than 5 kilo-bytes (which is already a lot of
742 | text).
743 |
744 | .. code::
745 |
746 | # match content-length larger than 5kB
747 | acl request-too-big hdr_val(content-length) gt 5000
748 | http-request deny if METH_POST request-too-big
749 |
750 | Marking instead of blocking
751 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~
752 |
753 | Blocking requests may be the preferred behavior in production, but only after a
754 | grace period that allows you to build a traffic profile, and fine tune your
755 | configuration. Instead of using `http-request deny` statements in the ACLs, we
756 | can insert a header with a description of the blocking decision. This header
757 | will be logged, and can be analyzed to verify that no legitimate traffic would
758 | be blocked.
759 |
760 | As discussed in `Logging in a separate frontend`, HAProxy is unable to log
761 | request header that it has set itself. So make sure to log in a separate
762 | frontend if you use this technique.
763 |
764 | The configuration below uses a custom header `X-Haproxy-ACL`. If an ACL matches,
765 | the header is set to the name of the ACL that matched. If several ACLs match,
766 | each ACL name is appended to the header, and separated by a comma.
767 |
768 | At the end of the ACL evaluation, if this header is present in the request, we
769 | know that the request should be blocked.
770 |
771 | In the `fxa-https` frontend, we replace the `http-request deny` paramameters with the
772 | following logic:
773 |
774 | .. code::
775 |
776 | # ~~~ Requests validation using ACLs ~~~
777 | # block requests that don't match the predefined endpoints
778 | acl valid-get path -f /etc/haproxy/get_endpoints.lst
779 | acl valid-post path -f /etc/haproxy/post_endpoints.lst
780 | http-request add-header X-Haproxy-ACL %[req.fhdr(X-Haproxy-ACL,-1)]invalid-endpoint, unless METH_GET valid-get or METH_POST valid-post
781 |
782 | # block requests on verify_email that do not have the correct params
783 | acl endpoint-verify_email path /v1/verify_email
784 | acl param-code urlp_reg(code) [0-9a-fA-F]{1,32}
785 | acl param-uid urlp_reg(uid) [0-9a-fA-F]{1,32}
786 | http-request add-header X-Haproxy-ACL %[req.fhdr(X-Haproxy-ACL,-1)]invalid-parameters, if endpoint-verify_email !param-code or endpoint-verify_email !param-uid
787 |
788 | # block requests on complete_reset_password that do not have the correct params
789 | acl endpoint-complete_reset_password path /v1/complete_reset_password
790 | acl param-email urlp_reg(email) -i ^[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}$
791 | acl param-token urlp_reg(token) [0-9a-fA-F]{1,64}
792 | http-request add-header X-Haproxy-ACL %[req.fhdr(X-Haproxy-ACL,-1)]invalid-parameters, if endpoint-complete_reset_password !param-email or endpoint-complete_reset_password !param-token or endpoint-complete_reset_password !param-code
793 |
794 | # block content-length larger than 500kB
795 | acl request-too-big hdr_val(content-length) gt 5000
796 | http-request add-header X-Haproxy-ACL %[req.fhdr(X-Haproxy-ACL,-1)]request-too-big, if METH_POST request-too-big
797 |
798 | Note the `%[req.fhdr(X-Haproxy-ACL,-1)]` parameter, that retrieves the value of
799 | the latest occurence of the `X-Haproxy-ACL` header, so we can append to it and
800 | store it again. However, this will create multiple headers if more than one ACL
801 | is matched, but that's OK because:
802 | - we can delete them before sending the request to the backend, using `reqdel`
803 | - the logging directive `capture request header` will only log the last occurence
804 |
805 | .. code::
806 |
807 | X-Haproxy-ACL: over-100-active-connections,
808 | X-Haproxy-ACL: over-100-active-connections,over-100-connections-in-10-seconds,
809 | X-Haproxy-ACL: over-100-active-connections,over-100-connections-in-10-seconds,high-error-rate,
810 | X-Haproxy-ACL: over-100-active-connections,over-100-connections-in-10-seconds,high-error-rate,high-request-rate,
811 |
812 | Then, in the logger frontend, we check the value of the header, and block if
813 | needed.
814 |
815 | .. code::
816 |
817 | # frontend use to log acl activity
818 | frontend logger
819 | ...
820 | # if previous ACL didn't pass, and IP isn't whitelisted, block the request
821 | acl whitelisted req.fhdr(X-Haproxy-ACL) -m beg whitelisted,
822 | acl fail-validation req.fhdr(X-Haproxy-ACL) -m found
823 | http-request deny if !whitelisted fail-validation
824 |
825 | HAProxy management
826 | ------------------
827 |
828 | Enabling the stat socket
829 | ~~~~~~~~~~~~~~~~~~~~~~~~
830 |
831 | Collecting statistics
832 | ~~~~~~~~~~~~~~~~~~~~~
833 |
834 | Analyzing errors
835 | ~~~~~~~~~~~~~~~~
836 |
837 | Parsing performance metrics from the logs
838 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
839 |
840 | Soft reload
841 | ~~~~~~~~~~~
842 | HAProxy supports soft configuration reload, that doesn't drop connections. To
843 | perform a soft reload, call haproxy with the following command:
844 |
845 | .. code:: bash
846 |
847 | $ sudo /opt/haproxy -f /etc/haproxy/haproxy.cfg -sf $(pidof haproxy)
848 |
849 | The old process will be replaced with a new one, that uses a fresh
850 | configuration. The logs will show the reload:
851 |
852 | ::
853 |
854 | Mar 6 12:59:41 localhost haproxy[7603]: Proxy https started.
855 | Mar 6 12:59:41 localhost haproxy[7603]: Proxy app started.
856 | Mar 6 12:59:41 localhost haproxy[5763]: Stopping frontend https in 0 ms.
857 | Mar 6 12:59:41 localhost haproxy[5763]: Stopping backend app in 0 ms.
858 | Mar 6 12:59:41 localhost haproxy[5763]: Proxy https stopped (FE: 29476 conns, BE: 0 conns).
859 | Mar 6 12:59:41 localhost haproxy[5763]: Proxy app stopped (FE: 0 conns, BE: 1746 conns).
860 |
861 | Full HAProxy configuration
862 | --------------------------
863 |
864 | .. include:: haproxy.cfg
865 | :code: bash
866 |
867 | Building process
868 | ----------------
869 |
870 | Static build
871 | ~~~~~~~~~~~~
872 | The script `build_static_haproxy.sh`_ builds haproxy with statically linked
873 | OpenSSL and PCRE support.
874 |
875 | .. _`build_static_haproxy.sh`: build_static_haproxy.sh
876 |
877 | .. include:: build_static_haproxy.sh
878 | :code: bash
879 |
880 | Dynamic build
881 | ~~~~~~~~~~~~~
882 | The script `build_dynamic_haproxy.sh`_ does the same as above, but links to
883 | PCRE and OpenSSL dynamically.
884 |
885 | .. _`build_dynamic_haproxy.sh`: build_dynamic_haproxy.sh
886 |
887 | .. include:: build_dynamic_haproxy.sh
888 | :code: bash
889 |
890 | RPM build
891 | ~~~~~~~~~
892 | Using the spec file in `haproxy.spec`_ and bash scripts in `build_rpm.sh`_,
893 | we can build a RPM package using for the latest development version of HAProxy.
894 |
895 | .. _`haproxy.spec`: haproxy.spec
896 |
897 | .. _`build_rpm.sh`: build_rpm.sh
898 |
899 | `haproxy.spec`
900 |
901 | .. include:: haproxy.spec
902 | :code: bash
903 |
904 | `build_rpm.sh`
905 |
906 | .. include:: build_rpm.sh
907 | :code: bash
908 |
--------------------------------------------------------------------------------
/build_dynamic_haproxy.sh:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env bash
2 |
3 | # some dependencies for rhel/fedora/centos/sl
4 | [ -e /etc/redhat-release ] && sudo yum install pcre-devel pcre-static gcc make openssl-devel
5 |
6 | # create a new dir for the build
7 | BD="build$(date +%s)"
8 | echo working in $BD
9 | mkdir $BD
10 | cd $BD
11 |
12 | #-- Build static haproxy
13 | wget http://www.haproxy.org/download/1.5/src/haproxy-1.5.1.tar.gz
14 | tar -xzvf haproxy-1.5.1.tar.gz
15 | cd haproxy-1.5.1
16 | make clean
17 | make TARGET=linux2628 USE_OPENSSL=1
18 | ./haproxy -vv
19 |
20 | [ $? -lt 1 ] && echo haproxy successfully built at $BD/haproxy-1.5.1/haproxy
21 |
--------------------------------------------------------------------------------
/build_rpm.sh:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env bash
2 |
3 | haproxyversion="1.5.1"
4 |
5 | echo Installing dependencies
6 | sudo yum -y install rpmdevtools pcre-devel openssl-devel gcc make
7 |
8 | echo Backup up previous build dir
9 | [ -e ~/rpmbuild ] && mv ~/rpmbuild rpmbuild-$(date +%s)
10 |
11 | echo Initializing build dir
12 | rpmdev-setuptree
13 |
14 | echo Download HAProxy source
15 | wget http://haproxy.1wt.eu/download/1.5/src/haproxy-$haproxyversion.tar.gz -O ~/rpmbuild/SOURCES/haproxy-$haproxyversion.tar.gz
16 |
17 | echo Copying SPEC file over to build dir
18 | cp haproxy.spec ~/rpmbuild/SPECS/
19 |
20 | echo Building RPM
21 | rpmbuild -bb ~/rpmbuild/SPECS/haproxy.spec
22 |
23 | echo Moving RPMs to local dir
24 | find ~/rpmbuild/RPMS/ -type f -name *.rpm -exec cp {} . \;
25 |
26 | echo Backing up build dir
27 | mv ~/rpmbuild rpmbuild-$(date +%s)
28 |
--------------------------------------------------------------------------------
/build_static_haproxy.sh:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env bash
2 |
3 | # some dependencies for rhel/fedora/centos/sl
4 | [ -e /etc/redhat-release ] && sudo yum install pcre-devel pcre-static gcc make
5 |
6 | # create a new dir for the build
7 | BD="build$(date +%s)"
8 | echo working in $BD
9 | mkdir $BD
10 | cd $BD
11 |
12 | #-- Build static openssl
13 | wget http://www.openssl.org/source/openssl-1.0.1h.tar.gz
14 | tar -xzvf openssl-1.0.1h.tar.gz
15 | cd openssl-1.0.1h
16 | export STATICLIBSSL="../staticlibssl"
17 | rm -rf "$STATICLIBSSL"
18 | mkdir "$STATICLIBSSL"
19 | make clean
20 | ./config --prefix=$STATICLIBSSL no-shared enable-ec_nistp_64_gcc_128
21 | make depend
22 | make
23 | make install_sw
24 |
25 | #-- Build static haproxy
26 | cd ..
27 | wget http://haproxy.1wt.eu/download/1.5/src/haproxy-1.5.1.tar.gz
28 | tar -xzvf haproxy-1.5.1.tar.gz
29 | cd haproxy-1.5.1
30 | make clean
31 | make TARGET=linux2628 USE_STATIC_PCRE=1 USE_OPENSSL=1 SSL_INC=$STATICLIBSSL/include SSL_LIB="$STATICLIBSSL/lib -ldl"
32 | ./haproxy -vv
33 |
34 | [ $? -lt 1 ] && echo haproxy successfully built at $BD/haproxy-1.5.1/haproxy
35 |
--------------------------------------------------------------------------------
/get_endpoints.lst:
--------------------------------------------------------------------------------
1 | /v1/
2 | /v1/account/devices
3 | /v1/account/keys
4 | /v1/complete_reset_password
5 | /v1/recovery_email/status
6 | /v1/verify_email
7 | /v1/.well-known/browserid
8 | /v1/.well-known/browserid/provision.html
9 | /v1/.well-known/browserid/sign_in.html
10 |
--------------------------------------------------------------------------------
/haproxy-aws-arch-diagram.png:
--------------------------------------------------------------------------------
https://raw.githubusercontent.com/jvehent/haproxy-aws/cdfe45235bcbeafbbad8723ba8eda39f398b17f3/haproxy-aws-arch-diagram.png
--------------------------------------------------------------------------------
/haproxy.cfg:
--------------------------------------------------------------------------------
1 | global
2 | # to have these messages end up in /var/log/haproxy.log you will
3 | # need to:
4 | #
5 | # 1) configure syslog to accept network log events. This is done
6 | # by adding the '-r' option to the SYSLOGD_OPTIONS in
7 | # /etc/sysconfig/syslog
8 | #
9 | # 2) configure local2 events to go to the /var/log/haproxy.log
10 | # file. A line like the following can be added to
11 | # /etc/sysconfig/syslog
12 | #
13 | # local2.* /var/log/haproxy.log
14 | #
15 | log 127.0.0.1 local2
16 | chroot /var/lib/haproxy
17 | pidfile /var/run/haproxy.pid
18 | user haproxy
19 | group haproxy
20 | daemon
21 |
22 | # turn on stats unix socket
23 | stats socket ipv4@127.0.0.1:51415 level admin
24 | stats timeout 2m
25 |
26 | #---------------------------------------------------------------------
27 | # common defaults that all the 'listen' and 'backend' sections will
28 | # use if not designated in their block
29 | #---------------------------------------------------------------------
30 | defaults
31 | mode http
32 | log global
33 | no option dontlognull
34 | option splice-auto
35 | option http-keep-alive
36 | option redispatch
37 | retries 3
38 | # disconnect slow handshake clients early, protect from
39 | # resources exhaustion attacks
40 | timeout http-request 5s
41 | timeout queue 1m
42 | timeout connect 5s
43 | timeout client 1m
44 | timeout server 1m
45 | timeout http-keep-alive 10s
46 | timeout check 10s
47 | maxconn 100000
48 |
49 | # frontend used to return health status without requiring SSL
50 | frontend haproxy_status
51 | bind 0.0.0.0:34180 # 34180 means EALTH ;)
52 | log-format [%pid]\ [%Ts.%ms]\ %ac/%fc/%bc/%bq/%sc/%sq/%rc\ %Tq/%Tw/%Tc/%Tr/%Tt\ %tsc\ %ci:%cp\ %fi:%fp\ %si:%sp\ %ft\ %sslc\ %sslv\ %{+Q}r\ %ST\ %b:%s\ "%CC"\ "%hr"\ "%CS"\ "%hs"\ req_size=%U\ resp_size=%B
53 | # create a status URI in /haproxy_status that will return
54 | # a 200 is backend is healthy, and 503 if it isn't. This
55 | # URI is queried by the ELB.
56 | acl backend_dead nbsrv(fxa-nodejs) lt 1
57 | monitor-uri /haproxy_status
58 | monitor fail if backend_dead
59 |
60 | # reject non status requests
61 | #acl status_req path /haproxy_status
62 | #tcp-request content reject if ! status_req
63 | default_backend fxa-nodejs
64 |
65 | # main frontend with SSL and PROXY protocol
66 | frontend fxa-https
67 | bind 0.0.0.0:443 accept-proxy ssl crt /etc/haproxy/bundle.pem ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:ECDHE-RSA-RC4-SHA:ECDHE-ECDSA-RC4-SHA:AES128:AES256:RC4-SHA:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
68 |
69 | # Blacklist: Deny access to some IPs before anything else is checked
70 | tcp-request content reject if { src -f /etc/haproxy/blacklist.lst }
71 |
72 | # Whitelist: Allow IPs to bypass the filters
73 | http-request add-header X-Haproxy-ACL %[req.fhdr(X-Haproxy-ACL,-1)]whitelisted, if { src -f /etc/haproxy/whitelist.lst }
74 | http-request allow if { src -f /etc/haproxy/whitelist.lst }
75 |
76 | # ~~~ Requests validation using ACLs ~~~
77 | # block requests that don't match the predefined endpoints
78 | acl valid-get path -f /etc/haproxy/get_endpoints.lst
79 | acl valid-post path -f /etc/haproxy/post_endpoints.lst
80 | http-request add-header X-Haproxy-ACL %[req.fhdr(X-Haproxy-ACL,-1)]invalid-endpoint, unless METH_GET valid-get or METH_POST valid-post
81 |
82 | # block requests on verify_email that do not have the correct params
83 | acl endpoint-verify_email path /v1/verify_email
84 | acl param-code urlp_reg(code) [0-9a-fA-F]{1,32}
85 | acl param-uid urlp_reg(uid) [0-9a-fA-F]{1,32}
86 | http-request add-header X-Haproxy-ACL %[req.fhdr(X-Haproxy-ACL,-1)]invalid-parameters, if endpoint-verify_email !param-code or endpoint-verify_email !param-uid
87 |
88 | # block requests on complete_reset_password that do not have the correct params
89 | acl endpoint-complete_reset_password path /v1/complete_reset_password
90 | acl param-email urlp_reg(email) -i ^[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}$
91 | acl param-token urlp_reg(token) [0-9a-fA-F]{1,64}
92 | http-request add-header X-Haproxy-ACL %[req.fhdr(X-Haproxy-ACL,-1)]invalid-parameters, if endpoint-complete_reset_password !param-email or endpoint-complete_reset_password !param-token or endpoint-complete_reset_password !param-code
93 |
94 | # block content-length larger than 500kB
95 | acl request-too-big hdr_val(content-length) gt 5000
96 | http-request add-header X-Haproxy-ACL %[req.fhdr(X-Haproxy-ACL,-1)]request-too-big, if METH_POST request-too-big
97 |
98 | # ~~~ DDoS protection ~~~
99 | # HAproxy tracks client IPs into a global stick table. Each IP is
100 | # stored for a limited amount of time, with several counters attached
101 | # to it. When a new connection comes in, the stick table is evaluated
102 | # to verify that the new connection from this client is allowed to
103 | # continue.
104 |
105 | # enable tracking of counters for src ip in the default stick-table
106 | tcp-request content track-sc0 src
107 |
108 | # Stick Table Definitions
109 | # - conn_cur: count active connections
110 | # - conn_rate(10s): average incoming connection rate over 10 seconds
111 | # - http_req_rate(10s): Monitors the number of request sent by an IP over a period of 10 seconds
112 | # - http_err_rate(10s): Monitors the number of errors generated by an IP over a period of 10 seconds
113 | stick-table type ip size 500k expire 30s store conn_cur,conn_rate(10s),http_req_rate(10s),http_err_rate(10s)
114 |
115 | # Reject the new connection if the client already has 100 opened
116 | http-request add-header X-Haproxy-ACL %[req.fhdr(X-Haproxy-ACL,-1)]over-100-active-connections, if { src_conn_cur ge 100 }
117 |
118 | # Reject the new connection if the client has opened more than 100 connections in 10 seconds
119 | http-request add-header X-Haproxy-ACL %[req.fhdr(X-Haproxy-ACL,-1)]over-100-connections-in-10-seconds, if { src_conn_rate ge 100 }
120 |
121 | # Reject the connection if the client has passed the HTTP error rate
122 | http-request add-header X-Haproxy-ACL %[req.fhdr(X-Haproxy-ACL,-1)]high-error-rate, if { sc0_http_err_rate() gt 100 }
123 |
124 | # Reject the connection if the client has passed the HTTP request rate
125 | http-request add-header X-Haproxy-ACL %[req.fhdr(X-Haproxy-ACL,-1)]high-request-rate, if { sc0_http_req_rate() gt 500 }
126 |
127 | # ~~~ Logging ~~~
128 | # Insert a unique request identifier is the headers of the request
129 | # passed to the backend
130 | unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
131 | unique-id-header X-Unique-ID
132 |
133 | # Add the X-Forwarded-For header
134 | option forwardfor except 127.0.0.0/8
135 |
136 | # Don't log here, log into logger frontend
137 | no log
138 | default_backend logger
139 |
140 | backend logger
141 | server localhost localhost:55555 send-proxy
142 |
143 | # frontend use to log acl activity
144 | frontend logger
145 | bind localhost:55555 accept-proxy
146 | log-format [%pid]\ [%Ts.%ms]\ %ac/%fc/%bc/%bq/%sc/%sq/%rc\ %Tq/%Tw/%Tc/%Tr/%Tt\ %tsc\ %ci:%cp\ %fi:%fp\ %si:%sp\ %ft\ %sslc\ %sslv\ %{+Q}r\ %ST\ %b:%s\ "%CC"\ "%hr"\ "%CS"\ "%hs"\ ireq_size=%U\ resp_size=%B
147 | capture request header Referrer len 64
148 | capture request header Content-Length len 10
149 | capture request header User-Agent len 64
150 | capture request header X-Haproxy-ACL len 256
151 | capture request header X-Unique-ID len 64
152 |
153 | # if previous ACL didn't pass, and IP isn't whitelisted, block the request
154 | acl whitelisted req.fhdr(X-Haproxy-ACL) -m beg whitelisted,
155 | acl fail-validation req.fhdr(X-Haproxy-ACL) -m found
156 | http-request deny if !whitelisted fail-validation
157 |
158 | default_backend fxa-nodejs
159 |
160 | backend fxa-nodejs
161 | # Remove the ACL header, it's not useful to NodeJS
162 | reqdel ^X-Haproxy-ACL
163 | option httpchk GET /__heartbeat__ HTTP/1.0
164 | http-check expect status 200
165 | server nodejs1 localhost:8000 check inter 1000
166 |
--------------------------------------------------------------------------------
/haproxy.spec:
--------------------------------------------------------------------------------
1 | # To Build:
2 | #
3 | # sudo yum -y install rpmdevtools && rpmdev-setuptree
4 | # sudo yum -y install pcre-devel openssl-devel
5 | # wget https://raw.github.com/nmilford/rpm-haproxy/master/haproxy.spec -O ./rpmbuild/SPECS/haproxy.spec
6 | # wget http://haproxy.1wt.eu/download/1.5/src/haproxy-1.5.1.tar.gz -O ./rpmbuild/SOURCES/haproxy-1.5.1.tar.gz
7 | # rpmbuild -bb ./rpmbuild/SPECS/haproxy.spec
8 |
9 | %define version 1.5.1
10 | %define release 1.5.1
11 |
12 | Summary: HA-Proxy is a TCP/HTTP reverse proxy for high availability environments
13 | Name: haproxy
14 | Version: %{version}
15 | Release: %{release}
16 | License: GPL
17 | Group: System Environment/Daemons
18 | URL: http://haproxy.1wt.eu
19 | Source0: http://haproxy.1wt.eu/download/1.5/src/%{name}-%{version}.tar.gz
20 | BuildRoot: %{_tmppath}/%{name}-%{version}-root
21 | BuildRequires: pcre-devel openssl-devel
22 | Requires: /sbin/chkconfig, /sbin/service
23 |
24 | %description
25 | HAProxy is a TCP/HTTP reverse proxy which is particularly suited for high
26 | availability environments. Indeed, it can:
27 | - route HTTP requests depending on statically assigned cookies
28 | - spread the load among several servers while assuring server persistence
29 | through the use of HTTP cookies
30 | - switch to backup servers in the event a main one fails
31 | - accept connections to special ports dedicated to service monitoring
32 | - stop accepting connections without breaking existing ones
33 | - add/modify/delete HTTP headers both ways
34 | - block requests matching a particular pattern
35 |
36 | It needs very little resource. Its event-driven architecture allows it to easily
37 | handle thousands of simultaneous connections on hundreds of instances without
38 | risking the system's stability.
39 |
40 | %prep
41 | %setup -n %{name}-%{version}
42 |
43 | # We don't want any perl dependecies in this RPM:
44 | %define __perl_requires /bin/true
45 |
46 | %build
47 | %{__make} USE_PCRE=1 TARGET=linux2628 USE_OPENSSL=1
48 |
49 | %install
50 | [ "%{buildroot}" != "/" ] && %{__rm} -rf %{buildroot}
51 |
52 | %{__install} -d %{buildroot}%{_sbindir}
53 | %{__install} -d %{buildroot}%{_sysconfdir}/rc.d/init.d
54 | %{__install} -d %{buildroot}%{_sysconfdir}/%{name}
55 | %{__install} -d %{buildroot}%{_mandir}/man1/
56 |
57 | %{__install} -s %{name} %{buildroot}%{_sbindir}/
58 | %{__install} -c -m 644 examples/%{name}.cfg %{buildroot}%{_sysconfdir}/%{name}/
59 | %{__install} -c -m 755 examples/%{name}.init %{buildroot}%{_sysconfdir}/rc.d/init.d/%{name}
60 | %{__install} -c -m 755 doc/%{name}.1 %{buildroot}%{_mandir}/man1/
61 |
62 | %clean
63 | [ "%{buildroot}" != "/" ] && %{__rm} -rf %{buildroot}
64 |
65 | %post
66 | /sbin/chkconfig --add %{name}
67 |
68 | %preun
69 | if [ $1 = 0 ]; then
70 | /sbin/service %{name} stop >/dev/null 2>&1 || :
71 | /sbin/chkconfig --del %{name}
72 | fi
73 |
74 | %postun
75 | if [ "$1" -ge "1" ]; then
76 | /sbin/service %{name} condrestart >/dev/null 2>&1 || :
77 | fi
78 |
79 | %files
80 | %defattr(-,root,root)
81 | %doc CHANGELOG examples/*.cfg doc/haproxy-en.txt doc/haproxy-fr.txt doc/architecture.txt doc/configuration.txt
82 | %doc %{_mandir}/man1/%{name}.1*
83 |
84 | %attr(0755,root,root) %{_sbindir}/%{name}
85 | %dir %{_sysconfdir}/%{name}
86 | %attr(0644,root,root) %config(noreplace) %{_sysconfdir}/%{name}/%{name}.cfg
87 | %attr(0755,root,root) %config %{_sysconfdir}/rc.d/init.d/%{name}
88 |
89 | %changelog
90 | * Sat Jun 28 2014 Julien Vehent
91 | - Build RPM. see changelog at http://haproxy.1wt.eu/git?p=haproxy.git;a=log
92 |
--------------------------------------------------------------------------------
/log_parse_ddos_protection.sh:
--------------------------------------------------------------------------------
1 | #!/usr/bin/env bash
2 | if [[ ! -n $1 || ! -n $2 || ! -n $3 ]]
3 | then
4 | echo
5 | echo "$0 "
6 | echo "parse haproxy logs to count success and failures per seconds"
7 | echo "between two time periods"
8 | echo
9 | echo "example: $0 88.191.125.180 'Fri Jan 17 17:30:00 UTC 2014' 'Fri Jan 17 17:40:00 UTC 2014'"
10 | echo
11 | exit 1
12 | fi
13 | OIFS=$IFS
14 | IFS=$'\n'
15 | SRCIP=$1
16 | STARTTS=$(date -d "$2" +%s)
17 | if [ $? -gt 0 ]; then echo "invalid date $2"; exit 1;fi
18 | STOPTS=$(date -d "$3" +%s)
19 | if [ $? -gt 0 ]; then echo "invalid date $3"; exit 1;fi
20 | TS=$STARTTS
21 | while [ $TS -le $STOPTS ]
22 | do
23 | cts=$(date -d"@$TS" +%d/%b/%Y:%H:%M:%S)
24 | ctr_success=0
25 | ctr_failure=0
26 | for line in $(grep "$cts" /var/log/messages|grep "$SRCIP"|grep "haproxy")
27 | do
28 | # status code is at col 11
29 | st=$(echo $line|awk '{print $11}')
30 | if [ $st -lt 0 ]
31 | then
32 | ctr_failure=$((ctr_failure + 1))
33 | else
34 | ctr_success=$((ctr_success + 1))
35 | fi
36 | done
37 | echo $cts $ctr_success $ctr_failure
38 | # go to next second
39 | TS=$((TS + 1 ))
40 | done
41 | IFS=$OIFS
42 |
--------------------------------------------------------------------------------
/post_endpoints.lst:
--------------------------------------------------------------------------------
1 | /v1/account/create
2 | /v1/account/destroy
3 | /v1/account/login
4 | /v1/account/reset
5 | /v1/certificate/sign
6 | /v1/get_random_bytes
7 | /v1/password/change/finish
8 | /v1/password/change/start
9 | /v1/password/forgot/resend_code
10 | /v1/password/forgot/send_code
11 | /v1/password/forgot/verify_code
12 | /v1/recovery_email/resend_code
13 | /v1/recovery_email/verify_code
14 | /v1/session/destroy
15 |
--------------------------------------------------------------------------------