├── LICENSE ├── README.md ├── blue-green ├── blue.json └── green.json ├── canary ├── canary-check.sh ├── simpleservice-blueprint.yaml ├── vamp-es.json ├── vamp-gateway.json └── vamp.json ├── default ├── base-health.json ├── base-ready.json └── base.json ├── event-bus-log-base-min-over.txt └── img ├── base-health-update-deployment-step0.png ├── base-health-update-deployment-step1.png ├── base-health-update-deployment-step2.png ├── base-health-update-deployment-step3.png ├── base-health-update-deployment-step4.png ├── base-health-update-deployment.gif ├── base-health.png ├── base-min-over-step.png ├── base-update-deployment.png ├── base-update.png ├── haproxy-blue.png ├── haproxy-green.png ├── haproxy-idle.png ├── vamp-deployments.png ├── vamp-gateways.png └── vamp-idle.png /LICENSE: -------------------------------------------------------------------------------- 1 | Apache License 2 | Version 2.0, January 2004 3 | http://www.apache.org/licenses/ 4 | 5 | TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 6 | 7 | 1. Definitions. 8 | 9 | "License" shall mean the terms and conditions for use, reproduction, 10 | and distribution as defined by Sections 1 through 9 of this document. 11 | 12 | "Licensor" shall mean the copyright owner or entity authorized by 13 | the copyright owner that is granting the License. 14 | 15 | "Legal Entity" shall mean the union of the acting entity and all 16 | other entities that control, are controlled by, or are under common 17 | control with that entity. For the purposes of this definition, 18 | "control" means (i) the power, direct or indirect, to cause the 19 | direction or management of such entity, whether by contract or 20 | otherwise, or (ii) ownership of fifty percent (50%) or more of the 21 | outstanding shares, or (iii) beneficial ownership of such entity. 22 | 23 | "You" (or "Your") shall mean an individual or Legal Entity 24 | exercising permissions granted by this License. 25 | 26 | "Source" form shall mean the preferred form for making modifications, 27 | including but not limited to software source code, documentation 28 | source, and configuration files. 29 | 30 | "Object" form shall mean any form resulting from mechanical 31 | transformation or translation of a Source form, including but 32 | not limited to compiled object code, generated documentation, 33 | and conversions to other media types. 34 | 35 | "Work" shall mean the work of authorship, whether in Source or 36 | Object form, made available under the License, as indicated by a 37 | copyright notice that is included in or attached to the work 38 | (an example is provided in the Appendix below). 39 | 40 | "Derivative Works" shall mean any work, whether in Source or Object 41 | form, that is based on (or derived from) the Work and for which the 42 | editorial revisions, annotations, elaborations, or other modifications 43 | represent, as a whole, an original work of authorship. For the purposes 44 | of this License, Derivative Works shall not include works that remain 45 | separable from, or merely link (or bind by name) to the interfaces of, 46 | the Work and Derivative Works thereof. 47 | 48 | "Contribution" shall mean any work of authorship, including 49 | the original version of the Work and any modifications or additions 50 | to that Work or Derivative Works thereof, that is intentionally 51 | submitted to Licensor for inclusion in the Work by the copyright owner 52 | or by an individual or Legal Entity authorized to submit on behalf of 53 | the copyright owner. For the purposes of this definition, "submitted" 54 | means any form of electronic, verbal, or written communication sent 55 | to the Licensor or its representatives, including but not limited to 56 | communication on electronic mailing lists, source code control systems, 57 | and issue tracking systems that are managed by, or on behalf of, the 58 | Licensor for the purpose of discussing and improving the Work, but 59 | excluding communication that is conspicuously marked or otherwise 60 | designated in writing by the copyright owner as "Not a Contribution." 61 | 62 | "Contributor" shall mean Licensor and any individual or Legal Entity 63 | on behalf of whom a Contribution has been received by Licensor and 64 | subsequently incorporated within the Work. 65 | 66 | 2. Grant of Copyright License. Subject to the terms and conditions of 67 | this License, each Contributor hereby grants to You a perpetual, 68 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 69 | copyright license to reproduce, prepare Derivative Works of, 70 | publicly display, publicly perform, sublicense, and distribute the 71 | Work and such Derivative Works in Source or Object form. 72 | 73 | 3. Grant of Patent License. Subject to the terms and conditions of 74 | this License, each Contributor hereby grants to You a perpetual, 75 | worldwide, non-exclusive, no-charge, royalty-free, irrevocable 76 | (except as stated in this section) patent license to make, have made, 77 | use, offer to sell, sell, import, and otherwise transfer the Work, 78 | where such license applies only to those patent claims licensable 79 | by such Contributor that are necessarily infringed by their 80 | Contribution(s) alone or by combination of their Contribution(s) 81 | with the Work to which such Contribution(s) was submitted. If You 82 | institute patent litigation against any entity (including a 83 | cross-claim or counterclaim in a lawsuit) alleging that the Work 84 | or a Contribution incorporated within the Work constitutes direct 85 | or contributory patent infringement, then any patent licenses 86 | granted to You under this License for that Work shall terminate 87 | as of the date such litigation is filed. 88 | 89 | 4. Redistribution. You may reproduce and distribute copies of the 90 | Work or Derivative Works thereof in any medium, with or without 91 | modifications, and in Source or Object form, provided that You 92 | meet the following conditions: 93 | 94 | (a) You must give any other recipients of the Work or 95 | Derivative Works a copy of this License; and 96 | 97 | (b) You must cause any modified files to carry prominent notices 98 | stating that You changed the files; and 99 | 100 | (c) You must retain, in the Source form of any Derivative Works 101 | that You distribute, all copyright, patent, trademark, and 102 | attribution notices from the Source form of the Work, 103 | excluding those notices that do not pertain to any part of 104 | the Derivative Works; and 105 | 106 | (d) If the Work includes a "NOTICE" text file as part of its 107 | distribution, then any Derivative Works that You distribute must 108 | include a readable copy of the attribution notices contained 109 | within such NOTICE file, excluding those notices that do not 110 | pertain to any part of the Derivative Works, in at least one 111 | of the following places: within a NOTICE text file distributed 112 | as part of the Derivative Works; within the Source form or 113 | documentation, if provided along with the Derivative Works; or, 114 | within a display generated by the Derivative Works, if and 115 | wherever such third-party notices normally appear. The contents 116 | of the NOTICE file are for informational purposes only and 117 | do not modify the License. You may add Your own attribution 118 | notices within Derivative Works that You distribute, alongside 119 | or as an addendum to the NOTICE text from the Work, provided 120 | that such additional attribution notices cannot be construed 121 | as modifying the License. 122 | 123 | You may add Your own copyright statement to Your modifications and 124 | may provide additional or different license terms and conditions 125 | for use, reproduction, or distribution of Your modifications, or 126 | for any such Derivative Works as a whole, provided Your use, 127 | reproduction, and distribution of the Work otherwise complies with 128 | the conditions stated in this License. 129 | 130 | 5. Submission of Contributions. Unless You explicitly state otherwise, 131 | any Contribution intentionally submitted for inclusion in the Work 132 | by You to the Licensor shall be under the terms and conditions of 133 | this License, without any additional terms or conditions. 134 | Notwithstanding the above, nothing herein shall supersede or modify 135 | the terms of any separate license agreement you may have executed 136 | with Licensor regarding such Contributions. 137 | 138 | 6. Trademarks. This License does not grant permission to use the trade 139 | names, trademarks, service marks, or product names of the Licensor, 140 | except as required for reasonable and customary use in describing the 141 | origin of the Work and reproducing the content of the NOTICE file. 142 | 143 | 7. Disclaimer of Warranty. Unless required by applicable law or 144 | agreed to in writing, Licensor provides the Work (and each 145 | Contributor provides its Contributions) on an "AS IS" BASIS, 146 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or 147 | implied, including, without limitation, any warranties or conditions 148 | of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A 149 | PARTICULAR PURPOSE. You are solely responsible for determining the 150 | appropriateness of using or redistributing the Work and assume any 151 | risks associated with Your exercise of permissions under this License. 152 | 153 | 8. Limitation of Liability. In no event and under no legal theory, 154 | whether in tort (including negligence), contract, or otherwise, 155 | unless required by applicable law (such as deliberate and grossly 156 | negligent acts) or agreed to in writing, shall any Contributor be 157 | liable to You for damages, including any direct, indirect, special, 158 | incidental, or consequential damages of any character arising as a 159 | result of this License or out of the use or inability to use the 160 | Work (including but not limited to damages for loss of goodwill, 161 | work stoppage, computer failure or malfunction, or any and all 162 | other commercial damages or losses), even if such Contributor 163 | has been advised of the possibility of such damages. 164 | 165 | 9. Accepting Warranty or Additional Liability. While redistributing 166 | the Work or Derivative Works thereof, You may choose to offer, 167 | and charge a fee for, acceptance of support, warranty, indemnity, 168 | or other liability obligations and/or rights consistent with this 169 | License. However, in accepting such obligations, You may act only 170 | on Your own behalf and on Your sole responsibility, not on behalf 171 | of any other Contributor, and only if You agree to indemnify, 172 | defend, and hold each Contributor harmless for any liability 173 | incurred by, or claims asserted against, such Contributor by reason 174 | of your accepting any such warranty or additional liability. 175 | 176 | END OF TERMS AND CONDITIONS 177 | 178 | APPENDIX: How to apply the Apache License to your work. 179 | 180 | To apply the Apache License to your work, attach the following 181 | boilerplate notice, with the fields enclosed by brackets "{}" 182 | replaced with your own identifying information. (Don't include 183 | the brackets!) The text should be enclosed in the appropriate 184 | comment syntax for the file format. We also recommend that a 185 | file or class name and description of purpose be included on the 186 | same "printed page" as the copyright notice for easier 187 | identification within third-party archives. 188 | 189 | Copyright {yyyy} {name of copyright owner} 190 | 191 | Licensed under the Apache License, Version 2.0 (the "License"); 192 | you may not use this file except in compliance with the License. 193 | You may obtain a copy of the License at 194 | 195 | http://www.apache.org/licenses/LICENSE-2.0 196 | 197 | Unless required by applicable law or agreed to in writing, software 198 | distributed under the License is distributed on an "AS IS" BASIS, 199 | WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 200 | See the License for the specific language governing permissions and 201 | limitations under the License. 202 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Zero Downtime Deployments Lab 2 | 3 | This Zero Downtime Deployments (ZDD) lab aims at providing an introduction to [DC/OS](https://dcos.io/) service deployments. 4 | It serves as a step-wise guide how to deploy new versions of a DC/OS service without causing downtimes. 5 | 6 | We will do the following in the ZDD lab: 7 | 8 | 1. A rolling upgrade using the [default behaviour](#default-behaviour) 9 | 1. [Without health checks](#without-health-checks) 10 | 1. [With health checks](#with-health-checks) 11 | 1. [With readiness checks](#with-readiness-checks) 12 | 1. A rolling upgrade with [minimal overcapacity](#minimal-overcapacity) 13 | 1. A [canary deployment](#canary-deployment) 14 | 1. A [Blue-Green deployment](#blue-green-deployment) 15 | 16 | ## Preparation 17 | 18 | Throughout the ZDD lab we will be using [simpleservice](https://github.com/mhausenblas/simpleservice), a 19 | simple test service, allowing us to simulate certain behaviour such as reporting a certain version and health check delays. 20 | 21 | If you want to follow along and try out the described steps yourself, here are the prerequisites: 22 | 23 | - A running [DC/OS 1.8](https://dcos.io/releases/1.8.4/) cluster with at least one private agent, see also [installation](https://dcos.io/install/) if you don't have one yet. 24 | - The [DC/OS CLI](https://dcos.io/docs/1.8/usage/cli/) installed and configured. 25 | - The [jq](https://stedolan.github.io/jq/) tool, command-line JSON processor, installed. 26 | 27 | Finally, as a preparation you should have a (quick) look at the following docs (in the order we are using it in the ZDD lab): 28 | 29 | - [health checks](https://mesosphere.github.io/marathon/docs/health-checks.html) 30 | - [deployments](https://mesosphere.github.io/marathon/docs/deployments.html) 31 | - [readiness checks](https://mesosphere.github.io/marathon/docs/readiness-checks.html) (OPTIONAL) 32 | - [load balancing with HAProxy](https://serversforhackers.com/load-balancing-with-haproxy) (OPTIONAL) 33 | - [Marathon Blue-Green deployments](https://mesosphere.github.io/marathon/docs/blue-green-deploy.html) 34 | - [Marathon-LB ZDD](https://github.com/mesosphere/marathon-lb#zero-downtime-deployments) 35 | 36 | ## Default behaviour 37 | 38 | The default behaviour of DC/OS service deployments is a rolling upgrade, that is, DC/OS launches instances of the 39 | new version of your service while shutting down (killing) instances with the old version. How exactly this takes place 40 | depends on how much information (about the status of your service) you provide to DC/OS. This status info is called health 41 | and readiness checks in DC/OS and in the following we will walk through each of the basic cases. 42 | 43 | ### Without health checks 44 | 45 | To explore the default deployment behaviour of DC/OS services we're using [base.json](default/base.json). 46 | This launches a service with the ID `/zdd/base` with 4 instances of `simpleservice`, without health checking, and initially in the version `0.9`: 47 | 48 | $ dcos marathon app add default/base.json 49 | 50 | Now we should be able to verify that `simpleservice` is running and there are indeed 4 instances (tasks) available: 51 | 52 | $ dcos marathon task list /zdd/base 53 | APP HEALTHY STARTED HOST ID 54 | /zdd/base True 2016-10-12T11:38:56.845Z 10.0.3.192 zdd_base.75440e42-9070-11e6-aae4-3a4b79075094 55 | /zdd/base True 2016-10-12T11:38:56.861Z 10.0.3.193 zdd_base.75443553-9070-11e6-aae4-3a4b79075094 56 | /zdd/base True 2016-10-12T11:38:56.878Z 10.0.3.193 zdd_base.754546c5-9070-11e6-aae4-3a4b79075094 57 | /zdd/base True 2016-10-12T11:38:56.884Z 10.0.3.192 zdd_base.7544f8a4-9070-11e6-aae4-3a4b79075094 58 | 59 | The last column in above output is the so called `task ID` which we will be using in the following to refer to a single instance of `simpleservice`. 60 | 61 | Next, let's see what version of `simpleservice` is running. For this we need to invoke one of the 4 instances of `simpleservice`, so we pick a random one and try to discover where it is available: 62 | 63 | $ dcos marathon task show zdd_base.75443553-9070-11e6-aae4-3a4b79075094 64 | { 65 | "appId": "/zdd/base", 66 | "host": "10.0.3.193", 67 | "id": "zdd_base.75443553-9070-11e6-aae4-3a4b79075094", 68 | "ipAddresses": [ 69 | { 70 | "ipAddress": "10.0.3.193", 71 | "protocol": "IPv4" 72 | } 73 | ], 74 | "ports": [ 75 | 1765 76 | ], 77 | "servicePorts": [ 78 | 10000 79 | ], 80 | "slaveId": "145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S1", 81 | "stagedAt": "2016-10-12T11:38:55.952Z", 82 | "startedAt": "2016-10-12T11:38:56.861Z", 83 | "state": "TASK_RUNNING", 84 | "version": "2016-10-12T11:38:55.934Z" 85 | } 86 | 87 | From the above output we learn that the instance `zdd_base.75443553-9070-11e6-aae4-3a4b79075094` of `simpleservice` is available via `10.0.3.193:1765`. Since we didn't deploy the `simpleservice` onto a public agent, it is only available and accessible from with the cluster. We hence ssh into the DC/OS cluster to invoke the previously mentioned instance, for example like so: 88 | 89 | $ ssh -A core@$MASTER_IP_ADDRESS 90 | CoreOS stable (1068.9.0) 91 | Last login: Wed Oct 12 10:39:38 2016 from 46.7.174.29 92 | Update Strategy: No Reboots 93 | Failed Units: 1 94 | update-engine.service 95 | core@ip-10-0-6-211 ~ $ curl 10.0.3.193:1765/endpoint0 96 | {"host": "10.0.3.193:1765", "version": "0.9", "result": "all is well"} 97 | 98 | So we see from above output that indeed all is well and `simpleservice` is serving in version `0.9`. At the same time, we can have a look at the logs of this instance to verify that it has been invoked (in a new terminal): 99 | 100 | $ dcos task log --follow zdd_base.75443553-9070-11e6-aae4-3a4b79075094 stderr 101 | I1012 11:38:56.152595 27678 docker.cpp:815] Running docker -H unix:///var/run/docker.sock run --cpu-shares 102 --memory 33554432 -e MARATHON_APP_VERSION=2016-10-12T11:38:55.934Z -e HOST=10.0.3.193 -e MARATHON_APP_RESOURCE_CPUS=0.1 -e SIMPLE_SERVICE_VERSION=0.9 -e MARATHON_APP_RESOURCE_GPUS=0 -e HEALTH_MAX=5000 -e MARATHON_APP_DOCKER_IMAGE=mhausenblas/simpleservice:0.4.0 -e PORT_10000=1765 -e MESOS_TASK_ID=zdd_base.75443553-9070-11e6-aae4-3a4b79075094 -e PORT=1765 -e MARATHON_APP_RESOURCE_MEM=32.0 -e PORTS=1765 -e MARATHON_APP_RESOURCE_DISK=0.0 -e HEALTH_MIN=1000 -e MARATHON_APP_LABELS= -e MARATHON_APP_ID=/zdd/base -e PORT0=1765 -e LIBPROCESS_IP=10.0.3.193 -e MESOS_SANDBOX=/mnt/mesos/sandbox -e MESOS_CONTAINER_NAME=mesos-145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S1.76d75960-dd4d-49c1-b320-b8f466353927 -v /var/lib/mesos/slave/slaves/145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S1/frameworks/145f052d-8bcb-457f-b1e6-b1b4e2cdf787-0000/executors/zdd_base.75443553-9070-11e6-aae4-3a4b79075094/runs/76d75960-dd4d-49c1-b320-b8f466353927:/mnt/mesos/sandbox --net host --name mesos-145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S1.76d75960-dd4d-49c1-b320-b8f466353927 mhausenblas/simpleservice:0.4.0 102 | 2016-10-12T11:38:56 INFO This is simple service in version v0.9 listening on port 1765 [at line 101] 103 | 2016-10-12T12:00:36 INFO /endpoint0 serving from 10.0.3.193:1765 has been invoked from 10.0.6.211 [at line 59] 104 | 2016-10-12T12:00:36 INFO 200 GET /endpoint0 (10.0.6.211) 1.04ms [at line 1946] 105 | 106 | Now, we update the version of `simpleservice` by changing `SIMPLE_SERVICE_VERSION` to `1.0`, either through locally editing `base.json` and using the CLI command `dcos marathon app update /zdd/base < default/base.json` or via the DC/OS UI as shown in the following: 107 | 108 | ![Upgrading simpleservice with default behaviour](img/base-update.png) 109 | 110 | Once you hit the `Deploy Changes` button you should see something like the following: 111 | 112 | ![Deployment of upgraded simpleservice with default behaviour](img/base-update-deployment.png) 113 | 114 | Notice the old (`0.9`) instances being killed and the new (`1.0`) ones running, overall we have 8 tasks active. To verify if the new version is available we again (from within the cluster) invoke one of the instances as shown previously: 115 | 116 | core@ip-10-0-6-211 ~ $ curl 10.0.3.193:27670/endpoint0 117 | {"host": "10.0.3.193:27670", "version": "1.0", "result": "all is well"} 118 | 119 | Also, notice that none of the instances in the DC/OS UI is showing healthy. This is because DC/OS doesn't know anything about the health status. Let's change that. 120 | 121 | Note also that if you only want to scale the app (keeping the same version) you can use the following CLI command: `dcos marathon app update /zdd/base instances=5` to scale to 5 instances. 122 | 123 | ### With health checks 124 | 125 | To explore the default deployment behaviour of DC/OS services with health checks, we're using [base-health.json](default/base-health.json). 126 | This launches a service with the ID `/zdd/base-health` with 4 instances of `simpleservice`, with health checking, and initially in the version `0.9`: 127 | 128 | $ dcos marathon app add default/base-health.json 129 | 130 | What we now see in the DC/OS UI is the following: 131 | 132 | ![simpleservice with health checks](img/base-health.png) 133 | 134 | And indeed, as expected, DC/OS can now tell that all instances are healthy, thanks to the following snippet in `base-health.json` (note that besides `path` all other fields are actually the default values): 135 | 136 | "healthChecks": [{ 137 | "protocol": "HTTP", 138 | "path": "/health", 139 | "gracePeriodSeconds": 300, 140 | "intervalSeconds": 60, 141 | "timeoutSeconds": 20, 142 | "maxConsecutiveFailures": 3, 143 | "ignoreHttp1xx": false 144 | }] 145 | 146 | Alternatively, you can check the health of the `/zdd/base-health` service using the DC/OS CLI and `jq` like so: 147 | 148 | $ dcos marathon app show /zdd/base-health | jq '.tasks[].healthCheckResults[]' 149 | { 150 | "alive": true, 151 | "consecutiveFailures": 0, 152 | "firstSuccess": "2016-10-12T13:20:03.005Z", 153 | "lastFailure": null, 154 | "lastFailureCause": null, 155 | "lastSuccess": "2016-10-12T13:27:01.323Z", 156 | "taskId": "zdd_base-health.90056376-907e-11e6-aae4-3a4b79075094" 157 | } 158 | { 159 | "alive": true, 160 | "consecutiveFailures": 0, 161 | "firstSuccess": "2016-10-12T13:20:03.434Z", 162 | "lastFailure": null, 163 | "lastFailureCause": null, 164 | "lastSuccess": "2016-10-12T13:27:01.795Z", 165 | "taskId": "zdd_base-health.90056377-907e-11e6-aae4-3a4b79075094" 166 | } 167 | { 168 | "alive": true, 169 | "consecutiveFailures": 0, 170 | "firstSuccess": "2016-10-12T13:20:03.602Z", 171 | "lastFailure": null, 172 | "lastFailureCause": null, 173 | "lastSuccess": "2016-10-12T13:27:00.691Z", 174 | "taskId": "zdd_base-health.9004a024-907e-11e6-aae4-3a4b79075094" 175 | } 176 | { 177 | "alive": true, 178 | "consecutiveFailures": 0, 179 | "firstSuccess": "2016-10-12T13:20:00.989Z", 180 | "lastFailure": null, 181 | "lastFailureCause": null, 182 | "lastSuccess": "2016-10-12T13:27:02.398Z", 183 | "taskId": "zdd_base-health.9004c735-907e-11e6-aae4-3a4b79075094" 184 | } 185 | 186 | And, of course, as in the first case without health checks we can use `dcos marathon task list /zdd/base-health` to explore the 4 instances and verify if they serve the right version (`0.9`). Note, however, that in contrast to the previous case, the tasks now have a `healthCheckResults` array which provides you with details on what is going on concerning the health checks DC/OS performs. 187 | 188 | Let's now simulate a case where the health checks fail (time out), for example, because of an internal service failure or an integration point not being available. For this, we need to change two things: the `healthChecks` in `base-health.json` (either locally + CLI command `dcos marathon app update /zdd/base-health < default/base-health.json` or via the DC/OS UI) and as well as the `HEALTH_MIN`, `HEALTH_MAX`, and `SIMPLE_SERVICE_VERSION` env variables, resulting in: 189 | 190 | { 191 | "id": "/zdd/base-health", 192 | "instances": 4, 193 | "cpus": 0.1, 194 | "mem": 32, 195 | "container": { 196 | "type": "DOCKER", 197 | "docker": { 198 | "image": "mhausenblas/simpleservice:0.4.0", 199 | "network": "HOST" 200 | } 201 | }, 202 | "env": { 203 | "HEALTH_MIN": "1000", 204 | "HEALTH_MAX": "5000", 205 | "SIMPLE_SERVICE_VERSION": "1.0" 206 | }, 207 | "healthChecks": [{ 208 | "protocol": "HTTP", 209 | "path": "/health", 210 | "gracePeriodSeconds": 300, 211 | "intervalSeconds": 30, 212 | "timeoutSeconds": 4, 213 | "maxConsecutiveFailures": 20, 214 | "ignoreHttp1xx": false 215 | }] 216 | } 217 | 218 | Note that we've changed `timeoutSeconds` to `4`, meaning that if it takes longer than 4 sec for the `/health` endpoint to respond with `200` the instance is considered unhealthy. Since `HEALTH_MIN` is set to `1000` there should be at least one instance randomly assigned with a delay below 4 sec and hence we expect at least one unhealthy task. If you see all healthy, repeat the deployment or change the values so that it's more likely to happen. 219 | 220 | Further, note that we changed `SIMPLE_SERVICE_VERSION` to `1.0`, hence rolling out a new version, as well as increased `maxConsecutiveFailures` to `20` to give DC/OS enough opportunities to launch healthy instances and finally decreased `intervalSeconds` to `30` to perform the checks faster. 221 | 222 | Once the deployment has been kicked off, you should see a sequence like the following (note that the actual sequence will differ, depending on how many instances have been randomly assigned time outs above the 4 sec threshold and hence are not considered not healthy by DC/OS): 223 | 224 | ![Deployment of upgraded simpleservice with health checks](img/base-health-update-deployment.gif) 225 | 226 | [STEP 0](img/base-health-update-deployment-step0.png) | [STEP 1](img/base-health-update-deployment-step1.png) | [STEP 2](img/base-health-update-deployment-step2.png) | [STEP 3](img/base-health-update-deployment-step3.png) | [STEP 4](img/base-health-update-deployment-step4.png) 227 | 228 | Now, what happened? We requested 4 running, healthy instances of `simpleservice`. DC/OS recognizes the unhealthy instances and re-starts them until it has achieved the goal. 229 | 230 | ### With readiness checks 231 | 232 | So far we've been focusing on `healthChecks`, which are typically used to periodically check the health of a running service. In the deployment phase, for example, in the initial deployment or when you do a rolling upgrade via `dcos marathon app update`, there may be the need to realize when a service is ready to serve traffic. This could be the case for stateful services (a database) or if there are integration points calling out to 3rd party services such as AWS S3 or Azure Event Bus. The difference between `healthChecks` and `readinessChecks` is essentially that if a health check for a task fails, DC/OS will replace that task, whereas in the case of the readiness check failing DC/OS will wait until it succeeds before continuing with the deployment. 233 | 234 | To use a `readinessChecks` use something like shown in [base-ready.json](default/base-ready.json) (note that you MUST specify a `portDefinitions` in the spec and give it a name that you then reference in `portName`, otherwise it will not work): 235 | 236 | $ dcos marathon app add default/base-ready.json 237 | $ dcos marathon app show /zdd/base-ready | jq '.readinessChecks' 238 | [ 239 | { 240 | "httpStatusCodesForReady": [ 241 | 200 242 | ], 243 | "intervalSeconds": 30, 244 | "name": "readinessCheck", 245 | "path": "/health", 246 | "portName": "main-api", 247 | "preserveLastResponse": false, 248 | "protocol": "HTTP", 249 | "timeoutSeconds": 10 250 | } 251 | ] 252 | 253 | Note that `readinessChecks` result is a global property of the service (not on a task level). Note also that it's orthogonal to the `healthChecks`, that is, `dcos marathon app show /zdd/base-ready | jq '.tasks[].healthCheckResults[]'` will return an empty result and also the DC/OS UI will only show the tasks `Running` and not `Healthy`. 254 | 255 | Recommendation: use this property only if you really need fine-grained control over the deployment process, for example, in the context of a framework scheduler. 256 | 257 | ## Minimal overcapacity 258 | 259 | Using the defaults (as described in the section [default behaviour](#default-behaviour)) the DC/OS service deployments have the following implicit settings: 260 | 261 | "upgradeStrategy": { 262 | "minimumHealthCapacity": 1.0, 263 | "maximumOverCapacity": 1.0 264 | } 265 | 266 | In other words, the defaults of the [upgrade strategy](https://mesosphere.github.io/marathon/docs/rest-api.html#upgrade-strategy) mean that it results in a rather safe but somewhat resource-intensive upgrade. 267 | 268 | Formally, the meaning of `minimumHealthCapacity` and `maximumOverCapacity` is as follows: 269 | 270 | - `minimumHealthCapacity` … a floating point value between 0 and 1 (which defaults to `1`), specifying the % of instances to maintain healthy during deployment; with `0` meaning all old instances are stopped before the new version is deployed and `1` meaning all instances of the new version are deployed side by side with the old one before it is stopped. 271 | - `maximumOverCapacity` … a floating point value between 0 and 1 (which defaults to `1`), specifying the max. % of instances over capacity during deployment; with `0` meaning that during the upgrade process no additional capacity than may be used for old and new instances ( only when an old version is stopped, a new instance can be deployed) and `1` meaning that all old and new instances can co-exist during the upgrade process. 272 | 273 | Now, it's not always the case that there are sufficient spare capacities in the DC/OS cluster available. To carry out a deployment that uses minimal overcapacity, we could do the following: 274 | 275 | { 276 | "id": "/zdd/base-min-over", 277 | "instances": 4, 278 | "cpus": 0.1, 279 | "mem": 32, 280 | "container": { 281 | "type": "DOCKER", 282 | "docker": { 283 | "image": "mhausenblas/simpleservice:0.4.0", 284 | "network": "HOST" 285 | } 286 | }, 287 | "env": { 288 | "HEALTH_MIN": "1000", 289 | "HEALTH_MAX": "5000", 290 | "SIMPLE_SERVICE_VERSION": "0.9" 291 | }, 292 | "healthChecks": [{ 293 | "protocol": "HTTP", 294 | "path": "/health", 295 | "gracePeriodSeconds": 300, 296 | "intervalSeconds": 60, 297 | "timeoutSeconds": 20, 298 | "maxConsecutiveFailures": 3, 299 | "ignoreHttp1xx": false 300 | }], 301 | "upgradeStrategy": { 302 | "minimumHealthCapacity": 0.25, 303 | "maximumOverCapacity": 0.25 304 | } 305 | } 306 | 307 | In above example, notice the `"minimumHealthCapacity": 0.25` and `"maximumOverCapacity": 0.25`. 308 | Let's now have a look at how this might play out, step-by-step, when we simulate the upgrade to 309 | version `1.0` by changing `SIMPLE_SERVICE_VERSION` to `1.0`: 310 | 311 | ``` 312 | T0: [0.9] [0.9] [0.9] [0.9] 313 | 314 | T1: deployment kicks off 315 | 316 | T2: [0.9] [0.9] [0.9] [0.9] [1.0] 317 | | 318 | T3: [0.9] [0.9] [0.9] [1.0] [1.0] 319 | | 320 | T4: [0.9] [0.9] [1.0] [1.0] [1.0] 321 | | 322 | T5: [0.9] [1.0] [1.0] [1.0] [1.0] 323 | | 324 | T6: [1.0] [1.0] [1.0] [1.0] 325 | 326 | T7: deployment done 327 | ``` 328 | 329 | A `minimumHealthCapacity` of `0.25` means that 25% or exactly one instance (in our case, since we have specified 4) always needs to run on a certain version. I other words, at no time in the deployment can the app have less than one instance running with any given version, say, `0.9`. 330 | 331 | Up to and incl. timepoint `T0` the current version of the app was `0.9`. When the deployment kicks off at timepoint `T1` the `maximumOverCapacity` attribute becomes important: since we've set it to `0.25` it means no more than 25% (or: exactly one instance in our case) can be run in addition to the already running instances. In other words: with this setting, no more than 5 instances of the app (in whatever version they might be in) can ever run at the same time. 332 | 333 | At `T2` one instance at version `1.0` comes up, satisfying both capacity requirements; in the DC/OS UI this would, for example, look something like the following (note that when you look at the right-most `VERSION` column you see 4 instances with `10/13/2016, 1:36:22 PM` which corresponds to the `0.9` service version and 1 instance with `10/13/2016, 1:36:48 PM`, corresponding to `1.0`): 334 | 335 | ![simpleservice with minimal overcapacity](img/base-min-over-step.png) 336 | 337 | At `T3`, one `0.9` instance is stopped and replaced by a `1.0` instance; at `T4` the same happens again and with the `T5-T6` transition the last remaining `0.9` instance is stopped and since we now have 4 instances of `1.0` running all is good and as expected at `T7`. 338 | 339 | Lesson learned: certain combinations of `minimumHealthCapacity` and `maximumOverCapacity` make sense while others are not satisfiable, meaning that you can specify them, just the deployment will never be carried out. For example, a `"minimumHealthCapacity": 0.5` and `"maximumOverCapacity": 0.1` would be unsatisfiable, since you want to keep at least half of your instances around but only allow 10% overcapacity. To make this latter deployment satisfiable you'd need to change it to `"maximumOverCapacity": 0.5`. 340 | 341 | Tip: If you want to see the exact sequence of events happening, use the [Event Bus](https://mesosphere.github.io/marathon/docs/event-bus.html), like so (note that this is executed from within the DC/OS cluster): 342 | 343 | $ curl -H "Accept: text/event-stream" leader.mesos:8080/v2/events 344 | event: event_stream_attached 345 | data: {"remoteAddress":"10.0.6.211","eventType":"event_stream_attached","timestamp":"2016-10-13T13:29:08.959Z"} 346 | 347 | A recording of an example session for the above case (`/zdd/base-min-over`) is available here: [event-bus-log-base-min-over.txt](event-bus-log-base-min-over.txt). 348 | 349 | ## Canary deployment 350 | 351 | The deployments discussed so far all allowed us to do rolling upgrades of a service without causing any downtimes. That is, at any point in time, clients of the `simpleservice` would be served with some version of the service. However, there is one drawback with the deployments so far: clients of the service will potentially see different versions during the deployment in an uncontrolled manner until the point in time all new instances of the service would turn healthy. 352 | 353 | In a more realistic setup one would use a load balancer in front of the service instances: on the one hand, this would more evenly distribute the load amongst the service instances and on the other hand it allows us to carry out more advanced ZDD such as the one we're discussing in the following: a [canary deployment](http://martinfowler.com/bliki/CanaryRelease.html). The basic idea behind it is to expose a small fraction of the clients to a new version of the service. Once you're confident it works as expected you roll out the new version to all users. If you take this a step further, for example, by having multiple versions of the service you can do also A/B testing with it. 354 | 355 | We now have a look at a canary deployment with DC/OS: we will have 3 instances serving version `0.9` of `simpleservice` and 1 instance serving version `1.0` and want 80% of the traffic to be served by the former and 20% by the latter, the canary. In addition and in contrast to the previous cases want to expose the service to the outside world. That is, `simpleservice` should not only be available to clients within the DC/OS cluster but publicly available, from the wider Internet. So we aim to end up with the following situation: 356 | 357 | +----------+ 358 | | | 359 | | v0.9 +----+ 360 | | | | 361 | +----------+ | 362 | | +----------+ 363 | +----------+ | | | 364 | | | | 80% | | 365 | | v0.9 +----------------------+ | 366 | | | | | | 367 | +----------+ | | | 368 | | | | <-------------+ clients 369 | +----------+ | | | 370 | | | | 20% | | 371 | | v0.9 +----+ +--------+ | 372 | | | | | | 373 | +----------+ | | | 374 | | +----------+ 375 | +----------+ | 376 | | | | 377 | | v1.0 +-------------+ 378 | | | 379 | +----------+ 380 | 381 | Enter [VAMP](http://vamp.io/). VAMP is a platform for managing containerized microservices, supporting canary releases, route updates, metrics collection and service discovery. Note that while VAMP is conveniently available as a [package in the DC/OS Universe](https://github.com/mesosphere/universe/tree/version-3.x/repo/packages/V/vamp/) we will install a more recent version manually in the following to address a dependencies such as Elasticsearch and Logstash better and have a finer-grained control over how we want to use VAMP. 382 | 383 | You can either set up VAMP in an automated fashion, using a [DC/OS Jobs-based installer](https://gist.github.com/mhausenblas/bb967625088902874d631eaa502573cb) or manually, carrying out the following steps: 384 | 385 | 1. Deploy [vamp-es.json](canary/vamp-es.json) 386 | 1. Deploy [vamp.json](canary/vamp.json) 387 | 1. Deploy [vamp-gateway.json](canary/vamp-gateway.json) 388 | 389 | Deploy above either via the `dcos marathon app add` command or using the DC/OS UI and note that in `vamp-gateway.json` you need to change the `instances` to the number of agents you have in your cluster (find that out via `dcos node`): 390 | 391 | ... 392 | "instances": 3, 393 | ... 394 | 395 | Now, head over to `http://$PUBLIC_AGENT:8080`, in my case `http://52.25.126.14:8080/` and you should see: 396 | 397 | ![VAMP idle](img/vamp-idle.png) 398 | 399 | Now you can define a VAMP blueprint (also available via [simpleservice-blueprint.yaml](canary/simpleservice-blueprint.yaml)) by pasting it in the VAMP UI under the `Blueprints` tab and hit `Create` or use the VAMP [HTTP API](http://vamp.io/documentation/api-reference/) to submit it: 400 | 401 | --- 402 | name: simpleservice 403 | gateways: 404 | 10099: simpleservice/port 405 | clusters: 406 | simpleservice: 407 | gateways: 408 | routes: 409 | simpleservice:0.9: 410 | weight: 80% 411 | simpleservice:1.0: 412 | weight: 20% 413 | services: 414 | - 415 | breed: 416 | name: simpleservice:0.9 417 | deployable: mhausenblas/simpleservice:0.4.0 418 | ports: 419 | port: 0/http 420 | env: 421 | SIMPLE_SERVICE_VERSION: "0.9" 422 | scale: 423 | cpu: 0.1 424 | memory: 32MB 425 | instances: 3 426 | - 427 | breed: 428 | name: simpleservice:1.0 429 | deployable: mhausenblas/simpleservice:0.4.0 430 | ports: 431 | port: 0/http 432 | env: 433 | SIMPLE_SERVICE_VERSION: "1.0" 434 | scale: 435 | cpu: 0.1 436 | memory: 32MB 437 | instances: 1 438 | 439 | To use the above blueprint, hit the `Deploy as` button and you should see the following in the `Deployments` tab: 440 | 441 | ![VAMP simpleservice deployments](img/vamp-deployments.png) 442 | 443 | As well as the following under the `Gateways` tab: 444 | 445 | ![VAMP simpleservice gateways](img/vamp-gateways.png) 446 | 447 | We can now check which version clients of `simpleservice` see, using the [canary-check.sh](canary/canary-check.sh) test script as shown in the following (with the public agent, that is, `http://$PUBLIC_AGENT` as the first argument and the number of clients as the optional second argument, `10` in this case): 448 | 449 | $ ./canary-check.sh http://52.25.126.14 10 450 | Invoking simpleservice: 0 451 | Invoking simpleservice: 1 452 | Invoking simpleservice: 2 453 | Invoking simpleservice: 3 454 | Invoking simpleservice: 4 455 | Invoking simpleservice: 5 456 | Invoking simpleservice: 6 457 | Invoking simpleservice: 7 458 | Invoking simpleservice: 8 459 | Invoking simpleservice: 9 460 | Out of 10 clients of simpleservice 8 saw version 0.9 and 2 saw version 1.0 461 | 462 | As expected, now 80% of the clients see version `0.9` and 20% are served by version `1.0`. 463 | 464 | Tip: If you want to simulate more clients here, pass in the number of clients as the second argument, as in `./canary-check.sh http://52.25.126.14 100` to simulate 100 clients, for example. 465 | 466 | With this we conclude the canary deployment section and if you want to learn more, you might also want to check out the [VAMP tutorial on this topic](http://vamp.io/documentation/guides/getting-started-tutorial/2-canary-release/). 467 | 468 | ## Blue-Green deployment 469 | 470 | Another popular form of ZDD supported by DC/OS is the [Blue-Green deployment](http://martinfowler.com/bliki/BlueGreenDeployment.html). Here, the idea is basically to have two versions of your service (unsurprisingly called `blue` and `green`): let's say that `blue` is the live one, serving production traffic and `green` is the new version to be rolled out. Once all instances of `green` are healthy, a load balancer is reconfigured to cut over from `blue` to `green` and if necessary (to roll back) one can do the same in the reverse direction. 471 | 472 | Essentially, we want the following. We start out with `blue` being active: 473 | 474 | +----------------+ 475 | | | 476 | | | +----------+ 477 | | blue (v0.9) +------+ | | 478 | | | | | | 479 | | | +---------+ | 480 | +----------------+ | | 481 | | | 482 | | | <-------------+ clients 483 | | | 484 | +----------------+ | | 485 | | | | | 486 | | | | | 487 | | green (v1.0) | | | 488 | | | +----------+ 489 | | | 490 | +----------------+ 491 | 492 | And once `green` is healthy, we cut over to it by updating the routing: 493 | 494 | +----------------+ 495 | | | 496 | | | +----------+ 497 | | blue (v0.9) | | | 498 | | | | | 499 | | | | | 500 | +----------------+ | | 501 | | | 502 | | | <-------------+ clients 503 | | | 504 | +----------------+ | | 505 | | | +---------+ | 506 | | | | | | 507 | | green (v1.0) +------+ | | 508 | | | +----------+ 509 | | | 510 | +----------------+ 511 | 512 | As a first step, we need a load balancer. For this we install [Marathon-LB](https://dcos.io/docs/1.8/usage/service-discovery/marathon-lb/) (MLB for short) from the Universe: 513 | 514 | $ dcos package install marathon-lb 515 | We recommend a minimum of 0.5 CPUs and 256 MB of RAM available for the Marathon-LB DCOS Service. 516 | Continue installing? [yes/no] yes 517 | Installing Marathon app for package [marathon-lb] version [1.4.1] 518 | Marathon-lb DC/OS Service has been successfully installed! 519 | See https://github.com/mesosphere/marathon-lb for documentation. 520 | 521 | In its default configuration, just as we did with the `dcos package install` command above, MLB runs on a public agent, acting as an edge router and allows us to expose a DC/OS service to the outside world. The MLB default config looks like the following: 522 | 523 | { 524 | "marathon-lb": { 525 | "auto-assign-service-ports": false, 526 | "bind-http-https": true, 527 | "cpus": 2, 528 | "haproxy-group": "external", 529 | "haproxy-map": true, 530 | "instances": 1, 531 | "mem": 1024, 532 | "minimumHealthCapacity": 0.5, 533 | "maximumOverCapacity": 0.2, 534 | "name": "marathon-lb", 535 | "role": "slave_public", 536 | "sysctl-params": "net.ipv4.tcp_tw_reuse=1 net.ipv4.tcp_fin_timeout=30 net.ipv4.tcp_max_syn_backlog=10240 net.ipv4.tcp_max_tw_buckets=400000 net.ipv4.tcp_max_orphans=60000 net.core.somaxconn=10000", 537 | "marathon-uri": "http://master.mesos:8080" 538 | } 539 | } 540 | 541 | MLB is using [HAProxy](http://www.haproxy.org/) under the hood and gets the information it needs to re-write the mappings from frontends to backends from the Marathon event bus. Once MLB is installed, you need to [locate the public agent](https://dcos.io/docs/1.8/administration/locate-public-agent/) it runs on, let's say `$PUBLIC_AGENT` is the resulting IP. Now, to see the HAProxy MLB has under management in action, visit the URL `http://$PUBLIC_AGENT:9090/haproxy?stats` and you should see something like the following: 542 | 543 | ![MLB HAProxy idle](img/haproxy-idle.png) 544 | 545 | In the following we will walk through a manual sequence how to achieve the Blue-Green deployment, however in practice an automated approach is recommended (and pointed out at the end of this section). 546 | 547 | So, let's dive into it. First we set up the `blue` version of `simpleservice` via MLB we're using [blue.json](blue-green/blue.json). In the following is the new section highlighted that has been added to `base-health.json` to make this happen: 548 | 549 | "labels": { 550 | "HAPROXY_GROUP": "external" 551 | "HAPROXY_0_PORT": "10080", 552 | "HAPROXY_0_VHOST": "http://ec2-52-25-126-14.us-west-2.compute.amazonaws.com" 553 | } 554 | 555 | The semantics of the added labels from above is as follows: 556 | 557 | - `HAPROXY_GROUP` is set to expose it on the (edge-routing) MLB we installed in the previous step. 558 | - `HAPROXY_0_PORT` defines`10080` as the external, public port we want version `0.9` of `simpleservice` to be available. 559 | - `HAPROXY_0_VHOST` is the virtual host to be used for the edge routing, in my case the FQDN of the public agent, see also the [MLB docs](https://dcos.io/docs/1.8/usage/service-discovery/marathon-lb/usage/). 560 | 561 | Note that the labels you specify here actually define [service-level HAProxy configurations](https://github.com/mesosphere/marathon-lb/blob/master/Longhelp.md#templates) under the hood. 562 | 563 | Let's check what's going on in HAProxy now: 564 | 565 | ![MLB HAProxy blue](img/haproxy-blue.png) 566 | 567 | In above HAProxy screen shot we can see the `blue` frontend `zdd_blue_10080` for our service, serving on `52.25.126.14:10099` (with `52.25.126.14` being the IP of my public agent) as well as the `blue`backend `zdd_blue_10080`, corresponding to the four instances DC/OS has launched as requested. To verify the ports we can use Mesos-DNS from within the cluster: 568 | 569 | core@ip-10-0-6-211 ~ $ dig _blue-zdd._tcp.marathon.mesos SRV 570 | 571 | ; <<>> DiG 9.10.2-P4 <<>> _blue-zdd._tcp.marathon.mesos SRV 572 | ;; global options: +cmd 573 | ;; Got answer: 574 | ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31245 575 | ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4 576 | 577 | ;; QUESTION SECTION: 578 | ;_blue-zdd._tcp.marathon.mesos. IN SRV 579 | 580 | ;; ANSWER SECTION: 581 | _blue-zdd._tcp.marathon.mesos. 60 IN SRV 0 0 19301 blue-zdd-rrf4y-s2.marathon.mesos. 582 | _blue-zdd._tcp.marathon.mesos. 60 IN SRV 0 0 9383 blue-zdd-8sqqy-s2.marathon.mesos. 583 | _blue-zdd._tcp.marathon.mesos. 60 IN SRV 0 0 3238 blue-zdd-4hzbx-s2.marathon.mesos. 584 | _blue-zdd._tcp.marathon.mesos. 60 IN SRV 0 0 10164 blue-zdd-xu4a3-s2.marathon.mesos. 585 | 586 | ;; ADDITIONAL SECTION: 587 | blue-zdd-xu4a3-s2.marathon.mesos. 60 IN A 10.0.3.192 588 | blue-zdd-8sqqy-s2.marathon.mesos. 60 IN A 10.0.3.192 589 | blue-zdd-rrf4y-s2.marathon.mesos. 60 IN A 10.0.3.192 590 | blue-zdd-4hzbx-s2.marathon.mesos. 60 IN A 10.0.3.192 591 | 592 | ;; Query time: 1 msec 593 | ;; SERVER: 198.51.100.1#53(198.51.100.1) 594 | ;; WHEN: Sat Oct 15 09:15:28 UTC 2016 595 | ;; MSG SIZE rcvd: 263 596 | 597 | We're now in the position that we can access version `0.9` of `simpleservice` from outside the cluster: 598 | 599 | $ curl http://52.25.126.14:10080/endpoint0 600 | {"host": "52.25.126.14:10080", "version": "0.9", "result": "all is well"} 601 | 602 | Next, we deploy version `1.0` of `simpleservice`, using [green.json](blue-green/green.json). Note that nothing has changed so far in HAProxy (check it out, you'll still see the `blue` frontend and backend), however, we have `green` now available within the cluster: 603 | 604 | core@ip-10-0-6-211 ~ $ dig _green-zdd._tcp.marathon.mesos SRV 605 | 606 | ; <<>> DiG 9.10.2-P4 <<>> _green-zdd._tcp.marathon.mesos SRV 607 | ;; global options: +cmd 608 | ;; Got answer: 609 | ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 32879 610 | ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 4 611 | 612 | ;; QUESTION SECTION: 613 | ;_green-zdd._tcp.marathon.mesos. IN SRV 614 | 615 | ;; ANSWER SECTION: 616 | _green-zdd._tcp.marathon.mesos. 60 IN SRV 0 0 30238 green-zdd-re77j-s2.marathon.mesos. 617 | _green-zdd._tcp.marathon.mesos. 60 IN SRV 0 0 7077 green-zdd-c8oxq-s2.marathon.mesos. 618 | _green-zdd._tcp.marathon.mesos. 60 IN SRV 0 0 3409 green-zdd-657om-s2.marathon.mesos. 619 | _green-zdd._tcp.marathon.mesos. 60 IN SRV 0 0 19658 green-zdd-w5mkc-s2.marathon.mesos. 620 | 621 | ;; ADDITIONAL SECTION: 622 | green-zdd-re77j-s2.marathon.mesos. 60 IN A 10.0.3.192 623 | green-zdd-657om-s2.marathon.mesos. 60 IN A 10.0.3.192 624 | green-zdd-c8oxq-s2.marathon.mesos. 60 IN A 10.0.3.192 625 | green-zdd-w5mkc-s2.marathon.mesos. 60 IN A 10.0.3.192 626 | 627 | ;; Query time: 1 msec 628 | ;; SERVER: 198.51.100.1#53(198.51.100.1) 629 | ;; WHEN: Sat Oct 15 09:19:49 UTC 2016 630 | ;; MSG SIZE rcvd: 268 631 | 632 | So we can test `green` cluster-internally, for example using the following command (executed from the Master, here): 633 | 634 | core@ip-10-0-6-211 ~ $ curl green-zdd.marathon.mesos:7077/endpoint0 635 | {"host": "green-zdd.marathon.mesos:7077", "version": "1.0", "result": "all is well"} 636 | 637 | Now let's say we're satisfied with `green`, all instances are healthy so we update it with below snippet, effectively exposing it via MLB, while simultaneously scaling back `blue` to `0` instances: 638 | 639 | "labels": { 640 | "HAPROXY_GROUP": "external", 641 | "HAPROXY_0_PORT": "10080", 642 | "HAPROXY_0_VHOST": "http://ec2-52-25-126-14.us-west-2.compute.amazonaws.com" 643 | } 644 | 645 | As a result `green` should be available via MLB, so let's check what's going on in HAProxy now: 646 | 647 | ![MLB HAProxy green](img/haproxy-green.png) 648 | 649 | Once we're done scaling down `blue` we want to verify if we can access version `1.0` of `simpleservice` from outside the cluster: 650 | 651 | $ curl http://52.25.126.14:10080/endpoint0 652 | {"host": "52.25.126.14:10080", "version": "1.0", "result": "all is well"} 653 | 654 | And indeed we can. Since the exact mechanics of the deployment orchestration are rather complex, I recommend using [zdd.py](https://github.com/mesosphere/marathon-lb#zero-downtime-deployments) a script that makes respective API calls to the DC/OS System Marathon as well as takes care of gracefully terminating instances using the HAProxy stats endpoint. 655 | -------------------------------------------------------------------------------- /blue-green/blue.json: -------------------------------------------------------------------------------- 1 | { 2 | "id": "/zdd/blue", 3 | "instances": 4, 4 | "cpus": 0.1, 5 | "mem": 32, 6 | "container": { 7 | "type": "DOCKER", 8 | "docker": { 9 | "image": "mhausenblas/simpleservice:0.4.0", 10 | "network": "HOST" 11 | } 12 | }, 13 | "env": { 14 | "HEALTH_MIN": "1000", 15 | "HEALTH_MAX": "5000", 16 | "SIMPLE_SERVICE_VERSION": "0.9" 17 | }, 18 | "healthChecks": [{ 19 | "protocol": "HTTP", 20 | "path": "/health", 21 | "gracePeriodSeconds": 300, 22 | "intervalSeconds": 60, 23 | "timeoutSeconds": 20, 24 | "maxConsecutiveFailures": 3, 25 | "ignoreHttp1xx": false 26 | }], 27 | "labels": { 28 | "HAPROXY_GROUP": "external", 29 | "HAPROXY_0_PORT": "10080", 30 | "HAPROXY_0_VHOST": "http://ec2-52-25-126-14.us-west-2.compute.amazonaws.com" 31 | } 32 | } 33 | -------------------------------------------------------------------------------- /blue-green/green.json: -------------------------------------------------------------------------------- 1 | { 2 | "id": "/zdd/green", 3 | "instances": 4, 4 | "cpus": 0.1, 5 | "mem": 32, 6 | "container": { 7 | "type": "DOCKER", 8 | "docker": { 9 | "image": "mhausenblas/simpleservice:0.4.0", 10 | "network": "HOST" 11 | } 12 | }, 13 | "env": { 14 | "HEALTH_MIN": "1000", 15 | "HEALTH_MAX": "5000", 16 | "SIMPLE_SERVICE_VERSION": "1.0" 17 | }, 18 | "healthChecks": [{ 19 | "protocol": "HTTP", 20 | "path": "/health", 21 | "gracePeriodSeconds": 300, 22 | "intervalSeconds": 60, 23 | "timeoutSeconds": 20, 24 | "maxConsecutiveFailures": 3, 25 | "ignoreHttp1xx": false 26 | }] 27 | } 28 | -------------------------------------------------------------------------------- /canary/canary-check.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | COUNTER=0 4 | ENDPOINT=$1 5 | INVOCATIONS="${2:-10}" 6 | RESULTFILE=simpleservice.result 7 | 8 | if [ -f $RESULTFILE ]; then 9 | rm $RESULTFILE 10 | fi 11 | 12 | while [ $COUNTER -lt $INVOCATIONS ]; do 13 | echo Invoking simpleservice: $COUNTER 14 | let COUNTER=COUNTER+1 15 | curl -s $ENDPOINT:10099/endpoint0 >> $RESULTFILE 16 | echo "" >> $RESULTFILE 17 | sleep .1 18 | done 19 | 20 | NUM_v09=`cat $RESULTFILE | grep \"0.9 | wc -l` 21 | NUM_v10=`cat $RESULTFILE | grep \"1.0 | wc -l` 22 | 23 | echo Out of $INVOCATIONS clients of simpleservice $NUM_v09 saw version 0.9 and $NUM_v10 saw version 1.0 -------------------------------------------------------------------------------- /canary/simpleservice-blueprint.yaml: -------------------------------------------------------------------------------- 1 | --- 2 | name: simpleservice 3 | gateways: 4 | 10099: simpleservice/port 5 | clusters: 6 | simpleservice: 7 | gateways: 8 | routes: 9 | simpleservice:0.9: 10 | weight: 80% 11 | simpleservice:1.0: 12 | weight: 20% 13 | services: 14 | - 15 | breed: 16 | name: simpleservice:0.9 17 | deployable: mhausenblas/simpleservice:0.4.0 18 | ports: 19 | port: 0/http 20 | env: 21 | SIMPLE_SERVICE_VERSION: "0.9" 22 | scale: 23 | cpu: 0.1 24 | memory: 32MB 25 | instances: 3 26 | - 27 | breed: 28 | name: simpleservice:1.0 29 | deployable: mhausenblas/simpleservice:0.4.0 30 | ports: 31 | port: 0/http 32 | env: 33 | SIMPLE_SERVICE_VERSION: "1.0" 34 | scale: 35 | cpu: 0.1 36 | memory: 32MB 37 | instances: 1 -------------------------------------------------------------------------------- /canary/vamp-es.json: -------------------------------------------------------------------------------- 1 | { 2 | "id": "elasticsearch", 3 | "instances": 1, 4 | "cpus": 0.2, 5 | "mem": 512, 6 | "container": { 7 | "docker": { 8 | "image": "mesos/elasticsearch-scheduler", 9 | "network": "HOST", 10 | "forcePullImage": true 11 | } 12 | }, 13 | "args": [ 14 | "--zookeeperMesosUrl", "zk://zk-1.zk:2181/mesos", 15 | "--elasticsearchDockerImage", "magneticio/elastic:2.2", 16 | "--elasticsearchRam", "1024", 17 | "--elasticsearchPorts", "9200,9300" 18 | ], 19 | "env": { 20 | "JAVA_OPTS": "-Xms128m -Xmx256m" 21 | } 22 | } 23 | -------------------------------------------------------------------------------- /canary/vamp-gateway.json: -------------------------------------------------------------------------------- 1 | { 2 | "id": "/vamp/vamp-gateway-agent", 3 | "args": [ 4 | "--storeType=zookeeper", 5 | "--storeConnection=zk-1.zk:2181", 6 | "--storeKey=/vamp/haproxy/1.6", 7 | "--logstash=elasticsearch-executor.elasticsearch.mesos:10001" 8 | ], 9 | "cpus": 0.2, 10 | "mem": 256.0, 11 | "instances": 3, 12 | "acceptedResourceRoles": [ 13 | "slave_public", 14 | "*" 15 | ], 16 | "container": { 17 | "type": "DOCKER", 18 | "docker": { 19 | "image": "magneticio/vamp-gateway-agent:0.9.0", 20 | "network": "HOST", 21 | "portMappings": [], 22 | "privileged": true, 23 | "parameters": [] 24 | } 25 | }, 26 | "env": {}, 27 | "constraints": [ 28 | [ 29 | "hostname", 30 | "UNIQUE" 31 | ] 32 | ] 33 | } -------------------------------------------------------------------------------- /canary/vamp.json: -------------------------------------------------------------------------------- 1 | { 2 | "id": "vamp/vamp", 3 | "instances": 1, 4 | "cpus": 0.5, 5 | "mem": 1024, 6 | "container": { 7 | "type": "DOCKER", 8 | "docker": { 9 | "image": "magneticio/vamp:0.9.0-dcos", 10 | "network": "HOST", 11 | "forcePullImage": true 12 | } 13 | }, 14 | "acceptedResourceRoles": [ 15 | "slave_public" 16 | ] 17 | } 18 | -------------------------------------------------------------------------------- /default/base-health.json: -------------------------------------------------------------------------------- 1 | { 2 | "id": "/zdd/base-health", 3 | "instances": 4, 4 | "cpus": 0.1, 5 | "mem": 32, 6 | "container": { 7 | "type": "DOCKER", 8 | "docker": { 9 | "image": "mhausenblas/simpleservice:0.4.0", 10 | "network": "HOST" 11 | } 12 | }, 13 | "env": { 14 | "HEALTH_MIN": "1000", 15 | "HEALTH_MAX": "5000", 16 | "SIMPLE_SERVICE_VERSION": "0.9" 17 | }, 18 | "healthChecks": [{ 19 | "protocol": "HTTP", 20 | "path": "/health", 21 | "gracePeriodSeconds": 300, 22 | "intervalSeconds": 60, 23 | "timeoutSeconds": 20, 24 | "maxConsecutiveFailures": 3, 25 | "ignoreHttp1xx": false 26 | }] 27 | } 28 | -------------------------------------------------------------------------------- /default/base-ready.json: -------------------------------------------------------------------------------- 1 | { 2 | "id": "/zdd/base-ready", 3 | "instances": 4, 4 | "cpus": 0.1, 5 | "mem": 32, 6 | "container": { 7 | "type": "DOCKER", 8 | "docker": { 9 | "image": "mhausenblas/simpleservice:0.4.0", 10 | "network": "HOST" 11 | } 12 | }, 13 | "env": { 14 | "HEALTH_MIN": "1000", 15 | "HEALTH_MAX": "5000", 16 | "SIMPLE_SERVICE_VERSION": "0.9" 17 | }, 18 | "portDefinitions": [{ 19 | "protocol": "tcp", 20 | "name": "main-api", 21 | "port": 0 22 | }], 23 | "readinessChecks": [{ 24 | "protocol": "HTTP", 25 | "path": "/health", 26 | "portName": "main-api", 27 | "intervalSeconds": 30, 28 | "timeoutSeconds": 10, 29 | "httpStatusCodesForReady": [ 30 | 200 31 | ] 32 | }] 33 | } 34 | -------------------------------------------------------------------------------- /default/base.json: -------------------------------------------------------------------------------- 1 | { 2 | "id": "/zdd/base", 3 | "instances": 4, 4 | "cpus": 0.1, 5 | "mem": 32, 6 | "container": { 7 | "type": "DOCKER", 8 | "docker": { 9 | "image": "mhausenblas/simpleservice:0.4.0", 10 | "network": "HOST" 11 | } 12 | }, 13 | "env": { 14 | "HEALTH_MIN": "1000", 15 | "HEALTH_MAX": "5000", 16 | "SIMPLE_SERVICE_VERSION": "0.9" 17 | } 18 | } 19 | -------------------------------------------------------------------------------- /event-bus-log-base-min-over.txt: -------------------------------------------------------------------------------- 1 | event: deployment_info 2 | data: {"plan":{"id":"d36e08e3-a92c-4bbc-a32b-92649229acd1","original":{"id":"/","apps":[],"groups":[{"id":"/zdd","apps":[{"id":"/zdd/base-health","cmd":null,"args":null,"user":null,"env":{"HEALTH_MIN":"1000","HEALTH_MAX":"5000","SIMPLE_SERVICE_VERSION":"1.0"},"instances":4,"cpus":0.1,"mem":32,"disk":0,"gpus":0,"executor":"","constraints":[],"uris":[],"fetch":[],"storeUrls":[],"backoffSeconds":1,"backoffFactor":1.15,"maxLaunchDelaySeconds":3600,"container":{"type":"DOCKER","volumes":[],"docker":{"image":"mhausenblas/simpleservice:0.4.0","network":"HOST","portMappings":null,"privileged":false,"parameters":[],"forcePullImage":false}},"healthChecks":[{"path":"/health","protocol":"HTTP","portIndex":0,"gracePeriodSeconds":300,"intervalSeconds":60,"timeoutSeconds":20,"maxConsecutiveFailures":3,"ignoreHttp1xx":false}],"readinessChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":0.25,"maximumOverCapacity":0.25},"labels":{},"acceptedResourceRoles":null,"ipAddress":null,"version":"2016-10-13T12:34:03.838Z","residency":null,"secrets":{},"taskKillGracePeriodSeconds":null,"ports":[10000],"portDefinitions":[{"port":10000,"protocol":"tcp","labels":{}}],"requirePorts":false,"versionInfo":{"lastScalingAt":"2016-10-13T12:34:03.838Z","lastConfigChangeAt":"2016-10-13T12:34:03.838Z"}},{"id":"/zdd/base-min-over","cmd":null,"args":null,"user":null,"env":{"HEALTH_MIN":"1000","HEALTH_MAX":"5000","SIMPLE_SERVICE_VERSION":"1.0"},"instances":4,"cpus":0.1,"mem":32,"disk":0,"gpus":0,"executor":"","constraints":[],"uris":[],"fetch":[],"storeUrls":[],"backoffSeconds":1,"backoffFactor":1.15,"maxLaunchDelaySeconds":3600,"container":{"type":"DOCKER","volumes":[],"docker":{"image":"mhausenblas/simpleservice:0.4.0","network":"HOST","portMappings":null,"privileged":false,"parameters":[],"forcePullImage":false}},"healthChecks":[{"path":"/health","protocol":"HTTP","portIndex":0,"gracePeriodSeconds":300,"intervalSeconds":60,"timeoutSeconds":20,"maxConsecutiveFailures":3,"ignoreHttp1xx":false}],"readinessChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":0.25,"maximumOverCapacity":0.25},"labels":{},"acceptedResourceRoles":null,"ipAddress":null,"version":"2016-10-13T12:36:48.366Z","residency":null,"secrets":{},"taskKillGracePeriodSeconds":null,"ports":[10000],"portDefinitions":[{"port":10000,"protocol":"tcp","labels":{}}],"requirePorts":false,"versionInfo":{"lastScalingAt":"2016-10-13T12:36:48.366Z","lastConfigChangeAt":"2016-10-13T12:36:48.366Z"}}],"groups":[],"dependencies":[],"version":"2016-10-13T12:36:48.366Z"}],"dependencies":[],"version":"2016-10-13T12:36:48.366Z"},"target":{"id":"/","apps":[],"groups":[{"id":"/zdd","apps":[{"id":"/zdd/base-health","cmd":null,"args":null,"user":null,"env":{"HEALTH_MIN":"1000","HEALTH_MAX":"5000","SIMPLE_SERVICE_VERSION":"1.0"},"instances":4,"cpus":0.1,"mem":32,"disk":0,"gpus":0,"executor":"","constraints":[],"uris":[],"fetch":[],"storeUrls":[],"backoffSeconds":1,"backoffFactor":1.15,"maxLaunchDelaySeconds":3600,"container":{"type":"DOCKER","volumes":[],"docker":{"image":"mhausenblas/simpleservice:0.4.0","network":"HOST","portMappings":null,"privileged":false,"parameters":[],"forcePullImage":false}},"healthChecks":[{"path":"/health","protocol":"HTTP","portIndex":0,"gracePeriodSeconds":300,"intervalSeconds":60,"timeoutSeconds":20,"maxConsecutiveFailures":3,"ignoreHttp1xx":false}],"readinessChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":0.25,"maximumOverCapacity":0.25},"labels":{},"acceptedResourceRoles":null,"ipAddress":null,"version":"2016-10-13T12:34:03.838Z","residency":null,"secrets":{},"taskKillGracePeriodSeconds":null,"ports":[10000],"portDefinitions":[{"port":10000,"protocol":"tcp","labels":{}}],"requirePorts":false,"versionInfo":{"lastScalingAt":"2016-10-13T12:34:03.838Z","lastConfigChangeAt":"2016-10-13T12:34:03.838Z"}},{"id":"/zdd/base-min-over","cmd":null,"args":null,"user":null,"env":{"HEALTH_MIN":"1000","HEALTH_MAX":"5000","SIMPLE_SERVICE_VERSION":"0.9"},"instances":4,"cpus":0.1,"mem":32,"disk":0,"gpus":0,"executor":"","constraints":[],"uris":[],"fetch":[],"storeUrls":[],"backoffSeconds":1,"backoffFactor":1.15,"maxLaunchDelaySeconds":3600,"container":{"type":"DOCKER","volumes":[],"docker":{"image":"mhausenblas/simpleservice:0.4.0","network":"HOST","portMappings":null,"privileged":false,"parameters":[],"forcePullImage":false}},"healthChecks":[{"path":"/health","protocol":"HTTP","portIndex":0,"gracePeriodSeconds":300,"intervalSeconds":60,"timeoutSeconds":20,"maxConsecutiveFailures":3,"ignoreHttp1xx":false}],"readinessChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":0.25,"maximumOverCapacity":0.25},"labels":{},"acceptedResourceRoles":null,"ipAddress":null,"version":"2016-10-13T13:31:39.055Z","residency":null,"secrets":{},"taskKillGracePeriodSeconds":null,"ports":[10000],"portDefinitions":[{"port":10000,"protocol":"tcp","labels":{}}],"requirePorts":false,"versionInfo":{"lastScalingAt":"2016-10-13T13:31:39.055Z","lastConfigChangeAt":"2016-10-13T13:31:39.055Z"}}],"groups":[],"dependencies":[],"version":"2016-10-13T13:31:39.055Z"}],"dependencies":[],"version":"2016-10-13T13:31:39.055Z"},"steps":[{"actions":[{"action":"RestartApplication","app":"/zdd/base-min-over"}]}],"version":"2016-10-13T13:31:39.055Z"},"currentStep":{"actions":[{"action":"RestartApplication","app":"/zdd/base-min-over"}]},"eventType":"deployment_info","timestamp":"2016-10-13T13:31:39.067Z"} 3 | 4 | event: add_health_check_event 5 | data: {"appId":"/zdd/base-min-over","version":"2016-10-13T13:31:39.055Z","healthCheck":{"path":"/health","protocol":"HTTP","portIndex":0,"gracePeriodSeconds":300,"intervalSeconds":60,"timeoutSeconds":20,"maxConsecutiveFailures":3,"ignoreHttp1xx":false},"eventType":"add_health_check_event","timestamp":"2016-10-13T13:31:39.067Z"} 6 | 7 | event: group_change_success 8 | data: {"groupId":"/zdd","version":"2016-10-13T13:31:39.055Z","eventType":"group_change_success","timestamp":"2016-10-13T13:31:39.085Z"} 9 | 10 | event: api_post_event 11 | data: {"clientIp":"10.0.6.211","uri":"/v2/apps/zdd/base-min-over","appDefinition":{"id":"/zdd/base-min-over","cmd":null,"args":null,"user":null,"env":{"HEALTH_MIN":"1000","HEALTH_MAX":"5000","SIMPLE_SERVICE_VERSION":"0.9"},"instances":4,"cpus":0.1,"mem":32,"disk":0,"gpus":0,"executor":"","constraints":[],"uris":[],"fetch":[],"storeUrls":[],"backoffSeconds":1,"backoffFactor":1.15,"maxLaunchDelaySeconds":3600,"container":{"type":"DOCKER","volumes":[],"docker":{"image":"mhausenblas/simpleservice:0.4.0","network":"HOST","portMappings":null,"privileged":false,"parameters":[],"forcePullImage":false}},"healthChecks":[{"path":"/health","protocol":"HTTP","portIndex":0,"gracePeriodSeconds":300,"intervalSeconds":60,"timeoutSeconds":20,"maxConsecutiveFailures":3,"ignoreHttp1xx":false}],"readinessChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":0.25,"maximumOverCapacity":0.25},"labels":{},"acceptedResourceRoles":null,"ipAddress":null,"version":"2016-10-13T13:31:39.055Z","residency":null,"secrets":{},"taskKillGracePeriodSeconds":null,"ports":[10000],"portDefinitions":[{"port":10000,"protocol":"tcp","labels":{}}],"requirePorts":false,"versionInfo":{"lastScalingAt":"2016-10-13T13:31:39.055Z","lastConfigChangeAt":"2016-10-13T13:31:39.055Z"}},"eventType":"api_post_event","timestamp":"2016-10-13T13:31:39.085Z"} 12 | 13 | event: status_update_event 14 | data: {"slaveId":"145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S2","taskId":"zdd_base-min-over.c475e0d3-9141-11e6-aae4-3a4b79075094","taskStatus":"TASK_KILLING","message":"","appId":"/zdd/base-min-over","host":"10.0.3.192","ipAddresses":[{"ipAddress":"10.0.3.192","protocol":"IPv4"}],"ports":[18107],"version":"2016-10-13T12:36:48.366Z","eventType":"status_update_event","timestamp":"2016-10-13T13:31:39.088Z"} 15 | 16 | event: status_update_event 17 | data: {"slaveId":"145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S1","taskId":"zdd_base-min-over.b5689380-9141-11e6-aae4-3a4b79075094","taskStatus":"TASK_KILLING","message":"","appId":"/zdd/base-min-over","host":"10.0.3.193","ipAddresses":[{"ipAddress":"10.0.3.193","protocol":"IPv4"}],"ports":[31419],"version":"2016-10-13T12:36:48.366Z","eventType":"status_update_event","timestamp":"2016-10-13T13:31:39.088Z"} 18 | 19 | event: status_update_event 20 | data: {"slaveId":"145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S1","taskId":"zdd_base-min-over.5ed13e24-9149-11e6-aae4-3a4b79075094","taskStatus":"TASK_STAGING","message":"","appId":"/zdd/base-min-over","host":"10.0.3.193","ports":[16643],"version":"2016-10-13T13:31:39.055Z","eventType":"status_update_event","timestamp":"2016-10-13T13:31:39.088Z"} 21 | 22 | event: status_update_event 23 | data: {"slaveId":"145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S2","taskId":"zdd_base-min-over.c475b9c2-9141-11e6-aae4-3a4b79075094","taskStatus":"TASK_KILLING","message":"","appId":"/zdd/base-min-over","host":"10.0.3.192","ipAddresses":[{"ipAddress":"10.0.3.192","protocol":"IPv4"}],"ports":[11218],"version":"2016-10-13T12:36:48.366Z","eventType":"status_update_event","timestamp":"2016-10-13T13:31:39.092Z"} 24 | 25 | event: status_update_event 26 | data: {"slaveId":"145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S1","taskId":"zdd_base-min-over.5ed13e24-9149-11e6-aae4-3a4b79075094","taskStatus":"TASK_RUNNING","message":"","appId":"/zdd/base-min-over","host":"10.0.3.193","ipAddresses":[{"ipAddress":"10.0.3.193","protocol":"IPv4"}],"ports":[16643],"version":"2016-10-13T13:31:39.055Z","eventType":"status_update_event","timestamp":"2016-10-13T13:31:39.986Z"} 27 | 28 | event: health_status_changed_event 29 | data: {"appId":"/zdd/base-min-over","taskId":"zdd_base-min-over.5ed13e24-9149-11e6-aae4-3a4b79075094","version":"2016-10-13T13:31:39.055Z","alive":true,"eventType":"health_status_changed_event","timestamp":"2016-10-13T13:31:46.710Z"} 30 | 31 | event: status_update_event 32 | data: {"slaveId":"145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S1","taskId":"zdd_base-min-over.c176aa91-9141-11e6-aae4-3a4b79075094","taskStatus":"TASK_KILLING","message":"","appId":"/zdd/base-min-over","host":"10.0.3.193","ipAddresses":[{"ipAddress":"10.0.3.193","protocol":"IPv4"}],"ports":[26681],"version":"2016-10-13T12:36:48.366Z","eventType":"status_update_event","timestamp":"2016-10-13T13:31:46.719Z"} 33 | 34 | 35 | 36 | event: status_update_event 37 | data: {"slaveId":"145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S1","taskId":"zdd_base-min-over.b5689380-9141-11e6-aae4-3a4b79075094","taskStatus":"TASK_KILLED","message":"Container exited with status 137","appId":"/zdd/base-min-over","host":"10.0.3.193","ipAddresses":[{"ipAddress":"10.0.3.193","protocol":"IPv4"}],"ports":[31419],"version":"2016-10-13T12:36:48.366Z","eventType":"status_update_event","timestamp":"2016-10-13T13:31:59.208Z"} 38 | 39 | event: status_update_event 40 | data: {"slaveId":"145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S2","taskId":"zdd_base-min-over.6ad14b75-9149-11e6-aae4-3a4b79075094","taskStatus":"TASK_STAGING","message":"","appId":"/zdd/base-min-over","host":"10.0.3.192","ports":[6518],"version":"2016-10-13T13:31:39.055Z","eventType":"status_update_event","timestamp":"2016-10-13T13:31:59.217Z"} 41 | 42 | event: status_update_event 43 | data: {"slaveId":"145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S2","taskId":"zdd_base-min-over.c475b9c2-9141-11e6-aae4-3a4b79075094","taskStatus":"TASK_KILLED","message":"Container exited with status 137","appId":"/zdd/base-min-over","host":"10.0.3.192","ipAddresses":[{"ipAddress":"10.0.3.192","protocol":"IPv4"}],"ports":[11218],"version":"2016-10-13T12:36:48.366Z","eventType":"status_update_event","timestamp":"2016-10-13T13:31:59.366Z"} 44 | 45 | event: status_update_event 46 | data: {"slaveId":"145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S2","taskId":"zdd_base-min-over.c475e0d3-9141-11e6-aae4-3a4b79075094","taskStatus":"TASK_KILLED","message":"Container exited with status 137","appId":"/zdd/base-min-over","host":"10.0.3.192","ipAddresses":[{"ipAddress":"10.0.3.192","protocol":"IPv4"}],"ports":[18107],"version":"2016-10-13T12:36:48.366Z","eventType":"status_update_event","timestamp":"2016-10-13T13:31:59.369Z"} 47 | 48 | event: status_update_event 49 | data: {"slaveId":"145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S2","taskId":"zdd_base-min-over.6ad14b75-9149-11e6-aae4-3a4b79075094","taskStatus":"TASK_RUNNING","message":"","appId":"/zdd/base-min-over","host":"10.0.3.192","ipAddresses":[{"ipAddress":"10.0.3.192","protocol":"IPv4"}],"ports":[6518],"version":"2016-10-13T13:31:39.055Z","eventType":"status_update_event","timestamp":"2016-10-13T13:32:00.123Z"} 50 | 51 | event: status_update_event 52 | data: {"slaveId":"145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S2","taskId":"zdd_base-min-over.6dcf4936-9149-11e6-aae4-3a4b79075094","taskStatus":"TASK_STAGING","message":"","appId":"/zdd/base-min-over","host":"10.0.3.192","ports":[11179],"version":"2016-10-13T13:31:39.055Z","eventType":"status_update_event","timestamp":"2016-10-13T13:32:04.238Z"} 53 | 54 | event: status_update_event 55 | data: {"slaveId":"145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S1","taskId":"zdd_base-min-over.6dcf7047-9149-11e6-aae4-3a4b79075094","taskStatus":"TASK_STAGING","message":"","appId":"/zdd/base-min-over","host":"10.0.3.193","ports":[26829],"version":"2016-10-13T13:31:39.055Z","eventType":"status_update_event","timestamp":"2016-10-13T13:32:04.241Z"} 56 | 57 | event: status_update_event 58 | data: {"slaveId":"145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S1","taskId":"zdd_base-min-over.6dcf7047-9149-11e6-aae4-3a4b79075094","taskStatus":"TASK_RUNNING","message":"","appId":"/zdd/base-min-over","host":"10.0.3.193","ipAddresses":[{"ipAddress":"10.0.3.193","protocol":"IPv4"}],"ports":[26829],"version":"2016-10-13T13:31:39.055Z","eventType":"status_update_event","timestamp":"2016-10-13T13:32:05.079Z"} 59 | 60 | event: status_update_event 61 | data: {"slaveId":"145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S2","taskId":"zdd_base-min-over.6dcf4936-9149-11e6-aae4-3a4b79075094","taskStatus":"TASK_RUNNING","message":"","appId":"/zdd/base-min-over","host":"10.0.3.192","ipAddresses":[{"ipAddress":"10.0.3.192","protocol":"IPv4"}],"ports":[11179],"version":"2016-10-13T13:31:39.055Z","eventType":"status_update_event","timestamp":"2016-10-13T13:32:05.139Z"} 62 | 63 | event: status_update_event 64 | data: {"slaveId":"145f052d-8bcb-457f-b1e6-b1b4e2cdf787-S1","taskId":"zdd_base-min-over.c176aa91-9141-11e6-aae4-3a4b79075094","taskStatus":"TASK_KILLED","message":"Container exited with status 137","appId":"/zdd/base-min-over","host":"10.0.3.193","ipAddresses":[{"ipAddress":"10.0.3.193","protocol":"IPv4"}],"ports":[26681],"version":"2016-10-13T12:36:48.366Z","eventType":"status_update_event","timestamp":"2016-10-13T13:32:06.939Z"} 65 | 66 | 67 | 68 | 69 | 70 | event: health_status_changed_event 71 | data: {"appId":"/zdd/base-min-over","taskId":"zdd_base-min-over.6ad14b75-9149-11e6-aae4-3a4b79075094","version":"2016-10-13T13:31:39.055Z","alive":true,"eventType":"health_status_changed_event","timestamp":"2016-10-13T13:32:46.080Z"} 72 | 73 | event: health_status_changed_event 74 | data: {"appId":"/zdd/base-min-over","taskId":"zdd_base-min-over.6dcf7047-9149-11e6-aae4-3a4b79075094","version":"2016-10-13T13:31:39.055Z","alive":true,"eventType":"health_status_changed_event","timestamp":"2016-10-13T13:32:47.303Z"} 75 | 76 | event: health_status_changed_event 77 | data: {"appId":"/zdd/base-min-over","taskId":"zdd_base-min-over.6dcf4936-9149-11e6-aae4-3a4b79075094","version":"2016-10-13T13:31:39.055Z","alive":true,"eventType":"health_status_changed_event","timestamp":"2016-10-13T13:32:48.600Z"} 78 | 79 | event: deployment_step_success 80 | data: {"plan":{"id":"d36e08e3-a92c-4bbc-a32b-92649229acd1","original":{"id":"/","apps":[],"groups":[{"id":"/zdd","apps":[{"id":"/zdd/base-health","cmd":null,"args":null,"user":null,"env":{"HEALTH_MIN":"1000","HEALTH_MAX":"5000","SIMPLE_SERVICE_VERSION":"1.0"},"instances":4,"cpus":0.1,"mem":32,"disk":0,"gpus":0,"executor":"","constraints":[],"uris":[],"fetch":[],"storeUrls":[],"backoffSeconds":1,"backoffFactor":1.15,"maxLaunchDelaySeconds":3600,"container":{"type":"DOCKER","volumes":[],"docker":{"image":"mhausenblas/simpleservice:0.4.0","network":"HOST","portMappings":null,"privileged":false,"parameters":[],"forcePullImage":false}},"healthChecks":[{"path":"/health","protocol":"HTTP","portIndex":0,"gracePeriodSeconds":300,"intervalSeconds":60,"timeoutSeconds":20,"maxConsecutiveFailures":3,"ignoreHttp1xx":false}],"readinessChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":0.25,"maximumOverCapacity":0.25},"labels":{},"acceptedResourceRoles":null,"ipAddress":null,"version":"2016-10-13T12:34:03.838Z","residency":null,"secrets":{},"taskKillGracePeriodSeconds":null,"ports":[10000],"portDefinitions":[{"port":10000,"protocol":"tcp","labels":{}}],"requirePorts":false,"versionInfo":{"lastScalingAt":"2016-10-13T12:34:03.838Z","lastConfigChangeAt":"2016-10-13T12:34:03.838Z"}},{"id":"/zdd/base-min-over","cmd":null,"args":null,"user":null,"env":{"HEALTH_MIN":"1000","HEALTH_MAX":"5000","SIMPLE_SERVICE_VERSION":"1.0"},"instances":4,"cpus":0.1,"mem":32,"disk":0,"gpus":0,"executor":"","constraints":[],"uris":[],"fetch":[],"storeUrls":[],"backoffSeconds":1,"backoffFactor":1.15,"maxLaunchDelaySeconds":3600,"container":{"type":"DOCKER","volumes":[],"docker":{"image":"mhausenblas/simpleservice:0.4.0","network":"HOST","portMappings":null,"privileged":false,"parameters":[],"forcePullImage":false}},"healthChecks":[{"path":"/health","protocol":"HTTP","portIndex":0,"gracePeriodSeconds":300,"intervalSeconds":60,"timeoutSeconds":20,"maxConsecutiveFailures":3,"ignoreHttp1xx":false}],"readinessChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":0.25,"maximumOverCapacity":0.25},"labels":{},"acceptedResourceRoles":null,"ipAddress":null,"version":"2016-10-13T12:36:48.366Z","residency":null,"secrets":{},"taskKillGracePeriodSeconds":null,"ports":[10000],"portDefinitions":[{"port":10000,"protocol":"tcp","labels":{}}],"requirePorts":false,"versionInfo":{"lastScalingAt":"2016-10-13T12:36:48.366Z","lastConfigChangeAt":"2016-10-13T12:36:48.366Z"}}],"groups":[],"dependencies":[],"version":"2016-10-13T12:36:48.366Z"}],"dependencies":[],"version":"2016-10-13T12:36:48.366Z"},"target":{"id":"/","apps":[],"groups":[{"id":"/zdd","apps":[{"id":"/zdd/base-health","cmd":null,"args":null,"user":null,"env":{"HEALTH_MIN":"1000","HEALTH_MAX":"5000","SIMPLE_SERVICE_VERSION":"1.0"},"instances":4,"cpus":0.1,"mem":32,"disk":0,"gpus":0,"executor":"","constraints":[],"uris":[],"fetch":[],"storeUrls":[],"backoffSeconds":1,"backoffFactor":1.15,"maxLaunchDelaySeconds":3600,"container":{"type":"DOCKER","volumes":[],"docker":{"image":"mhausenblas/simpleservice:0.4.0","network":"HOST","portMappings":null,"privileged":false,"parameters":[],"forcePullImage":false}},"healthChecks":[{"path":"/health","protocol":"HTTP","portIndex":0,"gracePeriodSeconds":300,"intervalSeconds":60,"timeoutSeconds":20,"maxConsecutiveFailures":3,"ignoreHttp1xx":false}],"readinessChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":0.25,"maximumOverCapacity":0.25},"labels":{},"acceptedResourceRoles":null,"ipAddress":null,"version":"2016-10-13T12:34:03.838Z","residency":null,"secrets":{},"taskKillGracePeriodSeconds":null,"ports":[10000],"portDefinitions":[{"port":10000,"protocol":"tcp","labels":{}}],"requirePorts":false,"versionInfo":{"lastScalingAt":"2016-10-13T12:34:03.838Z","lastConfigChangeAt":"2016-10-13T12:34:03.838Z"}},{"id":"/zdd/base-min-over","cmd":null,"args":null,"user":null,"env":{"HEALTH_MIN":"1000","HEALTH_MAX":"5000","SIMPLE_SERVICE_VERSION":"0.9"},"instances":4,"cpus":0.1,"mem":32,"disk":0,"gpus":0,"executor":"","constraints":[],"uris":[],"fetch":[],"storeUrls":[],"backoffSeconds":1,"backoffFactor":1.15,"maxLaunchDelaySeconds":3600,"container":{"type":"DOCKER","volumes":[],"docker":{"image":"mhausenblas/simpleservice:0.4.0","network":"HOST","portMappings":null,"privileged":false,"parameters":[],"forcePullImage":false}},"healthChecks":[{"path":"/health","protocol":"HTTP","portIndex":0,"gracePeriodSeconds":300,"intervalSeconds":60,"timeoutSeconds":20,"maxConsecutiveFailures":3,"ignoreHttp1xx":false}],"readinessChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":0.25,"maximumOverCapacity":0.25},"labels":{},"acceptedResourceRoles":null,"ipAddress":null,"version":"2016-10-13T13:31:39.055Z","residency":null,"secrets":{},"taskKillGracePeriodSeconds":null,"ports":[10000],"portDefinitions":[{"port":10000,"protocol":"tcp","labels":{}}],"requirePorts":false,"versionInfo":{"lastScalingAt":"2016-10-13T13:31:39.055Z","lastConfigChangeAt":"2016-10-13T13:31:39.055Z"}}],"groups":[],"dependencies":[],"version":"2016-10-13T13:31:39.055Z"}],"dependencies":[],"version":"2016-10-13T13:31:39.055Z"},"steps":[{"actions":[{"action":"RestartApplication","app":"/zdd/base-min-over"}]}],"version":"2016-10-13T13:31:39.055Z"},"currentStep":{"actions":[{"action":"RestartApplication","app":"/zdd/base-min-over"}]},"eventType":"deployment_step_success","timestamp":"2016-10-13T13:32:48.601Z"} 81 | 82 | event: deployment_success 83 | data: {"id":"d36e08e3-a92c-4bbc-a32b-92649229acd1","plan":{"id":"d36e08e3-a92c-4bbc-a32b-92649229acd1","original":{"id":"/","apps":[],"groups":[{"id":"/zdd","apps":[{"id":"/zdd/base-health","cmd":null,"args":null,"user":null,"env":{"HEALTH_MIN":"1000","HEALTH_MAX":"5000","SIMPLE_SERVICE_VERSION":"1.0"},"instances":4,"cpus":0.1,"mem":32,"disk":0,"gpus":0,"executor":"","constraints":[],"uris":[],"fetch":[],"storeUrls":[],"backoffSeconds":1,"backoffFactor":1.15,"maxLaunchDelaySeconds":3600,"container":{"type":"DOCKER","volumes":[],"docker":{"image":"mhausenblas/simpleservice:0.4.0","network":"HOST","portMappings":null,"privileged":false,"parameters":[],"forcePullImage":false}},"healthChecks":[{"path":"/health","protocol":"HTTP","portIndex":0,"gracePeriodSeconds":300,"intervalSeconds":60,"timeoutSeconds":20,"maxConsecutiveFailures":3,"ignoreHttp1xx":false}],"readinessChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":0.25,"maximumOverCapacity":0.25},"labels":{},"acceptedResourceRoles":null,"ipAddress":null,"version":"2016-10-13T12:34:03.838Z","residency":null,"secrets":{},"taskKillGracePeriodSeconds":null,"ports":[10000],"portDefinitions":[{"port":10000,"protocol":"tcp","labels":{}}],"requirePorts":false,"versionInfo":{"lastScalingAt":"2016-10-13T12:34:03.838Z","lastConfigChangeAt":"2016-10-13T12:34:03.838Z"}},{"id":"/zdd/base-min-over","cmd":null,"args":null,"user":null,"env":{"HEALTH_MIN":"1000","HEALTH_MAX":"5000","SIMPLE_SERVICE_VERSION":"1.0"},"instances":4,"cpus":0.1,"mem":32,"disk":0,"gpus":0,"executor":"","constraints":[],"uris":[],"fetch":[],"storeUrls":[],"backoffSeconds":1,"backoffFactor":1.15,"maxLaunchDelaySeconds":3600,"container":{"type":"DOCKER","volumes":[],"docker":{"image":"mhausenblas/simpleservice:0.4.0","network":"HOST","portMappings":null,"privileged":false,"parameters":[],"forcePullImage":false}},"healthChecks":[{"path":"/health","protocol":"HTTP","portIndex":0,"gracePeriodSeconds":300,"intervalSeconds":60,"timeoutSeconds":20,"maxConsecutiveFailures":3,"ignoreHttp1xx":false}],"readinessChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":0.25,"maximumOverCapacity":0.25},"labels":{},"acceptedResourceRoles":null,"ipAddress":null,"version":"2016-10-13T12:36:48.366Z","residency":null,"secrets":{},"taskKillGracePeriodSeconds":null,"ports":[10000],"portDefinitions":[{"port":10000,"protocol":"tcp","labels":{}}],"requirePorts":false,"versionInfo":{"lastScalingAt":"2016-10-13T12:36:48.366Z","lastConfigChangeAt":"2016-10-13T12:36:48.366Z"}}],"groups":[],"dependencies":[],"version":"2016-10-13T12:36:48.366Z"}],"dependencies":[],"version":"2016-10-13T12:36:48.366Z"},"target":{"id":"/","apps":[],"groups":[{"id":"/zdd","apps":[{"id":"/zdd/base-health","cmd":null,"args":null,"user":null,"env":{"HEALTH_MIN":"1000","HEALTH_MAX":"5000","SIMPLE_SERVICE_VERSION":"1.0"},"instances":4,"cpus":0.1,"mem":32,"disk":0,"gpus":0,"executor":"","constraints":[],"uris":[],"fetch":[],"storeUrls":[],"backoffSeconds":1,"backoffFactor":1.15,"maxLaunchDelaySeconds":3600,"container":{"type":"DOCKER","volumes":[],"docker":{"image":"mhausenblas/simpleservice:0.4.0","network":"HOST","portMappings":null,"privileged":false,"parameters":[],"forcePullImage":false}},"healthChecks":[{"path":"/health","protocol":"HTTP","portIndex":0,"gracePeriodSeconds":300,"intervalSeconds":60,"timeoutSeconds":20,"maxConsecutiveFailures":3,"ignoreHttp1xx":false}],"readinessChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":0.25,"maximumOverCapacity":0.25},"labels":{},"acceptedResourceRoles":null,"ipAddress":null,"version":"2016-10-13T12:34:03.838Z","residency":null,"secrets":{},"taskKillGracePeriodSeconds":null,"ports":[10000],"portDefinitions":[{"port":10000,"protocol":"tcp","labels":{}}],"requirePorts":false,"versionInfo":{"lastScalingAt":"2016-10-13T12:34:03.838Z","lastConfigChangeAt":"2016-10-13T12:34:03.838Z"}},{"id":"/zdd/base-min-over","cmd":null,"args":null,"user":null,"env":{"HEALTH_MIN":"1000","HEALTH_MAX":"5000","SIMPLE_SERVICE_VERSION":"0.9"},"instances":4,"cpus":0.1,"mem":32,"disk":0,"gpus":0,"executor":"","constraints":[],"uris":[],"fetch":[],"storeUrls":[],"backoffSeconds":1,"backoffFactor":1.15,"maxLaunchDelaySeconds":3600,"container":{"type":"DOCKER","volumes":[],"docker":{"image":"mhausenblas/simpleservice:0.4.0","network":"HOST","portMappings":null,"privileged":false,"parameters":[],"forcePullImage":false}},"healthChecks":[{"path":"/health","protocol":"HTTP","portIndex":0,"gracePeriodSeconds":300,"intervalSeconds":60,"timeoutSeconds":20,"maxConsecutiveFailures":3,"ignoreHttp1xx":false}],"readinessChecks":[],"dependencies":[],"upgradeStrategy":{"minimumHealthCapacity":0.25,"maximumOverCapacity":0.25},"labels":{},"acceptedResourceRoles":null,"ipAddress":null,"version":"2016-10-13T13:31:39.055Z","residency":null,"secrets":{},"taskKillGracePeriodSeconds":null,"ports":[10000],"portDefinitions":[{"port":10000,"protocol":"tcp","labels":{}}],"requirePorts":false,"versionInfo":{"lastScalingAt":"2016-10-13T13:31:39.055Z","lastConfigChangeAt":"2016-10-13T13:31:39.055Z"}}],"groups":[],"dependencies":[],"version":"2016-10-13T13:31:39.055Z"}],"dependencies":[],"version":"2016-10-13T13:31:39.055Z"},"steps":[{"actions":[{"action":"RestartApplication","app":"/zdd/base-min-over"}]}],"version":"2016-10-13T13:31:39.055Z"},"eventType":"deployment_success","timestamp":"2016-10-13T13:32:48.601Z"} -------------------------------------------------------------------------------- /img/base-health-update-deployment-step0.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mhausenblas/zdd-lab/2d29ce6130698f4437445f67faa0029255fa9088/img/base-health-update-deployment-step0.png -------------------------------------------------------------------------------- /img/base-health-update-deployment-step1.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mhausenblas/zdd-lab/2d29ce6130698f4437445f67faa0029255fa9088/img/base-health-update-deployment-step1.png -------------------------------------------------------------------------------- /img/base-health-update-deployment-step2.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mhausenblas/zdd-lab/2d29ce6130698f4437445f67faa0029255fa9088/img/base-health-update-deployment-step2.png -------------------------------------------------------------------------------- /img/base-health-update-deployment-step3.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mhausenblas/zdd-lab/2d29ce6130698f4437445f67faa0029255fa9088/img/base-health-update-deployment-step3.png -------------------------------------------------------------------------------- /img/base-health-update-deployment-step4.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mhausenblas/zdd-lab/2d29ce6130698f4437445f67faa0029255fa9088/img/base-health-update-deployment-step4.png -------------------------------------------------------------------------------- /img/base-health-update-deployment.gif: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mhausenblas/zdd-lab/2d29ce6130698f4437445f67faa0029255fa9088/img/base-health-update-deployment.gif -------------------------------------------------------------------------------- /img/base-health.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mhausenblas/zdd-lab/2d29ce6130698f4437445f67faa0029255fa9088/img/base-health.png -------------------------------------------------------------------------------- /img/base-min-over-step.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mhausenblas/zdd-lab/2d29ce6130698f4437445f67faa0029255fa9088/img/base-min-over-step.png -------------------------------------------------------------------------------- /img/base-update-deployment.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mhausenblas/zdd-lab/2d29ce6130698f4437445f67faa0029255fa9088/img/base-update-deployment.png -------------------------------------------------------------------------------- /img/base-update.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mhausenblas/zdd-lab/2d29ce6130698f4437445f67faa0029255fa9088/img/base-update.png -------------------------------------------------------------------------------- /img/haproxy-blue.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mhausenblas/zdd-lab/2d29ce6130698f4437445f67faa0029255fa9088/img/haproxy-blue.png -------------------------------------------------------------------------------- /img/haproxy-green.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mhausenblas/zdd-lab/2d29ce6130698f4437445f67faa0029255fa9088/img/haproxy-green.png -------------------------------------------------------------------------------- /img/haproxy-idle.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mhausenblas/zdd-lab/2d29ce6130698f4437445f67faa0029255fa9088/img/haproxy-idle.png -------------------------------------------------------------------------------- /img/vamp-deployments.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mhausenblas/zdd-lab/2d29ce6130698f4437445f67faa0029255fa9088/img/vamp-deployments.png -------------------------------------------------------------------------------- /img/vamp-gateways.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mhausenblas/zdd-lab/2d29ce6130698f4437445f67faa0029255fa9088/img/vamp-gateways.png -------------------------------------------------------------------------------- /img/vamp-idle.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/mhausenblas/zdd-lab/2d29ce6130698f4437445f67faa0029255fa9088/img/vamp-idle.png --------------------------------------------------------------------------------