├── LICENSE
└── README.md


/LICENSE:
--------------------------------------------------------------------------------
 1 | The MIT License (MIT)
 2 | 
 3 | Copyright (c) 2016 
 4 | 
 5 | Permission is hereby granted, free of charge, to any person obtaining a copy
 6 | of this software and associated documentation files (the "Software"), to deal
 7 | in the Software without restriction, including without limitation the rights
 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 | 
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 | 
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | # dcos-rexray
  2 | 
  3 | DCOS with EMC {code} RexRay & DVDI for persistent storage
  4 | 
  5 | * DCOS v1.7
  6 | * DVDCLI v0.2.0
  7 | * DVDI v0.4.2
  8 | * rexray v0.3.3
  9 | 
 10 | ## Overview
 11 | This walkthrough is meant to be used by those who want to play with DCOS 1.7 (not the Enterprise Ed). 
 12 | Everything described here is for demo purposes and purely experimental. 
 13 | 
 14 | Tested on AWS with DCOS Community Edition, one Mesos Master.
 15 | 
 16 | DCOS 1.7 comes with Docker 1.7.1 on CoreOS. According to Mesosphere there is no support for Docker containers with external volumes for DCOS installations running Docker older than 1.8.
 17 | I have experimented with modifying the original DCOS CloudFormation template to use the latest stable CoreOS which comes with Docker 1.9.1. Just to be able to use Docker/rexray/DVDI until a new DCOS version comes out.
 18 | 
 19 | ## Install the components
 20 | 
 21 | To ssh on each Mesos agent and on the master(s), you may want to check this link:
 22 | 
 23 | https://docs.mesosphere.com/administration/sshcluster/ 
 24 | 
 25 | To ssh on agents:
 26 | 
 27 | `dcos node ssh --master-proxy --mesos-id=<mesos-id>`
 28 | 
 29 | To ssh on the master:
 30 | 
 31 | `dcos node ssh --master-proxy --leader`
 32 | 
 33 | 
 34 | #### Install rexray on Mesos agents
 35 | 
 36 | `curl -sSL https://dl.bintray.com/emccode/rexray/install | sh -s stable`
 37 | 
 38 | This walkthrough should work with rexray 0.3.3. 
 39 | 
 40 | Create and edit the config.yml:
 41 | 
 42 | `sudo vi /etc/rexray/config.yml`
 43 | 
 44 | ```yml
 45 | rexray:
 46 |   storageDrivers:
 47 |   - ec2
 48 |   logLevel: info
 49 |   volume:
 50 |     remove:
 51 |       disable: true
 52 |     create:
 53 |       disable: false
 54 |     mount:
 55 |       preempt: true
 56 |     unmount:
 57 |       ignoreUsedCount: true
 58 |     path:
 59 |       disableCache: true
 60 | linux:
 61 |   volume:
 62 |     rootPath: /data
 63 |     fileMode: 0700
 64 | ```
 65 | 
 66 | This particular setup will pre-emptively detach any existing attachments to other instances before attempting a mount.
 67 | 
 68 | Check the config options here: http://rexray.readthedocs.io/en/stable/user-guide/config/#advanced-configuration
 69 | 
 70 | Test it. You should see the available EBS volumes:
 71 | 
 72 | `rexray volume ls`
 73 | 
 74 | Start the rexray service:
 75 | 
 76 | `sudo rexray start`
 77 | 
 78 | #### Install DVDCLI on the Mesos agents
 79 | More details about DVDCLI may be found here: https://github.com/emccode/dvdcli
 80 | 
 81 | DVDCLI stands for Docker Volume Driver CLI and it is exactly that: a CLI implementation of the Docker Volume Driver.
 82 | 
 83 | `curl -sSL https://dl.bintray.com/emccode/dvdcli/install | sh -s stable`
 84 | 
 85 | Test it:
 86 | 
 87 | `sudo dvdcli mount --volumedriver=rexray --volumename=test1`
 88 | 
 89 | rexray will mount the volume here: `/var/lib/rexray/volumes/test1`
 90 | 
 91 | Check the EC2 dashboard - you should see the volume in use by one of the Mesos Agents. 
 92 | 
 93 | This will unmount the volume:
 94 | 
 95 | `sudo dvdcli unmount --volumedriver=rexray --volumename=test1`
 96 | 
 97 | #### Install mesos-module-dvdi on the Mesos agents
 98 | DVDI stands for Docker Volume Driver Interface. To understand it better, I recommend checking this material: https://github.com/emccode/mesos-module-dvdi/blob/master/DVDI%20Isolator%20Overview%20and%20Roadmap.md 
 99 | 
100 | DCOS 1.7 comes with Mesos 0.28.1 hence this command should get you the binaries:
101 | 
102 | `curl -L -O https://github.com/emccode/mesos-module-dvdi/releases/download/v0.4.2/libmesos_dvdi_isolator-0.28.1.so`
103 | 
104 | For me it didn't work. First there is a problem with calling the DVDCLI on CoreOS, for which I opened an issue (https://github.com/emccode/mesos-module-dvdi/issues/99). I also got a `invalid ELF header` when trying to use the above binaries. Then I decided to build it myself. My build is publicly available from an S3 location: 
105 | 
106 | `curl -O https://s3.amazonaws.com/dcos-mitel/libmesos_dvdi_isolator-0.28.1.so`
107 | 
108 | If you want to build it yourself, here is how I did it (right on one of the Mesos agents):
109 | 
110 | ```bash
111 | # clone the git repo:
112 | git clone https://github.com/emccode/mesos-module-dvdi.git
113 | cd mesos-module-dvdi
114 | 
115 | # edit and change /usr/bin to /opt/bin for the dvdcli executable
116 | # that's because dvdcli installs by default in /opt/bin on CoreOS
117 | vi isolator/isolator/docker_volume_driver_isolator.hpp
118 | 
119 | # build the docker image
120 | docker build -t mitelone/mesos-module-dvdi-dev:0.28.1 - < Dockerfile-mesos-module-dvdi-dev
121 | 
122 | # mount the isolator folder and run the build inside a Docker container
123 | cd isolator
124 | docker run -ti --volume=$(pwd):/isolator mitelone/mesos-module-dvdi-dev:0.28.1
125 | 
126 | # you will find the build here:
127 | cd /build/.libs
128 | ```
129 | 
130 | After downloading or building the .so file, copy it somewhere, eg: /home/core/dvdi/
131 | 
132 | Add the library to the list in mesos-slave-modules.json, following the json syntax:
133 | 
134 | `sudo vi /opt/mesosphere/etc/mesos-slave-modules.json` 
135 | 
136 | ```json
137 | {
138 |   "libraries": [
139 |     { ... },
140 |     {
141 |       "file": "/home/core/dvdi/libmesos_dvdi_isolator-0.28.1.so",
142 |       "modules": [
143 |         {
144 |           "name": "com_emccode_mesos_DockerVolumeDriverIsolator",
145 |           "parameters": [
146 |             {
147 |               "key": "isolator_command",
148 |               "value": "/emc/dvdi_isolator"
149 |             }
150 |           ]
151 |         }
152 |       ]
153 |     }
154 |   ]
155 | }
156 | ```
157 | 
158 | Add `com_emccode_mesos_DockerVolumeDriverIsolator` to the existing `MESOS_ISOLATION` list:
159 | 
160 | `sudo vi /opt/mesosphere/etc/mesos-slave-common`
161 | 
162 | It should look like this:
163 | 
164 | `MESOS_ISOLATION=cgroups/cpu,cgroups/mem,posix/disk,com_emccode_mesos_DockerVolumeDriverIsolator`
165 | 
166 | Now it should make some more sense to you why DVDI/rexray is so important - storage becomes a Mesos-managed resource along with CPU and Memory.
167 | 
168 | Stop the Mesos Agent:
169 | 
170 | `sudo systemctl stop dcos-mesos-slave.service`
171 | 
172 | After all these modifications, starting the service via systemctl fails for some reason I could not identify yet, hence I had to start it manually once, like this:
173 | 
174 | `sudo /opt/mesosphere/packages/mesos--0335ca0d3700ea88ad8b808f3b1b84d747ed07f0/bin/mesos-slave --modules=file:///opt/mesosphere/etc/mesos-slave-modules.json --isolation=cgroups/cpu,cgroups/mem,posix/disk,com_emccode_mesos_DockerVolumeDriverIsolator --master=zk://leader.mesos:2181/mesos --containerizers=docker,mesos --log_dir=/var/log/mesos`
175 | 
176 | It should start succesfully (check your `mesos-slave` executable location, might be different than the one above), then you can CTRL-C and then start it under systemd:
177 | 
178 | `sudo systemctl start dcos-mesos-slave.service`
179 | 
180 | Verify the service status:
181 | 
182 | `sudo systemctl status dcos-mesos-slave.service`
183 | 
184 | #### Setup Marathon for persistent volumes:
185 | 	
186 | `dcos node ssh --master-proxy --leader`
187 | 
188 | Add `external_volumes` to `--enable_features` list:
189 | 
190 | `sudo systemctl edit --full dcos-marathon.service`
191 | 
192 | It should look like this now:
193 | 
194 | `--enable_features "vips,task_killing,external_volumes"`
195 | 
196 | Restart Marathon:
197 | 
198 | `sudo systemctl reload-or-restart dcos-marathon.service`
199 | 
200 | #### Even more tweaking
201 | After running the above steps you should be able to run stateful apps using the Mesos containerizer.
202 | 
203 | ..but not yet the Docker containerizer due to the Docker version used on the CoreOS version used by DCOS 1.7.
204 | Quite frustrating :) So I modified the CloudFormation template to use a newer CoreOS version that comes with Docker > 1.8.
205 | Check it out and/or use it from
206 | 
207 | https://s3.amazonaws.com/dcos-mitel/dcos_single_master_23.04_coreos_899.17.0.json (docker 1.9.1)
208 | 
209 | https://s3.amazonaws.com/dcos-mitel/dcos_single_master_23.04_coreos_1010.3.0.json (docker 1.10.3)
210 | 
211 | You can provide this link when deploying DCOS on AWS (US-East-1 region only - I used an AMI available in that region).
212 | 
213 | Have fun!
214 | 


--------------------------------------------------------------------------------