├── LICENSE ├── README.md └── hodor /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2014 Sergey Khaladzinski 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining 4 | a copy of this software and associated documentation files (the 5 | "Software"), to deal in the Software without restriction, including 6 | without limitation the rights to use, copy, modify, merge, publish, 7 | distribute, sublicense, and/or sell copies of the Software, and to 8 | permit persons to whom the Software is furnished to do so, subject to 9 | the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be 12 | included in all copies or substantial portions of the Software. 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 14 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 15 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 16 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 17 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 18 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 19 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 20 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | hodor 2 | ===== 3 | 4 | Small utility to streamline dev process with Docker on OS X (Boot2Docker/Docker Machine) and Linux 5 | 6 | Blog posts with more info: 7 | 8 | * [Docker VM shortcomings and how hodor can help](http://distinctplace.com/infrastructure/2014/09/24/docker-vm-shortcomings-and-how-hodor-can-help/) 9 | * [Docker + Hodor for simple and reliable dev setup](http://distinctplace.com/infrastructure/2015/06/18/docker--hodor-to-simplify-app-setup/) 10 | 11 | features 12 | ===== 13 | * same workflow for Linux and OS X 14 | * volume sharing support for current project dir (with unison and fswatch to keep host and VM in sync) 15 | * exposed docker ports are automatically mapped to your host, it just works (VirtualBox only) 16 | * ssh key sharing support between host and containers (ex: pull from github inside of the container without typing ssh key passwords) 17 | * containers orchestration with deps management, so we can start containers in particular order 18 | * separate containers list per project (with `.hodorfile`) 19 | 20 | Hodor Hodor Hodor 21 | 22 | unsupported 23 | ===== 24 | * Windows OS 25 | * `docker-compose` toolchain 26 | 27 | PRs, suggestions, and contributions are welcomed. 28 | 29 | requirements 30 | ===== 31 | * ruby >= 1.9.3 32 | * docker / boot2docker >= v1.1.2 / docker-machine 33 | * unison = 2.40.x (for two-way sync on OS X, not required for Linux) 34 | - `brew install homebrew/versions/unison240` 35 | * fswatch >= 1.4.3.1 (for automatic volumes sync on project file change on OS X, not required for Linux) 36 | - `brew install fswatch`. 37 | - Make sure you `fswatch` version has -o option (--one-per-batch option). If it's missing, you will need the newer version. 38 | 39 | how to use 40 | ===== 41 | 42 | First you need to clone this repo, `chmod +x hodor` and copy hodor script to `/usr/local/bin` 43 | 44 | In your project create file called `.hodorfile` (now processed through ERB) with following structure ( I'll explain options later, but everything should be pretty obvious from this example file: 45 | 46 | ```yaml 47 | # (Optional. Default: boot2docker) 48 | host-manager: docker-machine 49 | # (Optional. VirtualBox vm name. 50 | # Defaults: boot2docker-vm (boot2docker), default (docker-machine)) 51 | host: default 52 | 53 | containers: 54 | redis: 55 | background: true 56 | image: gansbrest/redis 57 | ports: -P 58 | onetime: false 59 | volumes: 60 | __PROJECT__/conf: /data/conf 61 | jetpack: 62 | background: false 63 | image: gansbrest/fc_jetpack 64 | ports: 65 | - 49000:8880 66 | # (**note:** if you use `net: host`, but still want to start conainers in 67 | # particular order, you should use **depends** keyword with the list of continers instead) 68 | links: 69 | redis: redis 70 | environment: 71 | AWS_ACCESS_KEY_ID: <%= ENV['AWS_ACCESS_KEY_ID'] %> 72 | AWS_SECRET_ACCESS_KEY: <%= ENV['AWS_SECRET_ACCESS_KEY'] %> 73 | workdir: "/data" 74 | onetime: true 75 | 76 | tasks: 77 | test: 78 | sync_project_to: /data/slot-fc1 79 | cmd: "cd /data/slot-fc1 && bash" 80 | container: "jetpack" 81 | npm: 82 | sync_project_to: /data/slot-fc1 83 | cmd: "cd /data/slot-fc1 && npm" 84 | container: "jetpack" 85 | grunt: 86 | sync_project_to: /data/slot-fc1 87 | cmd: "cd /data/slot-fc1 && grunt" 88 | container: "jetpack" 89 | run: 90 | sync_project_to: /data/slot-fc1 91 | cmd: "cd /data/slot-fc1 && ./simple_server" 92 | container: "jetpack" 93 | default: run #(if no task is given, the run task is executed) 94 | ``` 95 | 96 | Run `hodor` with one of the tasks you specified while you are in project dir. 97 | 98 | For example `hodor grunt build` - it will execute grunt build inside of the container and auto sync your host and VM after it's done! 99 | 100 | or 101 | 102 | `hodor run` - will run `simple_server` (which in the above example is a wrapper for nodemon and some other stuff) inside of the container. You will see nodemon output. Any files modified on host will automatically sync to container because of fswatch! 103 | 104 | This project is very raw. Report any issues, questions or suggestions. 105 | 106 | Hope it will save you time and allow to fully enjoy docker! 107 | 108 | 109 | why not fuse or VirtualBox Guest Additions? 110 | ===== 111 | 112 | Fuse is just very slow for big projects and inconvenient to setup. VirtualBox Guest Additions is, well, slow. I was not able to use it for our project with 17k small files.. Some people find it useful for small projects though and it's backed into new Boot2Docker, or that I've heard. 113 | 114 | why not Vagrant, it does it all already? 115 | ===== 116 | 117 | Well, let me put it this way: I tried Vagrant and it wasn't good for I needed. I just felt that everything was too abstracted away. Plus file sharing is slow and not even sure for ssh forwarding. Also not a fan of installing sshd on my containers. 118 | 119 | Not that I advice against using it, I just felt that I need something different and lightweight. 120 | 121 | Read more here: 122 | 123 | Docker and Vagrant: [http://stackoverflow.com/questions/16647069/should-i-use-vagrant-or-docker-io-for-creating-an-isolated-environment](http://stackoverflow.com/questions/16647069/should-i-use-vagrant-or-docker-io-for-creating-an-isolated-environment) 124 | 125 | SSHD in your containers: [http://jpetazzo.github.io/2014/06/23/docker-ssh-considered-evil/](http://jpetazzo.github.io/2014/06/23/docker-ssh-considered-evil/) 126 | 127 | motivation 128 | ===== 129 | 130 | Once I started working with Docker, my initial reaction was, "Hey, that's great, everything just works and I can even share my volumes and ports with host machine easily..." 131 | 132 | I forgot to mention that I use Ubuntu as my desktop OS... So I spent about a week to create multiple containers for our FastCompany stack (node, redis etc) and decided to convince our devs to integrate Docker as part of the new development process. Everything was supposed to be so much better. Boy, was I surprised. 133 | 134 | Most of our dev team members use Mac OS X. I always knew that. I think I was just misled by Docker docs, where they would say "It works everywhere, even on Windows! You just need this tiny VM to make it all happen." Well, let me make it short for you - Docker is PAIN in the b..t when you want to share volumes or ports, basically use it indirectly through any particular VM. 135 | 136 | I decided to create this tool called Hodor with one goal in mind - to create reliable helper/proxy, which allows to use of Docker same way on Linux or OS X, where volume sharing and ports are just working. 137 | -------------------------------------------------------------------------------- /hodor: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | require 'rbconfig' 4 | require 'yaml' 5 | require 'erb' 6 | 7 | opts = ARGV 8 | op = opts[0] 9 | 10 | # Remove op from args list 11 | opts.shift 12 | 13 | class Hodor 14 | 15 | def initialize(opts, op) 16 | @os = get_os() 17 | @op = op 18 | @opts = opts 19 | @dir_path = Dir.pwd 20 | @dir_name = File.basename(Dir.pwd) 21 | @uid = `id -u $USER`.to_i 22 | @containers = [] 23 | @tasks = [] 24 | @fileshare_name = "#{@dir_name}_fileshare" 25 | @hostmanager = "boot2docker" 26 | @host = "boot2docker-vm" 27 | @ssh_file = "~/.ssh/id_boot2docker" 28 | 29 | # For non-linux envs 30 | @exposed_ports = [] 31 | 32 | process_project_config 33 | 34 | @docker_prefix = "sudo " 35 | if @os != :linux 36 | @docker_prefix = "" 37 | non_linux_init 38 | end 39 | 40 | end 41 | 42 | def process_project_config 43 | if !File.exist? File.join(@dir_path, '.hodorfile') 44 | abort(".hodorfile was not found in your project, please create it and try again") 45 | end 46 | 47 | conf = YAML.load(ERB.new(File.read('.hodorfile')).result) 48 | @containers = conf['containers'] 49 | @tasks = conf['tasks'] 50 | if !@op && @tasks['default'] 51 | @op = @tasks['default'] 52 | end 53 | 54 | case conf['host-manager'] 55 | when "docker-machine" 56 | @hostmanager = "docker-machine" 57 | @host = conf['host'] || ENV['DOCKER_MACHINE_NAME'] || 'default' 58 | @ssh_file = "~/.docker/machine/machines/#{@host}/id_rsa" 59 | end 60 | end 61 | 62 | def non_linux_init 63 | # Check if hostmanager is started 64 | # if not display error 65 | if @hostmanager == "boot2docker" 66 | status = `boot2docker status 2>&1`.strip! 67 | elsif @hostmanager == "docker-machine" 68 | status = `docker-machine status #{@host} 2>&1`.downcase.strip! 69 | end 70 | if status != "running" 71 | if @hostmanager == "boot2docker" 72 | abort("Seems like boot2docker was not started, try to run 'boot2docker init' and then 'boot2docker up'") 73 | elsif @hostmanager == "docker-machine" 74 | abort("Seems like docker-machine was not started, try to run 'docker-machine start @host'") 75 | end 76 | end 77 | 78 | if ENV['DOCKER_HOST'].nil? 79 | if @hostmanager == "boot2docker" 80 | abort("You need to run 'export DOCKER_HOST=tcp://$(boot2docker ip 2>/dev/null):2375' so we know where your docker host is"); 81 | elsif @hostmanager == "docker-machine" 82 | abort("You need to run 'export DOCKER_HOST=tcp://$(docker-machine ip #{@host} 2>/dev/null):2375' so we know where your docker host is"); 83 | end 84 | end 85 | 86 | if @hostmanager == "boot2docker" 87 | @docker_host_ip = `boot2docker ip 2>/dev/null`.chomp 88 | elsif @hostmanager == "docker-machine" 89 | @docker_host_ip = `docker-machine ip #{@host} 2>/dev/null`.chomp 90 | end 91 | 92 | # Prepare container 93 | fileshare_running = `#{@docker_prefix} docker inspect -f "{{ .State.Running }}" #{@fileshare_name} 2>&1`.strip! 94 | 95 | status = $?.exitstatus 96 | if status != 0 || fileshare_running == '' || fileshare_running == "false" 97 | 98 | # Kill leftover containers 99 | if fileshare_running == "false" 100 | `#{@docker_prefix} docker rm -f #{@fileshare_name} 2>&1` 101 | end 102 | 103 | # Copy ssh info to allow ssh forwarding 104 | `find ~/.ssh -maxdepth 1 -type f ! -name 'authorized_keys' ! -name 'config' -exec scp -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i #{@ssh_file} {} docker@#{@docker_host_ip}:~/.ssh \\\; 2>&1 ` 105 | 106 | # Store ssh key in ssh-agent on VM 107 | # @TODO - get key name from user input 108 | # and capture key pass in more secured way 109 | if File.exists? (File.join(File.expand_path('~'), ".ssh/id_rsa")) 110 | system("ssh -t -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i #{@ssh_file} docker@#{@docker_host_ip} 'rm -f /tmp/hodor_docker_socket && eval `ssh-agent -a /tmp/hodor_docker_socket` && ssh-add'") 111 | if $?.exitstatus != 0 112 | abort("Your ssh key was not loaded correctly, try one more time.") 113 | end 114 | end 115 | 116 | # Clean screen from previous stuff 117 | system "clear" 118 | 119 | end 120 | end 121 | 122 | def get_os 123 | @os ||= ( 124 | host_os = RbConfig::CONFIG['host_os'] 125 | case host_os 126 | when /mswin|msys|mingw|cygwin|bccwin|wince|emc/ 127 | :windows 128 | when /darwin|mac os/ 129 | :macosx 130 | when /linux/ 131 | :linux 132 | when /solaris|bsd/ 133 | :unix 134 | else 135 | raise Error::WebDriverError, "unknown os: #{host_os.inspect}" 136 | end 137 | ) 138 | end 139 | 140 | def fileshare_port 141 | `#{@docker_prefix} docker port #{@fileshare_name} 45678 2>&1`.strip!.split(':')[1] 142 | end 143 | 144 | def run 145 | 146 | cmd = get_task_prop("cmd") 147 | sync_to = get_task_prop("sync_project_to") 148 | container_name = get_task_prop("container") 149 | 150 | pre_launch_ops(sync_to) 151 | launch_container(cmd, sync_to, container_name) 152 | post_launch_ops(sync_to) 153 | 154 | end 155 | 156 | def get_task_prop(prop) 157 | begin 158 | return @tasks[@op].fetch(prop) 159 | rescue NoMethodError, KeyError 160 | abort("Either your task or #{prop} property not found in the config") 161 | end 162 | end 163 | 164 | def pre_launch_ops(sync_to) 165 | 166 | if @os == :macosx 167 | fileshare_running = `#{@docker_prefix} docker inspect -f "{{ .State.Running }}" #{@fileshare_name} 2>&1`.strip! 168 | 169 | status = $?.exitstatus 170 | if status != 0 || fileshare_running == '' || fileshare_running == "false" 171 | # Special tricks for sharing volumes with docker host 172 | # Mac only 173 | share_volumes_opts = [] 174 | 175 | ## Volumes support 176 | @containers.each do |container_data| 177 | name = container_data.first 178 | unless @containers[name]['volumes'].nil? 179 | @containers[name]['volumes'].each do |src_vol, dst_vol| 180 | if src_vol.include? "__PROJECT__" 181 | src_vol = src_vol.sub("__PROJECT__", "/data/#{@dir_path}") 182 | share_volumes_opts << "-v #{src_vol}:#{dst_vol}" 183 | end 184 | end 185 | end 186 | end 187 | 188 | if sync_to 189 | share_volumes_opts << "-v /data/#{@dir_path}:#{sync_to}" 190 | end 191 | 192 | # Initialize unison container 193 | system("#{@docker_prefix} docker run #{share_volumes_opts.join(" ")} -p 45678 -d --name #{@fileshare_name} gansbrest/fs-base") 194 | end 195 | 196 | unison_sync(sync_to) 197 | 198 | # Run fswatch and unison on the background 199 | pipe_cmd_in, pipe_cmd_out = IO.pipe 200 | @fswatch_pid = Process.spawn("fswatch -o #{@dir_path}", :out => pipe_cmd_out) 201 | Process.spawn("xargs -n1 -I{} unison -silent -ignore 'Name {.git,*.swp}' -batch -confirmbigdel=false -ui text -sshargs '-i #{@ssh_file}' #{@dir_path} socket://#{@docker_host_ip}:#{fileshare_port}/#{sync_to}", :in => pipe_cmd_in, :out => "/dev/null") 202 | 203 | Process.detach @fswatch_pid 204 | 205 | end 206 | end 207 | 208 | def unison_sync(sync_folder) 209 | 210 | # Copying files to the VM 211 | puts "Synchronize #{@dir_path} with VM.." 212 | `unison -ignore "Name {.git,*.swp}" -batch -confirmbigdel=false -ui text -sshargs '-i #{@ssh_file}' #{@dir_path} socket://#{@docker_host_ip}:#{fileshare_port}/#{sync_folder} 2>&1` 213 | `ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -i #{@ssh_file} docker@#{@docker_host_ip} "sudo chown -R docker /data/#{@dir_path}" 2>&1` 214 | 215 | end 216 | 217 | def launch_container(cmd, sync_to, container_name) 218 | begin 219 | container_data = @containers.fetch(container_name) 220 | 221 | docker_opts = [] 222 | 223 | docker_opts << "run" 224 | 225 | # General stuff to make sure ssh forwarding works inside of the container 226 | # so you don't need to retype your private key password / load it every time 227 | if @os != :linux 228 | docker_opts << "--volumes-from #{@fileshare_name}" 229 | docker_opts << "-v /tmp/hodor_docker_socket:/tmp/hodor_docker_socket" 230 | docker_opts << "-e SSH_AUTH_SOCK=/tmp/hodor_docker_socket" 231 | else 232 | docker_opts << "-e SSH_AUTH_SOCK=$(echo $SSH_AUTH_SOCK)" 233 | docker_opts << "-v $(dirname $SSH_AUTH_SOCK):$(dirname $SSH_AUTH_SOCK)" 234 | end 235 | 236 | # Launch container deps first if any 237 | if container_data.include? "links" 238 | container_data["links"].each do |linked_container, link_alias| 239 | launch_container(container_data["cmd"], nil, linked_container) 240 | docker_opts << "--link #{linked_container}:#{link_alias}" 241 | end 242 | end 243 | 244 | # Sometimes we need deps to be started in particular order without links 245 | if container_data.include? "depends" 246 | container_data["depends"].each do |linked_container| 247 | launch_container(container_data["cmd"], nil, linked_container) 248 | end 249 | end 250 | 251 | docker_opts << "--name #{container_name}" 252 | 253 | if container_data["ports"] 254 | if container_data["ports"].kind_of?(Array) 255 | container_data["ports"].each do |ports_str| 256 | docker_opts << "-p #{ports_str}" 257 | port_parts = ports_str.split(":") 258 | 259 | puts "#### #{container_name.upcase} IS EXPOSED UNDER 127.0.0.1:#{port_parts[0]} ADDRESS ####" 260 | if @os == :macosx 261 | # Store ports, so we can later get rid of them 262 | @exposed_ports << port_parts[0] 263 | 264 | # Expose ports 265 | `VBoxManage controlvm #{@host} natpf1 "tcp-port#{port_parts[0]},tcp,,#{port_parts[0]},,#{port_parts[0]}"` 266 | `VBoxManage controlvm #{@host} natpf1 "udp-port#{port_parts[0]},udp,,#{port_parts[0]},,#{port_parts[0]}"` 267 | end 268 | end 269 | else 270 | docker_opts << "#{container_data["ports"]}" 271 | end 272 | end 273 | 274 | if container_data['workdir'] 275 | docker_opts << "-w #{container_data['workdir']}" 276 | end 277 | 278 | if container_data['background'] 279 | docker_opts << "-d" 280 | else 281 | docker_opts << "-it" 282 | end 283 | 284 | if container_data['net'] 285 | docker_opts << "--net #{container_data['net']}" 286 | end 287 | 288 | ## All volumes are only working withing current project dir 289 | # because .hodorfile is meant to be commited to the repo 290 | # and evey dev can have diff folders structure 291 | if container_data.include? "volumes" 292 | container_data["volumes"].each do |src_vol, dst_vol| 293 | if src_vol.include? "__PROJECT__" 294 | src_vol = src_vol.sub("__PROJECT__", @dir_path) 295 | docker_opts << "-v #{src_vol}:#{dst_vol}" 296 | end 297 | end 298 | end 299 | 300 | if sync_to 301 | docker_opts << "-v #{@dir_path}:#{sync_to}" 302 | end 303 | 304 | if container_data.include? "environment" 305 | container_data["environment"].each do |var_name, var_value| 306 | docker_opts << "-e #{var_name}='#{var_value}'" 307 | end 308 | end 309 | 310 | puts "Launching #{container_name} container" 311 | # Check if we should keep container around 312 | if container_data['onetime'] 313 | # Make sure we remove old containers if those are stuck 314 | `#{@docker_prefix} docker rm -f #{container_name} 2>&1` 315 | docker_opts << "--rm" 316 | else 317 | running = `#{@docker_prefix} docker inspect -f "{{ .State.Running }}" #{container_name} 2>&1`.strip! 318 | 319 | status = $?.exitstatus 320 | # Container exist and stopped - restart it 321 | if running == "false" && status == 0 322 | docker_opts.clear 323 | docker_opts << "start" 324 | docker_opts << container_name 325 | return system("#{@docker_prefix} docker #{docker_opts.join(" ")}") 326 | elsif running == "true" 327 | # Do nothing, already running 328 | docker_opts.clear 329 | end 330 | end 331 | 332 | if !container_data['background'] && cmd 333 | cmd = "sh -c '[ -d ~/.ssh ] || mkdir ~/.ssh && echo \"Host github.com\nStrictHostKeyChecking no\n\" >> ~/.ssh/config && #{cmd} #{@opts.join(" ")} '" 334 | end 335 | 336 | # If any options present at this point run docker container 337 | if docker_opts.any? 338 | docker_exec_str = "#{@docker_prefix} docker #{docker_opts.join(" ")} #{container_data.fetch("image")} #{cmd}" 339 | system(docker_exec_str) 340 | end 341 | 342 | rescue NoMethodError, KeyError => e 343 | abort("Problem with #{container_name} container config definition. #{e.message}") 344 | end 345 | end 346 | 347 | def post_launch_ops(sync_to) 348 | if @os == :macosx 349 | unison_sync(sync_to) 350 | 351 | # Cleanup exposed ports 352 | if @exposed_ports.any? 353 | @exposed_ports.each do |port_num| 354 | `VBoxManage controlvm #{@host} natpf1 delete tcp-port#{port_num}` 355 | `VBoxManage controlvm #{@host} natpf1 delete udp-port#{port_num}` 356 | end 357 | end 358 | 359 | if @fswatch_pid > 0 360 | Process.kill("SIGTERM", @fswatch_pid) 361 | end 362 | end 363 | end 364 | 365 | end 366 | 367 | hodor = Hodor.new(opts, op) 368 | hodor.run 369 | --------------------------------------------------------------------------------