├── .gitignore ├── .travis.yml ├── Gemfile ├── LICENSE ├── README.md ├── Rakefile ├── bin ├── console └── setup ├── capistrano-stretcher.gemspec ├── lib ├── capistrano-stretcher.rb └── capistrano │ ├── stretcher.rb │ ├── tasks │ └── stretcher.rake │ └── templates │ └── manifest.yml.erb └── test ├── capistrano └── stretcher_test.rb └── test_helper.rb /.gitignore: -------------------------------------------------------------------------------- 1 | /.bundle/ 2 | /.yardoc 3 | /Gemfile.lock 4 | /_yardoc/ 5 | /coverage/ 6 | /doc/ 7 | /pkg/ 8 | /spec/reports/ 9 | /tmp/ 10 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: ruby 2 | rvm: 3 | - 2.2.5 4 | - 2.3.1 5 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | source 'https://rubygems.org' 2 | 3 | gemspec 4 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2015- GMO Pepabo, Inc. 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining a copy 4 | of this software and associated documentation files (the "Software"), to deal 5 | in the Software without restriction, including without limitation the rights 6 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 7 | copies of the Software, and to permit persons to whom the Software is 8 | furnished to do so, subject to the following conditions: 9 | 10 | The above copyright notice and this permission notice shall be included in 11 | all copies or substantial portions of the Software. 12 | 13 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 14 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 15 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 16 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 17 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 18 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 19 | THE SOFTWARE. 20 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Capistrano::Stretcher 2 | 3 | capistrano task for stretcher. 4 | 5 | ## Installation 6 | 7 | Add this line to your application's Gemfile: 8 | 9 | ```ruby 10 | gem 'capistrano-stretcher' 11 | ``` 12 | 13 | And then execute: 14 | 15 | $ bundle 16 | 17 | Or install it yourself as: 18 | 19 | $ gem install capistrano-stretcher 20 | 21 | ## Requirements 22 | 23 | capistrano-stretcher requires target server for building to application assets. This server should be installed the following packages: 24 | 25 | * git 26 | * rsync 27 | * tar 28 | * gzip 29 | * awk 30 | * openssl 31 | * aws-cli 32 | * consul 33 | * pv 34 | 35 | target server builds assets, uploads assets to AWS S3 and invokes `consul event` automatically. So target server can access AWS s3 via aws-cli and join your deployment consul cluster. 36 | 37 | If you want to use non-s3 (e.g. private DC), upload assets to your server with rsync and download from http(s). 38 | 39 | ## Usage 40 | 41 | You need to add `require "capistrano/stretcher"` to Capfile and add `config/deploy.rb` following variables: 42 | 43 | ```ruby 44 | role :build, ['your-target-server.lan'], :no_release => true 45 | 46 | # If your build server is accessible to consul cluster, then set the same server 47 | # If different, set a server that can access consul cluster 48 | # just to kick consul event! 49 | role :consul, ['your-target-server.lan'], :no_release => true 50 | 51 | set :application, 'your-application' 52 | set :deploy_to, '/var/www' 53 | set :deploy_roles, 'www,batch' 54 | set :stretcher_hooks, 'config/stretcher.yml.erb' 55 | set :local_tarball_name, 'rails-applicaiton.tar.gz' 56 | set :stretcher_src, "s3://your-deployment-bucket/assets/rails-application-#{env.now}.tgz" 57 | set :manifest_path, "s3://your-deployment-bucket/manifests/" 58 | # Optional, if you want to use mv 59 | set :stretcher_sync_strategy, "mv" 60 | 61 | # Optinal, if you want to http(s) in stretcher_src, manifest_path 62 | set :rsync_ssh_option, "-p 22" 63 | set :rsync_ssh_user, "MY_USER" # if undefined, use current user on build server 64 | set :rsync_host, "xxx.xxx.xxx.xxx" 65 | set :rsync_stretcher_src_path, "/var/www/resource/assets/rails-application-#{env.now}.tgz" 66 | set :rsync_manifest_path, "/var/www/resource/manifests" 67 | ``` 68 | 69 | and write hooks for stretcher to `config/stretcher.yml.erb` 70 | 71 | ```yaml 72 | default: &default 73 | pre: 74 | - 75 | success: 76 | - 77 | failure: 78 | - cat >> /tmp/failure 79 | www: 80 | <<: *default 81 | post: 82 | - ln -nfs <%= fetch(:deploy_to) %>/shared/data <%= fetch(:deploy_to) %>/current/data 83 | - sudo systemctl reload unicorn 84 | batch: 85 | <<: *default 86 | post: 87 | - ln -nfs <%= fetch(:deploy_to) %>/shared/data <%= fetch(:deploy_to) %>/current/data 88 | ``` 89 | 90 | above hooks is extracted to manifest.yml for stretcher. If you have "www,batch" roles and stages named staging and production, capistrano-stretcher extract to following yaml from configuration. 91 | 92 | * `manifest_www_staging.yml` 93 | * `manifest_batch_staging.yml` 94 | 95 | and invoke 96 | 97 | * `consul event -name deploy_www_staging s3://.../manifest_www.yml` 98 | * `consul event -name deploy_batch_staging s3://.../manifest_batch.yml` 99 | 100 | with `cap staging stretcher:deploy` command on target server. When it's invoked with `cap production stretcher:deploy`, capistrano-stretcher replace suffix `staging` to `production`. 101 | 102 | ## Related Projects 103 | 104 | * [capistrano-stretcher-rails](https://github.com/pepabo/capistrano-stretcher-rails) 105 | * [capistrano-stretcher-npm](https://github.com/pepabo/capistrano-stretcher-npm) 106 | 107 | ## Development 108 | 109 | After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment. 110 | 111 | To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org). 112 | 113 | ## Contributing 114 | 115 | Bug reports and pull requests are welcome on GitHub at https://github.com/pepabo/capistrano-stretcher. 116 | 117 | ## LICENSE 118 | 119 | The MIT License (MIT) 120 | 121 | Copyright (c) 2015- GMO Pepabo, Inc. 122 | -------------------------------------------------------------------------------- /Rakefile: -------------------------------------------------------------------------------- 1 | require "bundler/gem_tasks" 2 | require "rake/testtask" 3 | 4 | Rake::TestTask.new(:test) do |t| 5 | t.libs << "test" 6 | t.libs << "lib" 7 | t.test_files = FileList['test/**/*_test.rb'] 8 | end 9 | 10 | task :default => :test 11 | -------------------------------------------------------------------------------- /bin/console: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | require "bundler/setup" 4 | require "capistrano/stretcher" 5 | 6 | # You can add fixtures and/or initialization code here to make experimenting 7 | # with your gem easier. You can also use a different console, if you like. 8 | 9 | # (If you use this, don't forget to add pry to your Gemfile!) 10 | # require "pry" 11 | # Pry.start 12 | 13 | require "irb" 14 | IRB.start 15 | -------------------------------------------------------------------------------- /bin/setup: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -euo pipefail 3 | IFS=$'\n\t' 4 | 5 | bundle install 6 | 7 | # Do any other automated setup that you need to do here 8 | -------------------------------------------------------------------------------- /capistrano-stretcher.gemspec: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | lib = File.expand_path('../lib', __FILE__) 3 | $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib) 4 | 5 | Gem::Specification.new do |spec| 6 | spec.name = "capistrano-stretcher" 7 | spec.version = "0.5.4" 8 | spec.authors = ["SHIBATA Hiroshi", "Uchio Kondo"] 9 | spec.email = ["hsbt@ruby-lang.org", "udzura@udzura.jp"] 10 | 11 | spec.summary = %q{capistrano task for stretcher.} 12 | spec.description = %q{capistrano task for stretcher.} 13 | spec.homepage = "https://github.com/pepabo/capistrano-stretcher" 14 | 15 | spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) } 16 | spec.bindir = "exe" 17 | spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) } 18 | spec.require_paths = ["lib"] 19 | 20 | spec.add_dependency 'capistrano', '>= 3' 21 | 22 | spec.add_development_dependency "bundler" 23 | spec.add_development_dependency "rake" 24 | spec.add_development_dependency "minitest" 25 | end 26 | -------------------------------------------------------------------------------- /lib/capistrano-stretcher.rb: -------------------------------------------------------------------------------- 1 | # this file is dummy file for Bundler.setup 2 | -------------------------------------------------------------------------------- /lib/capistrano/stretcher.rb: -------------------------------------------------------------------------------- 1 | load File.expand_path("../tasks/stretcher.rake", __FILE__) 2 | -------------------------------------------------------------------------------- /lib/capistrano/tasks/stretcher.rake: -------------------------------------------------------------------------------- 1 | # -*- coding: utf-8; mode: ruby -*- 2 | require 'erb' 3 | require 'yaml' 4 | require 'tempfile' 5 | 6 | namespace :load do 7 | task :defaults do 8 | set :gzip_archiver, "gzip" 9 | set :gzip_archiver_option, "" 10 | end 11 | end 12 | 13 | namespace :stretcher do 14 | set :exclude_dirs, ["tmp"] 15 | 16 | def local_working_path_base 17 | @_local_working_path_base ||= fetch(:local_working_path_base, "/var/tmp/#{fetch :application}") 18 | end 19 | 20 | def local_repo_path 21 | "#{local_working_path_base}/repo" 22 | end 23 | 24 | def local_checkout_path 25 | "#{local_working_path_base}/checkout" 26 | end 27 | 28 | def local_build_path 29 | "#{local_working_path_base}/build" 30 | end 31 | 32 | def local_tarball_path 33 | "#{local_working_path_base}/tarballs" 34 | end 35 | 36 | def application_builder_roles 37 | roles(fetch(:application_builder_roles, [:build])) 38 | end 39 | 40 | def consul_roles 41 | roles(fetch(:consul_roles, [:consul])) 42 | end 43 | 44 | # upload to s3 45 | def upload_s3(local_src_path, remote_dst_path) 46 | execute :aws, :s3, :cp, local_src_path, remote_dst_path 47 | end 48 | 49 | # upload to resource server with rsync 50 | def upload_resource(local_src_path, remote_dst_path) 51 | rsync_ssh_command = "ssh" 52 | rsync_ssh_command << " " + fetch(:rsync_ssh_option) if fetch(:rsync_ssh_option) 53 | rsync_ssh_user = fetch(:rsync_ssh_user) { capture(:whoami).strip } 54 | 55 | execute :rsync, "-ave", %Q("#{rsync_ssh_command}"), 56 | local_src_path, 57 | "#{rsync_ssh_user}@#{fetch(:rsync_host)}:#{remote_dst_path}" 58 | end 59 | 60 | task :mark_deploying do 61 | set :deploying, true 62 | end 63 | 64 | desc "Create a tarball that is set up for deploy" 65 | task :archive_project => 66 | [:ensure_directories, :checkout_local, 67 | :create_tarball, :upload_tarball, 68 | :create_and_upload_manifest, :cleanup_dirs] 69 | 70 | task :ensure_directories do 71 | on application_builder_roles do 72 | execute :mkdir, '-p', local_repo_path, local_checkout_path, local_build_path, local_tarball_path 73 | end 74 | end 75 | 76 | task :checkout_local do 77 | on application_builder_roles do 78 | if test("[ -f #{local_repo_path}/HEAD ]") 79 | within local_repo_path do 80 | execute :git, :remote, :update 81 | end 82 | else 83 | execute :git, :clone, '--mirror', repo_url, local_repo_path 84 | end 85 | 86 | within local_repo_path do 87 | execute :mkdir, '-p', "#{local_checkout_path}/#{env.now}" 88 | execute :git, :archive, fetch(:branch), "| tar -x -C", "#{local_checkout_path}/#{env.now}" 89 | set :current_revision, capture(:git, 'rev-parse', fetch(:branch)).chomp 90 | 91 | execute :echo, fetch(:current_revision), "> #{local_checkout_path}/#{env.now}/REVISION" 92 | 93 | execute :rsync, "-av", "--delete", 94 | *fetch(:exclude_dirs, ["tmp"]).map{|d| ['--exclude', d].join(' ')}, 95 | "#{local_checkout_path}/#{env.now}/", "#{local_build_path}/", 96 | "| pv -l -s $( find #{local_checkout_path}/#{env.now}/ -type f | wc -l ) >/dev/null" 97 | end 98 | end 99 | end 100 | 101 | task :create_tarball do 102 | on application_builder_roles do 103 | within local_build_path do 104 | archiver = fetch(:gzip_archiver, "gzip") 105 | archiver_option = fetch(:gzip_archiver_option, "") 106 | execute :mkdir, '-p', "#{local_tarball_path}/#{env.now}" 107 | execute :tar, '-cf', '-', 108 | "--exclude tmp", "--exclude spec", "./", 109 | "| pv -s $( du -sb ./ | awk '{print $1}' )", 110 | "| #{archiver} #{archiver_option} > #{local_tarball_path}/#{env.now}/#{fetch(:local_tarball_name)}" 111 | end 112 | within local_tarball_path do 113 | execute :rm, '-f', 'current' 114 | execute :ln, '-sf', env.now, 'current' 115 | end 116 | end 117 | end 118 | 119 | task :upload_tarball do 120 | on application_builder_roles do 121 | as fetch(:stretcher_user) || 'root' do 122 | local_tarball_file = "#{local_tarball_path}/current/#{fetch(:local_tarball_name)}" 123 | 124 | if fetch(:stretcher_src).start_with?("s3://") 125 | # upload to s3 126 | upload_s3(local_tarball_file, fetch(:stretcher_src)) 127 | else 128 | # upload to resource server with rsync 129 | upload_resource(local_tarball_file, fetch(:rsync_stretcher_src_path)) 130 | end 131 | end 132 | end 133 | end 134 | 135 | task :create_and_upload_manifest do 136 | on application_builder_roles do 137 | as fetch(:stretcher_user) || 'root' do 138 | failure_message = "Deploy failed at *$(hostname)* :fire:" 139 | checksum = capture("openssl sha256 #{local_tarball_path}/current/#{fetch(:local_tarball_name)} | gawk -F' ' '{print $2}'").chomp 140 | src = fetch(:stretcher_src) 141 | template = File.read(File.expand_path('../../templates/manifest.yml.erb', __FILE__)) 142 | yaml = YAML.load(ERB.new(capture(:cat, "#{local_build_path}/#{fetch(:stretcher_hooks)}")).result(binding)) 143 | release_timestamp = fetch(:release_timestamp) || env.now 144 | fetch(:deploy_roles).split(',').each do |role| 145 | hooks = yaml[role] 146 | yml = ERB.new(template).result(binding) 147 | tempfile_path = Tempfile.open("manifest_#{role}") do |t| 148 | t.write yml 149 | t.path 150 | end 151 | 152 | local_manifest_file = "#{local_tarball_path}/current/manifest_#{role}_#{fetch(:stage)}.yml" 153 | 154 | upload! tempfile_path, local_manifest_file 155 | 156 | if fetch(:manifest_path).start_with?("s3://") 157 | # upload to s3 158 | upload_s3(local_manifest_file, "#{fetch(:manifest_path)}/manifest_#{role}_#{fetch(:stage)}.yml") 159 | else 160 | # upload to resource server with rsync 161 | execute :chmod, "644", local_manifest_file 162 | upload_resource(local_manifest_file, "#{fetch(:rsync_manifest_path)}/manifest_#{role}_#{fetch(:stage)}.yml") 163 | end 164 | end 165 | end 166 | end 167 | end 168 | 169 | # refs https://github.com/capistrano/capistrano/blob/master/lib/capistrano/tasks/deploy.rake#L138 170 | task :cleanup_dirs do 171 | on application_builder_roles do 172 | releases = capture(:ls, '-tr', "#{local_tarball_path}", "| grep -v current").split 173 | checkouts = capture(:ls, '-tr', "#{local_checkout_path}").split 174 | 175 | if releases.count >= fetch(:keep_releases) 176 | info t(:keeping_releases, host: host.to_s, keep_releases: fetch(:keep_releases), releases: releases.count) 177 | directories = ((releases | checkouts) - releases.last(fetch(:keep_releases))) 178 | unless directories.empty? 179 | directories_str = directories.map do |release| 180 | "#{local_tarball_path}/#{release} #{local_checkout_path}/#{release}" 181 | end.join(" ") 182 | execute :rm, '-rf', directories_str 183 | else 184 | info t(:no_old_releases, host: host.to_s, keep_releases: fetch(:keep_releases)) 185 | end 186 | end 187 | end 188 | end 189 | 190 | desc "Kick the stretcher's deploy event via Consul" 191 | task :kick_stretcher do 192 | fetch(:deploy_roles).split(',').each do |target_role| 193 | on consul_roles do 194 | opts = ["-name deploy_#{target_role}_#{fetch(:stage)}"] 195 | opts << "-node #{ENV['TARGET_HOSTS']}" if ENV['TARGET_HOSTS'] 196 | opts << "#{fetch(:manifest_path)}/manifest_#{target_role}_#{fetch(:stage)}.yml" 197 | execute :consul, :event, *opts 198 | end 199 | end 200 | end 201 | 202 | desc 'Deploy via Stretcher' 203 | task :deploy => ["stretcher:mark_deploying", "stretcher:archive_project", "stretcher:kick_stretcher"] 204 | end 205 | -------------------------------------------------------------------------------- /lib/capistrano/templates/manifest.yml.erb: -------------------------------------------------------------------------------- 1 | src: <%= src %> 2 | checksum: <%= checksum %> 3 | dest: <%= fetch(:deploy_to) %>/releases/<%= env.now %> 4 | commands: 5 | pre: 6 | <% hooks["pre"].each do |c| %> 7 | - <%= c %> 8 | <% end %> 9 | post: 10 | - ln -nfs <%= fetch(:deploy_to) %>/releases/<%= release_timestamp %> <%= fetch(:deploy_to) %>/current 11 | - rm -rf <%= fetch(:deploy_to) %>/current/log 12 | - ln -nfs <%= fetch(:deploy_to) %>/shared/log <%= fetch(:deploy_to) %>/current/log 13 | - mkdir -p <%= fetch(:deploy_to) %>/current/tmp 14 | - ln -nfs <%= fetch(:deploy_to) %>/shared/pids <%= fetch(:deploy_to) %>/current/tmp/pids 15 | <% hooks["post"].each do |c| %> 16 | - <%= c %> 17 | <% end %> 18 | success: 19 | <% hooks["success"].each do |c| %> 20 | - <%= c %> 21 | <% end %> 22 | failure: 23 | <% hooks["failure"].each do |c| %> 24 | - <%= c %> 25 | <% end %> 26 | excludes: 27 | - "*.pid" 28 | - "*.socket" 29 | <% if strategy = fetch(:stretcher_sync_strategy, false) %> 30 | sync_strategy: <%= strategy %> 31 | <% end %> 32 | -------------------------------------------------------------------------------- /test/capistrano/stretcher_test.rb: -------------------------------------------------------------------------------- 1 | require 'test_helper' 2 | 3 | class Capistrano::StretcherTest < Minitest::Test 4 | def test_that_it_has_a_version_number 5 | refute_nil ::Capistrano::Stretcher::VERSION 6 | end 7 | end 8 | -------------------------------------------------------------------------------- /test/test_helper.rb: -------------------------------------------------------------------------------- 1 | $LOAD_PATH.unshift File.expand_path('../../lib', __FILE__) 2 | require 'capistrano/stretcher' 3 | 4 | require 'minitest/autorun' 5 | --------------------------------------------------------------------------------