├── .gitignore ├── LICENSE ├── README.rdoc ├── cached_externals.gemspec ├── lib ├── cached_externals.rb ├── cached_externals │ └── git-hooks │ │ ├── post-checkout │ │ └── post-merge └── tasks │ └── git.rake └── recipes └── cached_externals.rb /.gitignore: -------------------------------------------------------------------------------- 1 | *.swp 2 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | Copyright (c) 2008 by 37signals, LLC 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining 4 | a copy of this software and associated documentation files (the 5 | "Software"), to deal in the Software without restriction, including 6 | without limitation the rights to use, copy, modify, merge, publish, 7 | distribute, sublicense, and/or sell copies of the Software, and to 8 | permit persons to whom the Software is furnished to do so, subject to 9 | the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be 12 | included in all copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 15 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 16 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 17 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 18 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 19 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 20 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /README.rdoc: -------------------------------------------------------------------------------- 1 | = Cached Externals 2 | 3 | This Capistrano extension provides yet another way to manage your application's external dependencies. 4 | 5 | External dependencies are a necessary evil. If you're doing Rails development, you'll automatically be dependent on the Rails framework itself, at the very least, and you probably have a handful or two of plugins that you're using as well. Traditionally, the solution to these has been to either install those dependencies globally (which breaks down when you have multiple apps that need different versions of Rails or the plugins), or to bundle them into your application directly via freeze:gems or the like (which tends to bloat your repository). 6 | 7 | Now, bloating a repository is, in itself, not a big deal. However, when it comes time to deploy your application, more data in the repository means more data to check out, resulting in longer deploy times. Some deployment strategies (like remote_cache) can help, but you're _still_ going to be copying all of that data every time you deploy your application. 8 | 9 | One solution is to use a deployment strategy like FastRemoteCache (http://github.com/37signals/fast_remote_cache), which employs hard-links instead of copying files. But your deploys will go _even faster_ if you didn't have to worry about moving or linking all the files in your external dependencies. 10 | 11 | That's where this Cached Externals plugin comes in. It capitalizes on one key concept: your external dependencies are unlikely to change frequently. Because they don't change frequently, it makes more sense to just check them out once, and simply refer to that checkout via symlinks. 12 | 13 | This means your deploys only have to add a single symbolic link for each external dependency, which can mean orders of magnitude less work for a deploy to do. 14 | 15 | == Dependencies 16 | 17 | * Capistrano 2 or later (http://www.capify.org) 18 | 19 | == Assumptions 20 | 21 | The Cached Externals plugin assumes that your external dependencies are either a gem, or under version control somewhere, and the gem sources and repositories are accessible both locally and from your deployment targets. 22 | 23 | == Usage 24 | 25 | When using this with Rails applications, all you need to do is install this as a plugin in your application. Make sure your Capfile has a line like the following: 26 | 27 | Dir['vendor/plugins/*/recipes/*.rb'].each { |plugin| load(plugin) } 28 | 29 | It should be just before the line that loads the 'config/deploy' file. If that line is not there, add it (or rerun "capify ."). 30 | 31 | If you are not using this with a Rails application, you'll need to explicitly load recipes/cached_externals.rb in your Capfile. 32 | 33 | Next, tell Capistrano about your external dependencies. This is done via a YAML file: config/externals.yml. It describes a hash of path names that describe the external repository. For example: 34 | 35 | --- 36 | vendor/rails: 37 | :type: git 38 | :repository: git://github.com/rails/rails.git 39 | :revision: f2d8d13c6495f2a9b3bbf3b50d869c0e5b25c207 40 | vendor/plugins/exception_notification: 41 | :type: git 42 | :repository: git://github.com/rails/exception_notification.git 43 | :revision: ed0b914ff493f9137abc4f68ee08e3c3cd7a3211 44 | vendor/libs/tzinfo: 45 | :type: gem 46 | :version: 0.3.18 47 | 48 | Specify as many as you like. Although it is best to specify an exact revision, you can also give any revision identifier that capistrano understands (git branches, HEAD, etc.). If you do, though, Capistrano will have to resolve those pseudo-revision identifiers every time, which can slow things down a little. 49 | 50 | Once you've got your externals.yml in place, you'll need to clear away the cruft. For example, if you were going to put vendor/rails as a cached external dependency, you'd need to first remove vendor/rails from your working copy. After clearing away all of the in-repository copies of your external dependencies, you just tell capistrano to load up the cached copies: 51 | 52 | cap local externals:setup 53 | 54 | That will cause Capistrano to read your externals.yml, checkout your dependencies, and create symlinks to them. When run locally like that, the dependencies will be checked out to ../shared/externals (e.g., up a directory from your project root). 55 | 56 | Any time you update your config/externals.yml, you'll need to rerun that command. If an external hasn't changed revision, Capistrano will notice that and will not check it out again--only those that have changed will be updated. 57 | 58 | When you deploy your application, these externals will be checked out into #{shared_path}/externals, and again, nothing will be checked out if the dependency's revision matches what has already been cached. 59 | 60 | == Tips 61 | 62 | For the fastest possible deploys, always give an exact revision identifier. This way, Capistrano may not have to query the dependency's repository at all, if nothing has changed. 63 | 64 | Also, if you're using git, it can be a pain to change branches with this plugin, because different branches may have different dependencies, or (even worse) the same dependency but at different revisions. This plugin provides a Rake task, "git:hooks:install", that installs a couple of git hook scripts: post-checkout and post-merge. (If you already have scripts defined for those, back them up before running this task, because they'll get clobbered!) These hooks will then make it so that any time you switch branches, or do a "git pull" or "git merge", the "cap local externals:setup" task will also get run, keeping your external dependencies in sync. 65 | 66 | == License 67 | 68 | This code is released under the MIT license, and is copyright (c) 2008 by 37signals, LLC. Please see the accompanying LICENSE file for the full text of the license. 69 | -------------------------------------------------------------------------------- /cached_externals.gemspec: -------------------------------------------------------------------------------- 1 | Gem::Specification.new do |s| 2 | s.name = 'cached_externals' 3 | s.version = '1.0.0' 4 | s.date = '2010-03-29' 5 | s.summary = 'Symlink to external dependencies, rather than bloating your repositories with them' 6 | s.description = s.summary 7 | 8 | s.add_dependency('capistrano') 9 | 10 | s.files = Dir['lib/**/*'] 11 | 12 | s.author = 'Jamis Buck' 13 | s.email = 'jamis@jamisbuck.org' 14 | s.homepage = 'http://github.com/37signals/cached_externals' 15 | end 16 | -------------------------------------------------------------------------------- /lib/cached_externals.rb: -------------------------------------------------------------------------------- 1 | # --------------------------------------------------------------------------- 2 | # This is a recipe definition file for Capistrano. The tasks are documented 3 | # below. 4 | # --------------------------------------------------------------------------- 5 | # This file is distributed under the terms of the MIT license by 37signals, 6 | # LLC, and is copyright (c) 2008 by the same. See the LICENSE file distributed 7 | # with this file for the complete text of the license. 8 | # --------------------------------------------------------------------------- 9 | 10 | Capistrano::Configuration.instance.load do 11 | # The :external_modules variable is used internally to load and contain the 12 | # contents of the config/externals.yml file. Although you _could_ set the 13 | # variable yourself (to bypass the need for a config/externals.yml file, for 14 | # instance), you'll rarely (if ever) want to. 15 | # 16 | # If ONLY_MODS is set to a comma-delimited string, you can specify which 17 | # modules to process explicitly. 18 | # 19 | # If EXCEPT_MODS is set to a comma-delimited string, the specified modules 20 | # will be ignored. 21 | set(:external_modules) do 22 | require 'yaml' 23 | 24 | modules = YAML.load_file("config/externals.yml") rescue {} 25 | 26 | if ENV['ONLY_MODS'] 27 | patterns = ENV['ONLY_MODS'].split(/,/).map { |s| Regexp.new(s) } 28 | modules = Hash[modules.select { |k,v| patterns.any? { |p| k.to_s =~ p } }] 29 | end 30 | 31 | if ENV['EXCEPT_MODS'] 32 | patterns = ENV['EXCEPT_MODS'].split(/,/).map { |s| Regexp.new(s) } 33 | modules = Hash[modules.reject { |k,v| patterns.any? { |p| k.to_s =~ p } }] 34 | end 35 | 36 | modules.each do |path, options| 37 | strings = options.select { |k, v| String === k } 38 | raise ArgumentError, "the externals.yml file must use symbols for the option keys (found #{strings.inspect} under #{path})" if strings.any? 39 | end 40 | end 41 | 42 | def in_local_stage? 43 | exists?(:stage) && stage == :local 44 | end 45 | 46 | set(:shared_externals_dir) do 47 | if in_local_stage? 48 | File.expand_path("../shared/externals") 49 | else 50 | File.join(shared_path, "externals") 51 | end 52 | end 53 | 54 | set(:shared_gems_dir) do 55 | if in_local_stage? 56 | File.expand_path("../shared/gems") 57 | else 58 | File.join(shared_path, "gems") 59 | end 60 | end 61 | 62 | def process_external(path, options) 63 | puts "configuring #{path}" 64 | shared_dir = File.join(shared_externals_dir, path) 65 | 66 | if options[:type] == 'gem' 67 | process_external_gem(path, shared_dir, options) 68 | else 69 | process_external_scm(path, shared_dir, options) 70 | end 71 | end 72 | 73 | def process_external_gem(path, shared_dir, options) 74 | name = options[:name] || File.basename(path) 75 | base = File.dirname(path) 76 | 77 | destination = File.join(shared_gems_dir, "gems/#{name}-#{options[:version]}") 78 | install_command = fetch(:gem, "gem") + " install --quiet --ignore-dependencies --no-ri --no-rdoc --install-dir='#{shared_gems_dir}' '#{name}' -v '#{options[:version]}'" 79 | 80 | if in_local_stage? 81 | FileUtils.rm_rf(path) 82 | FileUtils.mkdir_p(base) 83 | if !File.exists?(destination) 84 | FileUtils.mkdir_p(shared_gems_dir) 85 | system(install_command) or raise "error installing #{name}:#{options[:version]} gem" 86 | end 87 | FileUtils.ln_s(destination, path) 88 | else 89 | commands = [ 90 | "mkdir -p #{shared_gems_dir} #{latest_release}/#{base}", 91 | "if [ ! -d #{destination} ]; then (#{install_command}) || (rm -rf #{destination} && false); fi", 92 | "ln -nsf #{destination} #{latest_release}/#{path}" 93 | ] 94 | 95 | run(commands.join(" && ")) 96 | end 97 | end 98 | 99 | def process_external_scm(path, shared_dir, options) 100 | puts options 101 | if options[:type] == "github_https" 102 | options[:type] = "git" 103 | begin 104 | options[:scm_user] = ENV.fetch('GITHUB_TOKEN') 105 | options[:scm_password] = "x-oauth-basic" 106 | options[:repository] = options[:repository].gsub("github.com", "#{options[:scm_user]}@github.com") 107 | rescue 108 | $stderr.puts "ERROR: GITHUB_TOKEN environment variable is not set." 109 | exit! 110 | end 111 | end 112 | 113 | scm = Capistrano::Deploy::SCM.new(options[:type], options) 114 | revision = 115 | begin 116 | scm.query_revision(options[:revision]) { |cmd| `#{cmd}` } 117 | rescue => scm_error 118 | $stderr.puts scm_error 119 | return 120 | end 121 | 122 | destination = File.join(shared_dir, revision) 123 | 124 | if in_local_stage? 125 | FileUtils.rm_rf(path) 126 | FileUtils.mkdir_p(shared_dir) 127 | if !File.exists?(destination) 128 | unless system(scm.checkout(revision, destination)) 129 | FileUtils.rm_rf(destination) if File.exists?(destination) 130 | raise "Error checking out #{revision} to #{destination}" 131 | end 132 | end 133 | FileUtils.ln_s(destination, path) 134 | else 135 | run "rm -rf #{latest_release}/#{path} && mkdir -p #{shared_dir} && if [ ! -d #{destination} ]; then (#{scm.checkout(revision, destination)}) || rm -rf #{destination}; fi && ln -nsf #{destination} #{latest_release}/#{path}" 136 | end 137 | end 138 | 139 | desc "Indicate that externals should be applied locally. See externals:setup." 140 | task :local do 141 | set :stage, :local 142 | end 143 | 144 | namespace :externals do 145 | desc <<-DESC 146 | Set up all defined external modules. This will check to see if any of the 147 | modules need to be checked out (be they new or just updated), and will then 148 | create symlinks to them. If running in 'local' mode (see the :local task) 149 | then these will be created in a "../shared/externals" directory relative 150 | to the project root. Otherwise, these will be created on the remote 151 | machines under [shared_path]/externals. 152 | 153 | Specify ONLY_MODS to process only a subset of the defined modules, and 154 | EXCEPT_MODS to ignore certain modules for processing. 155 | 156 | $ cap local externals:setup ONLY_MODS=rails,solr 157 | $ cap local externals:setup EXCEPT_MODS=rails,solr 158 | DESC 159 | task :setup, :except => { :no_release => true } do 160 | require 'fileutils' 161 | require 'capistrano/recipes/deploy/scm' 162 | 163 | external_modules.each do |path, options| 164 | process_external(path, options) 165 | end 166 | end 167 | end 168 | 169 | # Need to do this before finalize_update, instead of after update_code, 170 | # because finalize_update tries to do a touch of all assets, and some 171 | # assets might be symlinks to files in plugins that have been externalized. 172 | # Updating those externals after finalize_update means that the plugins 173 | # haven't been set up yet when the touch occurs, causing the touch to 174 | # fail and leaving some assets temporally out of sync, potentially, with 175 | # the other servers. 176 | before "deploy:finalize_update", "externals:setup" 177 | end 178 | -------------------------------------------------------------------------------- /lib/cached_externals/git-hooks/post-checkout: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | if [[ "1" == $3 && -f config/externals.yml ]]; 4 | then 5 | cap -q local externals:setup 6 | fi; 7 | -------------------------------------------------------------------------------- /lib/cached_externals/git-hooks/post-merge: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | if [ -f config/externals.yml ]; 4 | then 5 | cap -q local externals:setup 6 | fi; 7 | -------------------------------------------------------------------------------- /lib/tasks/git.rake: -------------------------------------------------------------------------------- 1 | namespace :git do 2 | namespace :hooks do 3 | desc "Install some git hooks for updating cached externals" 4 | task :install do 5 | Dir["#{RAILS_ROOT}/vendor/plugins/cached_externals/script/git-hooks/*"].each do |hook| 6 | cp hook, ".git/hooks" 7 | chmod 0755, ".git/hooks/#{File.basename(hook)}" 8 | end 9 | end 10 | end 11 | end 12 | -------------------------------------------------------------------------------- /recipes/cached_externals.rb: -------------------------------------------------------------------------------- 1 | lib = File.expand_path('../../lib', __FILE__) 2 | $:.unshift(lib) unless $:.include?(lib) 3 | require 'cached_externals' 4 | --------------------------------------------------------------------------------