├── templates ├── cli │ └── utility │ │ ├── compressor │ │ ├── bzip2 │ │ ├── gzip │ │ ├── lzma │ │ ├── pbzip2 │ │ └── custom │ │ ├── storage │ │ ├── local │ │ ├── ninefold │ │ ├── rsync │ │ ├── scp │ │ ├── sftp │ │ ├── ftp │ │ ├── dropbox │ │ ├── s3 │ │ └── cloud_files │ │ ├── splitter │ │ ├── notifier │ │ ├── prowl │ │ ├── campfire │ │ ├── twitter │ │ ├── hipchat │ │ └── mail │ │ ├── encryptor │ │ ├── gpg │ │ └── openssl │ │ ├── syncer │ │ ├── rsync_local │ │ ├── rsync_pull │ │ ├── rsync_push │ │ ├── s3 │ │ └── cloud_files │ │ ├── database │ │ ├── riak │ │ ├── redis │ │ ├── postgresql │ │ ├── mongodb │ │ └── mysql │ │ ├── archive │ │ ├── model.erb │ │ └── config ├── general │ ├── version.erb │ └── links ├── storage │ └── dropbox │ │ ├── authorized.erb │ │ ├── authorization_url.erb │ │ └── cache_file_written.erb └── notifier │ └── mail │ ├── success.erb │ ├── failure.erb │ └── warning.erb ├── .gitignore ├── spec-live ├── .gitignore ├── README ├── compressor │ ├── gzip_spec.rb │ └── custom_spec.rb ├── backups │ └── config.yml.template ├── spec_helper.rb ├── storage │ ├── local_spec.rb │ └── dropbox_spec.rb ├── notifier │ └── mail_spec.rb └── syncer │ └── cloud │ └── s3_spec.rb ├── .travis.yml ├── bin └── backup ├── lib ├── backup │ ├── configuration │ │ └── store.rb │ ├── binder.rb │ ├── encryptor │ │ ├── base.rb │ │ ├── open_ssl.rb │ │ └── gpg.rb │ ├── compressor │ │ ├── base.rb │ │ ├── gzip.rb │ │ ├── bzip2.rb │ │ ├── lzma.rb │ │ ├── custom.rb │ │ └── pbzip2.rb │ ├── syncer │ │ ├── rsync │ │ │ ├── pull.rb │ │ │ ├── base.rb │ │ │ ├── local.rb │ │ │ └── push.rb │ │ ├── base.rb │ │ └── cloud │ │ │ ├── s3.rb │ │ │ └── cloud_files.rb │ ├── configuration.rb │ ├── version.rb │ ├── package.rb │ ├── template.rb │ ├── database │ │ ├── base.rb │ │ ├── riak.rb │ │ ├── redis.rb │ │ └── postgresql.rb │ ├── notifier │ │ ├── prowl.rb │ │ ├── twitter.rb │ │ ├── base.rb │ │ └── hipchat.rb │ ├── splitter.rb │ ├── storage │ │ ├── local.rb │ │ ├── cloudfiles.rb │ │ ├── s3.rb │ │ ├── scp.rb │ │ ├── base.rb │ │ ├── sftp.rb │ │ ├── ninefold.rb │ │ ├── ftp.rb │ │ ├── cycler.rb │ │ └── rsync.rb │ ├── cli │ │ └── helpers.rb │ ├── dependency.rb │ ├── archive.rb │ ├── errors.rb │ ├── packager.rb │ ├── cleaner.rb │ └── pipeline.rb └── backup.rb ├── spec ├── version_spec.rb ├── configuration │ └── store_spec.rb ├── encryptor │ └── base_spec.rb ├── spec_helper.rb ├── dependency_spec.rb ├── compressor │ ├── base_spec.rb │ ├── lzma_spec.rb │ └── custom_spec.rb ├── configuration_spec.rb ├── database │ └── base_spec.rb ├── package_spec.rb ├── syncer │ ├── rsync │ │ ├── base_spec.rb │ │ └── pull_spec.rb │ └── base_spec.rb ├── notifier │ ├── base_spec.rb │ └── prowl_spec.rb └── splitter_spec.rb ├── Gemfile ├── Guardfile ├── backup.gemspec └── LICENSE.md /templates/cli/utility/compressor/bzip2: -------------------------------------------------------------------------------- 1 | ## 2 | # Bzip2 [Compressor] 3 | # 4 | compress_with Bzip2 5 | -------------------------------------------------------------------------------- /templates/cli/utility/compressor/gzip: -------------------------------------------------------------------------------- 1 | ## 2 | # Gzip [Compressor] 3 | # 4 | compress_with Gzip 5 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .DS_Store 2 | *.gem 3 | .rvmrc 4 | .bundle/ 5 | vendor/bundle 6 | *.swp 7 | *.swo 8 | Gemfile.lock 9 | -------------------------------------------------------------------------------- /spec-live/.gitignore: -------------------------------------------------------------------------------- 1 | backups/config.yml 2 | backups/data 3 | backups/log 4 | backups/.cache 5 | backups/.tmp 6 | tmp/ 7 | -------------------------------------------------------------------------------- /templates/general/version.erb: -------------------------------------------------------------------------------- 1 | Backup version <%= Backup::Version.current %> 2 | Ruby version <%= RUBY_DESCRIPTION %> 3 | -------------------------------------------------------------------------------- /templates/storage/dropbox/authorized.erb: -------------------------------------------------------------------------------- 1 | 2 | Backup has successfully been authorized! 3 | Writing session cache file to: 4 | <%= @cached_file %> 5 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | script: "bundle exec rspec ./spec/" 2 | rvm: 3 | - 1.8.7 4 | - 1.9.2 5 | - 1.9.3 6 | - ree 7 | branches: 8 | only: 9 | - master 10 | - develop -------------------------------------------------------------------------------- /templates/cli/utility/storage/local: -------------------------------------------------------------------------------- 1 | ## 2 | # Local (Copy) [Storage] 3 | # 4 | store_with Local do |local| 5 | local.path = "~/backups/" 6 | local.keep = 5 7 | end 8 | -------------------------------------------------------------------------------- /templates/storage/dropbox/authorization_url.erb: -------------------------------------------------------------------------------- 1 | 2 | Visit the following URL to authorize your Dropbox App with Backup: 3 | 4 | <%= @session.get_authorize_url %> 5 | 6 | Hit "Enter/Return" once you're authorized. 7 | -------------------------------------------------------------------------------- /templates/cli/utility/splitter: -------------------------------------------------------------------------------- 1 | ## 2 | # Split [Splitter] 3 | # 4 | # Split the backup file in to chunks of 250 megabytes 5 | # if the backup file size exceeds 250 megabytes 6 | # 7 | split_into_chunks_of 250 8 | -------------------------------------------------------------------------------- /spec-live/README: -------------------------------------------------------------------------------- 1 | == spec-live == 2 | 3 | This folder contains "Live" specs which test various features against 4 | the actual filesystem and/or real storage service accounts. 5 | 6 | These are only intended to be used by developers. 7 | Use at your own risk :) 8 | -------------------------------------------------------------------------------- /templates/cli/utility/notifier/prowl: -------------------------------------------------------------------------------- 1 | ## 2 | # Prowl [Notifier] 3 | # 4 | notify_by Prowl do |prowl| 5 | prowl.on_success = true 6 | prowl.on_warning = true 7 | prowl.on_failure = true 8 | 9 | prowl.application = "my_application" 10 | prowl.api_key = "my_api_key" 11 | end 12 | -------------------------------------------------------------------------------- /templates/cli/utility/compressor/lzma: -------------------------------------------------------------------------------- 1 | ## 2 | # Lzma [Compressor] 3 | # 4 | # [DEPRECATED] 5 | # See the Wiki for more info. 6 | # https://github.com/meskyanichi/backup/wiki/Compressors 7 | compress_with Lzma do |compression| 8 | compression.best = true 9 | compression.fast = false 10 | end 11 | -------------------------------------------------------------------------------- /templates/cli/utility/storage/ninefold: -------------------------------------------------------------------------------- 1 | ## 2 | # Ninefold Cloud Storage [Storage] 3 | # 4 | store_with Ninefold do |nf| 5 | nf.storage_token = "my_storage_token" 6 | nf.storage_secret = "my_storage_secret" 7 | nf.path = "/path/to/my/backups" 8 | nf.keep = 10 9 | end 10 | -------------------------------------------------------------------------------- /templates/notifier/mail/success.erb: -------------------------------------------------------------------------------- 1 | 2 | Backup <%= @model.label %> (<%= @model.trigger %>) finished without any errors! 3 | 4 | =========================================================================== 5 | <%= Backup::Template.new.result("general/version.erb") %> 6 | 7 | <%= Backup::Template.new.result("general/links") %> 8 | -------------------------------------------------------------------------------- /templates/cli/utility/compressor/pbzip2: -------------------------------------------------------------------------------- 1 | ## 2 | # Pbzip2 [Compressor] 3 | # 4 | # [DEPRECATED] 5 | # See the Wiki for more info. 6 | # https://github.com/meskyanichi/backup/wiki/Compressors 7 | compress_with Pbzip2 do |compression| 8 | compression.best = true 9 | compression.fast = false 10 | end 11 | -------------------------------------------------------------------------------- /templates/general/links: -------------------------------------------------------------------------------- 1 | Backup's Ruby Gem releases: 2 | http://rubygems.org/gems/backup 3 | 4 | Backup's Git repository: 5 | https://github.com/meskyanichi/backup 6 | 7 | Backup's Issue Tracker: 8 | https://github.com/meskyanichi/backup/issues 9 | 10 | Backup's Wikipedia/Documentation/Guides: 11 | https://github.com/meskyanichi/backup/wiki -------------------------------------------------------------------------------- /templates/cli/utility/encryptor/gpg: -------------------------------------------------------------------------------- 1 | ## 2 | # GPG [Encryptor] 3 | # 4 | encrypt_with GPG do |encryption| 5 | encryption.key = <<-KEY 6 | -----BEGIN PGP PUBLIC KEY BLOCK----- 7 | Version: GnuPG v1.4.11 (Darwin) 8 | 9 | 10 | -----END PGP PUBLIC KEY BLOCK----- 11 | KEY 12 | end 13 | -------------------------------------------------------------------------------- /templates/cli/utility/storage/rsync: -------------------------------------------------------------------------------- 1 | ## 2 | # RSync [Storage] 3 | # 4 | store_with RSync do |server| 5 | server.username = "my_username" 6 | server.password = "my_password" 7 | server.ip = "123.45.678.90" 8 | server.port = 22 9 | server.path = "~/backups/" 10 | server.local = false 11 | end 12 | -------------------------------------------------------------------------------- /templates/cli/utility/storage/scp: -------------------------------------------------------------------------------- 1 | ## 2 | # SCP (Secure Copy) [Storage] 3 | # 4 | store_with SCP do |server| 5 | server.username = "my_username" 6 | server.password = "my_password" 7 | server.ip = "123.45.678.90" 8 | server.port = 22 9 | server.path = "~/backups/" 10 | server.keep = 5 11 | end 12 | -------------------------------------------------------------------------------- /templates/notifier/mail/failure.erb: -------------------------------------------------------------------------------- 1 | 2 | Backup <%= @model.label %> (<%= @model.trigger %>) Failed! 3 | 4 | See the attached backup log for details. 5 | 6 | =========================================================================== 7 | <%= Backup::Template.new.result("general/version.erb") %> 8 | 9 | <%= Backup::Template.new.result("general/links") %> 10 | -------------------------------------------------------------------------------- /templates/cli/utility/encryptor/openssl: -------------------------------------------------------------------------------- 1 | ## 2 | # OpenSSL [Encryptor] 3 | # 4 | encrypt_with OpenSSL do |encryption| 5 | encryption.password = "my_password" # From String 6 | encryption.password_file = "/path/to/password/file" # Or from File 7 | encryption.base64 = true 8 | encryption.salt = true 9 | end 10 | -------------------------------------------------------------------------------- /bin/backup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | # encoding: utf-8 3 | 4 | # Load the Backup core library 5 | require File.expand_path("../../lib/backup", __FILE__) 6 | 7 | # Load the Backup command line interface utility 8 | require File.expand_path("../../lib/backup/cli/utility", __FILE__) 9 | 10 | # Initialize the Backup command line utility 11 | Backup::CLI::Utility.start 12 | -------------------------------------------------------------------------------- /templates/cli/utility/syncer/rsync_local: -------------------------------------------------------------------------------- 1 | ## 2 | # RSync::Local [Syncer] 3 | # 4 | sync_with RSync::Local do |rsync| 5 | rsync.path = "~/backups/" 6 | rsync.mirror = true 7 | 8 | rsync.directories do |directory| 9 | directory.add "/var/apps/my_app/public/uploads" 10 | directory.add "/var/apps/my_app/logs" 11 | end 12 | end 13 | -------------------------------------------------------------------------------- /templates/notifier/mail/warning.erb: -------------------------------------------------------------------------------- 1 | 2 | Backup <%= @model.label %> (<%= @model.trigger %>) finished with warnings. 3 | 4 | See the attached backup log for details. 5 | 6 | =========================================================================== 7 | <%= Backup::Template.new.result("general/version.erb") %> 8 | 9 | <%= Backup::Template.new.result("general/links") %> 10 | -------------------------------------------------------------------------------- /templates/cli/utility/compressor/custom: -------------------------------------------------------------------------------- 1 | ## 2 | # Custom [Compressor] 3 | # 4 | # For information on using a Custom Compressor, 5 | # please see the following Wiki page: 6 | # https://github.com/meskyanichi/backup/wiki/Compressors 7 | # 8 | compress_with Custom do |compressor| 9 | compressor.command = 'gzip' 10 | compressor.extension = '.gz' 11 | end 12 | -------------------------------------------------------------------------------- /templates/cli/utility/storage/sftp: -------------------------------------------------------------------------------- 1 | ## 2 | # SFTP (Secure File Transfer Protocol) [Storage] 3 | # 4 | store_with SFTP do |server| 5 | server.username = "my_username" 6 | server.password = "my_password" 7 | server.ip = "123.45.678.90" 8 | server.port = 22 9 | server.path = "~/backups/" 10 | server.keep = 5 11 | end 12 | -------------------------------------------------------------------------------- /templates/cli/utility/database/riak: -------------------------------------------------------------------------------- 1 | ## 2 | # Riak [Database] 3 | # 4 | database Riak do |db| 5 | db.name = "hostname" 6 | db.node = "riak@hostname" 7 | db.cookie = "cookie" 8 | # Optional: Use to set the location of this utility 9 | # if it cannot be found by name in your $PATH 10 | # db.riak_admin_utility = '/opt/local/bin/riak-admin' 11 | end 12 | -------------------------------------------------------------------------------- /templates/cli/utility/notifier/campfire: -------------------------------------------------------------------------------- 1 | ## 2 | # Campfire [Notifier] 3 | # 4 | notify_by Campfire do |campfire| 5 | campfire.on_success = true 6 | campfire.on_warning = true 7 | campfire.on_failure = true 8 | 9 | campfire.api_token = "my_api_authentication_token" 10 | campfire.subdomain = "my_subdomain" 11 | campfire.room_id = "my_room_id" 12 | end 13 | -------------------------------------------------------------------------------- /templates/cli/utility/storage/ftp: -------------------------------------------------------------------------------- 1 | ## 2 | # FTP (File Transfer Protocol) [Storage] 3 | # 4 | store_with FTP do |server| 5 | server.username = "my_username" 6 | server.password = "my_password" 7 | server.ip = "123.45.678.90" 8 | server.port = 21 9 | server.path = "~/backups/" 10 | server.keep = 5 11 | server.passive_mode = false 12 | end 13 | -------------------------------------------------------------------------------- /templates/cli/utility/notifier/twitter: -------------------------------------------------------------------------------- 1 | ## 2 | # Twitter [Notifier] 3 | # 4 | notify_by Twitter do |tweet| 5 | tweet.on_success = true 6 | tweet.on_warning = true 7 | tweet.on_failure = true 8 | 9 | tweet.consumer_key = "my_consumer_key" 10 | tweet.consumer_secret = "my_consumer_secret" 11 | tweet.oauth_token = "my_oauth_token" 12 | tweet.oauth_token_secret = "my_oauth_token_secret" 13 | end 14 | -------------------------------------------------------------------------------- /templates/cli/utility/notifier/hipchat: -------------------------------------------------------------------------------- 1 | ## 2 | # Hipchat [Notifier] 3 | # 4 | notify_by Hipchat do |hipchat| 5 | hipchat.on_success = true 6 | hipchat.on_warning = true 7 | hipchat.on_failure = true 8 | 9 | hipchat.token = "token" 10 | hipchat.from = "DB Backup" 11 | hipchat.rooms_notified = ["activity"] 12 | hipchat.success_color = "green" 13 | hipchat.warning_color = "yellow" 14 | hipchat.failure_color = "red" 15 | end 16 | -------------------------------------------------------------------------------- /templates/storage/dropbox/cache_file_written.erb: -------------------------------------------------------------------------------- 1 | 2 | Cache data written! You will no longer need to manually authorize this 3 | Dropbox account via an authorization URL on this machine. 4 | 5 | Note: If you run Backup with this Dropbox account in another location, 6 | or on other machines, you will need to either authorize them the same 7 | way, or simply copy over the cache file from: 8 | <%= @cached_file %> 9 | to the cache directory on your other machines to use this Dropbox 10 | account there as well. 11 | -------------------------------------------------------------------------------- /lib/backup/configuration/store.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require 'ostruct' 4 | 5 | module Backup 6 | module Configuration 7 | class Store < OpenStruct 8 | 9 | ## 10 | # Returns an Array of all attribute method names 11 | # that default values were set for. 12 | def _attributes 13 | @table.keys 14 | end 15 | 16 | ## 17 | # Used only within the specs 18 | def reset! 19 | @table.clear 20 | end 21 | 22 | end 23 | end 24 | end 25 | -------------------------------------------------------------------------------- /templates/cli/utility/syncer/rsync_pull: -------------------------------------------------------------------------------- 1 | ## 2 | # RSync::Pull [Syncer] 3 | # 4 | sync_with RSync::Pull do |rsync| 5 | rsync.ip = "123.45.678.90" 6 | rsync.port = 22 7 | rsync.username = "my_username" 8 | rsync.password = "my_password" 9 | rsync.path = "~/backups/" 10 | rsync.mirror = true 11 | rsync.compress = true 12 | 13 | rsync.directories do |directory| 14 | directory.add "/var/apps/my_app/public/uploads" 15 | directory.add "/var/apps/my_app/logs" 16 | end 17 | end 18 | -------------------------------------------------------------------------------- /templates/cli/utility/syncer/rsync_push: -------------------------------------------------------------------------------- 1 | ## 2 | # RSync::Push [Syncer] 3 | # 4 | sync_with RSync::Push do |rsync| 5 | rsync.ip = "123.45.678.90" 6 | rsync.port = 22 7 | rsync.username = "my_username" 8 | rsync.password = "my_password" 9 | rsync.path = "~/backups/" 10 | rsync.mirror = true 11 | rsync.compress = true 12 | 13 | rsync.directories do |directory| 14 | directory.add "/var/apps/my_app/public/uploads" 15 | directory.add "/var/apps/my_app/logs" 16 | end 17 | end 18 | -------------------------------------------------------------------------------- /templates/cli/utility/storage/dropbox: -------------------------------------------------------------------------------- 1 | ## 2 | # Dropbox File Hosting Service [Storage] 3 | # 4 | # Access Type: 5 | # 6 | # - :app_folder (Default) 7 | # - :dropbox 8 | # 9 | # Note: 10 | # 11 | # Initial backup must be performed manually to authorize 12 | # this machine with your Dropbox account. 13 | # 14 | store_with Dropbox do |db| 15 | db.api_key = "my_api_key" 16 | db.api_secret = "my_api_secret" 17 | db.access_type = :app_folder 18 | db.path = "/path/to/my/backups" 19 | db.keep = 25 20 | end 21 | -------------------------------------------------------------------------------- /templates/cli/utility/storage/s3: -------------------------------------------------------------------------------- 1 | ## 2 | # Amazon Simple Storage Service [Storage] 3 | # 4 | # Available Regions: 5 | # 6 | # - ap-northeast-1 7 | # - ap-southeast-1 8 | # - eu-west-1 9 | # - us-east-1 10 | # - us-west-1 11 | # 12 | store_with S3 do |s3| 13 | s3.access_key_id = "my_access_key_id" 14 | s3.secret_access_key = "my_secret_access_key" 15 | s3.region = "us-east-1" 16 | s3.bucket = "bucket-name" 17 | s3.path = "/path/to/my/backups" 18 | s3.keep = 10 19 | end 20 | -------------------------------------------------------------------------------- /lib/backup/binder.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | class Binder 5 | 6 | ## 7 | # Creates a new Backup::Notifier::Binder instance. Loops through the provided 8 | # Hash to set instance variables 9 | def initialize(key_and_values) 10 | key_and_values.each do |key, value| 11 | instance_variable_set("@#{ key }", value) 12 | end 13 | end 14 | 15 | ## 16 | # Returns the binding (needs a wrapper method because #binding is a private method) 17 | def get_binding 18 | binding 19 | end 20 | 21 | end 22 | end -------------------------------------------------------------------------------- /templates/cli/utility/database/redis: -------------------------------------------------------------------------------- 1 | ## 2 | # Redis [Database] 3 | # 4 | database Redis do |db| 5 | db.name = "my_database_name" 6 | db.path = "/usr/local/var/db/redis" 7 | db.password = "my_password" 8 | db.host = "localhost" 9 | db.port = 5432 10 | db.socket = "/tmp/redis.sock" 11 | db.additional_options = [] 12 | db.invoke_save = true 13 | # Optional: Use to set the location of this utility 14 | # if it cannot be found by name in your $PATH 15 | # db.redis_cli_utility = "/opt/local/bin/redis-cli" 16 | end 17 | -------------------------------------------------------------------------------- /spec/version_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../spec_helper.rb', __FILE__) 4 | 5 | def set_version(major, minor, patch) 6 | Backup::Version.stubs(:major).returns(major) 7 | Backup::Version.stubs(:minor).returns(minor) 8 | Backup::Version.stubs(:patch).returns(patch) 9 | end 10 | 11 | describe Backup::Version do 12 | it 'should return a nicer gemspec output' do 13 | set_version(1,2,3) 14 | Backup::Version.current.should == '1.2.3' 15 | end 16 | 17 | it 'should return a nicer gemspec output with build' do 18 | set_version(4,5,6) 19 | Backup::Version.current.should == '4.5.6' 20 | end 21 | end 22 | -------------------------------------------------------------------------------- /templates/cli/utility/database/postgresql: -------------------------------------------------------------------------------- 1 | ## 2 | # PostgreSQL [Database] 3 | # 4 | database PostgreSQL do |db| 5 | db.name = "my_database_name" 6 | db.username = "my_username" 7 | db.password = "my_password" 8 | db.host = "localhost" 9 | db.port = 5432 10 | db.socket = "/tmp/pg.sock" 11 | db.skip_tables = ["skip", "these", "tables"] 12 | db.only_tables = ["only", "these" "tables"] 13 | db.additional_options = ["-xc", "-E=utf8"] 14 | # Optional: Use to set the location of this utility 15 | # if it cannot be found by name in your $PATH 16 | # db.pg_dump_utility = "/opt/local/bin/pg_dump" 17 | end 18 | -------------------------------------------------------------------------------- /templates/cli/utility/archive: -------------------------------------------------------------------------------- 1 | ## 2 | # Archive [Archive] 3 | # 4 | # Adding a file: 5 | # 6 | # archive.add "/path/to/a/file.rb" 7 | # 8 | # Adding an directory (including sub-directories): 9 | # 10 | # archive.add "/path/to/a/directory/" 11 | # 12 | # Excluding a file: 13 | # 14 | # archive.exclude "/path/to/an/excluded_file.rb" 15 | # 16 | # Excluding a directory (including sub-directories): 17 | # 18 | # archive.exclude "/path/to/an/excluded_directory/ 19 | # 20 | archive :my_archive do |archive| 21 | archive.add "/path/to/a/file.rb" 22 | archive.add "/path/to/a/folder/" 23 | archive.exclude "/path/to/a/excluded_file.rb" 24 | archive.exclude "/path/to/a/excluded_folder/" 25 | end 26 | -------------------------------------------------------------------------------- /templates/cli/utility/storage/cloud_files: -------------------------------------------------------------------------------- 1 | ## 2 | # Rackspace Cloud Files [Storage] 3 | # 4 | # Available Auth URLs: 5 | # 6 | # - https://auth.api.rackspacecloud.com (US - Default) 7 | # - https://lon.auth.api.rackspacecloud.com (UK) 8 | # 9 | # Servicenet: 10 | # 11 | # Set this to 'true' if Backup runs on a Rackspace server. It will avoid 12 | # transfer charges and it's more performant. 13 | # 14 | store_with CloudFiles do |cf| 15 | cf.api_key = "my_api_key" 16 | cf.username = "my_username" 17 | cf.container = "my_container" 18 | cf.path = "/path/to/my/backups" 19 | cf.keep = 5 20 | cf.auth_url = "lon.auth.api.rackspacecloud.com" 21 | cf.servicenet = false 22 | end 23 | -------------------------------------------------------------------------------- /templates/cli/utility/database/mongodb: -------------------------------------------------------------------------------- 1 | ## 2 | # MongoDB [Database] 3 | # 4 | database MongoDB do |db| 5 | db.name = "my_database_name" 6 | db.username = "my_username" 7 | db.password = "my_password" 8 | db.host = "localhost" 9 | db.port = 5432 10 | db.ipv6 = false 11 | db.only_collections = ["only", "these" "collections"] 12 | db.additional_options = [] 13 | db.lock = false 14 | # Optional: Use to set the location of these utilities 15 | # if they cannot be found by their name in your $PATH 16 | # db.mongodump_utility = "/opt/local/bin/mongodump" 17 | # db.mongo_utility = "/opt/local/bin/mongo" 18 | end 19 | -------------------------------------------------------------------------------- /lib/backup/encryptor/base.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Encryptor 5 | class Base 6 | include Backup::CLI::Helpers 7 | include Backup::Configuration::Helpers 8 | 9 | def initialize 10 | load_defaults! 11 | end 12 | 13 | private 14 | 15 | ## 16 | # Return the encryptor name, with Backup namespace removed 17 | def encryptor_name 18 | self.class.to_s.sub('Backup::', '') 19 | end 20 | 21 | ## 22 | # Logs a message to the console and log file to inform 23 | # the client that Backup is encrypting the archive 24 | def log! 25 | Logger.message "Using #{ encryptor_name } to encrypt the archive." 26 | end 27 | end 28 | end 29 | end 30 | -------------------------------------------------------------------------------- /Gemfile: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | # RubyGems Source 4 | source 'http://rubygems.org' 5 | 6 | # Include gem dependencies from the gemspec for development purposes 7 | gemspec 8 | 9 | # Dynamically define the dependencies specified in Backup::Dependency.all 10 | require File.expand_path("../lib/backup/dependency", __FILE__) 11 | Backup::Dependency.all.each do |name, gemspec| 12 | gem(name, gemspec[:version]) 13 | end 14 | 15 | # Define gems to be used in the 'test' environment 16 | group :test do 17 | gem 'rspec' 18 | gem 'mocha' 19 | gem 'timecop' 20 | gem 'fuubar' 21 | 22 | gem 'guard' 23 | gem 'guard-rspec' 24 | gem 'rb-fsevent' # guard notifications for osx 25 | gem 'growl' # $ brew install growlnotify 26 | gem 'rb-inotify' # guard notifications for linux 27 | gem 'libnotify' # $ apt-get install ??? 28 | end 29 | -------------------------------------------------------------------------------- /templates/cli/utility/notifier/mail: -------------------------------------------------------------------------------- 1 | ## 2 | # Mail [Notifier] 3 | # 4 | # The default delivery method for Mail Notifiers is 'SMTP'. 5 | # See the Wiki for other delivery options. 6 | # https://github.com/meskyanichi/backup/wiki/Notifiers 7 | # 8 | notify_by Mail do |mail| 9 | mail.on_success = true 10 | mail.on_warning = true 11 | mail.on_failure = true 12 | 13 | mail.from = "sender@email.com" 14 | mail.to = "receiver@email.com" 15 | mail.address = "smtp.gmail.com" 16 | mail.port = 587 17 | mail.domain = "your.host.name" 18 | mail.user_name = "sender@email.com" 19 | mail.password = "my_password" 20 | mail.authentication = "plain" 21 | mail.enable_starttls_auto = true 22 | end 23 | -------------------------------------------------------------------------------- /Guardfile: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | # To run the test suite against all 3 rubies: 1.9.3, 1.9.2, and 1.8.7, simply run the following command: 4 | # 5 | # $ guard start 6 | # 7 | # Be use you are using RVM and have Ruby 1.9.3, 1.9.2, 1.8.7 installed as well as all of Backup's gem 8 | # dependencies for each of these Ruby versions. 9 | # 10 | # Be sure to run `bundle install` against every Ruby version, as well as `gem install thor POpen4` 11 | 12 | guard "rspec", 13 | :version => 2, 14 | :rvm => ["1.9.3", "1.9.2", "1.8.7"], 15 | :cli => "--color --format Fuubar", 16 | :notification => false, 17 | :all_after_pass => false, 18 | :all_on_start => false do 19 | 20 | watch("lib/backup.rb") { "spec" } 21 | watch("spec/spec_helper.rb") { "spec" } 22 | watch(%r{^lib/backup/(.+)\.rb}) {|m| "spec/#{ m[1] }_spec.rb" } 23 | watch(%r{^spec/.+_spec\.rb}) 24 | end 25 | -------------------------------------------------------------------------------- /spec-live/compressor/gzip_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../../spec_helper.rb', __FILE__) 4 | 5 | describe 'Compressor::Gzip' do 6 | 7 | def archive_file_for(model) 8 | File.join( 9 | Backup::SpecLive::TMP_PATH, 10 | "#{model.trigger}", model.time, "#{model.trigger}.tar" 11 | ) 12 | end 13 | 14 | def archive_contents_for(model) 15 | archive_file = archive_file_for(model) 16 | %x{ tar -tvf #{archive_file} } 17 | end 18 | 19 | it 'should compress an archive' do 20 | model = h_set_trigger('compressor_gzip_archive_local') 21 | model.perform! 22 | archive_file = archive_file_for(model) 23 | File.exist?(archive_file).should be_true 24 | archive_contents_for(model).should match( 25 | /compressor_gzip_archive_local\/archives\/test_archive\.tar\.gz/ 26 | ) 27 | File.stat(archive_file).size.should be > 0 28 | end 29 | 30 | end 31 | -------------------------------------------------------------------------------- /spec-live/compressor/custom_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../../spec_helper.rb', __FILE__) 4 | 5 | describe 'Compressor::Custom' do 6 | 7 | def archive_file_for(model) 8 | File.join( 9 | Backup::SpecLive::TMP_PATH, 10 | "#{model.trigger}", model.time, "#{model.trigger}.tar" 11 | ) 12 | end 13 | 14 | def archive_contents_for(model) 15 | archive_file = archive_file_for(model) 16 | %x{ tar -tvf #{archive_file} } 17 | end 18 | 19 | it 'should compress an archive' do 20 | model = h_set_trigger('compressor_custom_archive_local') 21 | model.perform! 22 | archive_file = archive_file_for(model) 23 | File.exist?(archive_file).should be_true 24 | archive_contents_for(model).should match( 25 | /compressor_custom_archive_local\/archives\/test_archive\.tar\.foo/ 26 | ) 27 | File.stat(archive_file).size.should be > 0 28 | end 29 | 30 | end 31 | -------------------------------------------------------------------------------- /lib/backup/compressor/base.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Compressor 5 | class Base 6 | include Backup::CLI::Helpers 7 | include Backup::Configuration::Helpers 8 | 9 | ## 10 | # Yields to the block the compressor command and filename extension. 11 | def compress_with 12 | log! 13 | yield @cmd, @ext 14 | end 15 | 16 | private 17 | 18 | ## 19 | # Return the compressor name, with Backup namespace removed 20 | def compressor_name 21 | self.class.to_s.sub('Backup::', '') 22 | end 23 | 24 | ## 25 | # Logs a message to the console and log file to inform 26 | # the client that Backup is using the compressor 27 | def log! 28 | Logger.message "Using #{ compressor_name } for compression.\n" + 29 | " Command: '#{ @cmd }'\n" + 30 | " Ext: '#{ @ext }'" 31 | end 32 | 33 | end 34 | end 35 | end 36 | -------------------------------------------------------------------------------- /lib/backup/syncer/rsync/pull.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Syncer 5 | module RSync 6 | class Pull < Push 7 | 8 | ## 9 | # Performs the RSync::Pull operation 10 | # debug options: -vhP 11 | def perform! 12 | write_password_file! 13 | 14 | @directories.each do |directory| 15 | Logger.message("#{ syncer_name } started syncing '#{ directory }'.") 16 | run("#{ utility(:rsync) } #{ options } " + 17 | "'#{ username }@#{ ip }:#{ directory.sub(/^\~\//, '') }' " + 18 | "'#{ dest_path }'") 19 | end 20 | 21 | ensure 22 | remove_password_file! 23 | end 24 | 25 | private 26 | 27 | ## 28 | # Return expanded @path, since this path is local 29 | def dest_path 30 | @dest_path ||= File.expand_path(@path) 31 | end 32 | 33 | end 34 | end 35 | end 36 | end 37 | -------------------------------------------------------------------------------- /lib/backup/configuration.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | %w[helpers store].each do |file| 4 | require File.expand_path("../configuration/#{file}", __FILE__) 5 | end 6 | 7 | # Temporary measure for deprecating the use of Configuration 8 | # namespaced classes for setting pre-configured defaults. 9 | module Backup 10 | module Configuration 11 | extend self 12 | 13 | ## 14 | # Pass calls on to the proper class and log a warning 15 | def defaults(&block) 16 | klass = eval(self.to_s.sub('Configuration::', '')) 17 | Logger.warn Errors::ConfigurationError.new <<-EOS 18 | [DEPRECATION WARNING] 19 | #{ self }.defaults is being deprecated. 20 | To set pre-configured defaults for #{ klass }, use: 21 | #{ klass }.defaults 22 | EOS 23 | klass.defaults(&block) 24 | end 25 | 26 | private 27 | 28 | def const_missing(const) 29 | const_set(const, Module.new { extend Configuration }) 30 | end 31 | 32 | end 33 | end 34 | -------------------------------------------------------------------------------- /templates/cli/utility/database/mysql: -------------------------------------------------------------------------------- 1 | ## 2 | # MySQL [Database] 3 | # 4 | database MySQL do |db| 5 | # To dump all databases, set `db.name = :all` (or leave blank) 6 | db.name = "my_database_name" 7 | db.username = "my_username" 8 | db.password = "my_password" 9 | db.host = "localhost" 10 | db.port = 3306 11 | db.socket = "/tmp/mysql.sock" 12 | # Note: when using `skip_tables` with the `db.name = :all` option, 13 | # table names should be prefixed with a database name. 14 | # e.g. ["db_name.table_to_skip", ...] 15 | db.skip_tables = ["skip", "these", "tables"] 16 | db.only_tables = ["only", "these" "tables"] 17 | db.additional_options = ["--quick", "--single-transaction"] 18 | # Optional: Use to set the location of this utility 19 | # if it cannot be found by name in your $PATH 20 | # db.mysqldump_utility = "/opt/local/bin/mysqldump" 21 | end 22 | -------------------------------------------------------------------------------- /templates/cli/utility/model.erb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | ## 4 | # Backup Generated: <%= @options[:trigger] %> 5 | # Once configured, you can run the backup with the following command: 6 | # 7 | # $ backup perform -t <%= @options[:trigger] %> [-c ] 8 | # 9 | Backup::Model.new(:<%= @options[:trigger] %>, 'Description for <%= @options[:trigger] %>') do 10 | <% if @options[:splitter] %> 11 | <%= Backup::Template.new.result("cli/utility/splitter") %> 12 | <% end; if @options[:archives] %> 13 | <%= Backup::Template.new.result("cli/utility/archive") %> 14 | <% end; [:databases, :storages, :syncers, :encryptors, :compressors, :notifiers].each do |item| 15 | if @options[item] 16 | @options[item].split(',').map(&:strip).uniq.each do |entry| 17 | if File.exist?(File.join(Backup::TEMPLATE_PATH, 'cli', 'utility', item.to_s[0..-2], entry)) %> 18 | <%= Backup::Template.new.result("cli/utility/#{item.to_s[0..-2]}/#{entry}") %> 19 | <% end 20 | end 21 | end 22 | end %> 23 | end 24 | -------------------------------------------------------------------------------- /spec-live/backups/config.yml.template: -------------------------------------------------------------------------------- 1 | ## 2 | # config.yml template 3 | # for usage, see: 4 | # spec-live/spec_helper.rb 5 | # spec-live/backups/config.rb 6 | ## 7 | --- 8 | storage: 9 | scp: 10 | specs_enabled: false 11 | username: 12 | password: 13 | ip: localhost 14 | port: 22 15 | path: /absolute/path/to/spec-live/tmp 16 | dropbox: 17 | specs_enabled: false 18 | api_key: 19 | api_secret: 20 | path: 21 | timeout: 22 | notifier: 23 | mail: 24 | specs_enabled: false 25 | delivery_method: smtp 26 | from: 27 | to: 28 | address: smtp.gmail.com 29 | port: 587 30 | user_name: 31 | password: 32 | authentication: plain 33 | enable_starttls_auto: true 34 | sendmail: 35 | sendmail_args: 36 | syncer: 37 | cloud: 38 | s3: 39 | specs_enabled: false 40 | access_key_id: 41 | secret_access_key: 42 | bucket: 43 | region: 44 | -------------------------------------------------------------------------------- /templates/cli/utility/config: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | ## 4 | # Backup 5 | # Generated Main Config Template 6 | # 7 | # For more information: 8 | # 9 | # View the Git repository at https://github.com/meskyanichi/backup 10 | # View the Wiki/Documentation at https://github.com/meskyanichi/backup/wiki 11 | # View the issue log at https://github.com/meskyanichi/backup/issues 12 | 13 | ## 14 | # Global Configuration 15 | # Add more (or remove) global configuration below 16 | # 17 | # Backup::Storage::S3.defaults do |s3| 18 | # s3.access_key_id = "my_access_key_id" 19 | # s3.secret_access_key = "my_secret_access_key" 20 | # end 21 | # 22 | # Backup::Encryptor::OpenSSL.defaults do |encryption| 23 | # encryption.password = "my_password" 24 | # encryption.base64 = true 25 | # encryption.salt = true 26 | # end 27 | 28 | ## 29 | # Load all models from the models directory (after the above global configuration blocks) 30 | Dir[File.join(File.dirname(Config.config_file), "models", "*.rb")].each do |model| 31 | instance_eval(File.read(model)) 32 | end 33 | -------------------------------------------------------------------------------- /lib/backup/version.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | class Version 5 | 6 | ## 7 | # Change the MAJOR, MINOR and PATCH constants below 8 | # to adjust the version of the Backup gem 9 | # 10 | # MAJOR: 11 | # Defines the major version 12 | # MINOR: 13 | # Defines the minor version 14 | # PATCH: 15 | # Defines the patch version 16 | MAJOR, MINOR, PATCH = 3, 0, 25 17 | 18 | ## 19 | # Returns the major version ( big release based off of multiple minor releases ) 20 | def self.major 21 | MAJOR 22 | end 23 | 24 | ## 25 | # Returns the minor version ( small release based off of multiple patches ) 26 | def self.minor 27 | MINOR 28 | end 29 | 30 | ## 31 | # Returns the patch version ( updates, features and (crucial) bug fixes ) 32 | def self.patch 33 | PATCH 34 | end 35 | 36 | ## 37 | # Returns the current version of the Backup gem ( qualified for the gemspec ) 38 | def self.current 39 | "#{major}.#{minor}.#{patch}" 40 | end 41 | 42 | end 43 | end 44 | -------------------------------------------------------------------------------- /spec/configuration/store_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../../spec_helper.rb', __FILE__) 4 | 5 | describe 'Backup::Configuration::Store' do 6 | let(:store) { Backup::Configuration::Store.new } 7 | 8 | before do 9 | store.foo = 'one' 10 | store.bar = 'two' 11 | end 12 | 13 | it 'should be a subclass of OpenStruct' do 14 | Backup::Configuration::Store.superclass.should == OpenStruct 15 | end 16 | 17 | it 'should return nil for unset attributes' do 18 | store.foobar.should be_nil 19 | end 20 | 21 | describe '#_attribues' do 22 | it 'should return an array of attribute names' do 23 | store._attributes.should be_an Array 24 | store._attributes.count.should be(2) 25 | store._attributes.should include(:foo, :bar) 26 | end 27 | end 28 | 29 | describe '#reset!' do 30 | it 'should clear all attributes set' do 31 | store.reset! 32 | store._attributes.should be_an Array 33 | store._attributes.should be_empty 34 | store.foo.should be_nil 35 | store.bar.should be_nil 36 | end 37 | end 38 | 39 | end 40 | -------------------------------------------------------------------------------- /lib/backup/syncer/base.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Syncer 5 | class Base 6 | include Backup::CLI::Helpers 7 | include Backup::Configuration::Helpers 8 | 9 | ## 10 | # Path to store the synced files/directories to 11 | attr_accessor :path 12 | 13 | ## 14 | # Flag for mirroring the files/directories 15 | attr_accessor :mirror 16 | 17 | def initialize 18 | load_defaults! 19 | 20 | @path ||= 'backups' 21 | @mirror ||= false 22 | @directories = Array.new 23 | end 24 | 25 | ## 26 | # Syntactical suger for the DSL for adding directories 27 | def directories(&block) 28 | return @directories unless block_given? 29 | instance_eval(&block) 30 | end 31 | 32 | ## 33 | # Adds a path to the @directories array 34 | def add(path) 35 | @directories << path 36 | end 37 | 38 | private 39 | 40 | def syncer_name 41 | self.class.to_s.sub('Backup::', '') 42 | end 43 | 44 | end 45 | end 46 | end 47 | -------------------------------------------------------------------------------- /lib/backup/package.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | class Package 5 | 6 | ## 7 | # The time when the backup initiated (in format: 2011.02.20.03.29.59) 8 | attr_reader :time 9 | 10 | ## 11 | # The trigger which initiated the backup process 12 | attr_reader :trigger 13 | 14 | ## 15 | # Extension for the final archive file(s) 16 | attr_accessor :extension 17 | 18 | ## 19 | # Set by the Splitter if the final archive was "chunked" 20 | attr_accessor :chunk_suffixes 21 | 22 | ## 23 | # The version of Backup used to create the package 24 | attr_reader :version 25 | 26 | def initialize(model) 27 | @time = model.time 28 | @trigger = model.trigger 29 | @extension = 'tar' 30 | @chunk_suffixes = Array.new 31 | @version = Backup::Version.current 32 | end 33 | 34 | def filenames 35 | if chunk_suffixes.empty? 36 | [basename] 37 | else 38 | chunk_suffixes.map {|suffix| "#{ basename }-#{ suffix }" } 39 | end 40 | end 41 | 42 | def basename 43 | "#{ time }.#{ trigger }.#{ extension }" 44 | end 45 | 46 | end 47 | end 48 | -------------------------------------------------------------------------------- /spec/encryptor/base_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../../spec_helper.rb', __FILE__) 4 | 5 | describe Backup::Encryptor::Base do 6 | let(:base) { Backup::Encryptor::Base.new } 7 | 8 | it 'should include CLI::Helpers' do 9 | Backup::Encryptor::Base. 10 | include?(Backup::CLI::Helpers).should be_true 11 | end 12 | 13 | it 'should include Configuration::Helpers' do 14 | Backup::Encryptor::Base. 15 | include?(Backup::Configuration::Helpers).should be_true 16 | end 17 | 18 | describe '#initialize' do 19 | it 'should load defaults' do 20 | Backup::Encryptor::Base.any_instance.expects(:load_defaults!) 21 | base 22 | end 23 | end 24 | 25 | describe '#encryptor_name' do 26 | it 'should return class name with Backup namespace removed' do 27 | base.send(:encryptor_name).should == 'Encryptor::Base' 28 | end 29 | end 30 | 31 | describe '#log!' do 32 | it 'should log a message' do 33 | base.expects(:encryptor_name).returns('Encryptor Name') 34 | Backup::Logger.expects(:message).with( 35 | 'Using Encryptor Name to encrypt the archive.' 36 | ) 37 | base.send(:log!) 38 | end 39 | end 40 | end 41 | -------------------------------------------------------------------------------- /templates/cli/utility/syncer/s3: -------------------------------------------------------------------------------- 1 | ## 2 | # Amazon S3 [Syncer] 3 | # 4 | # Available Regions: 5 | # 6 | # - ap-northeast-1 7 | # - ap-southeast-1 8 | # - eu-west-1 9 | # - us-east-1 10 | # - us-west-1 11 | # 12 | # Mirroring: 13 | # 14 | # When enabled it will keep an exact mirror of your filesystem on S3. 15 | # This means that when you remove a file from the filesystem, 16 | # it will also remote it from S3. 17 | # 18 | # Concurrency: 19 | # 20 | # `concurrency_type` may be set to: 21 | # 22 | # - false (default) 23 | # - :threads 24 | # - :processes 25 | # 26 | # Set `concurrency_level` to the number of threads/processes to use. 27 | # Defaults to 2. 28 | # 29 | sync_with Cloud::S3 do |s3| 30 | s3.access_key_id = "my_access_key_id" 31 | s3.secret_access_key = "my_secret_access_key" 32 | s3.bucket = "my-bucket" 33 | s3.region = "us-east-1" 34 | s3.path = "/backups" 35 | s3.mirror = true 36 | s3.concurrency_type = false 37 | s3.concurrency_level = 2 38 | 39 | s3.directories do |directory| 40 | directory.add "/path/to/directory/to/sync" 41 | directory.add "/path/to/other/directory/to/sync" 42 | end 43 | end 44 | -------------------------------------------------------------------------------- /lib/backup/template.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require 'erb' 4 | 5 | module Backup 6 | class Template 7 | 8 | # Holds a binding object. Nil if not provided. 9 | attr_accessor :binding 10 | 11 | ## 12 | # Creates a new instance of the Backup::Template class 13 | # and optionally takes an argument that can be either a binding object, a Hash or nil 14 | def initialize(object = nil) 15 | if object.is_a?(Binding) 16 | @binding = object 17 | elsif object.is_a?(Hash) 18 | @binding = Backup::Binder.new(object).get_binding 19 | else 20 | @binding = nil 21 | end 22 | end 23 | 24 | ## 25 | # Renders the provided file (in the context of the binding if any) to the console 26 | def render(file) 27 | puts result(file) 28 | end 29 | 30 | ## 31 | # Returns a String object containing the contents of the file (in the context of the binding if any) 32 | def result(file) 33 | ERB.new(file_contents(file), nil, '<>').result(binding) 34 | end 35 | 36 | private 37 | 38 | ## 39 | # Reads and returns the contents of the provided file path, 40 | # relative from the Backup::TEMPLATE_PATH 41 | def file_contents(file) 42 | File.read(File.join(Backup::TEMPLATE_PATH, file)) 43 | end 44 | 45 | end 46 | end 47 | -------------------------------------------------------------------------------- /lib/backup/compressor/gzip.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Compressor 5 | class Gzip < Base 6 | 7 | ## 8 | # Specify the level of compression to use. 9 | # 10 | # Values should be a single digit from 1 to 9. 11 | # Note that setting the level to either extreme may or may not 12 | # give the desired result. Be sure to check the documentation 13 | # for the compressor being used. 14 | # 15 | # The default `level` is 6. 16 | attr_accessor :level 17 | 18 | attr_deprecate :fast, :version => '3.0.24', 19 | :replacement => :level, 20 | :value => lambda {|val| val ? 1 : nil } 21 | attr_deprecate :best, :version => '3.0.24', 22 | :replacement => :level, 23 | :value => lambda {|val| val ? 9 : nil } 24 | 25 | ## 26 | # Creates a new instance of Backup::Compressor::Gzip 27 | def initialize(&block) 28 | load_defaults! 29 | 30 | @level ||= false 31 | 32 | instance_eval(&block) if block_given? 33 | 34 | @cmd = "#{ utility(:gzip) }#{ options }" 35 | @ext = '.gz' 36 | end 37 | 38 | private 39 | 40 | def options 41 | " -#{ @level }" if @level 42 | end 43 | 44 | end 45 | end 46 | end 47 | -------------------------------------------------------------------------------- /lib/backup/compressor/bzip2.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Compressor 5 | class Bzip2 < Base 6 | 7 | ## 8 | # Specify the level of compression to use. 9 | # 10 | # Values should be a single digit from 1 to 9. 11 | # Note that setting the level to either extreme may or may not 12 | # give the desired result. Be sure to check the documentation 13 | # for the compressor being used. 14 | # 15 | # The default `level` is 9. 16 | attr_accessor :level 17 | 18 | attr_deprecate :fast, :version => '3.0.24', 19 | :replacement => :level, 20 | :value => lambda {|val| val ? 1 : nil } 21 | attr_deprecate :best, :version => '3.0.24', 22 | :replacement => :level, 23 | :value => lambda {|val| val ? 9 : nil } 24 | 25 | ## 26 | # Creates a new instance of Backup::Compressor::Bzip2 27 | def initialize(&block) 28 | load_defaults! 29 | 30 | @level ||= false 31 | 32 | instance_eval(&block) if block_given? 33 | 34 | @cmd = "#{ utility(:bzip2) }#{ options }" 35 | @ext = '.bz2' 36 | end 37 | 38 | private 39 | 40 | def options 41 | " -#{ @level }" if @level 42 | end 43 | 44 | end 45 | end 46 | end 47 | -------------------------------------------------------------------------------- /backup.gemspec: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('lib/backup/version') 4 | 5 | Gem::Specification.new do |gem| 6 | 7 | ## 8 | # General configuration / information 9 | gem.name = 'backup' 10 | gem.version = Backup::Version.current 11 | gem.platform = Gem::Platform::RUBY 12 | gem.authors = 'Michael van Rooijen' 13 | gem.email = 'meskyanichi@gmail.com' 14 | gem.homepage = 'http://rubygems.org/gems/backup' 15 | gem.summary = 'Backup is a RubyGem, written for UNIX-like operating systems, that allows you to easily perform backup operations on both your remote and local environments. It provides you with an elegant DSL in Ruby for modeling your backups. Backup has built-in support for various databases, storage protocols/services, syncers, compressors, encryptors and notifiers which you can mix and match. It was built with modularity, extensibility and simplicity in mind.' 16 | 17 | ## 18 | # Files and folder that need to be compiled in to the Ruby Gem 19 | gem.files = %x[git ls-files].split("\n") 20 | gem.test_files = %x[git ls-files -- {spec}/*].split("\n") 21 | gem.require_path = 'lib' 22 | 23 | ## 24 | # The Backup CLI executable 25 | gem.executables = ['backup'] 26 | 27 | ## 28 | # Gem dependencies 29 | gem.add_dependency 'thor', ['~> 0.15.4'] 30 | gem.add_dependency 'open4', ['~> 1.3.0'] 31 | 32 | end 33 | -------------------------------------------------------------------------------- /lib/backup/syncer/rsync/base.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Syncer 5 | module RSync 6 | class Base < Syncer::Base 7 | ## 8 | # Additional options for the rsync cli 9 | attr_accessor :additional_options 10 | 11 | ## 12 | # Instantiates a new RSync Syncer object 13 | # and sets the default configuration 14 | def initialize 15 | super 16 | 17 | @additional_options ||= Array.new 18 | end 19 | 20 | private 21 | 22 | ## 23 | # Returns the @directories as a space-delimited string of 24 | # single-quoted values for use in the `rsync` command line. 25 | # Each path is expanded, since these refer to local paths 26 | # for both RSync::Local and RSync::Push. 27 | # RSync::Pull does not use this method. 28 | def directories_option 29 | @directories.map do |directory| 30 | "'#{ File.expand_path(directory) }'" 31 | end.join(' ') 32 | end 33 | 34 | ## 35 | # Returns Rsync syntax for enabling mirroring 36 | def mirror_option 37 | '--delete' if @mirror 38 | end 39 | 40 | ## 41 | # Returns Rsync syntax for invoking "archive" mode 42 | def archive_option 43 | '--archive' 44 | end 45 | 46 | end 47 | end 48 | end 49 | end 50 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | 2 | Copyright (c) 2009-2011 Michael van Rooijen ( [@meskyanichi](http://twitter.com/#!/meskyanichi) ) 3 | ================================================================================================= 4 | 5 | The "Backup" RubyGem is released under the **MIT LICENSE** 6 | ---------------------------------------------------------- 7 | 8 | Permission is hereby granted, free of charge, to any person obtaining a copy 9 | of this software and associated documentation files (the "Software"), to deal 10 | in the Software without restriction, including without limitation the rights 11 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 12 | copies of the Software, and to permit persons to whom the Software is 13 | furnished to do so, subject to the following conditions: 14 | 15 | The above copyright notice and this permission notice shall be included in 16 | all copies or substantial portions of the Software. 17 | 18 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 19 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 20 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 21 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 22 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 23 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 24 | THE SOFTWARE. 25 | -------------------------------------------------------------------------------- /lib/backup/compressor/lzma.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Compressor 5 | class Lzma < Base 6 | 7 | ## 8 | # Tells Backup::Compressor::Lzma to compress 9 | # better (-9) rather than faster when set to true 10 | attr_accessor :best 11 | 12 | ## 13 | # Tells Backup::Compressor::Lzma to compress 14 | # faster (-1) rather than better when set to true 15 | attr_accessor :fast 16 | 17 | ## 18 | # Creates a new instance of Backup::Compressor::Lzma 19 | def initialize(&block) 20 | load_defaults! 21 | 22 | @best ||= false 23 | @fast ||= false 24 | 25 | instance_eval(&block) if block_given? 26 | 27 | @cmd = "#{ utility(:lzma) }#{ options }" 28 | @ext = '.lzma' 29 | end 30 | 31 | 32 | ## 33 | # Yields to the block the compressor command and filename extension. 34 | def compress_with 35 | Backup::Logger.warn( 36 | "[DEPRECATION WARNING]\n" + 37 | " Compressor::Lzma is being deprecated as of backup v.3.0.24\n" + 38 | " and will soon be removed. Please see the Compressors wiki page at\n" + 39 | " https://github.com/meskyanichi/backup/wiki/Compressors" 40 | ) 41 | super 42 | end 43 | 44 | private 45 | 46 | def options 47 | (' --best' if @best) || (' --fast' if @fast) 48 | end 49 | 50 | end 51 | end 52 | end 53 | -------------------------------------------------------------------------------- /lib/backup/compressor/custom.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Compressor 5 | class Custom < Base 6 | 7 | ## 8 | # Specify the system command to invoke a compressor, 9 | # including any command-line arguments. 10 | # e.g. @compressor.command = 'pbzip2 -p2 -4' 11 | # 12 | # The data to be compressed will be piped to the command's STDIN, 13 | # and it should write the compressed data to STDOUT. 14 | # i.e. `cat file.tar | %command% > file.tar.%extension%` 15 | attr_accessor :command 16 | 17 | ## 18 | # File extension to append to the compressed file's filename. 19 | # e.g. @compressor.extension = '.bz2' 20 | attr_accessor :extension 21 | 22 | ## 23 | # Initializes a new custom compressor. 24 | def initialize(&block) 25 | load_defaults! 26 | 27 | instance_eval(&block) if block_given? 28 | 29 | @cmd = set_cmd 30 | @ext = set_ext 31 | end 32 | 33 | private 34 | 35 | ## 36 | # Return the command line using the full path. 37 | # Ensures the command exists and is executable. 38 | def set_cmd 39 | parts = @command.to_s.split(' ') 40 | parts[0] = utility(parts[0]) 41 | parts.join(' ') 42 | end 43 | 44 | ## 45 | # Return the extension given without whitespace. 46 | # If extension was not set, return an empty string 47 | def set_ext 48 | @extension.to_s.strip 49 | end 50 | 51 | end 52 | end 53 | end 54 | -------------------------------------------------------------------------------- /lib/backup/database/base.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Database 5 | class Base 6 | include Backup::CLI::Helpers 7 | include Backup::Configuration::Helpers 8 | 9 | ## 10 | # Creates a new instance of the MongoDB database object 11 | # * Called using super(model) from subclasses * 12 | def initialize(model) 13 | @model = model 14 | load_defaults! 15 | end 16 | 17 | ## 18 | # Super method for all child (database) objects. Every database object's #perform! 19 | # method should call #super before anything else to prepare 20 | def perform! 21 | prepare! 22 | log! 23 | end 24 | 25 | private 26 | 27 | ## 28 | # Defines the @dump_path and ensures it exists by creating it 29 | def prepare! 30 | @dump_path = File.join( 31 | Config.tmp_path, 32 | @model.trigger, 33 | 'databases', 34 | self.class.name.split('::').last 35 | ) 36 | FileUtils.mkdir_p(@dump_path) 37 | end 38 | 39 | ## 40 | # Return the database name, with Backup namespace removed 41 | def database_name 42 | self.class.to_s.sub('Backup::', '') 43 | end 44 | 45 | ## 46 | # Logs a message to the console and log file to inform 47 | # the client that Backup is dumping the database 48 | def log! 49 | Logger.message "#{ database_name } started dumping and archiving '#{ name }'." 50 | end 51 | end 52 | end 53 | end 54 | -------------------------------------------------------------------------------- /templates/cli/utility/syncer/cloud_files: -------------------------------------------------------------------------------- 1 | ## 2 | # Rackspace Cloud Files [Syncer] 3 | # 4 | # Available Auth URLs: 5 | # 6 | # - https://auth.api.rackspacecloud.com (US - Default) 7 | # - https://lon.auth.api.rackspacecloud.com (UK) 8 | # 9 | # Servicenet: 10 | # 11 | # Set this to 'true' if Backup runs on a Rackspace server. 12 | # It will avoid transfer charges and it's more performant. 13 | # 14 | # Mirroring: 15 | # 16 | # When enabled it will keep an exact mirror of your filesystem on Cloud Files. 17 | # This means that when you remove a file from the filesystem, 18 | # it will also remote it from Cloud Files. 19 | # 20 | # Concurrency: 21 | # 22 | # `concurrency_type` may be set to: 23 | # 24 | # - false (default) 25 | # - :threads 26 | # - :processes 27 | # 28 | # Set `concurrency_level` to the number of threads/processes to use. 29 | # Defaults to 2. 30 | # 31 | sync_with Cloud::CloudFiles do |cf| 32 | cf.username = "my_username" 33 | cf.api_key = "my_api_key" 34 | cf.container = "my_container" 35 | cf.auth_url = "https://auth.api.rackspacecloud.com" 36 | cf.servicenet = false 37 | cf.path = "/backups" 38 | cf.mirror = true 39 | cf.concurrency_type = false 40 | cf.concurrency_level = 2 41 | 42 | cf.directories do |directory| 43 | directory.add "/path/to/directory/to/sync" 44 | directory.add "/path/to/other/directory/to/sync" 45 | end 46 | end 47 | -------------------------------------------------------------------------------- /spec/spec_helper.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | ## 4 | # Use Bundler 5 | require 'rubygems' if RUBY_VERSION < '1.9' 6 | require 'bundler/setup' 7 | 8 | ## 9 | # Load Backup 10 | require 'backup' 11 | 12 | require 'timecop' 13 | 14 | module Backup::ExampleHelpers 15 | # ripped from MiniTest :) 16 | # RSpec doesn't have a method for this? Am I missing something? 17 | def capture_io 18 | require 'stringio' 19 | 20 | orig_stdout, orig_stderr = $stdout, $stderr 21 | captured_stdout, captured_stderr = StringIO.new, StringIO.new 22 | $stdout, $stderr = captured_stdout, captured_stderr 23 | 24 | yield 25 | 26 | return captured_stdout.string, captured_stderr.string 27 | ensure 28 | $stdout = orig_stdout 29 | $stderr = orig_stderr 30 | end 31 | end 32 | 33 | require 'rspec/autorun' 34 | RSpec.configure do |config| 35 | ## 36 | # Use Mocha to mock with RSpec 37 | config.mock_with :mocha 38 | 39 | ## 40 | # Example Helpers 41 | config.include Backup::ExampleHelpers 42 | 43 | ## 44 | # Actions to perform before each example 45 | config.before(:each) do 46 | FileUtils.collect_method(:noop).each do |method| 47 | FileUtils.stubs(method).raises("Unexpected call to FileUtils.#{ method }") 48 | end 49 | 50 | Open4.stubs(:popen4).raises('Unexpected call to Open4::popen4()') 51 | 52 | [:message, :error, :warn, :normal, :silent].each do |method| 53 | Backup::Logger.stubs(method).raises("Unexpected call to Backup::Logger.#{ method }") 54 | end 55 | end 56 | end 57 | 58 | unless @_put_ruby_version 59 | puts @_put_ruby_version = "\n\nRuby version: #{ RUBY_DESCRIPTION }\n\n" 60 | end 61 | -------------------------------------------------------------------------------- /spec/dependency_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../spec_helper.rb', __FILE__) 4 | 5 | describe Backup::Dependency do 6 | before do 7 | Backup::Dependency.stubs(:all).returns( 8 | 'net-sftp' => { 9 | :require => 'net/sftp', 10 | :version => '~> 2.0.5', 11 | :for => 'SFTP Protocol (SFTP Storage)' 12 | }) 13 | end 14 | 15 | describe ".load" do 16 | it "should load and require given dependency" do 17 | Backup::Dependency.expects(:gem).with("net-sftp", "~> 2.0.5") 18 | Backup::Dependency.expects(:require).with("net/sftp") 19 | Backup::Dependency.load("net-sftp") 20 | end 21 | 22 | context "on a missing dependency" do 23 | before do 24 | Backup::Dependency.stubs(:gem).raises(LoadError) 25 | end 26 | 27 | it "should display error message" do 28 | Backup::Logger.expects(:error).with do |exception| 29 | exception.message.should == "Dependency::LoadError: Dependency missing 30 | Dependency required for: 31 | SFTP Protocol (SFTP Storage) 32 | To install the gem, issue the following command: 33 | > gem install net-sftp -v '~> 2.0.5' 34 | Please try again after installing the missing dependency." 35 | end 36 | 37 | expect do 38 | Backup::Dependency.load("net-sftp") 39 | end.to raise_error(SystemExit) 40 | end 41 | 42 | it "should exit with status code 1" do 43 | Backup::Logger.expects(:error) 44 | 45 | expect do 46 | Backup::Dependency.load("net-sftp") 47 | end.to raise_error { |exit| exit.status.should be(1) } 48 | end 49 | end 50 | end 51 | end 52 | -------------------------------------------------------------------------------- /lib/backup/syncer/rsync/local.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Syncer 5 | module RSync 6 | class Local < Base 7 | 8 | ## 9 | # Instantiates a new RSync::Local Syncer. 10 | # 11 | # Pre-configured defaults specified in 12 | # Configuration::Syncer::RSync::Local 13 | # are set via a super() call to RSync::Base, 14 | # which in turn will invoke Syncer::Base. 15 | # 16 | # Once pre-configured defaults and RSync specific defaults are set, 17 | # the block from the user's configuration file is evaluated. 18 | def initialize(&block) 19 | super 20 | 21 | instance_eval(&block) if block_given? 22 | end 23 | 24 | ## 25 | # Performs the RSync::Local operation 26 | # debug options: -vhP 27 | def perform! 28 | Logger.message( 29 | "#{ syncer_name } started syncing the following directories:\n\s\s" + 30 | @directories.join("\n\s\s") 31 | ) 32 | run("#{ utility(:rsync) } #{ options } " + 33 | "#{ directories_option } '#{ dest_path }'") 34 | end 35 | 36 | private 37 | 38 | ## 39 | # Return expanded @path 40 | def dest_path 41 | @dest_path ||= File.expand_path(@path) 42 | end 43 | 44 | ## 45 | # Returns all the specified Rsync::Local options, 46 | # concatenated, ready for the CLI 47 | def options 48 | ([archive_option, mirror_option] + 49 | additional_options).compact.join("\s") 50 | end 51 | 52 | end 53 | end 54 | end 55 | end 56 | -------------------------------------------------------------------------------- /spec/compressor/base_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../../spec_helper.rb', __FILE__) 4 | 5 | describe Backup::Compressor::Base do 6 | let(:compressor) { Backup::Compressor::Base.new } 7 | 8 | it 'should include CLI::Helpers' do 9 | Backup::Compressor::Base. 10 | include?(Backup::CLI::Helpers).should be_true 11 | end 12 | 13 | it 'should include Configuration::Helpers' do 14 | Backup::Compressor::Base. 15 | include?(Backup::Configuration::Helpers).should be_true 16 | end 17 | 18 | describe '#compress_with' do 19 | it 'should yield the compressor command and extension' do 20 | compressor.instance_variable_set(:@cmd, 'compressor command') 21 | compressor.instance_variable_set(:@ext, 'compressor extension') 22 | 23 | compressor.expects(:log!) 24 | 25 | compressor.compress_with do |cmd, ext| 26 | cmd.should == 'compressor command' 27 | ext.should == 'compressor extension' 28 | end 29 | end 30 | end 31 | 32 | describe '#compressor_name' do 33 | it 'should return class name with Backup namespace removed' do 34 | compressor.send(:compressor_name).should == 'Compressor::Base' 35 | end 36 | end 37 | 38 | describe '#log!' do 39 | it 'should log a message' do 40 | compressor.instance_variable_set(:@cmd, 'compressor command') 41 | compressor.instance_variable_set(:@ext, 'compressor extension') 42 | compressor.expects(:compressor_name).returns('Compressor Name') 43 | 44 | Backup::Logger.expects(:message).with( 45 | "Using Compressor Name for compression.\n" + 46 | " Command: 'compressor command'\n" + 47 | " Ext: 'compressor extension'" 48 | ) 49 | compressor.send(:log!) 50 | end 51 | end 52 | end 53 | -------------------------------------------------------------------------------- /lib/backup/compressor/pbzip2.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Compressor 5 | class Pbzip2 < Base 6 | 7 | ## 8 | # Tells Backup::Compressor::Lzma to compress 9 | # better (-9) rather than faster when set to true 10 | attr_accessor :best 11 | 12 | ## 13 | # Tells Backup::Compressor::Lzma to compress 14 | # faster (-1) rather than better when set to true 15 | attr_accessor :fast 16 | 17 | ## 18 | # Tells Backup::Compressor::Pbzip2 how many processors to use. 19 | # Autodetects the number of active CPUs by default. 20 | attr_accessor :processors 21 | 22 | ## 23 | # Creates a new instance of Backup::Compressor::Pbzip2 24 | def initialize(&block) 25 | load_defaults! 26 | 27 | @best ||= false 28 | @fast ||= false 29 | @processors ||= false 30 | 31 | instance_eval(&block) if block_given? 32 | 33 | @cmd = "#{ utility(:pbzip2) }#{ options }" 34 | @ext = '.bz2' 35 | end 36 | 37 | ## 38 | # Yields to the block the compressor command and filename extension. 39 | def compress_with 40 | Backup::Logger.warn( 41 | "[DEPRECATION WARNING]\n" + 42 | " Compressor::Pbzip2 is being deprecated as of backup v.3.0.24\n" + 43 | " and will soon be removed. Please see the Compressors wiki page at\n" + 44 | " https://github.com/meskyanichi/backup/wiki/Compressors" 45 | ) 46 | super 47 | end 48 | 49 | private 50 | 51 | def options 52 | level = (' --best' if @best) || (' --fast' if @fast) 53 | cpus = " -p#{ @processors }" if @processors 54 | "#{ level }#{ cpus }" 55 | end 56 | 57 | end 58 | end 59 | end 60 | -------------------------------------------------------------------------------- /spec/configuration_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../spec_helper.rb', __FILE__) 4 | 5 | describe 'Backup::Configuration' do 6 | 7 | after do 8 | Backup::Configuration.send(:remove_const, 'Foo') 9 | end 10 | 11 | it 'should create modules for missing constants' do 12 | Backup::Configuration::Foo.class.should == Module 13 | end 14 | 15 | describe 'a generated module' do 16 | 17 | before do 18 | module Backup 19 | class Foo; end 20 | end 21 | end 22 | 23 | after do 24 | Backup.send(:remove_const, 'Foo') 25 | end 26 | 27 | it 'should create modules for missing constants' do 28 | Backup::Configuration::Foo::A::B.class.should == Module 29 | end 30 | 31 | it 'should pass calls to .defaults to the proper class' do 32 | Backup::Logger.expects(:warn) 33 | Backup::Foo.expects(:defaults) 34 | Backup::Configuration::Foo.defaults 35 | end 36 | 37 | it 'should pass a given block to .defaults to the proper class' do 38 | Backup::Logger.expects(:warn) 39 | configuration = mock 40 | Backup::Foo.expects(:defaults).yields(configuration) 41 | configuration.expects(:foo=).with('bar') 42 | 43 | Backup::Configuration::Foo.defaults do |config| 44 | config.foo = 'bar' 45 | end 46 | end 47 | 48 | it 'should log a deprecation warning' do 49 | Backup::Foo.stubs(:defaults) 50 | Backup::Logger.expects(:warn).with do |err| 51 | err.message.should == 52 | "ConfigurationError: [DEPRECATION WARNING]\n" + 53 | " Backup::Configuration::Foo.defaults is being deprecated.\n" + 54 | " To set pre-configured defaults for Backup::Foo, use:\n" + 55 | " Backup::Foo.defaults" 56 | end 57 | Backup::Configuration::Foo.defaults 58 | end 59 | 60 | end 61 | 62 | end 63 | -------------------------------------------------------------------------------- /lib/backup/database/riak.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Database 5 | class Riak < Base 6 | 7 | ## 8 | # Name is the name of the backup 9 | attr_accessor :name 10 | 11 | ## 12 | # Node is the node from which to perform the backup. 13 | attr_accessor :node 14 | 15 | ## 16 | # Cookie is the Erlang cookie/shared secret used to connect to the node. 17 | attr_accessor :cookie 18 | 19 | ## 20 | # Path to riak-admin utility (optional) 21 | attr_accessor :riak_admin_utility 22 | 23 | attr_deprecate :utility_path, :version => '3.0.21', 24 | :replacement => :riak_admin_utility 25 | 26 | ## 27 | # Creates a new instance of the Riak adapter object 28 | def initialize(model, &block) 29 | super(model) 30 | 31 | instance_eval(&block) if block_given? 32 | 33 | @riak_admin_utility ||= utility('riak-admin') 34 | end 35 | 36 | ## 37 | # Performs the riak-admin command and outputs the 38 | # data to the specified path based on the 'trigger' 39 | def perform! 40 | super 41 | # have to make riak the owner since the riak-admin tool runs 42 | # as the riak user in a default setup. 43 | FileUtils.chown_R('riak', 'riak', @dump_path) 44 | 45 | backup_file = File.join(@dump_path, name) 46 | run("#{ riakadmin } #{ backup_file } node") 47 | 48 | if @model.compressor 49 | @model.compressor.compress_with do |command, ext| 50 | run("#{ command } -c #{ backup_file } > #{ backup_file + ext }") 51 | FileUtils.rm_f(backup_file) 52 | end 53 | end 54 | end 55 | 56 | private 57 | 58 | ## 59 | # Builds the full riak-admin string based on all attributes 60 | def riakadmin 61 | "#{ riak_admin_utility } backup #{ node } #{ cookie }" 62 | end 63 | 64 | end 65 | end 66 | end 67 | -------------------------------------------------------------------------------- /spec/database/base_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../../spec_helper.rb', __FILE__) 4 | 5 | describe Backup::Database::Base do 6 | let(:model) { Backup::Model.new('foo', 'foo') } 7 | let(:db) { Backup::Database::Base.new(model) } 8 | 9 | it 'should include CLI::Helpers' do 10 | Backup::Database::Base. 11 | include?(Backup::CLI::Helpers).should be_true 12 | end 13 | 14 | it 'should include Configuration::Helpers' do 15 | Backup::Database::Base. 16 | include?(Backup::Configuration::Helpers).should be_true 17 | end 18 | 19 | describe '#initialize' do 20 | it 'should load pre-configured defaults' do 21 | Backup::Database::Base.any_instance.expects(:load_defaults!) 22 | db 23 | end 24 | 25 | it 'should set a reference to the model' do 26 | db.instance_variable_get(:@model).should == model 27 | end 28 | end 29 | 30 | describe '#perform!' do 31 | it 'should invoke prepare! and log!' do 32 | s = sequence '' 33 | db.expects(:prepare!).in_sequence(s) 34 | db.expects(:log!).in_sequence(s) 35 | 36 | db.perform! 37 | end 38 | end 39 | 40 | describe '#prepare!' do 41 | it 'should set and create #dump_path' do 42 | model = stub(:trigger => 'test_trigger') 43 | db.instance_variable_set(:@model, model) 44 | FileUtils.expects(:mkdir_p).with( 45 | File.join(Backup::Config.tmp_path, 'test_trigger', 'databases', 'Base') 46 | ) 47 | db.send(:prepare!) 48 | db.instance_variable_get(:@dump_path).should == 49 | File.join(Backup::Config.tmp_path, 'test_trigger', 'databases', 'Base') 50 | end 51 | end 52 | 53 | describe '#log!' do 54 | it 'should use #database_name' do 55 | db.stubs(:name).returns('database_name') 56 | Backup::Logger.expects(:message).with( 57 | "Database::Base started dumping and archiving 'database_name'." 58 | ) 59 | 60 | db.send(:log!) 61 | end 62 | end 63 | end 64 | -------------------------------------------------------------------------------- /lib/backup/notifier/prowl.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | ## 4 | # Only load the Prowler gem when using Prowler notifications 5 | Backup::Dependency.load('prowler') 6 | 7 | module Backup 8 | module Notifier 9 | class Prowl < Base 10 | 11 | ## 12 | # Application name 13 | # Tell something like your server name. Example: "Server1 Backup" 14 | attr_accessor :application 15 | 16 | ## 17 | # API-Key 18 | # Create a Prowl account and request an API key on prowlapp.com. 19 | attr_accessor :api_key 20 | 21 | def initialize(model, &block) 22 | super(model) 23 | 24 | instance_eval(&block) if block_given? 25 | end 26 | 27 | private 28 | 29 | ## 30 | # Notify the user of the backup operation results. 31 | # `status` indicates one of the following: 32 | # 33 | # `:success` 34 | # : The backup completed successfully. 35 | # : Notification will be sent if `on_success` was set to `true` 36 | # 37 | # `:warning` 38 | # : The backup completed successfully, but warnings were logged 39 | # : Notification will be sent, including a copy of the current 40 | # : backup log, if `on_warning` was set to `true` 41 | # 42 | # `:failure` 43 | # : The backup operation failed. 44 | # : Notification will be sent, including the Exception which caused 45 | # : the failure, the Exception's backtrace, a copy of the current 46 | # : backup log and other information if `on_failure` was set to `true` 47 | # 48 | def notify!(status) 49 | name = case status 50 | when :success then 'Success' 51 | when :warning then 'Warning' 52 | when :failure then 'Failure' 53 | end 54 | message = '[Backup::%s]' % name 55 | send_message(message) 56 | end 57 | 58 | def send_message(message) 59 | client = Prowler.new(:application => application, :api_key => api_key) 60 | client.notify(message, "#{@model.label} (#{@model.trigger})") 61 | end 62 | 63 | end 64 | end 65 | end 66 | -------------------------------------------------------------------------------- /lib/backup/syncer/cloud/s3.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Syncer 5 | module Cloud 6 | class S3 < Base 7 | 8 | ## 9 | # Amazon Simple Storage Service (S3) Credentials 10 | attr_accessor :access_key_id, :secret_access_key 11 | 12 | ## 13 | # The S3 bucket to store files to 14 | attr_accessor :bucket 15 | 16 | ## 17 | # The AWS region of the specified S3 bucket 18 | attr_accessor :region 19 | 20 | ## 21 | # Instantiates a new Cloud::S3 Syncer. 22 | # 23 | # Pre-configured defaults specified in 24 | # Configuration::Syncer::Cloud::S3 25 | # are set via a super() call to Cloud::Base, 26 | # which in turn will invoke Syncer::Base. 27 | # 28 | # Once pre-configured defaults and Cloud specific defaults are set, 29 | # the block from the user's configuration file is evaluated. 30 | def initialize(&block) 31 | super 32 | 33 | instance_eval(&block) if block_given? 34 | @path = path.sub(/^\//, '') 35 | end 36 | 37 | private 38 | 39 | ## 40 | # Established and creates a new Fog storage object for S3. 41 | def connection 42 | @connection ||= Fog::Storage.new( 43 | :provider => provider, 44 | :aws_access_key_id => access_key_id, 45 | :aws_secret_access_key => secret_access_key, 46 | :region => region 47 | ) 48 | end 49 | 50 | ## 51 | # Creates a new @repository_object (bucket). 52 | # Fetches it from S3 if it already exists, 53 | # otherwise it will create it first and fetch use that instead. 54 | def repository_object 55 | @repository_object ||= connection.directories.get(bucket) || 56 | connection.directories.create(:key => bucket, :location => region) 57 | end 58 | 59 | ## 60 | # This is the provider that Fog uses for the Cloud Files 61 | def provider 62 | "AWS" 63 | end 64 | 65 | end # Class S3 < Base 66 | end # module Cloud 67 | end 68 | end 69 | -------------------------------------------------------------------------------- /spec/package_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../spec_helper.rb', __FILE__) 4 | 5 | describe Backup::Package do 6 | let(:model) { Backup::Model.new(:test_trigger, 'test label') } 7 | let(:package) { Backup::Package.new(model) } 8 | 9 | before do 10 | model.instance_variable_set(:@time, 'model_time') 11 | end 12 | 13 | describe '#initialize' do 14 | it 'should set all variables' do 15 | package.time.should == 'model_time' 16 | package.trigger.should == 'test_trigger' 17 | package.extension.should == 'tar' 18 | package.chunk_suffixes.should == [] 19 | package.version.should == Backup::Version.current 20 | end 21 | end 22 | 23 | describe '#filenames' do 24 | context 'when the package files were not split' do 25 | it 'should return an array with the single package filename' do 26 | package.filenames.should == ['model_time.test_trigger.tar'] 27 | end 28 | 29 | it 'should reflect changes in the extension' do 30 | package.extension << '.enc' 31 | package.filenames.should == ['model_time.test_trigger.tar.enc'] 32 | end 33 | end 34 | 35 | context 'when the package files were split' do 36 | before { package.chunk_suffixes = ['aa', 'ab'] } 37 | it 'should return an array of the package filenames' do 38 | package.filenames.should == ['model_time.test_trigger.tar-aa', 39 | 'model_time.test_trigger.tar-ab'] 40 | end 41 | 42 | it 'should reflect changes in the extension' do 43 | package.extension << '.enc' 44 | package.filenames.should == ['model_time.test_trigger.tar.enc-aa', 45 | 'model_time.test_trigger.tar.enc-ab'] 46 | end 47 | end 48 | end 49 | 50 | describe '#basename' do 51 | it 'should return the base filename for the package' do 52 | package.basename.should == 'model_time.test_trigger.tar' 53 | end 54 | 55 | it 'should reflect changes in the extension' do 56 | package.extension << '.enc' 57 | package.basename.should == 'model_time.test_trigger.tar.enc' 58 | end 59 | end 60 | 61 | end 62 | -------------------------------------------------------------------------------- /lib/backup/notifier/twitter.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | ## 4 | # Only load the Twitter gem when using Twitter notifications 5 | Backup::Dependency.load('twitter') 6 | 7 | module Backup 8 | module Notifier 9 | class Twitter < Base 10 | 11 | ## 12 | # Twitter consumer key credentials 13 | attr_accessor :consumer_key, :consumer_secret 14 | 15 | ## 16 | # OAuth credentials 17 | attr_accessor :oauth_token, :oauth_token_secret 18 | 19 | def initialize(model, &block) 20 | super(model) 21 | 22 | instance_eval(&block) if block_given? 23 | end 24 | 25 | private 26 | 27 | ## 28 | # Notify the user of the backup operation results. 29 | # `status` indicates one of the following: 30 | # 31 | # `:success` 32 | # : The backup completed successfully. 33 | # : Notification will be sent if `on_success` was set to `true` 34 | # 35 | # `:warning` 36 | # : The backup completed successfully, but warnings were logged 37 | # : Notification will be sent, including a copy of the current 38 | # : backup log, if `on_warning` was set to `true` 39 | # 40 | # `:failure` 41 | # : The backup operation failed. 42 | # : Notification will be sent, including the Exception which caused 43 | # : the failure, the Exception's backtrace, a copy of the current 44 | # : backup log and other information if `on_failure` was set to `true` 45 | # 46 | def notify!(status) 47 | name = case status 48 | when :success then 'Success' 49 | when :warning then 'Warning' 50 | when :failure then 'Failure' 51 | end 52 | message = "[Backup::%s] #{@model.label} (#{@model.trigger}) (@ #{@model.time})" % name 53 | send_message(message) 54 | end 55 | 56 | def send_message(message) 57 | ::Twitter.configure do |config| 58 | config.consumer_key = @consumer_key 59 | config.consumer_secret = @consumer_secret 60 | config.oauth_token = @oauth_token 61 | config.oauth_token_secret = @oauth_token_secret 62 | end 63 | 64 | client = ::Twitter::Client.new 65 | client.update(message) 66 | end 67 | 68 | end 69 | end 70 | end 71 | -------------------------------------------------------------------------------- /lib/backup/encryptor/open_ssl.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Encryptor 5 | class OpenSSL < Base 6 | 7 | ## 8 | # The password that'll be used to encrypt the backup. This 9 | # password will be required to decrypt the backup later on. 10 | attr_accessor :password 11 | 12 | ## 13 | # The password file to use to encrypt the backup. 14 | attr_accessor :password_file 15 | 16 | ## 17 | # Determines whether the 'base64' should be used or not 18 | attr_accessor :base64 19 | 20 | ## 21 | # Determines whether the 'salt' flag should be used 22 | attr_accessor :salt 23 | 24 | ## 25 | # Creates a new instance of Backup::Encryptor::OpenSSL and 26 | # sets the password attribute to what was provided 27 | def initialize(&block) 28 | super 29 | 30 | @base64 ||= false 31 | @salt ||= true 32 | @password_file ||= nil 33 | 34 | instance_eval(&block) if block_given? 35 | end 36 | 37 | ## 38 | # This is called as part of the procedure run by the Packager. 39 | # It sets up the needed options to pass to the openssl command, 40 | # then yields the command to use as part of the packaging procedure. 41 | # Once the packaging procedure is complete, it will return 42 | # so that any clean-up may be performed after the yield. 43 | def encrypt_with 44 | log! 45 | yield "#{ utility(:openssl) } #{ options }", '.enc' 46 | end 47 | 48 | private 49 | 50 | ## 51 | # Uses the 256bit AES encryption cipher, which is what the 52 | # US Government uses to encrypt information at the "Top Secret" level. 53 | # 54 | # The -base64 option will make the encrypted output base64 encoded, 55 | # this makes the encrypted file readable using text editors 56 | # 57 | # The -salt option adds strength to the encryption 58 | # 59 | # Always sets a password option, if even no password is given, 60 | # but will prefer the password_file option if both are given. 61 | def options 62 | opts = ['aes-256-cbc'] 63 | opts << '-base64' if @base64 64 | opts << '-salt' if @salt 65 | opts << ( @password_file.to_s.empty? ? 66 | "-k '#{@password}'" : "-pass file:#{@password_file}" ) 67 | opts.join(' ') 68 | end 69 | 70 | end 71 | end 72 | end 73 | -------------------------------------------------------------------------------- /spec-live/spec_helper.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | ## # Use Bundler 4 | require 'rubygems' if RUBY_VERSION < '1.9' 5 | require 'bundler/setup' 6 | 7 | ## 8 | # Load Backup 9 | require 'backup' 10 | 11 | module Backup 12 | module SpecLive 13 | PATH = File.expand_path('..', __FILE__) 14 | # to archive local backups, etc... 15 | TMP_PATH = PATH + '/tmp' 16 | SYNC_PATH = PATH + '/sync' 17 | 18 | config = PATH + '/backups/config.yml' 19 | if File.exist?(config) 20 | CONFIG = YAML.load_file(config) 21 | else 22 | puts "The 'spec-live/backups/config.yml' file is required." 23 | puts "Use 'spec-live/backups/config.yml.template' to create one" 24 | exit! 25 | end 26 | 27 | module ExampleHelpers 28 | 29 | def h_set_trigger(trigger) 30 | Backup::Logger.clear! 31 | Backup::Model.all.clear 32 | Backup::Config.load_config! 33 | FileUtils.mkdir_p(File.join(Backup::Config.data_path, trigger)) 34 | Backup::Model.find(trigger) 35 | end 36 | 37 | def h_clean_data_paths! 38 | paths = [:data_path, :log_path, :tmp_path ].map do |name| 39 | Backup::Config.send(name) 40 | end + [Backup::SpecLive::TMP_PATH] 41 | paths.each do |path| 42 | h_safety_check(path) 43 | FileUtils.rm_rf(path) 44 | FileUtils.mkdir_p(path) 45 | end 46 | end 47 | 48 | def h_safety_check(path) 49 | # Rule #1: Do No Harm. 50 | unless ( 51 | path.start_with?(Backup::SpecLive::PATH) && 52 | Backup::SpecLive::PATH.end_with?('spec-live') 53 | ) || path.include?('spec_live_test_dir') 54 | warn "\nSafety Check Failed:\nPath: #{path}\n\n" + 55 | caller(1).join("\n") 56 | exit! 57 | end 58 | end 59 | 60 | end # ExampleHelpers 61 | end 62 | 63 | Config.update(:root_path => SpecLive::PATH + '/backups') 64 | 65 | Logger.quiet = true unless ENV['VERBOSE'] 66 | end 67 | 68 | ## 69 | # Use Mocha to mock with RSpec 70 | require 'rspec' 71 | RSpec.configure do |config| 72 | config.mock_with :mocha 73 | config.include Backup::SpecLive::ExampleHelpers 74 | config.before(:each) do 75 | h_clean_data_paths! 76 | if ENV['VERBOSE'] 77 | /spec-live\/(.*):/ =~ self.example.metadata[:example_group][:block].inspect 78 | puts "\n\nSPEC: #{$1}" 79 | puts "DESC: #{self.example.metadata[:full_description]}" 80 | puts '-' * 78 81 | end 82 | end 83 | end 84 | 85 | puts "\n\nRuby version: #{RUBY_DESCRIPTION}\n\n" 86 | -------------------------------------------------------------------------------- /spec-live/storage/local_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../../spec_helper.rb', __FILE__) 4 | 5 | describe 'Storage::Local' do 6 | let(:trigger) { 'archive_local' } 7 | 8 | def archive_file_for(model) 9 | File.join( 10 | Backup::SpecLive::TMP_PATH, 11 | "#{model.trigger}", model.time, "#{model.trigger}.tar" 12 | ) 13 | end 14 | 15 | it 'should store a local archive' do 16 | model = h_set_trigger(trigger) 17 | model.perform! 18 | File.exist?(archive_file_for(model)).should be_true 19 | end 20 | 21 | describe 'Storage::Local Cycling' do 22 | 23 | context 'when archives exceed `keep` setting' do 24 | it 'should remove the oldest archive' do 25 | archives = [] 26 | 27 | model = h_set_trigger(trigger) 28 | model.perform! 29 | archives << archive_file_for(model) 30 | sleep 1 31 | 32 | model = h_set_trigger(trigger) 33 | model.perform! 34 | archives << archive_file_for(model) 35 | sleep 1 36 | 37 | model = h_set_trigger(trigger) 38 | model.perform! 39 | archives << archive_file_for(model) 40 | 41 | File.exist?(archives[0]).should be_false 42 | File.exist?(archives[1]).should be_true 43 | File.exist?(archives[2]).should be_true 44 | end 45 | end 46 | 47 | context 'when an archive to be removed does not exist' do 48 | it 'should log a warning and continue' do 49 | archives = [] 50 | 51 | model = h_set_trigger(trigger) 52 | model.perform! 53 | archives << archive_file_for(model) 54 | sleep 1 55 | 56 | model = h_set_trigger(trigger) 57 | model.perform! 58 | archives << archive_file_for(model) 59 | sleep 1 60 | 61 | File.exist?(archives[0]).should be_true 62 | File.exist?(archives[1]).should be_true 63 | # remove archive directory cycle! will attempt to remove 64 | dir = archives[0].split('/')[0...-1].join('/') 65 | h_safety_check(dir) 66 | FileUtils.rm_r(dir) 67 | File.exist?(archives[0]).should be_false 68 | 69 | expect do 70 | model = h_set_trigger(trigger) 71 | model.perform! 72 | archives << archive_file_for(model) 73 | end.not_to raise_error 74 | 75 | Backup::Logger.has_warnings?.should be_true 76 | 77 | File.exist?(archives[1]).should be_true 78 | File.exist?(archives[2]).should be_true 79 | end 80 | end 81 | 82 | end # describe 'Storage::Local Cycling' 83 | end 84 | -------------------------------------------------------------------------------- /lib/backup/syncer/cloud/cloud_files.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Syncer 5 | module Cloud 6 | class CloudFiles < Base 7 | 8 | ## 9 | # Rackspace CloudFiles Credentials 10 | attr_accessor :api_key, :username 11 | 12 | ## 13 | # Rackspace CloudFiles Container 14 | attr_accessor :container 15 | 16 | ## 17 | # Rackspace AuthURL allows you to connect 18 | # to a different Rackspace datacenter 19 | # - https://auth.api.rackspacecloud.com (Default: US) 20 | # - https://lon.auth.api.rackspacecloud.com (UK) 21 | attr_accessor :auth_url 22 | 23 | ## 24 | # Improve performance and avoid data transfer costs 25 | # by setting @servicenet to `true` 26 | # This only works if Backup runs on a Rackspace server 27 | attr_accessor :servicenet 28 | 29 | ## 30 | # Instantiates a new Cloud::CloudFiles Syncer. 31 | # 32 | # Pre-configured defaults specified in 33 | # Configuration::Syncer::Cloud::CloudFiles 34 | # are set via a super() call to Cloud::Base, 35 | # which in turn will invoke Syncer::Base. 36 | # 37 | # Once pre-configured defaults and Cloud specific defaults are set, 38 | # the block from the user's configuration file is evaluated. 39 | def initialize(&block) 40 | super 41 | 42 | instance_eval(&block) if block_given? 43 | @path = path.sub(/^\//, '') 44 | end 45 | 46 | private 47 | 48 | ## 49 | # Established and creates a new Fog storage object for CloudFiles. 50 | def connection 51 | @connection ||= Fog::Storage.new( 52 | :provider => provider, 53 | :rackspace_username => username, 54 | :rackspace_api_key => api_key, 55 | :rackspace_auth_url => auth_url, 56 | :rackspace_servicenet => servicenet 57 | ) 58 | end 59 | 60 | ## 61 | # Creates a new @repository_object (container). 62 | # Fetches it from Cloud Files if it already exists, 63 | # otherwise it will create it first and fetch use that instead. 64 | def repository_object 65 | @repository_object ||= connection.directories.get(container) || 66 | connection.directories.create(:key => container) 67 | end 68 | 69 | ## 70 | # This is the provider that Fog uses for the Cloud Files 71 | def provider 72 | "Rackspace" 73 | end 74 | 75 | end # class Cloudfiles < Base 76 | end # module Cloud 77 | end 78 | end 79 | -------------------------------------------------------------------------------- /lib/backup/splitter.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | class Splitter 5 | include Backup::CLI::Helpers 6 | 7 | def initialize(model, chunk_size) 8 | @model = model 9 | @chunk_size = chunk_size 10 | end 11 | 12 | ## 13 | # This is called as part of the procedure used to build the final 14 | # backup package file(s). It yields it's portion of the command line 15 | # for this procedure, which will split the data being piped into it 16 | # into multiple files, based on the @chunk_size. 17 | # Once the packaging procedure is complete, it will return and 18 | # @package.chunk_suffixes will be set based on the resulting files. 19 | def split_with 20 | before_packaging 21 | yield @split_command 22 | after_packaging 23 | end 24 | 25 | private 26 | 27 | ## 28 | # The `split` command reads from $stdin and will store it's output in 29 | # multiple files, based on the @chunk_size. The files will be 30 | # written using the given `prefix`, which is the full path to the 31 | # final @package.basename, plus a '-' separator. This `prefix` will then 32 | # be suffixed using 'aa', 'ab', and so on... for each file. 33 | def before_packaging 34 | @package = @model.package 35 | Logger.message "Splitter configured with a chunk size of " + 36 | "#{ @chunk_size }MB." 37 | 38 | @split_command = "#{ utility(:split) } -b #{ @chunk_size }m - " + 39 | "'#{ File.join(Config.tmp_path, @package.basename + '-') }'" 40 | end 41 | 42 | ## 43 | # Finds the resulting files from the packaging procedure 44 | # and stores an Array of suffixes used in @package.chunk_suffixes. 45 | # If the @chunk_size was never reached and only one file 46 | # was written, that file will be suffixed with '-aa'. 47 | # In which case, it will simply remove the suffix from the filename. 48 | def after_packaging 49 | suffixes = chunk_suffixes 50 | if suffixes == ['aa'] 51 | FileUtils.mv( 52 | File.join(Config.tmp_path, @package.basename + '-aa'), 53 | File.join(Config.tmp_path, @package.basename) 54 | ) 55 | else 56 | @package.chunk_suffixes = suffixes 57 | end 58 | end 59 | 60 | ## 61 | # Returns an array of suffixes for each chunk, in alphabetical order. 62 | # For example: [aa, ab, ac, ad, ae] 63 | def chunk_suffixes 64 | chunks.map {|chunk| File.extname(chunk).split('-').last }.sort 65 | end 66 | 67 | ## 68 | # Returns an array of full paths to the backup chunks. 69 | # Chunks are sorted in alphabetical order. 70 | def chunks 71 | Dir[File.join(Config.tmp_path, @package.basename + '-*')].sort 72 | end 73 | 74 | end 75 | end 76 | -------------------------------------------------------------------------------- /lib/backup/notifier/base.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Notifier 5 | class Base 6 | include Backup::Configuration::Helpers 7 | 8 | ## 9 | # When set to true, the user will be notified by email 10 | # when a backup process ends without raising any exceptions 11 | attr_accessor :on_success 12 | alias :notify_on_success? :on_success 13 | 14 | ## 15 | # When set to true, the user will be notified by email 16 | # when a backup process is successful, but has warnings 17 | attr_accessor :on_warning 18 | alias :notify_on_warning? :on_warning 19 | 20 | ## 21 | # When set to true, the user will be notified by email 22 | # when a backup process raises an exception before finishing 23 | attr_accessor :on_failure 24 | alias :notify_on_failure? :on_failure 25 | 26 | ## 27 | # Called with super(model) from subclasses 28 | def initialize(model) 29 | @model = model 30 | load_defaults! 31 | 32 | @on_success = true if on_success.nil? 33 | @on_warning = true if on_warning.nil? 34 | @on_failure = true if on_failure.nil? 35 | end 36 | 37 | ## 38 | # Performs the notification 39 | # Takes a flag to indicate that a failure has occured. 40 | # (this is only set from Model#perform! in the event of an error) 41 | # If this is the case it will set the 'action' to :failure. 42 | # Otherwise, it will set the 'action' to either :success or :warning, 43 | # depending on whether or not any warnings were sent to the Logger. 44 | # It will then invoke the notify! method with the 'action', 45 | # but only if the proper on_success, on_warning or on_failure flag is true. 46 | def perform!(failure = false) 47 | @template = Backup::Template.new({:model => @model}) 48 | 49 | action = false 50 | if failure 51 | action = :failure if notify_on_failure? 52 | else 53 | if notify_on_success? || (notify_on_warning? && Logger.has_warnings?) 54 | action = Logger.has_warnings? ? :warning : :success 55 | end 56 | end 57 | 58 | if action 59 | log! 60 | notify!(action) 61 | end 62 | end 63 | 64 | private 65 | 66 | ## 67 | # Return the notifier name, with Backup namespace removed 68 | def notifier_name 69 | self.class.to_s.sub('Backup::', '') 70 | end 71 | 72 | ## 73 | # Logs a message to the console and log file to inform 74 | # the client that Backup is notifying about the process 75 | def log! 76 | Logger.message "#{ notifier_name } started notifying about the process." 77 | end 78 | 79 | end 80 | end 81 | end 82 | -------------------------------------------------------------------------------- /lib/backup/storage/local.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Storage 5 | class Local < Base 6 | 7 | ## 8 | # Path where the backup will be stored. 9 | attr_accessor :path 10 | 11 | ## 12 | # Creates a new instance of the storage object 13 | def initialize(model, storage_id = nil, &block) 14 | super(model, storage_id) 15 | 16 | @path ||= File.join( 17 | File.expand_path(ENV['HOME'] || ''), 18 | 'backups' 19 | ) 20 | 21 | instance_eval(&block) if block_given? 22 | 23 | @path = File.expand_path(@path) 24 | end 25 | 26 | private 27 | 28 | ## 29 | # Transfers the archived file to the specified path 30 | def transfer! 31 | remote_path = remote_path_for(@package) 32 | FileUtils.mkdir_p(remote_path) 33 | 34 | files_to_transfer_for(@package) do |local_file, remote_file| 35 | Logger.message "#{storage_name} started transferring '#{ local_file }'." 36 | 37 | src_path = File.join(local_path, local_file) 38 | dst_path = File.join(remote_path, remote_file) 39 | FileUtils.send(transfer_method, src_path, dst_path) 40 | end 41 | end 42 | 43 | ## 44 | # Removes the transferred archive file(s) from the storage location. 45 | # Any error raised will be rescued during Cycling 46 | # and a warning will be logged, containing the error message. 47 | def remove!(package) 48 | remote_path = remote_path_for(package) 49 | 50 | messages = [] 51 | transferred_files_for(package) do |local_file, remote_file| 52 | messages << "#{storage_name} started removing '#{ local_file }'." 53 | end 54 | Logger.message messages.join("\n") 55 | 56 | FileUtils.rm_r(remote_path) 57 | end 58 | 59 | ## 60 | # Set and return the transfer method. 61 | # If this Local Storage is not the last Storage for the Model, 62 | # force the transfer to use a *copy* operation and issue a warning. 63 | def transfer_method 64 | return @transfer_method if @transfer_method 65 | 66 | if self == @model.storages.last 67 | @transfer_method = :mv 68 | else 69 | Logger.warn Errors::Storage::Local::TransferError.new(<<-EOS) 70 | Local File Copy Warning! 71 | The final backup file(s) for '#{@model.label}' (#{@model.trigger}) 72 | will be *copied* to '#{remote_path_for(@package)}' 73 | To avoid this, when using more than one Storage, the 'Local' Storage 74 | should be added *last* so the files may be *moved* to their destination. 75 | EOS 76 | @transfer_method = :cp 77 | end 78 | end 79 | 80 | end 81 | end 82 | end 83 | -------------------------------------------------------------------------------- /lib/backup/encryptor/gpg.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Encryptor 5 | class GPG < Base 6 | 7 | ## 8 | # The GPG Public key that'll be used to encrypt the backup 9 | attr_accessor :key 10 | 11 | ## 12 | # Creates a new instance of Backup::Encryptor::GPG and 13 | # sets the key to the provided GPG key. To enhance the DSL 14 | # the user may use tabs and spaces to indent the multi-line key string 15 | # since we gsub() every preceding 'space' and 'tab' on each line 16 | def initialize(&block) 17 | super 18 | 19 | instance_eval(&block) if block_given? 20 | end 21 | 22 | ## 23 | # This is called as part of the procedure run by the Packager. 24 | # It sets up the needed encryption_key_email to pass to the gpg command, 25 | # then yields the command to use as part of the packaging procedure. 26 | # Once the packaging procedure is complete, it will return 27 | # so that any clean-up may be performed after the yield. 28 | def encrypt_with 29 | log! 30 | extract_encryption_key_email! 31 | 32 | yield "#{ utility(:gpg) } #{ options }", '.gpg' 33 | end 34 | 35 | private 36 | 37 | ## 38 | # Imports the given encryption key to ensure it's available for use, 39 | # and extracts the email address used to create the key. 40 | # This is stored in '@encryption_key_email', to be used to specify 41 | # the --recipient when performing encryption so this key is used. 42 | def extract_encryption_key_email! 43 | if @encryption_key_email.to_s.empty? 44 | with_tmp_key_file do |tmp_file| 45 | @encryption_key_email = run( 46 | "#{ utility(:gpg) } --import '#{tmp_file}' 2>&1" 47 | ).match(/<(.+)>/)[1] 48 | end 49 | end 50 | end 51 | 52 | ## 53 | # GPG options 54 | # Sets the gpg mode to 'encrypt' and passes in the encryption_key_email 55 | def options 56 | "-e --trust-model always -r '#{ @encryption_key_email }'" 57 | end 58 | 59 | ## 60 | # Writes the provided public gpg key to a temp file, 61 | # yields the path, then deletes the file when the block returns. 62 | def with_tmp_key_file 63 | tmp_file = Tempfile.new('backup.pub') 64 | FileUtils.chown(Config.user, nil, tmp_file.path) 65 | FileUtils.chmod(0600, tmp_file.path) 66 | tmp_file.write(encryption_key) 67 | tmp_file.close 68 | yield tmp_file.path 69 | tmp_file.delete 70 | end 71 | 72 | ## 73 | # Returns the encryption key with preceding spaces and tabs removed 74 | def encryption_key 75 | key.gsub(/^[[:blank:]]+/, '') 76 | end 77 | 78 | end 79 | end 80 | end 81 | -------------------------------------------------------------------------------- /lib/backup/storage/cloudfiles.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | ## 4 | # Only load the Fog gem when the Backup::Storage::CloudFiles class is loaded 5 | Backup::Dependency.load('fog') 6 | 7 | module Backup 8 | module Storage 9 | class CloudFiles < Base 10 | 11 | ## 12 | # Rackspace Cloud Files Credentials 13 | attr_accessor :username, :api_key, :auth_url 14 | 15 | ## 16 | # Rackspace Service Net 17 | # (LAN-based transfers to avoid charges and improve performance) 18 | attr_accessor :servicenet 19 | 20 | ## 21 | # Rackspace Cloud Files container name and path 22 | attr_accessor :container, :path 23 | 24 | ## 25 | # Creates a new instance of the storage object 26 | def initialize(model, storage_id = nil, &block) 27 | super(model, storage_id) 28 | 29 | @servicenet ||= false 30 | @path ||= 'backups' 31 | 32 | instance_eval(&block) if block_given? 33 | end 34 | 35 | private 36 | 37 | ## 38 | # This is the provider that Fog uses for the Cloud Files Storage 39 | def provider 40 | 'Rackspace' 41 | end 42 | 43 | ## 44 | # Establishes a connection to Rackspace Cloud Files 45 | def connection 46 | @connection ||= Fog::Storage.new( 47 | :provider => provider, 48 | :rackspace_username => username, 49 | :rackspace_api_key => api_key, 50 | :rackspace_auth_url => auth_url, 51 | :rackspace_servicenet => servicenet 52 | ) 53 | end 54 | 55 | ## 56 | # Transfers the archived file to the specified Cloud Files container 57 | def transfer! 58 | remote_path = remote_path_for(@package) 59 | 60 | files_to_transfer_for(@package) do |local_file, remote_file| 61 | Logger.message "#{storage_name} started transferring '#{ local_file }'." 62 | 63 | File.open(File.join(local_path, local_file), 'r') do |file| 64 | connection.put_object( 65 | container, File.join(remote_path, remote_file), file 66 | ) 67 | end 68 | end 69 | end 70 | 71 | ## 72 | # Removes the transferred archive file(s) from the storage location. 73 | # Any error raised will be rescued during Cycling 74 | # and a warning will be logged, containing the error message. 75 | def remove!(package) 76 | remote_path = remote_path_for(package) 77 | 78 | transferred_files_for(package) do |local_file, remote_file| 79 | Logger.message "#{storage_name} started removing '#{ local_file }' " + 80 | "from container '#{ container }'." 81 | connection.delete_object(container, File.join(remote_path, remote_file)) 82 | end 83 | end 84 | 85 | end 86 | end 87 | end 88 | -------------------------------------------------------------------------------- /lib/backup/storage/s3.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | ## 4 | # Only load the Fog gem when the Backup::Storage::S3 class is loaded 5 | Backup::Dependency.load('fog') 6 | 7 | module Backup 8 | module Storage 9 | class S3 < Base 10 | 11 | ## 12 | # Amazon Simple Storage Service (S3) Credentials 13 | attr_accessor :access_key_id, :secret_access_key 14 | 15 | ## 16 | # Amazon S3 bucket name and path 17 | attr_accessor :bucket, :path 18 | 19 | ## 20 | # Region of the specified S3 bucket 21 | attr_accessor :region 22 | 23 | ## 24 | # Creates a new instance of the storage object 25 | def initialize(model, storage_id = nil, &block) 26 | super(model, storage_id) 27 | 28 | @path ||= 'backups' 29 | 30 | instance_eval(&block) if block_given? 31 | end 32 | 33 | private 34 | 35 | ## 36 | # This is the provider that Fog uses for the S3 Storage 37 | def provider 38 | 'AWS' 39 | end 40 | 41 | ## 42 | # Establishes a connection to Amazon S3 43 | def connection 44 | @connection ||= Fog::Storage.new( 45 | :provider => provider, 46 | :aws_access_key_id => access_key_id, 47 | :aws_secret_access_key => secret_access_key, 48 | :region => region 49 | ) 50 | end 51 | 52 | def remote_path_for(package) 53 | super(package).sub(/^\//, '') 54 | end 55 | 56 | ## 57 | # Transfers the archived file to the specified Amazon S3 bucket 58 | def transfer! 59 | remote_path = remote_path_for(@package) 60 | 61 | connection.sync_clock 62 | 63 | files_to_transfer_for(@package) do |local_file, remote_file| 64 | Logger.message "#{storage_name} started transferring " + 65 | "'#{ local_file }' to bucket '#{ bucket }'." 66 | 67 | File.open(File.join(local_path, local_file), 'r') do |file| 68 | connection.put_object( 69 | bucket, File.join(remote_path, remote_file), file 70 | ) 71 | end 72 | end 73 | end 74 | 75 | ## 76 | # Removes the transferred archive file(s) from the storage location. 77 | # Any error raised will be rescued during Cycling 78 | # and a warning will be logged, containing the error message. 79 | def remove!(package) 80 | remote_path = remote_path_for(package) 81 | 82 | connection.sync_clock 83 | 84 | transferred_files_for(package) do |local_file, remote_file| 85 | Logger.message "#{storage_name} started removing " + 86 | "'#{ local_file }' from bucket '#{ bucket }'." 87 | 88 | connection.delete_object(bucket, File.join(remote_path, remote_file)) 89 | end 90 | end 91 | 92 | end 93 | end 94 | end 95 | -------------------------------------------------------------------------------- /spec-live/notifier/mail_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../../spec_helper.rb', __FILE__) 4 | 5 | describe 'Notifier::Mail', 6 | :if => Backup::SpecLive::CONFIG['notifier']['mail']['specs_enabled'] do 7 | describe 'Notifier::Mail :smtp' do 8 | let(:trigger) { 'notifier_mail' } 9 | 10 | it 'should send a success email' do 11 | model = h_set_trigger(trigger) 12 | expect do 13 | model.perform! 14 | end.not_to raise_error 15 | end 16 | 17 | it 'should send a warning email' do 18 | model = h_set_trigger(trigger) 19 | Backup::Logger.warn 'You have been warned!' 20 | expect do 21 | model.perform! 22 | end.not_to raise_error 23 | end 24 | 25 | it 'should send a failure email for non-fatal errors' do 26 | model = h_set_trigger(trigger) 27 | model.stubs(:databases).raises('A successful failure?') 28 | expect do 29 | model.perform! 30 | end.not_to raise_error 31 | end 32 | 33 | it 'should send a failure email fatal errors' do 34 | model = h_set_trigger(trigger) 35 | model.stubs(:databases).raises(NoMemoryError, 'with increasing frequency...') 36 | expect do 37 | model.perform! 38 | end.to raise_error 39 | end 40 | end # describe 'Notifier::Mail :smtp' 41 | 42 | describe 'Notifier::Mail :file' do 43 | let(:trigger) { 'notifier_mail_file' } 44 | let(:test_email) { File.join(Backup::SpecLive::TMP_PATH, 'test@backup') } 45 | 46 | it 'should send a success email' do 47 | model = h_set_trigger(trigger) 48 | expect do 49 | model.perform! 50 | end.not_to raise_error 51 | File.exist?(test_email).should be_true 52 | File.read(test_email).should match(/without any errors/) 53 | end 54 | 55 | it 'should send a warning email' do 56 | model = h_set_trigger(trigger) 57 | Backup::Logger.warn 'You have been warned!' 58 | expect do 59 | model.perform! 60 | end.not_to raise_error 61 | File.exist?(test_email).should be_true 62 | File.read(test_email).should match(/You have been warned/) 63 | end 64 | 65 | it 'should send a failure email for non-fatal errors' do 66 | model = h_set_trigger(trigger) 67 | model.stubs(:databases).raises('A successful failure?') 68 | expect do 69 | model.perform! 70 | end.not_to raise_error 71 | File.exist?(test_email).should be_true 72 | File.read(test_email).should match(/successful failure/) 73 | end 74 | 75 | it 'should send a failure email fatal errors' do 76 | model = h_set_trigger(trigger) 77 | model.stubs(:databases).raises(NoMemoryError, 'with increasing frequency...') 78 | expect do 79 | model.perform! 80 | end.to raise_error 81 | File.exist?(test_email).should be_true 82 | File.read(test_email).should match(/with increasing frequency/) 83 | end 84 | end # describe 'Notifier::Mail :file' 85 | end 86 | -------------------------------------------------------------------------------- /lib/backup/cli/helpers.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module CLI 5 | module Helpers 6 | UTILITY = {} 7 | 8 | ## 9 | # Runs a system command 10 | # 11 | # All messages generated by the command will be logged. 12 | # Messages on STDERR will be logged as warnings. 13 | # 14 | # If the command fails to execute, or returns a non-zero exit status 15 | # an Error will be raised. 16 | # 17 | # Returns STDOUT 18 | def run(command) 19 | name = command_name(command) 20 | Logger.message "Running system utility '#{ name }'..." 21 | 22 | begin 23 | out, err = '', '' 24 | ps = Open4.popen4(command) do |pid, stdin, stdout, stderr| 25 | stdin.close 26 | out, err = stdout.read.strip, stderr.read.strip 27 | end 28 | rescue Exception => e 29 | raise Errors::CLI::SystemCallError.wrap(e, <<-EOS) 30 | Failed to execute system command on #{ RUBY_PLATFORM } 31 | Command was: #{ command } 32 | EOS 33 | end 34 | 35 | if ps.success? 36 | unless out.empty? 37 | Logger.message( 38 | out.lines.map {|line| "#{ name }:STDOUT: #{ line }" }.join 39 | ) 40 | end 41 | 42 | unless err.empty? 43 | Logger.warn( 44 | err.lines.map {|line| "#{ name }:STDERR: #{ line }" }.join 45 | ) 46 | end 47 | 48 | return out 49 | else 50 | raise Errors::CLI::SystemCallError, <<-EOS 51 | '#{ name }' Failed on #{ RUBY_PLATFORM } 52 | The following information should help to determine the problem: 53 | Command was: #{ command } 54 | Exit Status: #{ ps.exitstatus } 55 | STDOUT Messages: #{ out.empty? ? 'None' : "\n#{ out }" } 56 | STDERR Messages: #{ err.empty? ? 'None' : "\n#{ err }" } 57 | EOS 58 | end 59 | end 60 | 61 | 62 | ## 63 | # Returns the full path to the specified utility. 64 | # Raises an error if utility can not be found in the system's $PATH 65 | def utility(name) 66 | name = name.to_s.strip 67 | raise Errors::CLI::UtilityNotFoundError, 68 | 'Utility Name Empty' if name.empty? 69 | 70 | path = UTILITY[name] || %x[which #{ name } 2>/dev/null].chomp 71 | if path.empty? 72 | raise Errors::CLI::UtilityNotFoundError, <<-EOS 73 | Could not locate '#{ name }'. 74 | Make sure the specified utility is installed 75 | and available in your system's $PATH. 76 | EOS 77 | end 78 | UTILITY[name] = path 79 | end 80 | 81 | ## 82 | # Returns the name of the command name from the given command line 83 | def command_name(command) 84 | i = command =~ /\s/ 85 | command = command.slice(0, i) if i 86 | command.split('/')[-1] 87 | end 88 | 89 | end 90 | end 91 | end 92 | -------------------------------------------------------------------------------- /lib/backup/notifier/hipchat.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | # Load the HipChat library from the gem 4 | Backup::Dependency.load('hipchat') 5 | 6 | module Backup 7 | module Notifier 8 | class Hipchat < Base 9 | 10 | ## 11 | # The Hipchat API token 12 | attr_accessor :token 13 | 14 | ## 15 | # Who the notification should appear from 16 | attr_accessor :from 17 | 18 | ## 19 | # The rooms that should be notified 20 | attr_accessor :rooms_notified 21 | 22 | ## 23 | # Notify users in the room 24 | attr_accessor :notify_users 25 | 26 | ## 27 | # The background color of a success message. 28 | # One of :yellow, :red, :green, :purple, or :random. (default: yellow) 29 | attr_accessor :success_color 30 | 31 | ## 32 | # The background color of a warning message. 33 | # One of :yellow, :red, :green, :purple, or :random. (default: yellow) 34 | attr_accessor :warning_color 35 | 36 | ## 37 | # The background color of an error message. 38 | # One of :yellow, :red, :green, :purple, or :random. (default: yellow) 39 | attr_accessor :failure_color 40 | 41 | def initialize(model, &block) 42 | super(model) 43 | 44 | @notify_users ||= false 45 | @rooms_notified ||= [] 46 | @success_color ||= 'yellow' 47 | @warning_color ||= 'yellow' 48 | @failure_color ||= 'yellow' 49 | 50 | instance_eval(&block) if block_given? 51 | end 52 | 53 | private 54 | 55 | ## 56 | # Notify the user of the backup operation results. 57 | # `status` indicates one of the following: 58 | # 59 | # `:success` 60 | # : The backup completed successfully. 61 | # : Notification will be sent if `on_success` was set to `true` 62 | # 63 | # `:warning` 64 | # : The backup completed successfully, but warnings were logged 65 | # : Notification will be sent, including a copy of the current 66 | # : backup log, if `on_warning` was set to `true` 67 | # 68 | # `:failure` 69 | # : The backup operation failed. 70 | # : Notification will be sent, including the Exception which caused 71 | # : the failure, the Exception's backtrace, a copy of the current 72 | # : backup log and other information if `on_failure` was set to `true` 73 | # 74 | def notify!(status) 75 | name, color = case status 76 | when :success then ['Success', success_color] 77 | when :warning then ['Warning', warning_color] 78 | when :failure then ['Failure', failure_color] 79 | end 80 | message = "[Backup::%s] #{@model.label} (#{@model.trigger})" % name 81 | send_message(message, color) 82 | end 83 | 84 | def send_message(msg, color) 85 | client = HipChat::Client.new(token) 86 | [rooms_notified].flatten.each do |room| 87 | client[room].send(from, msg, :color => color, :notify => notify_users) 88 | end 89 | end 90 | 91 | end 92 | end 93 | end 94 | -------------------------------------------------------------------------------- /lib/backup/storage/scp.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | ## 4 | # Only load the Net::SSH and Net::SCP library/gems 5 | # when the Backup::Storage::SCP class is loaded 6 | Backup::Dependency.load('net-ssh') 7 | Backup::Dependency.load('net-scp') 8 | 9 | module Backup 10 | module Storage 11 | class SCP < Base 12 | 13 | ## 14 | # Server credentials 15 | attr_accessor :username, :password 16 | 17 | ## 18 | # Server IP Address and SCP port 19 | attr_accessor :ip, :port 20 | 21 | ## 22 | # Path to store backups to 23 | attr_accessor :path 24 | 25 | ## 26 | # Creates a new instance of the storage object 27 | def initialize(model, storage_id = nil, &block) 28 | super(model, storage_id) 29 | 30 | @port ||= 22 31 | @path ||= 'backups' 32 | 33 | instance_eval(&block) if block_given? 34 | 35 | @path = path.sub(/^\~\//, '') 36 | end 37 | 38 | private 39 | 40 | ## 41 | # Establishes a connection to the remote server 42 | # and yields the Net::SSH connection. 43 | # Net::SCP will use this connection to transfer backups 44 | def connection 45 | Net::SSH.start( 46 | ip, username, :password => password, :port => port 47 | ) {|ssh| yield ssh } 48 | end 49 | 50 | ## 51 | # Transfers the archived file to the specified remote server 52 | def transfer! 53 | remote_path = remote_path_for(@package) 54 | 55 | connection do |ssh| 56 | ssh.exec!("mkdir -p '#{ remote_path }'") 57 | 58 | files_to_transfer_for(@package) do |local_file, remote_file| 59 | Logger.message "#{storage_name} started transferring " + 60 | "'#{local_file}' to '#{ip}'." 61 | 62 | ssh.scp.upload!( 63 | File.join(local_path, local_file), 64 | File.join(remote_path, remote_file) 65 | ) 66 | end 67 | end 68 | end 69 | 70 | ## 71 | # Removes the transferred archive file(s) from the storage location. 72 | # Any error raised will be rescued during Cycling 73 | # and a warning will be logged, containing the error message. 74 | def remove!(package) 75 | remote_path = remote_path_for(package) 76 | 77 | messages = [] 78 | transferred_files_for(package) do |local_file, remote_file| 79 | messages << "#{storage_name} started removing " + 80 | "'#{local_file}' from '#{ip}'." 81 | end 82 | Logger.message messages.join("\n") 83 | 84 | errors = [] 85 | connection do |ssh| 86 | ssh.exec!("rm -r '#{remote_path}'") do |ch, stream, data| 87 | errors << data if stream == :stderr 88 | end 89 | end 90 | unless errors.empty? 91 | raise Errors::Storage::SCP::SSHError, 92 | "Net::SSH reported the following errors:\n" + 93 | errors.join("\n") 94 | end 95 | end 96 | 97 | end 98 | end 99 | end 100 | -------------------------------------------------------------------------------- /lib/backup/storage/base.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Storage 5 | class Base 6 | include Backup::Configuration::Helpers 7 | 8 | ## 9 | # Sets the limit to how many backups to keep in the remote location. 10 | # If exceeded, the oldest will be removed to make room for the newest 11 | attr_accessor :keep 12 | 13 | ## 14 | # (Optional) 15 | # User-defined string used to uniquely identify multiple storages of the 16 | # same type. This will be appended to the YAML storage file used for 17 | # cycling backups. 18 | attr_accessor :storage_id 19 | 20 | ## 21 | # Creates a new instance of the storage object 22 | # * Called with super(model, storage_id) from each subclass 23 | def initialize(model, storage_id = nil) 24 | load_defaults! 25 | @model = model 26 | @storage_id = storage_id 27 | end 28 | 29 | ## 30 | # Performs the backup transfer 31 | def perform! 32 | @package = @model.package 33 | transfer! 34 | cycle! 35 | end 36 | 37 | private 38 | 39 | ## 40 | # Provider defaults to false. Overridden when using a service-based 41 | # storage such as Amazon S3, Rackspace Cloud Files or Dropbox 42 | def provider 43 | false 44 | end 45 | 46 | ## 47 | # Each subclass must define a +path+ where remote files will be stored 48 | def path; end 49 | 50 | ## 51 | # Return the storage name, with optional storage_id 52 | def storage_name 53 | self.class.to_s.sub('Backup::', '') + 54 | (storage_id ? " (#{storage_id})" : '') 55 | end 56 | 57 | ## 58 | # Returns the local path 59 | # This is where any Package to be transferred is located. 60 | def local_path 61 | Config.tmp_path 62 | end 63 | 64 | ## 65 | # Returns the remote path for the given Package 66 | # This is where the Package will be stored, or was previously stored. 67 | def remote_path_for(package) 68 | File.join(path, package.trigger, package.time) 69 | end 70 | 71 | ## 72 | # Yields two arguments to the given block: "local_file, remote_file" 73 | # The local_file is the full file name: 74 | # e.g. "2011.08.30.11.00.02.backup.tar.enc" 75 | # The remote_file is the full file name, minus the timestamp: 76 | # e.g. "backup.tar.enc" 77 | def files_to_transfer_for(package) 78 | package.filenames.each do |filename| 79 | yield filename, filename[20..-1] 80 | end 81 | end 82 | alias :transferred_files_for :files_to_transfer_for 83 | 84 | ## 85 | # Adds the current package being stored to the YAML cycle data file 86 | # and will remove any old Package file(s) when the storage limit 87 | # set by #keep is exceeded. Any errors raised while attempting to 88 | # remove older packages will be rescued and a warning will be logged 89 | # containing the original error message. 90 | def cycle! 91 | return unless keep.to_i > 0 92 | Logger.message "#{ storage_name }: Cycling Started..." 93 | Cycler.cycle!(self, @package) 94 | Logger.message "#{ storage_name }: Cycling Complete!" 95 | end 96 | 97 | end 98 | end 99 | end 100 | -------------------------------------------------------------------------------- /spec/syncer/rsync/base_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../../../spec_helper.rb', __FILE__) 4 | 5 | describe 'Backup::Syncer::RSync::Base' do 6 | let(:syncer) { Backup::Syncer::RSync::Base.new } 7 | 8 | it 'should be a subclass of Syncer::Base' do 9 | Backup::Syncer::RSync::Base. 10 | superclass.should == Backup::Syncer::Base 11 | end 12 | 13 | describe '#initialize' do 14 | after { Backup::Syncer::RSync::Base.clear_defaults! } 15 | 16 | it 'should load pre-configured defaults through Syncer::Base' do 17 | Backup::Syncer::RSync::Base.any_instance.expects(:load_defaults!) 18 | syncer 19 | end 20 | 21 | context 'when no pre-configured defaults have been set' do 22 | it 'should use default values' do 23 | syncer.path.should == 'backups' 24 | syncer.mirror.should == false 25 | syncer.directories.should == [] 26 | syncer.additional_options.should == [] 27 | end 28 | end # context 'when no pre-configured defaults have been set' 29 | 30 | context 'when pre-configured defaults have been set' do 31 | before do 32 | Backup::Syncer::RSync::Base.defaults do |rsync| 33 | rsync.path = 'some_path' 34 | rsync.mirror = 'some_mirror' 35 | rsync.additional_options = 'some_additional_options' 36 | end 37 | end 38 | 39 | it 'should use pre-configured defaults' do 40 | syncer.path.should == 'some_path' 41 | syncer.mirror.should == 'some_mirror' 42 | syncer.directories.should == [] 43 | syncer.additional_options.should == 'some_additional_options' 44 | end 45 | end # context 'when pre-configured defaults have been set' 46 | end # describe '#initialize' 47 | 48 | describe '#directory_options' do 49 | before do 50 | syncer.instance_variable_set( 51 | :@directories, ['/some/directory', '/another/directory'] 52 | ) 53 | end 54 | 55 | it 'should return the directories for use in the command line' do 56 | syncer.send(:directories_option).should == 57 | "'/some/directory' '/another/directory'" 58 | end 59 | 60 | context 'when @directories have relative paths' do 61 | before do 62 | syncer.instance_variable_set( 63 | :@directories, ['/some/directory', '/another/directory', 64 | 'relative/path', '~/home/path'] 65 | ) 66 | end 67 | it 'should expand relative paths' do 68 | syncer.send(:directories_option).should == 69 | "'/some/directory' '/another/directory' " + 70 | "'#{ File.expand_path('relative/path') }' " + 71 | "'#{ File.expand_path('~/home/path') }'" 72 | end 73 | end 74 | end 75 | 76 | describe '#mirror_option' do 77 | context 'when @mirror is true' do 78 | before { syncer.mirror = true } 79 | it 'should return the command line flag for mirroring' do 80 | syncer.send(:mirror_option).should == '--delete' 81 | end 82 | end 83 | 84 | context 'when @mirror is false' do 85 | before { syncer.mirror = false } 86 | it 'should return nil' do 87 | syncer.send(:mirror_option).should be_nil 88 | end 89 | end 90 | end 91 | 92 | describe '#archive_option' do 93 | it 'should return the command line flag for archiving' do 94 | syncer.send(:archive_option).should == '--archive' 95 | end 96 | end 97 | 98 | end 99 | -------------------------------------------------------------------------------- /spec/notifier/base_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../../spec_helper.rb', __FILE__) 4 | 5 | describe 'Backup::Notifier::Base' do 6 | let(:model) { Backup::Model.new(:test_trigger, 'test label') } 7 | let(:notifier) { Backup::Notifier::Base.new(model) } 8 | 9 | it 'should include Configuration::Helpers' do 10 | Backup::Notifier::Base. 11 | include?(Backup::Configuration::Helpers).should be_true 12 | end 13 | 14 | describe '#initialize' do 15 | after { Backup::Notifier::Base.clear_defaults! } 16 | 17 | it 'should load pre-configured defaults' do 18 | Backup::Notifier::Base.any_instance.expects(:load_defaults!) 19 | notifier 20 | end 21 | 22 | it 'should set a reference to the model' do 23 | notifier.instance_variable_get(:@model).should == model 24 | end 25 | 26 | context 'when no pre-configured defaults have been set' do 27 | it 'should set default values' do 28 | notifier.on_success.should == true 29 | notifier.on_warning.should == true 30 | notifier.on_failure.should == true 31 | end 32 | end # context 'when no pre-configured defaults have been set' 33 | 34 | context 'when pre-configured defaults have been set' do 35 | before do 36 | Backup::Notifier::Base.defaults do |n| 37 | n.on_success = false 38 | n.on_warning = false 39 | n.on_failure = false 40 | end 41 | end 42 | 43 | it 'should use pre-configured defaults' do 44 | notifier.on_success.should be_false 45 | notifier.on_warning.should be_false 46 | notifier.on_failure.should be_false 47 | end 48 | end # context 'when pre-configured defaults have been set' 49 | end # describe '#initialize' 50 | 51 | describe '#perform!' do 52 | before do 53 | notifier.expects(:log!) 54 | Backup::Template.expects(:new).with({:model => model}) 55 | end 56 | 57 | context 'when failure is false' do 58 | context 'when no warnings were issued' do 59 | before do 60 | Backup::Logger.expects(:has_warnings?).returns(false) 61 | end 62 | 63 | it 'should call #notify! with :success' do 64 | notifier.expects(:notify!).with(:success) 65 | notifier.perform! 66 | end 67 | end 68 | 69 | context 'when warnings were issued' do 70 | before do 71 | Backup::Logger.expects(:has_warnings?).returns(true) 72 | end 73 | 74 | it 'should call #notify! with :warning' do 75 | notifier.expects(:notify!).with(:warning) 76 | notifier.perform! 77 | end 78 | end 79 | end # context 'when failure is false' 80 | 81 | context 'when failure is true' do 82 | it 'should call #notify with :failure' do 83 | notifier.expects(:notify!).with(:failure) 84 | notifier.perform!(true) 85 | end 86 | end 87 | end # describe '#perform!' 88 | 89 | describe '#notifier_name' do 90 | it 'should return class name without Backup:: namespace' do 91 | notifier.send(:notifier_name).should == 'Notifier::Base' 92 | end 93 | end 94 | 95 | describe '#log!' do 96 | it 'should log a message' do 97 | Backup::Logger.expects(:message).with( 98 | "Notifier::Base started notifying about the process." 99 | ) 100 | notifier.send(:log!) 101 | end 102 | end 103 | 104 | end 105 | -------------------------------------------------------------------------------- /spec/syncer/rsync/pull_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../../../spec_helper.rb', __FILE__) 4 | 5 | describe Backup::Syncer::RSync::Pull do 6 | let(:syncer) do 7 | Backup::Syncer::RSync::Pull.new do |rsync| 8 | rsync.username = 'my_username' 9 | rsync.password = 'my_password' 10 | rsync.ip = '123.45.678.90' 11 | rsync.port = 22 12 | rsync.compress = true 13 | rsync.path = "~/my_backups" 14 | 15 | rsync.directories do |directory| 16 | directory.add "/some/directory" 17 | directory.add "~/home/directory" 18 | directory.add "another/directory" 19 | end 20 | 21 | rsync.mirror = true 22 | rsync.additional_options = ['--opt-a', '--opt-b'] 23 | end 24 | end 25 | 26 | it 'should be a subclass of RSync::Push' do 27 | Backup::Syncer::RSync::Pull.superclass.should == Backup::Syncer::RSync::Push 28 | end 29 | 30 | describe '#perform!' do 31 | let(:s) { sequence '' } 32 | 33 | it 'should perform the RSync::Pull operation on two directories' do 34 | syncer.expects(:utility).times(3).with(:rsync).returns('rsync') 35 | syncer.expects(:options).times(3).returns('options_output') 36 | 37 | syncer.expects(:write_password_file!).in_sequence(s) 38 | 39 | # first directory - uses the given full path 40 | Backup::Logger.expects(:message).in_sequence(s).with( 41 | "Syncer::RSync::Pull started syncing '/some/directory'." 42 | ) 43 | syncer.expects(:run).in_sequence(s).with( 44 | "rsync options_output 'my_username@123.45.678.90:/some/directory' " + 45 | "'#{ File.expand_path('~/my_backups') }'" 46 | ) 47 | 48 | # second directory - removes leading '~' 49 | Backup::Logger.expects(:message).in_sequence(s).with( 50 | "Syncer::RSync::Pull started syncing '~/home/directory'." 51 | ) 52 | syncer.expects(:run).in_sequence(s).with( 53 | "rsync options_output 'my_username@123.45.678.90:home/directory' " + 54 | "'#{ File.expand_path('~/my_backups') }'" 55 | ) 56 | 57 | # third directory - does not expand path 58 | Backup::Logger.expects(:message).in_sequence(s).with( 59 | "Syncer::RSync::Pull started syncing 'another/directory'." 60 | ) 61 | syncer.expects(:run).in_sequence(s).with( 62 | "rsync options_output 'my_username@123.45.678.90:another/directory' " + 63 | "'#{ File.expand_path('~/my_backups') }'" 64 | ) 65 | 66 | syncer.expects(:remove_password_file!).in_sequence(s) 67 | 68 | syncer.perform! 69 | end 70 | 71 | it 'should ensure passoword file removal' do 72 | syncer.expects(:write_password_file!).raises('error message') 73 | syncer.expects(:remove_password_file!) 74 | 75 | expect do 76 | syncer.perform! 77 | end.to raise_error(RuntimeError, 'error message') 78 | end 79 | end # describe '#perform!' 80 | 81 | describe '#dest_path' do 82 | it 'should return @path expanded' do 83 | syncer.send(:dest_path).should == File.expand_path('~/my_backups') 84 | end 85 | 86 | it 'should set @dest_path' do 87 | syncer.send(:dest_path) 88 | syncer.instance_variable_get(:@dest_path).should == 89 | File.expand_path('~/my_backups') 90 | end 91 | 92 | it 'should return @dest_path if already set' do 93 | syncer.instance_variable_set(:@dest_path, 'foo') 94 | syncer.send(:dest_path).should == 'foo' 95 | end 96 | end 97 | 98 | end 99 | -------------------------------------------------------------------------------- /lib/backup/storage/sftp.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | ## 4 | # Only load the Net::SFTP library/gem when the Backup::Storage::SFTP class is loaded 5 | Backup::Dependency.load('net-ssh') 6 | Backup::Dependency.load('net-sftp') 7 | 8 | module Backup 9 | module Storage 10 | class SFTP < Base 11 | 12 | ## 13 | # Server credentials 14 | attr_accessor :username, :password 15 | 16 | ## 17 | # Server IP Address and SFTP port 18 | attr_accessor :ip, :port 19 | 20 | ## 21 | # Path to store backups to 22 | attr_accessor :path 23 | 24 | ## 25 | # Creates a new instance of the storage object 26 | def initialize(model, storage_id = nil, &block) 27 | super(model, storage_id) 28 | 29 | @port ||= 22 30 | @path ||= 'backups' 31 | 32 | instance_eval(&block) if block_given? 33 | 34 | @path = path.sub(/^\~\//, '') 35 | end 36 | 37 | private 38 | 39 | ## 40 | # Establishes a connection to the remote server 41 | def connection 42 | Net::SFTP.start( 43 | ip, username, 44 | :password => password, 45 | :port => port 46 | ) {|sftp| yield sftp } 47 | end 48 | 49 | ## 50 | # Transfers the archived file to the specified remote server 51 | def transfer! 52 | remote_path = remote_path_for(@package) 53 | 54 | connection do |sftp| 55 | create_remote_path(remote_path, sftp) 56 | 57 | files_to_transfer_for(@package) do |local_file, remote_file| 58 | Logger.message "#{storage_name} started transferring " + 59 | "'#{ local_file }' to '#{ ip }'." 60 | 61 | sftp.upload!( 62 | File.join(local_path, local_file), 63 | File.join(remote_path, remote_file) 64 | ) 65 | end 66 | end 67 | end 68 | 69 | ## 70 | # Removes the transferred archive file(s) from the storage location. 71 | # Any error raised will be rescued during Cycling 72 | # and a warning will be logged, containing the error message. 73 | def remove!(package) 74 | remote_path = remote_path_for(package) 75 | 76 | connection do |sftp| 77 | transferred_files_for(package) do |local_file, remote_file| 78 | Logger.message "#{storage_name} started removing " + 79 | "'#{ local_file }' from '#{ ip }'." 80 | 81 | sftp.remove!(File.join(remote_path, remote_file)) 82 | end 83 | 84 | sftp.rmdir!(remote_path) 85 | end 86 | end 87 | 88 | ## 89 | # Creates (if they don't exist yet) all the directories on the remote 90 | # server in order to upload the backup file. Net::SFTP does not support 91 | # paths to directories that don't yet exist when creating new 92 | # directories. Instead, we split the parts up in to an array (for each 93 | # '/') and loop through that to create the directories one by one. 94 | # Net::SFTP raises an exception when the directory it's trying to create 95 | # already exists, so we have rescue it 96 | def create_remote_path(remote_path, sftp) 97 | path_parts = Array.new 98 | remote_path.split('/').each do |path_part| 99 | path_parts << path_part 100 | begin 101 | sftp.mkdir!(path_parts.join('/')) 102 | rescue Net::SFTP::StatusException; end 103 | end 104 | end 105 | 106 | end 107 | end 108 | end 109 | -------------------------------------------------------------------------------- /spec/syncer/base_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../../spec_helper.rb', __FILE__) 4 | 5 | describe Backup::Syncer::Base do 6 | let(:syncer) { Backup::Syncer::Base.new } 7 | 8 | it 'should include CLI::Helpers' do 9 | Backup::Syncer::Base. 10 | include?(Backup::CLI::Helpers).should be_true 11 | end 12 | 13 | it 'should include Configuration::Helpers' do 14 | Backup::Syncer::Base. 15 | include?(Backup::Configuration::Helpers).should be_true 16 | end 17 | 18 | describe '#initialize' do 19 | after { Backup::Syncer::Base.clear_defaults! } 20 | 21 | it 'should load pre-configured defaults through Base' do 22 | Backup::Syncer::Base.any_instance.expects(:load_defaults!) 23 | syncer 24 | end 25 | 26 | it 'should establish a new array for @directories' do 27 | syncer.directories.should == [] 28 | end 29 | 30 | context 'when no pre-configured defaults have been set' do 31 | it 'should set default values' do 32 | syncer.path.should == 'backups' 33 | syncer.mirror.should == false 34 | end 35 | end # context 'when no pre-configured defaults have been set' 36 | 37 | context 'when pre-configured defaults have been set' do 38 | before do 39 | Backup::Syncer::Base.defaults do |s| 40 | s.path = 'some_path' 41 | s.mirror = 'some_mirror' 42 | end 43 | end 44 | 45 | it 'should use pre-configured defaults' do 46 | syncer.path.should == 'some_path' 47 | syncer.mirror.should == 'some_mirror' 48 | end 49 | end # context 'when pre-configured defaults have been set' 50 | end # describe '#initialize' 51 | 52 | describe '#directories' do 53 | before do 54 | syncer.instance_variable_set( 55 | :@directories, ['/some/directory', '/another/directory'] 56 | ) 57 | end 58 | 59 | context 'when no block is given' do 60 | it 'should return @directories' do 61 | syncer.directories.should == 62 | ['/some/directory', '/another/directory'] 63 | end 64 | end 65 | 66 | context 'when a block is given' do 67 | it 'should evalute the block, allowing #add to add directories' do 68 | syncer.directories do 69 | add '/new/path' 70 | add '/another/new/path' 71 | end 72 | syncer.directories.should == [ 73 | '/some/directory', 74 | '/another/directory', 75 | '/new/path', 76 | '/another/new/path' 77 | ] 78 | end 79 | end 80 | end # describe '#directories' 81 | 82 | describe '#add' do 83 | before do 84 | syncer.instance_variable_set( 85 | :@directories, ['/some/directory', '/another/directory'] 86 | ) 87 | end 88 | 89 | it 'should add the given path to @directories' do 90 | syncer.add '/my/path' 91 | syncer.directories.should == 92 | ['/some/directory', '/another/directory', '/my/path'] 93 | end 94 | 95 | # Note: Each Syncer should handle this as needed. 96 | # For example, expanding these here would break RSync::Pull 97 | it 'should not expand the given paths' do 98 | syncer.add 'relative/path' 99 | syncer.directories.should == 100 | ['/some/directory', '/another/directory', 'relative/path'] 101 | end 102 | end 103 | 104 | describe '#syncer_name' do 105 | it 'should return the class name with the Backup:: namespace removed' do 106 | syncer.send(:syncer_name).should == 'Syncer::Base' 107 | end 108 | end 109 | end 110 | -------------------------------------------------------------------------------- /lib/backup/dependency.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | 5 | ## 6 | # A little self-contained gem manager for Backup. 7 | # Rather than specifying hard dependencies in the gemspec, forcing users 8 | # to install gems they do not want/need, Backup will notify them when a gem 9 | # has not been installed, or when the gem's version is incorrect, and provide the 10 | # command to install the gem. These dependencies are dynamically loaded in the Gemfile 11 | class Dependency 12 | 13 | ## 14 | # Returns a hash of dependencies that Backup requires 15 | # in order to run every available feature 16 | def self.all 17 | { 18 | 'fog' => { 19 | :require => 'fog', 20 | :version => '~> 1.4.0', 21 | :for => 'Amazon S3, Rackspace Cloud Files (S3, CloudFiles Storages)' 22 | }, 23 | 24 | 'dropbox-sdk' => { 25 | :require => 'dropbox_sdk', 26 | :version => '~> 1.2.0', 27 | :for => 'Dropbox Web Service (Dropbox Storage)' 28 | }, 29 | 30 | 'net-sftp' => { 31 | :require => 'net/sftp', 32 | :version => '~> 2.0.5', 33 | :for => 'SFTP Protocol (SFTP Storage)' 34 | }, 35 | 36 | 'net-scp' => { 37 | :require => 'net/scp', 38 | :version => '~> 1.0.4', 39 | :for => 'SCP Protocol (SCP Storage)' 40 | }, 41 | 42 | 'net-ssh' => { 43 | :require => 'net/ssh', 44 | :version => '~> 2.3.0', 45 | :for => 'SSH Protocol (SSH Storage)' 46 | }, 47 | 48 | 'mail' => { 49 | :require => 'mail', 50 | :version => '~> 2.4.0', 51 | :for => 'Sending Emails (Mail Notifier)' 52 | }, 53 | 54 | 'twitter' => { 55 | :require => 'twitter', 56 | :version => '>= 1.7.1', 57 | :for => 'Sending Twitter Updates (Twitter Notifier)' 58 | }, 59 | 60 | 'httparty' => { 61 | :require => 'httparty', 62 | :version => '~> 0.8.1', 63 | :for => 'Sending Http Updates' 64 | }, 65 | 66 | 'prowler' => { 67 | :require => 'prowler', 68 | :version => '>= 1.3.1', 69 | :for => 'Sending iOS push notifications (Prowl Notifier)' 70 | }, 71 | 72 | 'hipchat' => { 73 | :require => 'hipchat', 74 | :version => '~> 0.4.1', 75 | :for => 'Sending notifications to Hipchat' 76 | }, 77 | 78 | 'parallel' => { 79 | :require => 'parallel', 80 | :version => '~> 0.5.12', 81 | :for => 'Adding concurrency to Cloud-based syncers.' 82 | } 83 | } 84 | end 85 | 86 | ## 87 | # Attempts to load the specified gem (by name and version). 88 | # If the gem with the correct version cannot be found, it'll display a message 89 | # to the user with instructions on how to install the required gem 90 | def self.load(name) 91 | begin 92 | gem(name, all[name][:version]) 93 | require(all[name][:require]) 94 | rescue LoadError 95 | Logger.error Errors::Dependency::LoadError.new(<<-EOS) 96 | Dependency missing 97 | Dependency required for: 98 | #{all[name][:for]} 99 | To install the gem, issue the following command: 100 | > gem install #{name} -v '#{all[name][:version]}' 101 | Please try again after installing the missing dependency. 102 | EOS 103 | exit 1 104 | end 105 | end 106 | 107 | end 108 | end 109 | -------------------------------------------------------------------------------- /lib/backup/archive.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | class Archive 5 | include Backup::CLI::Helpers 6 | 7 | ## 8 | # Stores the name of the archive 9 | attr_accessor :name 10 | 11 | ## 12 | # Stores an array of different paths/files to store 13 | attr_accessor :paths 14 | 15 | ## 16 | # Stores an array of different paths/files to exclude 17 | attr_accessor :excludes 18 | 19 | ## 20 | # String of additional arguments for the `tar` command 21 | attr_accessor :tar_args 22 | 23 | ## 24 | # Takes the name of the archive and the configuration block 25 | def initialize(model, name, &block) 26 | @model = model 27 | @name = name.to_s 28 | @paths = Array.new 29 | @excludes = Array.new 30 | @tar_args = '' 31 | 32 | instance_eval(&block) if block_given? 33 | end 34 | 35 | ## 36 | # Adds new paths to the @paths instance variable array 37 | def add(path) 38 | path = File.expand_path(path) 39 | if File.exist?(path) 40 | @paths << path 41 | else 42 | Logger.warn Errors::Archive::NotFoundError.new(<<-EOS) 43 | The following path was not found: 44 | #{ path } 45 | This path will be omitted from the '#{ name }' Archive. 46 | EOS 47 | end 48 | end 49 | 50 | ## 51 | # Adds new paths to the @excludes instance variable array 52 | def exclude(path) 53 | @excludes << File.expand_path(path) 54 | end 55 | 56 | ## 57 | # Adds the given String of +options+ to the `tar` command. 58 | # e.g. '-h --xattrs' 59 | def tar_options(options) 60 | @tar_args = options 61 | end 62 | 63 | ## 64 | # Archives all the provided paths in to a single .tar file 65 | # and places that .tar file in the folder which later will be packaged 66 | # If the model is configured with a Compressor, the tar command output 67 | # will be piped through the Compressor command and the file extension 68 | # will be adjusted to indicate the type of compression used. 69 | def perform! 70 | Logger.message "#{ self.class } has started archiving:\n" + 71 | paths.map {|path| " #{path}" }.join("\n") 72 | 73 | archive_path = File.join(Config.tmp_path, @model.trigger, 'archives') 74 | FileUtils.mkdir_p(archive_path) 75 | 76 | archive_ext = 'tar' 77 | pipeline = Pipeline.new 78 | 79 | pipeline << "#{ utility(:tar) } #{ tar_args } -cPf - " + 80 | "#{ paths_to_exclude } #{ paths_to_package }" 81 | 82 | if @model.compressor 83 | @model.compressor.compress_with do |command, ext| 84 | pipeline << command 85 | archive_ext << ext 86 | end 87 | end 88 | 89 | pipeline << "cat > '#{ File.join(archive_path, "#{name}.#{archive_ext}") }'" 90 | pipeline.run 91 | if pipeline.success? 92 | Logger.message "#{ self.class } Complete!" 93 | else 94 | raise Errors::Archive::PipelineError, 95 | "Failed to Create Backup Archive\n" + 96 | pipeline.error_messages 97 | end 98 | end 99 | 100 | private 101 | 102 | ## 103 | # Returns a "tar-ready" string of all the specified paths combined 104 | def paths_to_package 105 | paths.map {|path| "'#{path}'" }.join(' ') 106 | end 107 | 108 | ## 109 | # Returns a "tar-ready" string of all the specified excludes combined 110 | def paths_to_exclude 111 | if excludes.any? 112 | excludes.map {|path| "--exclude='#{path}'" }.join(' ') 113 | end 114 | end 115 | 116 | end 117 | end 118 | -------------------------------------------------------------------------------- /lib/backup/syncer/rsync/push.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Syncer 5 | module RSync 6 | class Push < Base 7 | 8 | ## 9 | # Server credentials 10 | attr_accessor :username, :password 11 | 12 | ## 13 | # Server IP Address and SSH port 14 | attr_accessor :ip 15 | 16 | ## 17 | # The SSH port to connect to 18 | attr_accessor :port 19 | 20 | ## 21 | # Flag for compressing (only compresses for the transfer) 22 | attr_accessor :compress 23 | 24 | ## 25 | # Instantiates a new RSync::Push or RSync::Pull Syncer. 26 | # 27 | # Pre-configured defaults specified in 28 | # Configuration::Syncer::RSync::Push or 29 | # Configuration::Syncer::RSync::Pull 30 | # are set via a super() call to RSync::Base, 31 | # which in turn will invoke Syncer::Base. 32 | # 33 | # Once pre-configured defaults and RSync specific defaults are set, 34 | # the block from the user's configuration file is evaluated. 35 | def initialize(&block) 36 | super 37 | 38 | @port ||= 22 39 | @compress ||= false 40 | 41 | instance_eval(&block) if block_given? 42 | end 43 | 44 | ## 45 | # Performs the RSync:Push operation 46 | # debug options: -vhP 47 | def perform! 48 | write_password_file! 49 | 50 | Logger.message( 51 | "#{ syncer_name } started syncing the following directories:\n\s\s" + 52 | @directories.join("\n\s\s") 53 | ) 54 | run("#{ utility(:rsync) } #{ options } #{ directories_option } " + 55 | "'#{ username }@#{ ip }:#{ dest_path }'") 56 | 57 | ensure 58 | remove_password_file! 59 | end 60 | 61 | private 62 | 63 | ## 64 | # Return @path with any preceeding "~/" removed 65 | def dest_path 66 | @dest_path ||= @path.sub(/^\~\//, '') 67 | end 68 | 69 | ## 70 | # Returns all the specified Rsync::[Push/Pull] options, 71 | # concatenated, ready for the CLI 72 | def options 73 | ([archive_option, mirror_option, compress_option, port_option, 74 | password_option] + additional_options).compact.join("\s") 75 | end 76 | 77 | ## 78 | # Returns Rsync syntax for compressing the file transfers 79 | def compress_option 80 | '--compress' if @compress 81 | end 82 | 83 | ## 84 | # Returns Rsync syntax for defining a port to connect to 85 | def port_option 86 | "-e 'ssh -p #{@port}'" 87 | end 88 | 89 | ## 90 | # Returns Rsync syntax for setting a password (via a file) 91 | def password_option 92 | "--password-file='#{@password_file.path}'" if @password_file 93 | end 94 | 95 | ## 96 | # Writes the provided password to a temporary file so that 97 | # the rsync utility can read the password from this file 98 | def write_password_file! 99 | unless @password.nil? 100 | @password_file = Tempfile.new('backup-rsync-password') 101 | @password_file.write(@password) 102 | @password_file.close 103 | end 104 | end 105 | 106 | ## 107 | # Removes the previously created @password_file 108 | # (temporary file containing the password) 109 | def remove_password_file! 110 | @password_file.delete if @password_file 111 | @password_file = nil 112 | end 113 | end 114 | end 115 | end 116 | end 117 | -------------------------------------------------------------------------------- /lib/backup/storage/ninefold.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | ## 4 | # Only load the Fog gem when the Backup::Storage::Ninefold class is loaded 5 | Backup::Dependency.load('fog') 6 | 7 | module Backup 8 | module Storage 9 | class Ninefold < Base 10 | 11 | ## 12 | # Ninefold Credentials 13 | attr_accessor :storage_token, :storage_secret 14 | 15 | ## 16 | # Ninefold directory path 17 | attr_accessor :path 18 | 19 | ## 20 | # Creates a new instance of the storage object 21 | def initialize(model, storage_id = nil, &block) 22 | super(model, storage_id) 23 | 24 | @path ||= 'backups' 25 | 26 | instance_eval(&block) if block_given? 27 | end 28 | 29 | 30 | private 31 | 32 | ## 33 | # This is the provider that Fog uses for the Ninefold storage 34 | def provider 35 | 'Ninefold' 36 | end 37 | 38 | ## 39 | # Establishes a connection to Amazon S3 40 | def connection 41 | @connection ||= Fog::Storage.new( 42 | :provider => provider, 43 | :ninefold_storage_token => storage_token, 44 | :ninefold_storage_secret => storage_secret 45 | ) 46 | end 47 | 48 | ## 49 | # Queries the connection for the directory for the given +remote_path+ 50 | # Returns nil if not found, or creates the directory if +create+ is true. 51 | def directory_for(remote_path, create = false) 52 | directory = connection.directories.get(remote_path) 53 | if directory.nil? && create 54 | directory = connection.directories.create(:key => remote_path) 55 | end 56 | directory 57 | end 58 | 59 | def remote_path_for(package) 60 | super(package).sub(/^\//, '') 61 | end 62 | 63 | ## 64 | # Transfers the archived file to the specified directory 65 | def transfer! 66 | remote_path = remote_path_for(@package) 67 | 68 | directory = directory_for(remote_path, true) 69 | 70 | files_to_transfer_for(@package) do |local_file, remote_file| 71 | Logger.message "#{storage_name} started transferring '#{ local_file }'." 72 | 73 | File.open(File.join(local_path, local_file), 'r') do |file| 74 | directory.files.create(:key => remote_file, :body => file) 75 | end 76 | end 77 | end 78 | 79 | ## 80 | # Removes the transferred archive file(s) from the storage location. 81 | # Any error raised will be rescued during Cycling 82 | # and a warning will be logged, containing the error message. 83 | def remove!(package) 84 | remote_path = remote_path_for(package) 85 | 86 | if directory = directory_for(remote_path) 87 | not_found = [] 88 | 89 | transferred_files_for(package) do |local_file, remote_file| 90 | Logger.message "#{storage_name} started removing " + 91 | "'#{ local_file }' from Ninefold." 92 | 93 | if file = directory.files.get(remote_file) 94 | file.destroy 95 | else 96 | not_found << remote_file 97 | end 98 | end 99 | 100 | directory.destroy 101 | 102 | unless not_found.empty? 103 | raise Errors::Storage::Ninefold::NotFoundError, <<-EOS 104 | The following file(s) were not found in '#{ remote_path }' 105 | #{ not_found.join("\n") } 106 | EOS 107 | end 108 | else 109 | raise Errors::Storage::Ninefold::NotFoundError, 110 | "Directory at '#{remote_path}' not found" 111 | end 112 | end 113 | 114 | end 115 | end 116 | end 117 | -------------------------------------------------------------------------------- /lib/backup/errors.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | ## 5 | # - automatically defines module namespaces referenced under Backup::Errors 6 | # - any constant name referenced that ends with 'Error' will be created 7 | # as a subclass of Backup::Errors::Error 8 | # e.g. 9 | # err = Backup::Errors::Foo::Bar::FooError.new('error message') 10 | # err.message => "Foo::Bar::FooError: error message" 11 | # 12 | module ErrorsHelper 13 | def const_missing(const) 14 | if const.to_s.end_with?('Error') 15 | module_eval("class #{const} < Backup::Errors::Error; end") 16 | else 17 | module_eval("module #{const}; extend Backup::ErrorsHelper; end") 18 | end 19 | const_get(const) 20 | end 21 | end 22 | 23 | ## 24 | # provides cascading errors with formatted messages 25 | # see the specs for details 26 | # 27 | # e.g. 28 | # module Backup 29 | # begin 30 | # begin 31 | # begin 32 | # raise Errors::ZoneAError, 'an error occurred in Zone A' 33 | # rescue => err 34 | # raise Errors::ZoneBError.wrap(err, <<-EOS) 35 | # an error occurred in Zone B 36 | # 37 | # the following error should give a reason 38 | # EOS 39 | # end 40 | # rescue => err 41 | # raise Errors::ZoneCError.wrap(err) 42 | # end 43 | # rescue => err 44 | # puts Errors::ZoneDError.wrap(err, 'an error occurred in Zone D') 45 | # end 46 | # end 47 | # 48 | # Outputs: 49 | # ZoneDError: an error occurred in Zone D 50 | # Reason: ZoneCError 51 | # ZoneBError: an error occurred in Zone B 52 | # 53 | # the following error should give a reason 54 | # Reason: ZoneAError 55 | # an error occurred in Zone A 56 | # 57 | module Errors 58 | extend ErrorsHelper 59 | 60 | class Error < StandardError 61 | 62 | def self.wrap(orig_err, msg = nil) 63 | new(msg, orig_err) 64 | end 65 | 66 | def initialize(msg = nil, orig_err = nil) 67 | super(msg) 68 | set_backtrace(orig_err.backtrace) if @orig_err = orig_err 69 | end 70 | 71 | def to_s 72 | return @to_s if @to_s 73 | orig_to_s = super() 74 | 75 | if orig_to_s == self.class.to_s 76 | msg = orig_err_msg ? 77 | "#{orig_err_class}: #{orig_err_msg}" : orig_err_class 78 | else 79 | msg = format_msg(orig_to_s) 80 | msg << "\n Reason: #{orig_err_class}" + (orig_err_msg ? 81 | "\n #{orig_err_msg}" : ' (no message given)') if @orig_err 82 | end 83 | 84 | @to_s = msg ? msg_prefix + msg : class_name 85 | end 86 | 87 | private 88 | 89 | def msg_prefix 90 | @msg_prefix ||= class_name + ': ' 91 | end 92 | 93 | def orig_msg 94 | @orig_msg ||= to_s.sub(msg_prefix, '') 95 | end 96 | 97 | def class_name 98 | @class_name ||= self.class.to_s.sub('Backup::Errors::', '') 99 | end 100 | 101 | def orig_err_class 102 | return unless @orig_err 103 | 104 | @orig_err_class ||= @orig_err.is_a?(Errors::Error) ? 105 | @orig_err.send(:class_name) : @orig_err.class.to_s 106 | end 107 | 108 | def orig_err_msg 109 | return unless @orig_err 110 | return @orig_err_msg unless @orig_err_msg.nil? 111 | 112 | msg = @orig_err.is_a?(Errors::Error) ? 113 | @orig_err.send(:orig_msg) : @orig_err.to_s 114 | @orig_err_msg = (msg == orig_err_class) ? 115 | false : format_msg(msg) 116 | end 117 | 118 | def format_msg(msg) 119 | msg.gsub(/^ */, ' ').strip 120 | end 121 | end 122 | 123 | end 124 | end 125 | -------------------------------------------------------------------------------- /spec/compressor/lzma_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../../spec_helper.rb', __FILE__) 4 | 5 | describe Backup::Compressor::Lzma do 6 | before do 7 | Backup::Compressor::Lzma.any_instance.stubs(:utility).returns('lzma') 8 | end 9 | 10 | it 'should be a subclass of Compressor::Base' do 11 | Backup::Compressor::Lzma. 12 | superclass.should == Backup::Compressor::Base 13 | end 14 | 15 | describe '#initialize' do 16 | let(:compressor) { Backup::Compressor::Lzma.new } 17 | 18 | after { Backup::Compressor::Lzma.clear_defaults! } 19 | 20 | it 'should load pre-configured defaults' do 21 | Backup::Compressor::Lzma.any_instance.expects(:load_defaults!) 22 | compressor 23 | end 24 | 25 | context 'when no pre-configured defaults have been set' do 26 | it 'should use default values' do 27 | compressor.best.should be_false 28 | compressor.fast.should be_false 29 | end 30 | 31 | it 'should use the values given' do 32 | compressor = Backup::Compressor::Lzma.new do |c| 33 | c.best = true 34 | c.fast = true 35 | end 36 | 37 | compressor.best.should be_true 38 | compressor.fast.should be_true 39 | end 40 | end # context 'when no pre-configured defaults have been set' 41 | 42 | context 'when pre-configured defaults have been set' do 43 | before do 44 | Backup::Compressor::Lzma.defaults do |c| 45 | c.best = true 46 | c.fast = true 47 | end 48 | end 49 | 50 | it 'should use pre-configured defaults' do 51 | compressor.best.should be_true 52 | compressor.fast.should be_true 53 | end 54 | 55 | it 'should override pre-configured defaults' do 56 | compressor = Backup::Compressor::Lzma.new do |c| 57 | c.best = false 58 | c.fast = false 59 | end 60 | 61 | compressor.best.should be_false 62 | compressor.fast.should be_false 63 | end 64 | end # context 'when pre-configured defaults have been set' 65 | end # describe '#initialize' 66 | 67 | describe '#compress_with' do 68 | before do 69 | Backup::Compressor::Lzma.any_instance.expects(:log!) 70 | 71 | Backup::Logger.expects(:warn).with do |msg| 72 | msg.should match( 73 | /\[DEPRECATION WARNING\]\n Compressor::Lzma is being deprecated/ 74 | ) 75 | end 76 | end 77 | 78 | it 'should yield with the --best option' do 79 | compressor = Backup::Compressor::Lzma.new do |c| 80 | c.best = true 81 | end 82 | 83 | compressor.compress_with do |cmd, ext| 84 | cmd.should == 'lzma --best' 85 | ext.should == '.lzma' 86 | end 87 | end 88 | 89 | it 'should yield with the --fast option' do 90 | compressor = Backup::Compressor::Lzma.new do |c| 91 | c.fast = true 92 | end 93 | 94 | compressor.compress_with do |cmd, ext| 95 | cmd.should == 'lzma --fast' 96 | ext.should == '.lzma' 97 | end 98 | end 99 | 100 | it 'should prefer the --best option over --fast' do 101 | compressor = Backup::Compressor::Lzma.new do |c| 102 | c.best = true 103 | c.fast = true 104 | end 105 | 106 | compressor.compress_with do |cmd, ext| 107 | cmd.should == 'lzma --best' 108 | ext.should == '.lzma' 109 | end 110 | end 111 | 112 | it 'should yield with no options' do 113 | compressor = Backup::Compressor::Lzma.new 114 | 115 | compressor.compress_with do |cmd, ext| 116 | cmd.should == 'lzma' 117 | ext.should == '.lzma' 118 | end 119 | end 120 | 121 | end # describe '#compress_with' 122 | 123 | end 124 | -------------------------------------------------------------------------------- /lib/backup/packager.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Packager 5 | class << self 6 | include Backup::CLI::Helpers 7 | 8 | ## 9 | # Build the final package for the backup model. 10 | def package!(model) 11 | @package = model.package 12 | @encryptor = model.encryptor 13 | @splitter = model.splitter 14 | @pipeline = Pipeline.new 15 | 16 | Logger.message "Packaging the backup files..." 17 | procedure.call 18 | 19 | if @pipeline.success? 20 | Logger.message "Packaging Complete!" 21 | else 22 | raise Errors::Packager::PipelineError, 23 | "Failed to Create Backup Package\n" + 24 | @pipeline.error_messages 25 | end 26 | end 27 | 28 | private 29 | 30 | ## 31 | # Builds a chain of nested Procs which adds each command to a Pipeline 32 | # needed to package the final command to package the backup. 33 | # This is done so that the Encryptor and Splitter have the ability 34 | # to perform actions before and after the final command is executed. 35 | # No Encryptors currently utilize this, however the Splitter does. 36 | def procedure 37 | stack = [] 38 | 39 | ## 40 | # Initial `tar` command to package the temporary backup folder. 41 | # The command's output will then be either piped to the Encryptor 42 | # or the Splitter (if no Encryptor), or through `cat` into the final 43 | # output file if neither are configured. 44 | @pipeline << "#{ utility(:tar) } -cf - " + 45 | "-C '#{ Config.tmp_path }' '#{ @package.trigger }'" 46 | 47 | ## 48 | # If an Encryptor was configured, it will be called first 49 | # to add the encryption utility command to be piped through, 50 | # and amend the final package extension. 51 | # It's output will then be either piped into a Splitter, 52 | # or through `cat` into the final output file. 53 | if @encryptor 54 | stack << lambda do 55 | @encryptor.encrypt_with do |command, ext| 56 | @pipeline << command 57 | @package.extension << ext 58 | stack.shift.call 59 | end 60 | end 61 | end 62 | 63 | ## 64 | # If a Splitter was configured, the `split` utility command will be 65 | # added to the Pipeline to split the final output into multiple files. 66 | # Once the Proc executing the Pipeline has completed and returns back 67 | # to the Splitter, it will check the final output files to determine 68 | # if the backup was indeed split. 69 | # If so, it will set the package's chunk_suffixes. If not, it will 70 | # remove the '-aa' suffix from the only file created by `split`. 71 | # 72 | # If no Splitter was configured, the final file output will be 73 | # piped through `cat` into the final output file. 74 | if @splitter 75 | stack << lambda do 76 | @splitter.split_with do |command| 77 | @pipeline << command 78 | stack.shift.call 79 | end 80 | end 81 | else 82 | stack << lambda do 83 | outfile = File.join(Config.tmp_path, @package.basename) 84 | @pipeline << "cat > #{ outfile }" 85 | stack.shift.call 86 | end 87 | end 88 | 89 | ## 90 | # Last Proc to be called runs the Pipeline the procedure built. 91 | # Once complete, the call stack will unwind back through the 92 | # preceeding Procs in the stack (if any) 93 | stack << lambda { @pipeline.run } 94 | 95 | stack.shift 96 | end 97 | 98 | end 99 | end 100 | end 101 | -------------------------------------------------------------------------------- /spec-live/syncer/cloud/s3_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../../../spec_helper.rb', __FILE__) 4 | 5 | describe 'Syncer::Cloud::S3 - No Concurrency', 6 | :if => Backup::SpecLive::CONFIG['syncer']['cloud']['s3']['specs_enabled'] do 7 | let(:trigger) { 'syncer_cloud_s3' } 8 | let(:model) { h_set_trigger(trigger) } 9 | 10 | before do 11 | model # trigger model initialization so Fog is available 12 | create_sync_files 13 | clean_remote 14 | end 15 | 16 | after do 17 | clean_sync_dir 18 | clean_remote 19 | end 20 | 21 | it 'should work' do 22 | model.perform! 23 | remote_files.map {|file| [file.key, file.etag] }.sort.should == [ 24 | ["backups/dir_a/one.file", "d3b07384d113edec49eaa6238ad5ff00"], 25 | ["backups/dir_b/dir_c/three.file", "d3b07384d113edec49eaa6238ad5ff00"], 26 | ["backups/dir_b/two.file", "d3b07384d113edec49eaa6238ad5ff00"] 27 | ] 28 | 29 | update_sync_files 30 | 31 | model.perform! 32 | remote_files.map {|file| [file.key, file.etag] }.sort.should == [ 33 | ["backups/dir_a/dir_d/two.new", "14758f1afd44c09b7992073ccf00b43d"], 34 | ["backups/dir_a/one.file", "14758f1afd44c09b7992073ccf00b43d"], 35 | ["backups/dir_b/dir_c/three.file", "d3b07384d113edec49eaa6238ad5ff00"], 36 | ["backups/dir_b/one.new", "14758f1afd44c09b7992073ccf00b43d"] 37 | ] 38 | end 39 | 40 | private 41 | 42 | ## 43 | # Initial Files are MD5: d3b07384d113edec49eaa6238ad5ff00 44 | # 45 | # ├── dir_a 46 | # │   └── one.file 47 | # └── dir_b 48 | # ├── dir_c 49 | # │   └── three.file 50 | # ├── bad\xFFfile 51 | # └── two.file 52 | def create_sync_files 53 | clean_sync_dir 54 | 55 | %w{ dir_a dir_b/dir_c }.each do |dir| 56 | path = File.join(Backup::SpecLive::SYNC_PATH, dir) 57 | FileUtils.mkdir_p(path) 58 | end 59 | 60 | %W{ dir_a/one.file 61 | dir_b/two.file 62 | dir_b/bad\xFFfile 63 | dir_b/dir_c/three.file }.each do |file| 64 | path = File.join(Backup::SpecLive::SYNC_PATH, file) 65 | File.open(path, 'w') {|file| file.puts 'foo' } 66 | end 67 | end 68 | 69 | ## 70 | # Added/Updated Files are MD5: 14758f1afd44c09b7992073ccf00b43d 71 | # 72 | # ├── dir_a 73 | # │   ├── dir_d (add) 74 | # │   │   └── two.new (add) 75 | # │   └── one.file (update) 76 | # └── dir_b 77 | # ├── dir_c 78 | # │   └── three.file 79 | # ├── bad\377file 80 | # ├── one.new (add) 81 | # └── two.file (remove) 82 | def update_sync_files 83 | FileUtils.mkdir_p(File.join(Backup::SpecLive::SYNC_PATH, 'dir_a/dir_d')) 84 | %w{ dir_a/one.file 85 | dir_b/one.new 86 | dir_a/dir_d/two.new }.each do |file| 87 | path = File.join(Backup::SpecLive::SYNC_PATH, file) 88 | File.open(path, 'w') {|file| file.puts 'foobar' } 89 | end 90 | 91 | path = File.join(Backup::SpecLive::SYNC_PATH, 'dir_b/two.file') 92 | h_safety_check(path) 93 | FileUtils.rm(path) 94 | end 95 | 96 | def clean_sync_dir 97 | path = Backup::SpecLive::SYNC_PATH 98 | if File.directory?(path) 99 | h_safety_check(path) 100 | FileUtils.rm_r(path) 101 | end 102 | end 103 | 104 | # use a new connection for each request 105 | def connection 106 | @opts = Backup::SpecLive::CONFIG['syncer']['cloud']['s3'] 107 | Fog::Storage.new( 108 | :provider => 'AWS', 109 | :aws_access_key_id => @opts['access_key_id'], 110 | :aws_secret_access_key => @opts['secret_access_key'], 111 | :region => @opts['region'] 112 | ) 113 | end 114 | 115 | def remote_files 116 | bucket = connection.directories.get(@opts['bucket']) 117 | bucket.files.all(:prefix => 'backups') 118 | end 119 | 120 | def clean_remote 121 | remote_files.each {|file| file.destroy } 122 | end 123 | 124 | end 125 | -------------------------------------------------------------------------------- /lib/backup/storage/ftp.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | ## 4 | # Only load the Net::FTP library/gem when the Backup::Storage::FTP class is loaded 5 | require 'net/ftp' 6 | 7 | module Backup 8 | module Storage 9 | class FTP < Base 10 | 11 | ## 12 | # Server credentials 13 | attr_accessor :username, :password 14 | 15 | ## 16 | # Server IP Address and FTP port 17 | attr_accessor :ip, :port 18 | 19 | ## 20 | # Path to store backups to 21 | attr_accessor :path 22 | 23 | ## 24 | # use passive mode? 25 | attr_accessor :passive_mode 26 | 27 | ## 28 | # Creates a new instance of the storage object 29 | def initialize(model, storage_id = nil, &block) 30 | super(model, storage_id) 31 | 32 | @port ||= 21 33 | @path ||= 'backups' 34 | @passive_mode ||= false 35 | 36 | instance_eval(&block) if block_given? 37 | 38 | @path = path.sub(/^\~\//, '') 39 | end 40 | 41 | private 42 | 43 | ## 44 | # Establishes a connection to the remote server 45 | # 46 | # Note: 47 | # Since the FTP port is defined as a constant in the Net::FTP class, and 48 | # might be required to change by the user, we dynamically remove and 49 | # re-add the constant with the provided port value 50 | def connection 51 | if Net::FTP.const_defined?(:FTP_PORT) 52 | Net::FTP.send(:remove_const, :FTP_PORT) 53 | end; Net::FTP.send(:const_set, :FTP_PORT, port) 54 | 55 | Net::FTP.open(ip, username, password) do |ftp| 56 | ftp.passive = true if passive_mode 57 | yield ftp 58 | end 59 | end 60 | 61 | ## 62 | # Transfers the archived file to the specified remote server 63 | def transfer! 64 | remote_path = remote_path_for(@package) 65 | 66 | connection do |ftp| 67 | create_remote_path(remote_path, ftp) 68 | 69 | files_to_transfer_for(@package) do |local_file, remote_file| 70 | Logger.message "#{storage_name} started transferring " + 71 | "'#{ local_file }' to '#{ ip }'." 72 | ftp.put( 73 | File.join(local_path, local_file), 74 | File.join(remote_path, remote_file) 75 | ) 76 | end 77 | end 78 | end 79 | 80 | ## 81 | # Removes the transferred archive file(s) from the storage location. 82 | # Any error raised will be rescued during Cycling 83 | # and a warning will be logged, containing the error message. 84 | def remove!(package) 85 | remote_path = remote_path_for(package) 86 | 87 | connection do |ftp| 88 | transferred_files_for(package) do |local_file, remote_file| 89 | Logger.message "#{storage_name} started removing " + 90 | "'#{ local_file }' from '#{ ip }'." 91 | 92 | ftp.delete(File.join(remote_path, remote_file)) 93 | end 94 | 95 | ftp.rmdir(remote_path) 96 | end 97 | end 98 | 99 | ## 100 | # Creates (if they don't exist yet) all the directories on the remote 101 | # server in order to upload the backup file. Net::FTP does not support 102 | # paths to directories that don't yet exist when creating new 103 | # directories. Instead, we split the parts up in to an array (for each 104 | # '/') and loop through that to create the directories one by one. 105 | # Net::FTP raises an exception when the directory it's trying to create 106 | # already exists, so we have rescue it 107 | def create_remote_path(remote_path, ftp) 108 | path_parts = Array.new 109 | remote_path.split('/').each do |path_part| 110 | path_parts << path_part 111 | begin 112 | ftp.mkdir(path_parts.join('/')) 113 | rescue Net::FTPPermError; end 114 | end 115 | end 116 | 117 | end 118 | end 119 | end 120 | -------------------------------------------------------------------------------- /spec/compressor/custom_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../../spec_helper.rb', __FILE__) 4 | 5 | describe Backup::Compressor::Custom do 6 | let(:compressor) { Backup::Compressor::Custom.new } 7 | 8 | before(:all) do 9 | # CLI::Helpers#utility will raise an error 10 | # if the command is invalid or not set 11 | Backup::Compressor::Custom.send( 12 | :define_method, :utility, 13 | lambda {|arg| arg.to_s.empty? ? 'error' : "/path/to/#{ arg }" } 14 | ) 15 | end 16 | 17 | it 'should be a subclass of Compressor::Base' do 18 | Backup::Compressor::Custom. 19 | superclass.should == Backup::Compressor::Base 20 | end 21 | 22 | describe '#initialize' do 23 | let(:compressor) { Backup::Compressor::Custom.new } 24 | 25 | after { Backup::Compressor::Custom.clear_defaults! } 26 | 27 | it 'should load pre-configured defaults' do 28 | Backup::Compressor::Custom.any_instance.expects(:load_defaults!) 29 | compressor 30 | end 31 | 32 | it 'should call CLI::Helpers#utility to validate command' do 33 | Backup::Compressor::Custom.any_instance.expects(:utility) 34 | compressor 35 | end 36 | 37 | it 'should clean the command and extension for use with compress_with' do 38 | compressor = Backup::Compressor::Custom.new do |c| 39 | c.command = ' my_command --option foo ' 40 | c.extension = ' my_extension ' 41 | end 42 | 43 | compressor.command.should == ' my_command --option foo ' 44 | compressor.extension.should == ' my_extension ' 45 | 46 | compressor.expects(:log!) 47 | compressor.compress_with do |cmd, ext| 48 | cmd.should == '/path/to/my_command --option foo' 49 | ext.should == 'my_extension' 50 | end 51 | end 52 | 53 | context 'when no pre-configured defaults have been set' do 54 | it 'should use default values' do 55 | compressor.command.should be_nil 56 | compressor.extension.should be_nil 57 | 58 | compressor.instance_variable_get(:@cmd).should == 'error' 59 | compressor.instance_variable_get(:@ext).should == '' 60 | end 61 | 62 | it 'should use the values given' do 63 | compressor = Backup::Compressor::Custom.new do |c| 64 | c.command = 'my_command' 65 | c.extension = 'my_extension' 66 | end 67 | 68 | compressor.command.should == 'my_command' 69 | compressor.extension.should == 'my_extension' 70 | 71 | compressor.instance_variable_get(:@cmd).should == '/path/to/my_command' 72 | compressor.instance_variable_get(:@ext).should == 'my_extension' 73 | end 74 | end # context 'when no pre-configured defaults have been set' 75 | 76 | context 'when pre-configured defaults have been set' do 77 | before do 78 | Backup::Compressor::Custom.defaults do |c| 79 | c.command = 'default_command' 80 | c.extension = 'default_extension' 81 | end 82 | end 83 | 84 | it 'should use pre-configured defaults' do 85 | compressor.command.should == 'default_command' 86 | compressor.extension.should == 'default_extension' 87 | 88 | compressor.instance_variable_get(:@cmd).should == '/path/to/default_command' 89 | compressor.instance_variable_get(:@ext).should == 'default_extension' 90 | end 91 | 92 | it 'should override pre-configured defaults' do 93 | compressor = Backup::Compressor::Custom.new do |c| 94 | c.command = 'new_command' 95 | c.extension = 'new_extension' 96 | end 97 | 98 | compressor.command.should == 'new_command' 99 | compressor.extension.should == 'new_extension' 100 | 101 | compressor.instance_variable_get(:@cmd).should == '/path/to/new_command' 102 | compressor.instance_variable_get(:@ext).should == 'new_extension' 103 | end 104 | end # context 'when pre-configured defaults have been set' 105 | end # describe '#initialize' 106 | end 107 | -------------------------------------------------------------------------------- /spec/splitter_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../spec_helper.rb', __FILE__) 4 | 5 | describe Backup::Splitter do 6 | let(:model) { Backup::Model.new(:test_trigger, 'test label') } 7 | let(:splitter) { Backup::Splitter.new(model, 250) } 8 | let(:package) { mock } 9 | 10 | describe '#initialize' do 11 | it 'should set instance variables' do 12 | splitter.instance_variable_get(:@model).should be(model) 13 | splitter.instance_variable_get(:@chunk_size).should be(250) 14 | end 15 | end 16 | 17 | describe '#split_with' do 18 | it 'should yield the split command, performing before/after methods' do 19 | s = sequence '' 20 | given_block = mock 21 | block = lambda {|arg| given_block.got(arg) } 22 | splitter.instance_variable_set(:@split_command, 'split command') 23 | 24 | splitter.expects(:before_packaging).in_sequence(s) 25 | given_block.expects(:got).in_sequence(s).with('split command') 26 | splitter.expects(:after_packaging).in_sequence(s) 27 | 28 | splitter.split_with(&block) 29 | end 30 | end 31 | 32 | # Note: using a 'M' suffix for the byte size is not OSX compatible 33 | describe '#before_packaging' do 34 | before do 35 | model.instance_variable_set(:@package, package) 36 | splitter.expects(:utility).with(:split).returns('split') 37 | package.expects(:basename).returns('base_filename') 38 | end 39 | 40 | it 'should set @package and @split_command' do 41 | Backup::Logger.expects(:message).with( 42 | 'Splitter configured with a chunk size of 250MB.' 43 | ) 44 | splitter.send(:before_packaging) 45 | 46 | splitter.instance_variable_get(:@package).should be(package) 47 | 48 | split_suffix = File.join(Backup::Config.tmp_path, 'base_filename-') 49 | splitter.instance_variable_get(:@split_command).should == 50 | "split -b 250m - '#{ split_suffix }'" 51 | end 52 | end 53 | 54 | describe '#after_packaging' do 55 | before do 56 | splitter.instance_variable_set(:@package, package) 57 | end 58 | 59 | context 'when splitting occurred during packaging' do 60 | before do 61 | splitter.expects(:chunk_suffixes).returns(['aa', 'ab']) 62 | end 63 | 64 | it 'should set the chunk_suffixes for the package' do 65 | package.expects(:chunk_suffixes=).with(['aa', 'ab']) 66 | splitter.send(:after_packaging) 67 | end 68 | end 69 | 70 | context 'when splitting did not occur during packaging' do 71 | before do 72 | splitter.expects(:chunk_suffixes).returns(['aa']) 73 | package.expects(:basename).twice.returns('base_filename') 74 | end 75 | 76 | it 'should remove the suffix from the only package file' do 77 | package.expects(:chunk_suffixes=).never 78 | FileUtils.expects(:mv).with( 79 | File.join(Backup::Config.tmp_path, 'base_filename-aa'), 80 | File.join(Backup::Config.tmp_path, 'base_filename') 81 | ) 82 | splitter.send(:after_packaging) 83 | end 84 | end 85 | end # describe '#after_packaging' 86 | 87 | describe '#chunk_suffixes' do 88 | before do 89 | splitter.expects(:chunks).returns( 90 | ['/path/to/file.tar-aa', '/path/to/file.tar-ab'] 91 | ) 92 | end 93 | 94 | it 'should return an array of chunk suffixes' do 95 | splitter.send(:chunk_suffixes).should == ['aa', 'ab'] 96 | end 97 | end 98 | 99 | describe '#chunks' do 100 | before do 101 | splitter.instance_variable_set(:@package, package) 102 | package.expects(:basename).returns('base_filename') 103 | FileUtils.unstub(:touch) 104 | end 105 | 106 | it 'should return a sorted array of chunked file paths' do 107 | Dir.mktmpdir do |dir| 108 | Backup::Config.expects(:tmp_path).returns(dir) 109 | FileUtils.touch(File.join(dir, 'base_filename-aa')) 110 | FileUtils.touch(File.join(dir, 'base_filename-ab')) 111 | 112 | splitter.send(:chunks).should == [ 113 | File.join(dir, 'base_filename-aa'), 114 | File.join(dir, 'base_filename-ab') 115 | ] 116 | end 117 | end 118 | end 119 | 120 | end 121 | -------------------------------------------------------------------------------- /lib/backup/cleaner.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Cleaner 5 | class << self 6 | 7 | ## 8 | # Logs warnings if any temporary files still exist 9 | # from the last time this model/trigger was run, 10 | # then removes the files. 11 | def prepare(model) 12 | @model = model 13 | 14 | messages = [] 15 | if packaging_folder_dirty? 16 | messages << <<-EOS 17 | The temporary backup folder still contains files! 18 | '#{ File.join(Config.tmp_path, @model.trigger) }' 19 | These files will now be removed. 20 | EOS 21 | FileUtils.rm_rf(File.join(Config.tmp_path, @model.trigger)) 22 | end 23 | 24 | package_files = tmp_path_package_files 25 | unless package_files.empty? 26 | # the chances that tmp_path would be dirty 27 | # AND package files exist are practically nil 28 | messages << ('-' * 74) unless messages.empty? 29 | 30 | messages << <<-EOS 31 | The temporary backup folder '#{ Config.tmp_path }' 32 | appears to contain the package files from the previous backup! 33 | #{ package_files.join("\n") } 34 | These files will now be removed. 35 | EOS 36 | package_files.each {|file| FileUtils.rm_f(file) } 37 | end 38 | 39 | unless messages.empty? 40 | Logger.warn Errors::CleanerError.new(<<-EOS) 41 | Cleanup Warning 42 | #{ messages.join("\n") } 43 | Please check the log for messages and/or your notifications 44 | concerning this backup: '#{ @model.label } (#{ @model.trigger })' 45 | The temporary files which had to be removed should not have existed. 46 | EOS 47 | end 48 | end 49 | 50 | ## 51 | # Remove the temporary folder used during packaging 52 | def remove_packaging(model) 53 | Logger.message "Cleaning up the temporary files..." 54 | FileUtils.rm_rf(File.join(Config.tmp_path, model.trigger)) 55 | end 56 | 57 | ## 58 | # Remove the final package files from tmp_path 59 | # Note: 'force' is used, since a Local Storage may *move* these files. 60 | def remove_package(package) 61 | Logger.message "Cleaning up the package files..." 62 | package.filenames.each do |file| 63 | FileUtils.rm_f(File.join(Config.tmp_path, file)) 64 | end 65 | end 66 | 67 | ## 68 | # Logs warnings if any temporary files still exist 69 | # when errors occur during the backup 70 | def warnings(model) 71 | @model = model 72 | 73 | messages = [] 74 | if packaging_folder_dirty? 75 | messages << <<-EOS 76 | The temporary backup folder still contains files! 77 | '#{ File.join(Config.tmp_path, @model.trigger) }' 78 | This folder may contain completed Archives and/or Database backups. 79 | EOS 80 | end 81 | 82 | package_files = tmp_path_package_files 83 | unless package_files.empty? 84 | # the chances that tmp_path would be dirty 85 | # AND package files exist are practically nil 86 | messages << ('-' * 74) unless messages.empty? 87 | 88 | messages << <<-EOS 89 | The temporary backup folder '#{ Config.tmp_path }' 90 | appears to contain the backup files which were to be stored: 91 | #{ package_files.join("\n") } 92 | EOS 93 | end 94 | 95 | unless messages.empty? 96 | Logger.warn Errors::CleanerError.new(<<-EOS) 97 | Cleanup Warning 98 | #{ messages.join("\n") } 99 | Make sure you check these files before the next scheduled backup for 100 | '#{ @model.label } (#{ @model.trigger })' 101 | These files will be removed at that time! 102 | EOS 103 | end 104 | end 105 | 106 | private 107 | 108 | def packaging_folder_dirty? 109 | !Dir[File.join(Config.tmp_path, @model.trigger, '*')].empty? 110 | end 111 | 112 | def tmp_path_package_files 113 | Dir[File.join( 114 | Config.tmp_path, 115 | "????.??.??.??.??.??.#{ @model.trigger }.tar{,[.-]*}" 116 | )] 117 | end 118 | 119 | end 120 | end 121 | end 122 | -------------------------------------------------------------------------------- /lib/backup/database/redis.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Database 5 | class Redis < Base 6 | 7 | ## 8 | # Name of and path to the database that needs to get dumped 9 | attr_accessor :name, :path 10 | 11 | ## 12 | # Credentials for the specified database 13 | attr_accessor :password 14 | 15 | ## 16 | # Connectivity options 17 | attr_accessor :host, :port, :socket 18 | 19 | ## 20 | # Determines whether Backup should invoke the SAVE command through 21 | # the 'redis-cli' utility to persist the most recent data before 22 | # copying over the dump file 23 | attr_accessor :invoke_save 24 | 25 | ## 26 | # Additional "redis-cli" options 27 | attr_accessor :additional_options 28 | 29 | ## 30 | # Path to the redis-cli utility (optional) 31 | attr_accessor :redis_cli_utility 32 | 33 | attr_deprecate :utility_path, :version => '3.0.21', 34 | :replacement => :redis_cli_utility 35 | 36 | ## 37 | # Creates a new instance of the Redis database object 38 | def initialize(model, &block) 39 | super(model) 40 | 41 | @additional_options ||= Array.new 42 | 43 | instance_eval(&block) if block_given? 44 | 45 | @name ||= 'dump' 46 | @redis_cli_utility ||= utility('redis-cli') 47 | end 48 | 49 | ## 50 | # Performs the Redis backup by using the 'cp' unix utility 51 | # to copy the persisted Redis dump file to the Backup archive. 52 | # Additionally, when 'invoke_save' is set to true, it'll tell 53 | # the Redis server to persist the current state to the dump file 54 | # before copying the dump to get the most recent updates in to the backup 55 | def perform! 56 | super 57 | 58 | invoke_save! if invoke_save 59 | copy! 60 | end 61 | 62 | private 63 | 64 | ## 65 | # Tells Redis to persist the current state of the 66 | # in-memory database to the persisted dump file 67 | def invoke_save! 68 | response = run("#{ redis_cli_utility } #{ credential_options } " + 69 | "#{ connectivity_options } #{ user_options } SAVE") 70 | unless response =~ /OK/ 71 | raise Errors::Database::Redis::CommandError, <<-EOS 72 | Could not invoke the Redis SAVE command. 73 | The #{ database } file might not contain the most recent data. 74 | Please check if the server is running, the credentials (if any) are correct, 75 | and the host/port/socket are correct. 76 | 77 | Redis CLI response: #{ response } 78 | EOS 79 | end 80 | end 81 | 82 | ## 83 | # Performs the copy command to copy over the Redis dump file to the Backup archive 84 | def copy! 85 | src_path = File.join(path, database) 86 | unless File.exist?(src_path) 87 | raise Errors::Database::Redis::NotFoundError, <<-EOS 88 | Redis database dump not found 89 | File path was #{ src_path } 90 | EOS 91 | end 92 | 93 | dst_path = File.join(@dump_path, database) 94 | if @model.compressor 95 | @model.compressor.compress_with do |command, ext| 96 | run("#{ command } -c #{ src_path } > #{ dst_path + ext }") 97 | end 98 | else 99 | FileUtils.cp(src_path, dst_path) 100 | end 101 | end 102 | 103 | ## 104 | # Returns the Redis database file name 105 | def database 106 | "#{ name }.rdb" 107 | end 108 | 109 | ## 110 | # Builds the Redis credentials syntax to authenticate the user 111 | # to perform the database dumping process 112 | def credential_options 113 | password.to_s.empty? ? '' : "-a '#{password}'" 114 | end 115 | 116 | ## 117 | # Builds the Redis connectivity options syntax to connect the user 118 | # to perform the database dumping process 119 | def connectivity_options 120 | %w[host port socket].map do |option| 121 | next if send(option).to_s.empty? 122 | "-#{option[0,1]} '#{send(option)}'" 123 | end.compact.join(' ') 124 | end 125 | 126 | ## 127 | # Builds a Redis compatible string for the 128 | # additional options specified by the user 129 | def user_options 130 | @additional_options.join(' ') 131 | end 132 | 133 | end 134 | end 135 | end 136 | -------------------------------------------------------------------------------- /lib/backup/storage/cycler.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Storage 5 | module Cycler 6 | class << self 7 | 8 | ## 9 | # Adds the given +package+ to the YAML storage file corresponding 10 | # to the given +storage+ and Package#trigger (Model#trigger). 11 | # Then, calls the +storage+ to remove the files for any older 12 | # packages that were removed from the YAML storage file. 13 | def cycle!(storage, package) 14 | @storage, @package = storage, package 15 | @storage_file = storage_file 16 | 17 | update_storage_file! 18 | remove_packages! 19 | end 20 | 21 | private 22 | 23 | ## 24 | # Updates the YAML data file according to the #keep setting 25 | # for the storage and sets the @packages_to_remove 26 | def update_storage_file! 27 | packages = yaml_load.unshift(@package) 28 | excess = packages.count - @storage.keep.to_i 29 | @packages_to_remove = (excess > 0) ? packages.pop(excess) : [] 30 | yaml_save(packages) 31 | end 32 | 33 | ## 34 | # Calls the @storage to remove any old packages 35 | # which were cycled out of the storage file. 36 | def remove_packages! 37 | @packages_to_remove.each do |pkg| 38 | begin 39 | @storage.send(:remove!, pkg) 40 | rescue => err 41 | Logger.warn Errors::Storage::CyclerError.wrap(err, <<-EOS) 42 | There was a problem removing the following package: 43 | Trigger: #{pkg.trigger} :: Dated: #{pkg.time} 44 | Package included the following #{ pkg.filenames.count } file(s): 45 | #{ pkg.filenames.join("\n") } 46 | EOS 47 | end 48 | end 49 | end 50 | 51 | ## 52 | # Return full path to the YAML data file, 53 | # based on the current values of @storage and @package 54 | def storage_file 55 | type = @storage.class.to_s.split('::').last 56 | suffix = @storage.storage_id.to_s.strip.gsub(/[\W\s]/, '_') 57 | filename = suffix.empty? ? type : "#{type}-#{suffix}" 58 | File.join(Config.data_path, @package.trigger, "#{filename}.yml") 59 | end 60 | 61 | ## 62 | # Load Package objects from YAML file. 63 | # Returns an Array, sorted by @time descending. 64 | # i.e. most recent is objects[0] 65 | def yaml_load 66 | packages = [] 67 | if File.exist?(@storage_file) && !File.zero?(@storage_file) 68 | packages = check_upgrade( 69 | YAML.load_file(@storage_file).sort do |a, b| 70 | b.instance_variable_get(:@time) <=> a.instance_variable_get(:@time) 71 | end 72 | ) 73 | end 74 | packages 75 | end 76 | 77 | ## 78 | # Store the given package objects to the YAML data file. 79 | def yaml_save(packages) 80 | File.open(@storage_file, 'w') do |file| 81 | file.write(packages.to_yaml) 82 | end 83 | end 84 | 85 | ## 86 | # Upgrade the objects loaded from the YAML file, if needed. 87 | def check_upgrade(objects) 88 | if objects.any? {|obj| obj.class.to_s =~ /Backup::Storage/ } 89 | # Version <= 3.0.20 90 | model = @storage.instance_variable_get(:@model) 91 | v3_0_20 = objects.any? {|obj| obj.instance_variable_defined?(:@version) } 92 | objects.map! do |obj| 93 | if v3_0_20 # Version == 3.0.20 94 | filename = obj.instance_variable_get(:@filename)[20..-1] 95 | chunk_suffixes = obj.instance_variable_get(:@chunk_suffixes) 96 | else # Version <= 3.0.19 97 | filename = obj.instance_variable_get(:@remote_file)[20..-1] 98 | chunk_suffixes = [] 99 | end 100 | time = obj.instance_variable_get(:@time) 101 | extension = filename.match(/\.(tar.*)$/)[1] 102 | 103 | package = Backup::Package.new(model) 104 | package.instance_variable_set(:@time, time) 105 | package.extension = extension 106 | package.chunk_suffixes = chunk_suffixes 107 | 108 | package 109 | end 110 | end 111 | objects 112 | end 113 | 114 | end 115 | end 116 | end 117 | end 118 | -------------------------------------------------------------------------------- /lib/backup/pipeline.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | class Pipeline 5 | include Backup::CLI::Helpers 6 | 7 | attr_reader :stderr, :errors 8 | 9 | def initialize 10 | @commands = [] 11 | @errors = [] 12 | @stderr = '' 13 | end 14 | 15 | ## 16 | # Adds a command to be executed in the pipeline. 17 | # Each command will be run in the order in which it was added, 18 | # with it's output being piped to the next command. 19 | def <<(command) 20 | @commands << command 21 | end 22 | 23 | ## 24 | # Runs the command line from `#pipeline` and collects STDOUT/STDERR. 25 | # STDOUT is then parsed to determine the exit status of each command. 26 | # For each command with a non-zero exit status, a SystemCallError is 27 | # created and added to @errors. All STDERR output is set in @stderr. 28 | # 29 | # Note that there is no accumulated STDOUT from the commands themselves. 30 | # Also, the last command should not attempt to write to STDOUT. 31 | # Any output on STDOUT from the final command will be sent to STDERR. 32 | # This in itself will not cause #run to fail, but will log warnings 33 | # when all commands exit with non-zero status. 34 | # 35 | # Use `#success?` to determine if all commands in the pipeline succeeded. 36 | # If `#success?` returns `false`, use `#error_messages` to get an error report. 37 | def run 38 | Open4.popen4(pipeline) do |pid, stdin, stdout, stderr| 39 | pipestatus = stdout.read.gsub("\n", '').split(':').sort 40 | pipestatus.each do |status| 41 | index, exitstatus = status.split('|').map(&:to_i) 42 | if exitstatus > 0 43 | command = command_name(@commands[index]) 44 | @errors << SystemCallError.new( 45 | "'#{ command }' returned exit code: #{ exitstatus }", exitstatus 46 | ) 47 | end 48 | end 49 | @stderr = stderr.read.strip 50 | end 51 | Logger.warn(stderr_messages) if success? && stderr_messages 52 | rescue Exception => e 53 | raise Errors::Pipeline::ExecutionError.wrap(e) 54 | end 55 | 56 | def success? 57 | @errors.empty? 58 | end 59 | 60 | ## 61 | # Returns a multi-line String, reporting all STDERR messages received 62 | # from the commands in the pipeline (if any), along with the SystemCallError 63 | # (Errno) message for each command which had a non-zero exit status. 64 | # 65 | # Each error is wrapped by Backup::Errors to provide formatting. 66 | def error_messages 67 | @error_messages ||= (stderr_messages || '') + 68 | "The following system errors were returned:\n" + 69 | @errors.map {|err| Errors::Error.wrap(err).message }.join("\n") 70 | end 71 | 72 | private 73 | 74 | ## 75 | # Each command is added as part of the pipeline, grouped with an `echo` 76 | # command to pass along the command's index in @commands and it's exit status. 77 | # The command's STDERR is redirected to FD#4, and the `echo` command to 78 | # report the "index|exit status" is redirected to FD#3. 79 | # Each command's STDOUT will be connected to the STDIN of the next subshell. 80 | # The entire pipeline is run within a container group, which redirects 81 | # FD#3 to STDOUT and FD#4 to STDERR so these can be collected. 82 | # FD#1 is redirected to STDERR so that any output from the final command 83 | # on STDOUT will generate warnings, since the final command should not 84 | # attempt to write to STDOUT, as this would interfere with collecting 85 | # the exit statuses. 86 | # 87 | # There is no guarantee as to the order of this output, which is why the 88 | # command's index in @commands is passed along with it's exit status. 89 | # And, if multiple commands output messages on STDERR, those messages 90 | # may be interleaved. Interleaving of the "index|exit status" outputs 91 | # should not be an issue, given the small byte size of the data being written. 92 | def pipeline 93 | parts = [] 94 | @commands.each_with_index do |command, index| 95 | parts << %Q[{ #{ command } 2>&4 ; echo "#{ index }|$?:" >&3 ; }] 96 | end 97 | %Q[{ #{ parts.join(' | ') } } 3>&1 1>&2 4>&2] 98 | end 99 | 100 | def stderr_messages 101 | @stderr_messages ||= @stderr.empty? ? false : <<-EOS.gsub(/^ +/, ' ') 102 | Pipeline STDERR Messages: 103 | (Note: may be interleaved if multiple commands returned error messages) 104 | 105 | #{ @stderr } 106 | EOS 107 | end 108 | 109 | end 110 | end 111 | -------------------------------------------------------------------------------- /lib/backup/storage/rsync.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | ## 4 | # Only load the Net::SSH library when the Backup::Storage::RSync class is loaded 5 | Backup::Dependency.load('net-ssh') 6 | 7 | module Backup 8 | module Storage 9 | class RSync < Base 10 | include Backup::CLI::Helpers 11 | 12 | ## 13 | # Server credentials 14 | attr_accessor :username, :password 15 | 16 | ## 17 | # Server IP Address and SSH port 18 | attr_accessor :ip, :port 19 | 20 | ## 21 | # Path to store backups to 22 | attr_accessor :path 23 | 24 | ## 25 | # Flag to use local backups 26 | attr_accessor :local 27 | 28 | ## 29 | # Creates a new instance of the storage object 30 | def initialize(model, storage_id = nil, &block) 31 | super(model, storage_id) 32 | 33 | @port ||= 22 34 | @path ||= 'backups' 35 | @local ||= false 36 | 37 | instance_eval(&block) if block_given? 38 | 39 | @path = path.sub(/^\~\//, '') 40 | end 41 | 42 | private 43 | 44 | ## 45 | # This is the remote path to where the backup files will be stored 46 | # 47 | # Note: This overrides the superclass' method 48 | def remote_path_for(package) 49 | File.join(path, package.trigger) 50 | end 51 | 52 | ## 53 | # Establishes a connection to the remote server 54 | def connection 55 | Net::SSH.start( 56 | ip, username, :password => password, :port => port 57 | ) {|ssh| yield ssh } 58 | end 59 | 60 | ## 61 | # Transfers the archived file to the specified remote server 62 | def transfer! 63 | write_password_file! unless local 64 | 65 | remote_path = remote_path_for(@package) 66 | 67 | create_remote_path!(remote_path) 68 | 69 | files_to_transfer_for(@package) do |local_file, remote_file| 70 | if local 71 | Logger.message "#{storage_name} started transferring " + 72 | "'#{ local_file }' to '#{ remote_path }'." 73 | run( 74 | "#{ utility(:rsync) } '#{ File.join(local_path, local_file) }' " + 75 | "'#{ File.join(remote_path, remote_file) }'" 76 | ) 77 | else 78 | Logger.message "#{storage_name} started transferring " + 79 | "'#{ local_file }' to '#{ ip }'." 80 | run( 81 | "#{ utility(:rsync) } #{ rsync_options } #{ rsync_port } " + 82 | "#{ rsync_password_file } '#{ File.join(local_path, local_file) }' " + 83 | "'#{ username }@#{ ip }:#{ File.join(remote_path, remote_file) }'" 84 | ) 85 | end 86 | end 87 | 88 | ensure 89 | remove_password_file! unless local 90 | end 91 | 92 | ## 93 | # Note: Storage::RSync doesn't cycle 94 | def remove!; end 95 | 96 | ## 97 | # Creates (if they don't exist yet) all the directories on the remote 98 | # server in order to upload the backup file. 99 | def create_remote_path!(remote_path) 100 | if @local 101 | FileUtils.mkdir_p(remote_path) 102 | else 103 | connection do |ssh| 104 | ssh.exec!("mkdir -p '#{ remote_path }'") 105 | end 106 | end 107 | end 108 | 109 | ## 110 | # Writes the provided password to a temporary file so that 111 | # the rsync utility can read the password from this file 112 | def write_password_file! 113 | unless password.nil? 114 | @password_file = Tempfile.new('backup-rsync-password') 115 | @password_file.write(password) 116 | @password_file.close 117 | end 118 | end 119 | 120 | ## 121 | # Removes the previously created @password_file 122 | # (temporary file containing the password) 123 | def remove_password_file! 124 | @password_file.delete if @password_file 125 | @password_file = nil 126 | end 127 | 128 | ## 129 | # Returns Rsync syntax for using a password file 130 | def rsync_password_file 131 | "--password-file='#{@password_file.path}'" if @password_file 132 | end 133 | 134 | ## 135 | # Returns Rsync syntax for defining a port to connect to 136 | def rsync_port 137 | "-e 'ssh -p #{port}'" 138 | end 139 | 140 | ## 141 | # RSync options 142 | # -z = Compresses the bytes that will be transferred to reduce bandwidth usage 143 | def rsync_options 144 | "-z" 145 | end 146 | 147 | end 148 | end 149 | end 150 | -------------------------------------------------------------------------------- /spec/notifier/prowl_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../../spec_helper.rb', __FILE__) 4 | 5 | describe Backup::Notifier::Prowl do 6 | let(:model) { Backup::Model.new(:test_trigger, 'test label') } 7 | let(:notifier) do 8 | Backup::Notifier::Prowl.new(model) do |prowl| 9 | prowl.application = 'application' 10 | prowl.api_key = 'api_key' 11 | end 12 | end 13 | 14 | it 'should be a subclass of Notifier::Base' do 15 | Backup::Notifier::Prowl. 16 | superclass.should == Backup::Notifier::Base 17 | end 18 | 19 | describe '#initialize' do 20 | after { Backup::Notifier::Prowl.clear_defaults! } 21 | 22 | it 'should load pre-configured defaults through Base' do 23 | Backup::Notifier::Prowl.any_instance.expects(:load_defaults!) 24 | notifier 25 | end 26 | 27 | it 'should pass the model reference to Base' do 28 | notifier.instance_variable_get(:@model).should == model 29 | end 30 | 31 | context 'when no pre-configured defaults have been set' do 32 | it 'should use the values given' do 33 | notifier.application.should == 'application' 34 | notifier.api_key.should == 'api_key' 35 | 36 | notifier.on_success.should == true 37 | notifier.on_warning.should == true 38 | notifier.on_failure.should == true 39 | end 40 | 41 | it 'should use default values if none are given' do 42 | notifier = Backup::Notifier::Prowl.new(model) 43 | notifier.application.should be_nil 44 | notifier.api_key.should be_nil 45 | 46 | notifier.on_success.should == true 47 | notifier.on_warning.should == true 48 | notifier.on_failure.should == true 49 | end 50 | end # context 'when no pre-configured defaults have been set' 51 | 52 | context 'when pre-configured defaults have been set' do 53 | before do 54 | Backup::Notifier::Prowl.defaults do |n| 55 | n.application = 'default_app' 56 | n.api_key = 'default_api_key' 57 | 58 | n.on_success = false 59 | n.on_warning = false 60 | n.on_failure = false 61 | end 62 | end 63 | 64 | it 'should use pre-configured defaults' do 65 | notifier = Backup::Notifier::Prowl.new(model) 66 | notifier.application.should == 'default_app' 67 | notifier.api_key.should == 'default_api_key' 68 | 69 | notifier.on_success.should == false 70 | notifier.on_warning.should == false 71 | notifier.on_failure.should == false 72 | end 73 | 74 | it 'should override pre-configured defaults' do 75 | notifier = Backup::Notifier::Prowl.new(model) do |n| 76 | n.application = 'new_app' 77 | n.api_key = 'new_api_key' 78 | 79 | n.on_success = false 80 | n.on_warning = true 81 | n.on_failure = true 82 | end 83 | 84 | notifier.application.should == 'new_app' 85 | notifier.api_key.should == 'new_api_key' 86 | 87 | notifier.on_success.should == false 88 | notifier.on_warning.should == true 89 | notifier.on_failure.should == true 90 | end 91 | end # context 'when pre-configured defaults have been set' 92 | end # describe '#initialize' 93 | 94 | describe '#notify!' do 95 | context 'when status is :success' do 96 | it 'should send Success message' do 97 | notifier.expects(:send_message).with( 98 | '[Backup::Success]' 99 | ) 100 | notifier.send(:notify!, :success) 101 | end 102 | end 103 | 104 | context 'when status is :warning' do 105 | it 'should send Warning message' do 106 | notifier.expects(:send_message).with( 107 | '[Backup::Warning]' 108 | ) 109 | notifier.send(:notify!, :warning) 110 | end 111 | end 112 | 113 | context 'when status is :failure' do 114 | it 'should send Failure message' do 115 | notifier.expects(:send_message).with( 116 | '[Backup::Failure]' 117 | ) 118 | notifier.send(:notify!, :failure) 119 | end 120 | end 121 | end # describe '#notify!' 122 | 123 | describe '#send_message' do 124 | it 'should send the given message' do 125 | client = mock 126 | Prowler.expects(:new).with( 127 | :application => 'application', :api_key => 'api_key' 128 | ).returns(client) 129 | client.expects(:notify).with( 130 | 'a message', 131 | 'test label (test_trigger)' 132 | ) 133 | 134 | notifier.send(:send_message, 'a message') 135 | end 136 | end 137 | 138 | end 139 | -------------------------------------------------------------------------------- /lib/backup.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | # Load Ruby Core Libraries 4 | require 'rubygems' 5 | require 'fileutils' 6 | require 'tempfile' 7 | require 'yaml' 8 | require 'etc' 9 | 10 | require 'open4' 11 | require 'thor' 12 | 13 | ## 14 | # The Backup Ruby Gem 15 | module Backup 16 | 17 | ## 18 | # Backup's internal paths 19 | LIBRARY_PATH = File.join(File.dirname(__FILE__), 'backup') 20 | CLI_PATH = File.join(LIBRARY_PATH, 'cli') 21 | STORAGE_PATH = File.join(LIBRARY_PATH, 'storage') 22 | DATABASE_PATH = File.join(LIBRARY_PATH, 'database') 23 | COMPRESSOR_PATH = File.join(LIBRARY_PATH, 'compressor') 24 | ENCRYPTOR_PATH = File.join(LIBRARY_PATH, 'encryptor') 25 | NOTIFIER_PATH = File.join(LIBRARY_PATH, 'notifier') 26 | SYNCER_PATH = File.join(LIBRARY_PATH, 'syncer') 27 | TEMPLATE_PATH = File.expand_path('../../templates', __FILE__) 28 | 29 | ## 30 | # Autoload Backup CLI files 31 | module CLI 32 | autoload :Helpers, File.join(CLI_PATH, 'helpers') 33 | autoload :Utility, File.join(CLI_PATH, 'utility') 34 | end 35 | 36 | ## 37 | # Autoload Backup storage files 38 | module Storage 39 | autoload :Base, File.join(STORAGE_PATH, 'base') 40 | autoload :Cycler, File.join(STORAGE_PATH, 'cycler') 41 | autoload :S3, File.join(STORAGE_PATH, 's3') 42 | autoload :CloudFiles, File.join(STORAGE_PATH, 'cloudfiles') 43 | autoload :Ninefold, File.join(STORAGE_PATH, 'ninefold') 44 | autoload :Dropbox, File.join(STORAGE_PATH, 'dropbox') 45 | autoload :FTP, File.join(STORAGE_PATH, 'ftp') 46 | autoload :SFTP, File.join(STORAGE_PATH, 'sftp') 47 | autoload :SCP, File.join(STORAGE_PATH, 'scp') 48 | autoload :RSync, File.join(STORAGE_PATH, 'rsync') 49 | autoload :Local, File.join(STORAGE_PATH, 'local') 50 | end 51 | 52 | ## 53 | # Autoload Backup syncer files 54 | module Syncer 55 | autoload :Base, File.join(SYNCER_PATH, 'base') 56 | module Cloud 57 | autoload :Base, File.join(SYNCER_PATH, 'cloud', 'base') 58 | autoload :CloudFiles, File.join(SYNCER_PATH, 'cloud', 'cloud_files') 59 | autoload :S3, File.join(SYNCER_PATH, 'cloud', 's3') 60 | end 61 | module RSync 62 | autoload :Base, File.join(SYNCER_PATH, 'rsync', 'base') 63 | autoload :Local, File.join(SYNCER_PATH, 'rsync', 'local') 64 | autoload :Push, File.join(SYNCER_PATH, 'rsync', 'push') 65 | autoload :Pull, File.join(SYNCER_PATH, 'rsync', 'pull') 66 | end 67 | end 68 | 69 | ## 70 | # Autoload Backup database files 71 | module Database 72 | autoload :Base, File.join(DATABASE_PATH, 'base') 73 | autoload :MySQL, File.join(DATABASE_PATH, 'mysql') 74 | autoload :PostgreSQL, File.join(DATABASE_PATH, 'postgresql') 75 | autoload :MongoDB, File.join(DATABASE_PATH, 'mongodb') 76 | autoload :Redis, File.join(DATABASE_PATH, 'redis') 77 | autoload :Riak, File.join(DATABASE_PATH, 'riak') 78 | end 79 | 80 | ## 81 | # Autoload compressor files 82 | module Compressor 83 | autoload :Base, File.join(COMPRESSOR_PATH, 'base') 84 | autoload :Gzip, File.join(COMPRESSOR_PATH, 'gzip') 85 | autoload :Bzip2, File.join(COMPRESSOR_PATH, 'bzip2') 86 | autoload :Custom, File.join(COMPRESSOR_PATH, 'custom') 87 | autoload :Pbzip2, File.join(COMPRESSOR_PATH, 'pbzip2') 88 | autoload :Lzma, File.join(COMPRESSOR_PATH, 'lzma') 89 | end 90 | 91 | ## 92 | # Autoload encryptor files 93 | module Encryptor 94 | autoload :Base, File.join(ENCRYPTOR_PATH, 'base') 95 | autoload :OpenSSL, File.join(ENCRYPTOR_PATH, 'open_ssl') 96 | autoload :GPG, File.join(ENCRYPTOR_PATH, 'gpg') 97 | end 98 | 99 | ## 100 | # Autoload notification files 101 | module Notifier 102 | autoload :Base, File.join(NOTIFIER_PATH, 'base') 103 | autoload :Binder, File.join(NOTIFIER_PATH, 'binder') 104 | autoload :Mail, File.join(NOTIFIER_PATH, 'mail') 105 | autoload :Twitter, File.join(NOTIFIER_PATH, 'twitter') 106 | autoload :Campfire, File.join(NOTIFIER_PATH, 'campfire') 107 | autoload :Prowl, File.join(NOTIFIER_PATH, 'prowl') 108 | autoload :Hipchat, File.join(NOTIFIER_PATH, 'hipchat') 109 | end 110 | 111 | ## 112 | # Require Backup base files 113 | %w{ 114 | archive 115 | binder 116 | cleaner 117 | config 118 | configuration 119 | dependency 120 | errors 121 | logger 122 | model 123 | package 124 | packager 125 | pipeline 126 | splitter 127 | template 128 | version 129 | }.each {|lib| require File.join(LIBRARY_PATH, lib) } 130 | 131 | end 132 | -------------------------------------------------------------------------------- /lib/backup/database/postgresql.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | module Backup 4 | module Database 5 | class PostgreSQL < Base 6 | 7 | ## 8 | # Name of the database that needs to get dumped 9 | attr_accessor :name 10 | 11 | ## 12 | # Credentials for the specified database 13 | attr_accessor :username, :password 14 | 15 | ## 16 | # Connectivity options 17 | attr_accessor :host, :port, :socket 18 | 19 | ## 20 | # Tables to skip while dumping the database 21 | attr_accessor :skip_tables 22 | 23 | ## 24 | # Tables to dump, tables that aren't specified won't get dumped 25 | attr_accessor :only_tables 26 | 27 | ## 28 | # Additional "pg_dump" options 29 | attr_accessor :additional_options 30 | 31 | ## 32 | # Path to pg_dump utility (optional) 33 | attr_accessor :pg_dump_utility 34 | 35 | attr_deprecate :utility_path, :version => '3.0.21', 36 | :replacement => :pg_dump_utility 37 | 38 | ## 39 | # Creates a new instance of the PostgreSQL adapter object 40 | # Sets the PGPASSWORD environment variable to the password 41 | # so it doesn't prompt and hang in the process 42 | def initialize(model, &block) 43 | super(model) 44 | 45 | @skip_tables ||= Array.new 46 | @only_tables ||= Array.new 47 | @additional_options ||= Array.new 48 | 49 | instance_eval(&block) if block_given? 50 | 51 | @pg_dump_utility ||= utility(:pg_dump) 52 | end 53 | 54 | ## 55 | # Performs the pgdump command and outputs the 56 | # data to the specified path based on the 'trigger' 57 | def perform! 58 | super 59 | 60 | pipeline = Pipeline.new 61 | dump_ext = 'sql' 62 | 63 | pipeline << pgdump 64 | if @model.compressor 65 | @model.compressor.compress_with do |command, ext| 66 | pipeline << command 67 | dump_ext << ext 68 | end 69 | end 70 | pipeline << "cat > '#{ File.join(@dump_path, name) }.#{ dump_ext }'" 71 | 72 | pipeline.run 73 | if pipeline.success? 74 | Logger.message "#{ database_name } Complete!" 75 | else 76 | raise Errors::Database::PipelineError, 77 | "#{ database_name } Dump Failed!\n" + 78 | pipeline.error_messages 79 | end 80 | end 81 | 82 | ## 83 | # Builds the full pgdump string based on all attributes 84 | def pgdump 85 | "#{password_options}" + 86 | "#{ pg_dump_utility } #{ username_options } #{ connectivity_options } " + 87 | "#{ user_options } #{ tables_to_dump } #{ tables_to_skip } #{ name }" 88 | end 89 | 90 | ## 91 | # Builds the password syntax PostgreSQL uses to authenticate the user 92 | # to perform database dumping 93 | def password_options 94 | password.to_s.empty? ? '' : "PGPASSWORD='#{password}' " 95 | end 96 | 97 | ## 98 | # Builds the credentials PostgreSQL syntax to authenticate the user 99 | # to perform the database dumping process 100 | def username_options 101 | username.to_s.empty? ? '' : "--username='#{username}'" 102 | end 103 | 104 | ## 105 | # Builds the PostgreSQL connectivity options syntax to connect the user 106 | # to perform the database dumping process, socket gets gsub'd to host since 107 | # that's the option PostgreSQL takes for socket connections as well. In case 108 | # both the host and the socket are specified, the socket will take priority over the host 109 | def connectivity_options 110 | %w[host port socket].map do |option| 111 | next if send(option).to_s.empty? 112 | "--#{option}='#{send(option)}'".gsub('--socket=', '--host=') 113 | end.compact.join(' ') 114 | end 115 | 116 | ## 117 | # Builds a PostgreSQL compatible string for the additional options 118 | # specified by the user 119 | def user_options 120 | additional_options.join(' ') 121 | end 122 | 123 | ## 124 | # Builds the PostgreSQL syntax for specifying which tables to dump 125 | # during the dumping of the database 126 | def tables_to_dump 127 | only_tables.map do |table| 128 | "--table='#{table}'" 129 | end.join(' ') 130 | end 131 | 132 | ## 133 | # Builds the PostgreSQL syntax for specifying which tables to skip 134 | # during the dumping of the database 135 | def tables_to_skip 136 | skip_tables.map do |table| 137 | "--exclude-table='#{table}'" 138 | end.join(' ') 139 | end 140 | 141 | end 142 | end 143 | end 144 | -------------------------------------------------------------------------------- /spec-live/storage/dropbox_spec.rb: -------------------------------------------------------------------------------- 1 | # encoding: utf-8 2 | 3 | require File.expand_path('../../spec_helper.rb', __FILE__) 4 | 5 | describe 'Storage::Dropbox', 6 | :if => Backup::SpecLive::CONFIG['storage']['dropbox']['specs_enabled'] do 7 | let(:trigger) { 'archive_dropbox' } 8 | 9 | def remote_files_for(storage, package) 10 | remote_path = storage.send(:remote_path_for, package) 11 | 12 | files = [] 13 | storage.send(:transferred_files_for, package) do |local_file, remote_file| 14 | files << File.join(remote_path, remote_file) 15 | end 16 | files 17 | end 18 | 19 | def check_remote_for(storage, package, expectation = true) 20 | remote_path = storage.send(:remote_path_for, package) 21 | 22 | # search the remote_path folder for the trigger (base file name) 23 | metadata = storage.send(:connection).search( 24 | remote_path, package.trigger 25 | ) 26 | files_found = metadata.map {|entry| File.basename(entry['path']) } 27 | 28 | files = remote_files_for(storage, package).map {|file| File.basename(file) } 29 | 30 | if expectation 31 | files_found.sort.should == files.sort 32 | else 33 | files_found.should be_empty 34 | end 35 | end 36 | 37 | def clean_remote!(storage, package) 38 | storage.send(:remove!, package) 39 | end 40 | 41 | it 'should store the archive on the remote', :init => true do 42 | model = h_set_trigger(trigger) 43 | 44 | model.perform! 45 | 46 | storage = model.storages.first 47 | package = model.package 48 | files = remote_files_for(storage, package) 49 | files.count.should == 1 50 | 51 | check_remote_for(storage, package) 52 | 53 | clean_remote!(storage, package) 54 | end 55 | 56 | describe 'Storage::Dropbox Cycling' do 57 | context 'when archives exceed `keep` setting' do 58 | it 'should remove the oldest archive' do 59 | packages = [] 60 | 61 | model = h_set_trigger(trigger) 62 | storage = model.storages.first 63 | model.perform! 64 | package = model.package 65 | package.filenames.count.should == 1 66 | packages << package 67 | sleep 1 68 | 69 | check_remote_for(storage, packages[0]) 70 | 71 | model = h_set_trigger(trigger) 72 | storage = model.storages.first 73 | model.perform! 74 | package = model.package 75 | package.filenames.count.should == 1 76 | packages << package 77 | sleep 1 78 | 79 | check_remote_for(storage, packages[1]) 80 | 81 | model = h_set_trigger(trigger) 82 | storage = model.storages.first 83 | model.perform! 84 | package = model.package 85 | package.filenames.count.should == 1 86 | packages << package 87 | 88 | check_remote_for(storage, packages[2]) 89 | clean_remote!(storage, packages[2]) 90 | 91 | check_remote_for(storage, packages[1]) 92 | clean_remote!(storage, packages[1]) 93 | 94 | check_remote_for(storage, packages[0], false) 95 | end 96 | end 97 | 98 | context 'when an archive to be removed does not exist' do 99 | it 'should log a warning and continue' do 100 | packages = [] 101 | 102 | model = h_set_trigger(trigger) 103 | storage = model.storages.first 104 | model.perform! 105 | package = model.package 106 | package.filenames.count.should == 1 107 | packages << package 108 | sleep 1 109 | 110 | check_remote_for(storage, packages[0]) 111 | 112 | model = h_set_trigger(trigger) 113 | storage = model.storages.first 114 | model.perform! 115 | package = model.package 116 | package.filenames.count.should == 1 117 | packages << package 118 | 119 | check_remote_for(storage, packages[1]) 120 | 121 | # remove archive directory cycle! will attempt to remove 122 | clean_remote!(storage, packages[0]) 123 | 124 | check_remote_for(storage, packages[0], false) 125 | 126 | check_remote_for(storage, packages[1]) 127 | 128 | 129 | model = h_set_trigger(trigger) 130 | storage = model.storages.first 131 | expect do 132 | model.perform! 133 | end.not_to raise_error 134 | 135 | Backup::Logger.has_warnings?.should be_true 136 | 137 | package = model.package 138 | package.filenames.count.should == 1 139 | packages << package 140 | 141 | check_remote_for(storage, packages[1]) 142 | clean_remote!(storage, packages[1]) 143 | 144 | check_remote_for(storage, packages[2]) 145 | clean_remote!(storage, packages[2]) 146 | end 147 | end 148 | 149 | end # describe 'Storage::SCP Cycling' 150 | 151 | end 152 | --------------------------------------------------------------------------------