├── .editorconfig ├── .gitignore ├── LICENSE ├── README.md ├── bin └── pxcbackup ├── lib ├── pxcbackup.rb └── pxcbackup │ ├── application.rb │ ├── array.rb │ ├── backup.rb │ ├── backupper.rb │ ├── command.rb │ ├── logger.rb │ ├── mysql.rb │ ├── path_resolver.rb │ ├── remote_repo.rb │ ├── repo.rb │ └── version.rb └── pxcbackup.gemspec /.editorconfig: -------------------------------------------------------------------------------- 1 | root = true 2 | 3 | [*] 4 | indent_style = space 5 | indent_size = 2 6 | end_of_line = lf 7 | charset = utf-8 8 | trim_trailing_whitespace = true 9 | insert_final_newline = true 10 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.gem 2 | *.rbc 3 | .bundle 4 | .config 5 | .yardoc 6 | Gemfile.lock 7 | InstalledFiles 8 | _yardoc 9 | coverage 10 | doc/ 11 | lib/bundler/man 12 | pkg 13 | rdoc 14 | spec/reports 15 | test/tmp 16 | test/version_tmp 17 | tmp 18 | *.bundle 19 | *.so 20 | *.o 21 | *.a 22 | mkmf.log 23 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License 2 | 3 | Copyright (c) 2014 Robbert Klarenbeek 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in 13 | all copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN 21 | THE SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # PXCBackup 2 | 3 | [![Gem Version](https://badge.fury.io/rb/pxcbackup.svg)](http://badge.fury.io/rb/pxcbackup) 4 | 5 | PXCBackup is a database backup tool meant for [Percona XtraDB Cluster](http://www.percona.com/software/percona-xtradb-cluster) (PXC), although it could also be used on other related systems, like a [MariaDB](https://mariadb.org) [Galera](http://galeracluster.com/products/) cluster using [XtraBackup](http://www.percona.com/software/percona-xtrabackup), for example. 6 | 7 | The `innobackupex` script provided by Percona makes it very easy to create backups, however restoring backups can become quite complicated, since backups might need to be extracted, uncompressed, decrypted, before restoring they need to be prepared, incremental backups need to be applied on top of full backups, indexes might need to be rebuilt for compact backups, etc. Usually, backups need to be restored in stressful emergency situations, where all of these steps can slow you down quite a bit. 8 | 9 | PXCBackup does all of this for you! As a bonus, PXCBackup provides syncing backups to [Amazon S3](http://aws.amazon.com/s3/) and even restoring straight from S3. 10 | 11 | Since PXCBackup is meant for Galera clusters, it does a few additional things: 12 | 13 | * Run `innobackupex` with `--galera-info` and reconstructing `grastate.dat` when restoring a backup. This preserves the local node state, allowing new nodes to be added from a backup with just an IST! 14 | 15 | * Turning on `wsrep_desync` before a backup, and turning it off again after `wsrep_local_recv_queue` is empty. The reason for this is twofold: 16 | * It prevents flow control from kicking in when the backup node takes a performance hit because of the increased disk load (this is similar to what happens on a donor node, during an SST). 17 | * Secondly, it makes `clustercheck` report this node as unavailable, which can be very useful to let your loadbalancer(s) skip this node during the backup. This behavior can be turned off by setting `available_when_donor` to `1` in `clustercheck`. 18 | 19 | PXCBackup is basically a server command line tool, which means the following constraints were used: 20 | 21 | * Support ruby >= 1.8.7. Yes, 1.8.7 is EOL, but many cloud provider OS images still contain 1.8.7. 22 | * Have no external gem dependencies. This tool should be completely stand-alone, and only require certain command line tools. 23 | * Instead, execute command line tools. For example, it uses the `mysql` and `s3cmd` tools instead of modules / gems. 24 | 25 | ## Installation 26 | 27 | Simply install the gem: 28 | 29 | ```shell 30 | $ gem install pxcbackup 31 | ``` 32 | 33 | Of course, you need to have PXC (or similar) running, which provides most of the tools (`innobackupex`, `xtrabackup`, `xbstream`, `xbcrypt`). 34 | 35 | To sync to Amazon S3, make sure you have [S3cmd](http://s3tools.org/s3cmd) installed and configured (`s3cmd --configure`, which creates a file `~/.s3cfg`). 36 | 37 | ## Usage 38 | 39 | Just check the built in command line help: 40 | 41 | ```shell 42 | $ pxcbackup help 43 | ``` 44 | 45 | Aside from command line flags, you can specify additional options in `~/.pxcbackup`, or another 46 | config given by `-c`. Some commonly used settings are: 47 | 48 | ```yaml 49 | backup_dir: /path/to/local/backups/ 50 | remote: s3://my-aws-bucket/ 51 | mysql_username: root 52 | mysql_password: 53 | compact: false 54 | compress: true 55 | encrypt: AES256 56 | encrypt_key: 57 | retention: 100 58 | desync_wait: 30 59 | threads: 4 60 | memory: 1G 61 | ``` 62 | 63 | ## Wishlist 64 | 65 | * More complex rotation schemes 66 | * Separate rotation scheme for remote 67 | * Better error handling for shell commands 68 | * Code documentation (RDoc?) 69 | * Tests (RSpec?) 70 | * Different remote providers 71 | 72 | ## Authors 73 | 74 | * Robbert Klarenbeek, 75 | 76 | ## License 77 | 78 | DeployHook is published under the [MIT License](http://www.opensource.org/licenses/mit-license.php). 79 | -------------------------------------------------------------------------------- /bin/pxcbackup: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env ruby 2 | 3 | require 'pxcbackup' 4 | require 'pxcbackup/application' 5 | 6 | begin 7 | PXCBackup::Application.new(ARGV).run 8 | rescue => e 9 | abort "#{$0}: #{e.message}" 10 | end 11 | -------------------------------------------------------------------------------- /lib/pxcbackup.rb: -------------------------------------------------------------------------------- 1 | require 'pxcbackup/version' 2 | require 'pxcbackup/backupper' 3 | -------------------------------------------------------------------------------- /lib/pxcbackup/application.rb: -------------------------------------------------------------------------------- 1 | require 'optparse' 2 | require 'time' 3 | require 'yaml' 4 | 5 | require 'pxcbackup/backupper' 6 | require 'pxcbackup/logger' 7 | require 'pxcbackup/version' 8 | 9 | module PXCBackup 10 | class Application 11 | def initialize(argv) 12 | parse_options(argv) 13 | 14 | config = File.join(ENV['HOME'], '.pxcbackup') 15 | if @options[:config] 16 | config = @options[:config] 17 | raise 'cannot find given config file' unless File.file?(config) 18 | end 19 | if File.file?(config) 20 | config_options = YAML.load_file(config) 21 | config_options = config_options.inject({}) { |hash, (k, v)| hash[k.to_sym] = v; hash } 22 | @options = config_options.merge(@options) 23 | end 24 | end 25 | 26 | def run 27 | Logger.color_output = ENV['TERM'] && !@options[:no_color] 28 | backupper = Backupper.new(@options) 29 | 30 | case @command 31 | when 'create' 32 | backupper.make_backup(@options) 33 | when 'list' 34 | backupper.list_backups 35 | when 'restore' 36 | time = @arguments.any? ? Time.parse(@arguments.first) : Time.now 37 | backupper.restore_backup(time, @options) 38 | end 39 | end 40 | 41 | def parse_options(argv) 42 | @options ||= {} 43 | parser = OptionParser.new do |opt| 44 | opt.banner = "Usage: #{$0} COMMAND [OPTIONS]" 45 | opt.separator '' 46 | opt.separator 'Commands' 47 | opt.separator ' create create a new backup' 48 | opt.separator ' help show this help' 49 | opt.separator ' list list available backups' 50 | opt.separator ' restore [time] restore to a point in time' 51 | opt.separator '' 52 | opt.separator 'Options' 53 | 54 | opt.on('-c', '--config', '=CONFIG_FILE', 'config file to use instead of ~/.pxcbackup') do |config_file| 55 | @options[:config] = config_file 56 | end 57 | 58 | opt.on('-d', '--dir', '=BACKUP_DIR', 'local repository to store backups') do |backup_dir| 59 | @options[:backup_dir] = backup_dir 60 | end 61 | 62 | opt.on('-f', '--full', 'create a full backup') do 63 | @options[:type] = :full 64 | end 65 | 66 | opt.on('-i', '--incremental', 'create an incremental backup') do 67 | @options[:type] = :incremental 68 | end 69 | 70 | opt.on('-l', '--local', 'stay local, i.e. do not communicate with S3') do 71 | @options[:local] = true 72 | end 73 | 74 | opt.on('--no-color', 'disable color output') do 75 | @options[:no_color] = true 76 | end 77 | 78 | opt.on('-r', '--remote', '=REMOTE_URI', 'remote URI to sync backups to, e.g. s3://my-aws-bucket/') do |remote| 79 | @options[:remote] = remote 80 | end 81 | 82 | opt.on('-v', '--verbose', 'verbose output') do 83 | Logger.raise_verbosity 84 | end 85 | 86 | opt.on('--version', 'print version and exit') do 87 | puts "pxcbackup #{VERSION}" 88 | exit 89 | end 90 | 91 | opt.on('-y', '--yes', 'skip confirmation on backup restore') do 92 | @options[:skip_confirmation] = true 93 | end 94 | end 95 | 96 | begin 97 | @command, *@arguments = parser.parse(argv) 98 | if @command == 'help' 99 | puts parser 100 | exit 101 | end 102 | raise 'no command given' if @command.to_s == '' 103 | raise "invalid command #{@command}" unless ['create', 'list', 'restore'].include?(@command) 104 | rescue => e 105 | abort "#{$0}: #{e.message}\n#{parser}" 106 | end 107 | end 108 | end 109 | end 110 | -------------------------------------------------------------------------------- /lib/pxcbackup/array.rb: -------------------------------------------------------------------------------- 1 | class Array 2 | def uniq_by(&blk) 3 | transforms = {} 4 | select do |el| 5 | t = blk[el] 6 | should_keep = !transforms[t] 7 | transforms[t] = true 8 | should_keep 9 | end 10 | end 11 | end 12 | -------------------------------------------------------------------------------- /lib/pxcbackup/backup.rb: -------------------------------------------------------------------------------- 1 | module PXCBackup 2 | class Backup 3 | attr_reader :repo, :path 4 | 5 | def initialize(repo, path) 6 | @repo = repo 7 | @path = path 8 | raise 'invalid backup name' unless match 9 | end 10 | 11 | def self.regexp 12 | /\/(\d+)_(full|incr)\.(xbstream|tar)(\.xbcrypt)?$/ 13 | end 14 | 15 | def ==(other) 16 | @path == other.path && @repo == other.repo 17 | end 18 | 19 | def <=>(other) 20 | compare = time <=> other.time 21 | compare = remote? ? -1 : 1 if compare == 0 && remote? != other.remote? 22 | compare 23 | end 24 | 25 | def to_s 26 | "#{time} - #{type.to_s[0..3]} (#{remote? ? 'remote' : 'local'})" 27 | end 28 | 29 | def time 30 | Time.at(match[:timestamp].to_i) 31 | end 32 | 33 | def type 34 | type = match[:type] 35 | type = 'incremental' if type == 'incr' 36 | type.to_sym 37 | end 38 | 39 | def stream 40 | match[:stream].to_sym 41 | end 42 | 43 | def encrypted? 44 | match[:encrypted] 45 | end 46 | 47 | def full? 48 | type == :full 49 | end 50 | 51 | def incremental? 52 | type == :incremental 53 | end 54 | 55 | def remote? 56 | @repo.is_a? RemoteRepo 57 | end 58 | 59 | def delete 60 | @repo.delete(self) 61 | end 62 | 63 | def stream_command 64 | @repo.stream_command(self) 65 | end 66 | 67 | private 68 | 69 | def match 70 | match = self.class.regexp.match(@path) 71 | return nil unless match 72 | { 73 | :timestamp => match[1], 74 | :type => match[2], 75 | :stream => match[3], 76 | :encrypted => !!match[4], 77 | } 78 | end 79 | end 80 | end 81 | -------------------------------------------------------------------------------- /lib/pxcbackup/backupper.rb: -------------------------------------------------------------------------------- 1 | require 'fileutils' 2 | require 'tmpdir' 3 | 4 | require 'pxcbackup/array' 5 | require 'pxcbackup/backup' 6 | require 'pxcbackup/command' 7 | require 'pxcbackup/logger' 8 | require 'pxcbackup/mysql' 9 | require 'pxcbackup/path_resolver' 10 | require 'pxcbackup/remote_repo' 11 | require 'pxcbackup/repo' 12 | 13 | module PXCBackup 14 | class Backupper 15 | def initialize(options) 16 | @threads = options[:threads] || 1 17 | @memory = options[:memory] || '100M' 18 | 19 | @defaults_file = options[:defaults_file] || nil 20 | @throttle = options[:throttle] || nil 21 | @encrypt = options[:encrypt] || nil 22 | @encrypt_key = options[:encrypt_key] || nil 23 | 24 | @which = PathResolver.new(options) 25 | 26 | local_repo_path = options[:backup_dir] 27 | @local_repo = local_repo_path ? Repo.new(local_repo_path, options) : nil 28 | 29 | remote_repo_path = options[:remote] 30 | @remote_repo = remote_repo_path && !options[:local] ? RemoteRepo.new(remote_repo_path, options) : nil 31 | 32 | @mysql = MySQL.new(options) 33 | end 34 | 35 | def make_backup(options = {}) 36 | type = options[:type] || :full 37 | stream = options[:stream] || :xbstream 38 | compress = options[:compress] || false 39 | compact = options[:compact] || false 40 | desync_wait = options[:desync_wait] || 60 41 | retention = options[:retention] || 100 42 | 43 | raise 'cannot find backup dir' unless @local_repo && File.directory?(@local_repo.path) 44 | raise 'cannot enable encryption without encryption key' if @encrypt && !@encrypt_key 45 | 46 | arguments = [ 47 | @mysql.auth, 48 | '--no-timestamp', 49 | "--extra-lsndir=#{@local_repo.path}", 50 | "--stream=#{stream.to_s}", 51 | '--galera-info', 52 | ] 53 | 54 | if compress 55 | arguments << '--compress' 56 | end 57 | 58 | if compact 59 | arguments << '--compact' 60 | end 61 | 62 | if @encrypt 63 | arguments << "--encrypt=#{@encrypt.shellescape}" 64 | arguments << "--encrypt-key=#{@encrypt_key.shellescape}" 65 | end 66 | 67 | filename = "#{Time.now.to_i}" 68 | if type == :incremental 69 | last_info = read_backup_info(File.join(@local_repo.path, 'xtrabackup_checkpoints')) 70 | arguments << '--incremental' 71 | arguments << "--incremental-lsn=#{last_info[:to_lsn]}" 72 | filename << "_incr" 73 | else 74 | filename << '_full' 75 | end 76 | filename << ".#{stream.to_s}" 77 | filename << '.xbcrypt' if @encrypt 78 | 79 | desync_enable(desync_wait) 80 | 81 | Dir.mktmpdir('pxcbackup-') do |dir| 82 | arguments << dir.shellescape 83 | Logger.action "Creating backup #{filename}" do 84 | innobackupex(arguments, File.join(@local_repo.path, filename)) 85 | end 86 | end 87 | 88 | desync_disable 89 | rotate(retention) 90 | 91 | if @remote_repo 92 | Logger.action 'Syncing backups to remote repository' do 93 | @remote_repo.sync(@local_repo) 94 | end 95 | end 96 | end 97 | 98 | def restore_backup(time, options = {}) 99 | skip_confirmation = options[:skip_confirmation] || false 100 | mysql_start_command = options[:mysql_start_command] || "#{@which.service.shellescape} mysql start" 101 | mysql_stop_command = options[:mysql_stop_command] || "#{@which.service.shellescape} mysql stop" 102 | 103 | incremental_backups = [] 104 | all_backups.reverse_each do |backup| 105 | incremental_backups.unshift(backup) if backup.time <= time 106 | break if incremental_backups.any? && backup.full? 107 | end 108 | raise "cannot find any backup before #{time}" if incremental_backups.empty? 109 | raise "cannot find a full backup before #{time}" unless incremental_backups.first.full? 110 | restore_time = incremental_backups.last.time 111 | 112 | full_backup = incremental_backups.shift 113 | 114 | Logger.info "[1/#{incremental_backups.size + 1}] Processing #{full_backup.type.to_s} backup from #{full_backup.time}" 115 | Logger.increase_indentation 116 | with_extracted_backup(full_backup) do |full_backup_path, full_backup_info| 117 | raise 'unexpected backup type' unless full_backup_info[:backup_type] == full_backup.type 118 | raise 'unexpected start LSN' unless full_backup_info[:from_lsn] == 0 119 | 120 | compact = full_backup_info[:compact] 121 | 122 | if full_backup_info[:compress] 123 | Logger.action 'Decompressing' do 124 | innobackupex(['--decompress', full_backup_path.shellescape]) 125 | end 126 | end 127 | 128 | if incremental_backups.any? 129 | Logger.action "Preparing base backup (LSN #{full_backup_info[:to_lsn]})" do 130 | innobackupex(['--apply-log', '--redo-only', full_backup_path.shellescape]) 131 | end 132 | 133 | current_lsn = full_backup_info[:to_lsn] 134 | 135 | index = 2 136 | incremental_backups.each do |incremental_backup| 137 | Logger.decrease_indentation 138 | Logger.info "[#{index}/#{incremental_backups.size + 1}] Processing #{incremental_backup.type.to_s} backup from #{incremental_backup.time}" 139 | Logger.increase_indentation 140 | with_extracted_backup(incremental_backup) do |incremental_backup_path, incremental_backup_info| 141 | raise 'unexpected backup type' unless incremental_backup_info[:backup_type] == incremental_backup.type 142 | raise 'unexpected start LSN' unless incremental_backup_info[:from_lsn] == current_lsn 143 | 144 | compact ||= incremental_backup_info[:compact] 145 | 146 | if incremental_backup_info[:compress] 147 | Logger.action 'Decompressing' do 148 | innobackupex(['--decompress', incremental_backup_path.shellescape]) 149 | end 150 | end 151 | 152 | Logger.action "Applying increment (LSN #{incremental_backup_info[:from_lsn]} -> #{incremental_backup_info[:to_lsn]})" do 153 | innobackupex(['--apply-log', '--redo-only', full_backup_path.shellescape, "--incremental-dir=#{incremental_backup_path.shellescape}"]) 154 | end 155 | 156 | current_lsn = incremental_backup_info[:to_lsn] 157 | end 158 | index += 1 159 | end 160 | end 161 | 162 | Logger.decrease_indentation 163 | 164 | action = 'Final prepare' 165 | arguments = [ 166 | '--apply-log', 167 | ] 168 | 169 | if compact 170 | action << ' + rebuild indexes' 171 | arguments << '--rebuild-indexes' 172 | end 173 | 174 | Logger.action "#{action}" do 175 | arguments << full_backup_path.shellescape 176 | innobackupex(arguments) 177 | end 178 | 179 | Logger.action 'Attempting to restore Galera info' do 180 | restore_galera_info(full_backup_path) 181 | end 182 | 183 | mysql_datadir = @mysql.datadir.chomp('/') 184 | mysql_datadir_old = mysql_datadir + '_YYYYMMDDhhmmss' 185 | 186 | unless skip_confirmation 187 | puts 188 | puts ' BACKUP IS NOW READY TO BE RESTORED' 189 | puts " BACKUP TIMESTAMP: #{restore_time}" 190 | puts ' PLEASE CONFIRM THIS ACTION' 191 | puts 192 | puts ' This will:' 193 | puts ' - stop the MySQL server' 194 | puts " - move the current datadir to #{mysql_datadir_old}" 195 | puts " - restore the backup to #{mysql_datadir}" 196 | puts ' - start the MySQL server' 197 | puts 198 | puts ' Afterwards you will have to:' 199 | puts ' - confirm everything is working and synced correctly' 200 | puts ' - manually create a new full backup (to re-allow incremental backups)' 201 | puts 202 | puts ' If MySQL server cannot be started, this might be because this is the' 203 | puts ' only (remaining) Galera node. If so, manually bootstrap the cluster:' 204 | puts ' # service mysql bootstrap-pxc' 205 | puts 206 | print ' Please type "yes" to continue: ' 207 | confirmation = STDIN.gets.chomp 208 | puts 209 | raise 'did not confirm restore' unless confirmation == 'yes' 210 | end 211 | 212 | Logger.action 'Stopping MySQL server' do 213 | Command.run(mysql_stop_command) 214 | end 215 | 216 | stat = File.stat(mysql_datadir) 217 | uid = stat.uid 218 | gid = stat.gid 219 | 220 | mysql_datadir_old = mysql_datadir + '_' + Time.now.strftime('%Y%m%d%H%M%S') 221 | Logger.action "Moving current datadir to #{mysql_datadir_old}" do 222 | File.rename(mysql_datadir, mysql_datadir_old) 223 | end 224 | 225 | Logger.action "Restoring backup to #{mysql_datadir}" do 226 | Dir.mkdir(mysql_datadir) 227 | innobackupex(['--move-back', full_backup_path.shellescape]) 228 | end 229 | 230 | Logger.action "Chowning #{mysql_datadir}" do 231 | FileUtils.chown_R(uid, gid, mysql_datadir) 232 | end 233 | 234 | if @local_repo 235 | xtrabackup_checkpoints_file = File.join(@local_repo.path, 'xtrabackup_checkpoints') 236 | if File.file?(xtrabackup_checkpoints_file) 237 | Logger.action "Removing last backup info" do 238 | File.delete(xtrabackup_checkpoints_file) 239 | end 240 | end 241 | end 242 | 243 | Logger.action 'Starting MySQL server' do 244 | Command.run(mysql_start_command) 245 | end 246 | end 247 | end 248 | 249 | def list_backups 250 | all_backups.each { |backup| puts backup } 251 | end 252 | 253 | private 254 | 255 | def all_backups 256 | backups = [] 257 | backups += @local_repo.backups if @local_repo 258 | backups += @remote_repo.backups if @remote_repo 259 | backups = backups.uniq_by { |backup| backup.time } 260 | backups.sort 261 | end 262 | 263 | def desync_enable(wait = 60) 264 | Logger.info 'Setting wsrep_desync=ON' 265 | @mysql.set_variable('wsrep_desync', 'ON') 266 | Logger.action "Waiting for #{wait} seconds" do 267 | sleep(wait) 268 | end 269 | end 270 | 271 | def desync_disable 272 | Logger.action 'Waiting until wsrep_local_recv_queue is empty' do 273 | sleep(2) until @mysql.get_status('wsrep_local_recv_queue') == '0' 274 | end 275 | Logger.info 'Setting wsrep_desync=OFF' 276 | @mysql.set_variable('wsrep_desync', 'OFF') 277 | end 278 | 279 | def rotate(retention) 280 | Logger.action 'Checking if we have old backups to remove' do 281 | @local_repo.backups.each do |backup| 282 | days = (Time.now - backup.time) / 86400 283 | break if days < retention && backup.full? 284 | Logger.info "Deleting backup from #{backup.time}" 285 | backup.delete 286 | end 287 | end 288 | end 289 | 290 | def innobackupex(arguments, output_file = nil) 291 | command = @which.innobackupex.shellescape 292 | # --defaults-file has to be the first option passed! 293 | command << " --defaults-file=#{@defaults_file.shellescape}" if @defaults_file 294 | 295 | arguments.unshift( 296 | "--ibbackup=#{@which.xtrabackup.shellescape}", 297 | "--parallel=#{@threads}", 298 | "--compress-threads=#{@threads}", 299 | "--rebuild-threads #{@threads}", 300 | "--use-memory=#{@memory}", 301 | "--tmpdir=#{Dir.tmpdir.shellescape}", 302 | ) 303 | arguments << "--throttle=#{@throttle.shellescape}" if @throttle 304 | 305 | command << ' ' + arguments.join(' ') 306 | command << " > #{output_file.shellescape}" if output_file 307 | result = Command.run(command) 308 | 309 | unless result[:stderr].lines.to_a.last.match(/ completed OK!$/) 310 | # Uncomment next line to see what's going on 311 | #puts result 312 | raise 'unexpected output from innobackupex' 313 | end 314 | end 315 | 316 | def read_backup_info(file) 317 | raise "cannot open #{file}" unless File.file?(file) 318 | result = {} 319 | File.open(file, 'r') do |file| 320 | file.each_line do |line| 321 | key, value = line.chomp.split(/\s*=\s*/, 2) 322 | case key 323 | when 'backup_type' 324 | value = 'full' if value == 'full-backuped' 325 | value = value.to_sym 326 | when /_lsn$/ 327 | value = value.to_i 328 | when 'compact' 329 | value = (value == '1') 330 | end 331 | result[key.to_sym] = value 332 | end 333 | end 334 | result 335 | end 336 | 337 | def with_extracted_backup(backup) 338 | Dir.mktmpdir('pxcbackup-') do |dir| 339 | command = backup.stream_command 340 | action = 'Extracting' 341 | if backup.encrypted? 342 | raise 'need encryption algorithm and key to decrypt this backup' unless @encrypt && @encrypt_key 343 | command << " | #{@which.xbcrypt.shellescape} -d --encrypt-algo=#{@encrypt.shellescape} --encrypt-key=#{@encrypt_key.shellescape}" 344 | action << ' + decrypting' 345 | end 346 | command << 347 | case backup.stream 348 | when :xbstream 349 | " | #{@which.xbstream.shellescape} -x -C #{dir.shellescape}" 350 | when :tar 351 | " | #{@which.tar.shellescape} -ixf - -C #{dir.shellescape}" 352 | end 353 | Logger.action action do 354 | Command.run(command) 355 | end 356 | 357 | checkpoints_file = File.join(dir, 'xtrabackup_checkpoints') 358 | unless File.file?(checkpoints_file) 359 | Logger.warning 'Could not find xtrabackup_checkpoints: trying to skip faulty backup' 360 | return 361 | end 362 | 363 | xtrabackup_binary_file = File.join(dir, 'xtrabackup_binary') 364 | File.delete(xtrabackup_binary_file) if File.file?(xtrabackup_binary_file) 365 | 366 | info = read_backup_info(checkpoints_file) 367 | info[:compress] = Dir.glob(File.join(dir, '**', '*.qp')).any? 368 | 369 | yield(dir, info) 370 | end 371 | end 372 | 373 | def restore_galera_info(dir) 374 | galera_info_file = File.join(dir, 'xtrabackup_galera_info') 375 | return unless File.file?(galera_info_file) 376 | uuid, seqno = nil 377 | File.open(galera_info_file, 'r') do |file| 378 | uuid, seqno = file.gets.chomp.split(':') 379 | end 380 | 381 | version = @mysql.get_status('wsrep_provider_version') 382 | if version 383 | version = version.split('(').first 384 | else 385 | current_grastate_file = File.join(@mysql.datadir, 'grastate.dat') 386 | if File.file?(current_grastate_file) 387 | File.open(current_grastate_file, 'r') do |file| 388 | file.each_line do |line| 389 | match = line.match(/^version:\s+(.*)$/) 390 | if match 391 | version = match[1] 392 | break 393 | end 394 | end 395 | end 396 | end 397 | end 398 | return unless version 399 | 400 | File.open(File.join(dir, 'grastate.dat'), 'w') do |file| 401 | file.write("# GALERA saved state\n") 402 | file.write("version: #{version}\n") 403 | file.write("uuid: #{uuid}\n") 404 | file.write("seqno: #{seqno}\n") 405 | file.write("cert_index:\n") 406 | end 407 | end 408 | end 409 | end 410 | -------------------------------------------------------------------------------- /lib/pxcbackup/command.rb: -------------------------------------------------------------------------------- 1 | require 'open3' 2 | 3 | require 'pxcbackup/logger' 4 | 5 | module PXCBackup 6 | module Command 7 | def self.run(command, ignore_exit_status = false) 8 | Logger.debug "# #{command}" 9 | captured_stdout = '' 10 | captured_stderr = '' 11 | Open3.popen3(command) do |stdin, stdout, stderr| 12 | stdin.close 13 | until stdout.closed? && stderr.closed? 14 | sockets = [] 15 | sockets << stdout unless stdout.closed? 16 | sockets << stderr unless stderr.closed? 17 | IO.select(sockets).flatten.compact.each do |socket| 18 | begin 19 | data = socket.readpartial(1024) 20 | captured_stdout << data if socket == stdout 21 | captured_stderr << data if socket == stderr 22 | rescue EOFError 23 | socket.close 24 | end 25 | end 26 | end 27 | end 28 | raise 'command "#{command.split.first}" exited with a non-zero status' unless $?.success? || ignore_exception 29 | { :stdout => captured_stdout, :stderr => captured_stderr, :exit_status => $?.exitstatus } 30 | end 31 | end 32 | end 33 | -------------------------------------------------------------------------------- /lib/pxcbackup/logger.rb: -------------------------------------------------------------------------------- 1 | module PXCBackup 2 | module Logger 3 | @verbosity_level = 0 4 | @indentation = 0 5 | @color_output = false 6 | @partial = false 7 | 8 | def self.raise_verbosity 9 | @verbosity_level += 1 10 | end 11 | 12 | def self.increase_indentation 13 | @indentation += 1 14 | end 15 | 16 | def self.decrease_indentation 17 | @indentation -= 1 18 | end 19 | 20 | def self.color_output=(value) 21 | @color_output = value 22 | end 23 | 24 | def self.output(message, skip_newline = false) 25 | if @partial 26 | puts 27 | increase_indentation 28 | @partial = false 29 | end 30 | print ' ' * @indentation + message 31 | puts unless skip_newline 32 | $stdout.flush 33 | end 34 | 35 | def self.action_start(message) 36 | return unless @verbosity_level >= 1 37 | output "#{message}: ", true 38 | @partial = true 39 | end 40 | 41 | def self.action_end(message) 42 | return unless @verbosity_level >= 1 43 | if @partial 44 | puts message 45 | @partial = false 46 | else 47 | output message 48 | decrease_indentation 49 | end 50 | end 51 | 52 | def self.info(message) 53 | output message if @verbosity_level >= 1 54 | end 55 | 56 | def self.warning(message) 57 | output yellow(message) if @verbosity_level >= 1 58 | end 59 | 60 | def self.debug(message) 61 | output blue(message) if @verbosity_level >= 2 62 | end 63 | 64 | def self.action(message) 65 | return yield unless @verbosity_level >= 1 66 | 67 | action_start(message) 68 | t1 = Time.now 69 | begin 70 | result = yield 71 | rescue => e 72 | action_end(red('fail')) 73 | raise e 74 | end 75 | t2 = Time.now 76 | action_end(green('done') + ' (%.1fs)' % (t2 - t1)) 77 | result 78 | end 79 | 80 | def self.colorize(text, color_code) 81 | @color_output ? "\e[#{color_code}m#{text}\e[0m" : text 82 | end 83 | 84 | def self.red(text) 85 | colorize(text, 31); 86 | end 87 | 88 | def self.green(text) 89 | colorize(text, 32) 90 | end 91 | 92 | def self.yellow(text) 93 | colorize(text, 33); 94 | end 95 | 96 | def self.blue(text) 97 | colorize(text, 34); 98 | end 99 | end 100 | end 101 | -------------------------------------------------------------------------------- /lib/pxcbackup/mysql.rb: -------------------------------------------------------------------------------- 1 | require 'shellwords' 2 | 3 | require 'pxcbackup/command' 4 | 5 | module PXCBackup 6 | class MySQL 7 | attr_reader :datadir 8 | 9 | def initialize(options = {}) 10 | @which = PathResolver.new(options) 11 | @username = options[:mysql_user] || 'root' 12 | @password = options[:mysql_pass] || '' 13 | @datadir = options[:mysql_datadir] || get_variable('datadir') || '/var/lib/mysql' 14 | raise 'Could not find mysql data dir' unless File.directory?(@datadir) 15 | end 16 | 17 | def auth 18 | "--user=#{@username.shellescape} --password=#{@password.shellescape}" 19 | end 20 | 21 | def exec(query) 22 | output = Command.run("echo #{query.shellescape} | #{@which.mysql.shellescape} #{auth}", true) 23 | lines = output[:stdout].lines.to_a 24 | return nil if lines.empty? 25 | 26 | keys = lines.shift.chomp.split("\t") 27 | rows = [] 28 | lines.each do |line| 29 | values = line.chomp.split("\t") 30 | row = {} 31 | keys.each_with_index do |val, key| 32 | row[val] = values[key] 33 | end 34 | rows << row 35 | end 36 | rows 37 | end 38 | 39 | def get_variable(variable, scope = 'GLOBAL') 40 | result = exec("SHOW #{scope} VARIABLES LIKE '#{variable}'") 41 | result ? result.first['Value'] : nil 42 | end 43 | 44 | def set_variable(variable, value, scope = 'GLOBAL') 45 | exec("SET #{scope} #{variable}=#{value}") 46 | end 47 | 48 | def get_status(variable) 49 | result = exec("SHOW STATUS LIKE '#{variable}'") 50 | result ? result.first['Value'] : nil 51 | end 52 | end 53 | end 54 | -------------------------------------------------------------------------------- /lib/pxcbackup/path_resolver.rb: -------------------------------------------------------------------------------- 1 | require 'shellwords' 2 | 3 | module PXCBackup 4 | class PathResolver 5 | def initialize(options = {}) 6 | @options = options 7 | @paths = {} 8 | end 9 | 10 | def method_missing(name, *arguments) 11 | unless @paths[name] 12 | @paths[name] = @options["#{name.to_s}_path".to_sym] || `which #{name.to_s.shellescape}`.strip 13 | raise "cannot find path for '#{name.to_s}'" unless File.file?(@paths[name]) 14 | end 15 | @paths[name] 16 | end 17 | end 18 | end 19 | -------------------------------------------------------------------------------- /lib/pxcbackup/remote_repo.rb: -------------------------------------------------------------------------------- 1 | require 'shellwords' 2 | 3 | require 'pxcbackup/backup' 4 | require 'pxcbackup/command' 5 | require 'pxcbackup/repo' 6 | 7 | module PXCBackup 8 | class RemoteRepo < Repo 9 | def initialize(path, options = {}) 10 | super(path, options) 11 | @which.s3cmd 12 | end 13 | 14 | def backups 15 | backups = [] 16 | output = Command.run("#{@which.s3cmd.shellescape} ls #{@path.shellescape}") 17 | output[:stdout].lines.to_a.each do |line| 18 | path = line.chomp.split[3] 19 | next unless Backup.regexp.match(path) 20 | backups << Backup.new(self, path) 21 | end 22 | backups.sort 23 | end 24 | 25 | def sync(local_repo) 26 | source = File.join(local_repo.path, '/') 27 | target = File.join(path, '/') 28 | Command.run("#{@which.s3cmd.shellescape} sync --no-progress --delete-removed #{source.shellescape} #{target.shellescape}") 29 | end 30 | 31 | def delete(backup) 32 | verify(backup) 33 | Command.run("#{@which.s3cmd.shellescape} del #{backup.path.shellescape}") 34 | end 35 | 36 | def stream_command(backup) 37 | verify(backup) 38 | "#{@which.s3cmd.shellescape} get --no-progress #{backup.path.shellescape} -" 39 | end 40 | end 41 | end 42 | -------------------------------------------------------------------------------- /lib/pxcbackup/repo.rb: -------------------------------------------------------------------------------- 1 | require 'shellwords' 2 | 3 | require 'pxcbackup/backup' 4 | 5 | module PXCBackup 6 | class Repo 7 | attr_reader :path 8 | 9 | def initialize(path, options = {}) 10 | @path = path 11 | @which = PathResolver.new(options) 12 | end 13 | 14 | def backups 15 | backups = [] 16 | Dir.foreach(@path) do |file| 17 | path = File.join(@path, file) 18 | next unless File.file?(path) 19 | next unless Backup.regexp.match(path) 20 | backups << Backup.new(self, path) 21 | end 22 | backups.sort 23 | end 24 | 25 | def delete(backup) 26 | verify(backup) 27 | File.delete(backup.path) 28 | end 29 | 30 | def stream_command(backup) 31 | verify(backup) 32 | "#{@which.cat.shellescape} #{backup.path.shellescape}" 33 | end 34 | 35 | private 36 | 37 | def verify(backup) 38 | raise 'backup does not belong to this repo' if backup.repo != self 39 | end 40 | end 41 | end 42 | 43 | -------------------------------------------------------------------------------- /lib/pxcbackup/version.rb: -------------------------------------------------------------------------------- 1 | module PXCBackup 2 | VERSION = "0.2.3" 3 | end 4 | -------------------------------------------------------------------------------- /pxcbackup.gemspec: -------------------------------------------------------------------------------- 1 | # coding: utf-8 2 | lib = File.expand_path('../lib', __FILE__) 3 | $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib) 4 | require 'pxcbackup/version' 5 | 6 | Gem::Specification.new do |spec| 7 | spec.name = 'pxcbackup' 8 | spec.version = PXCBackup::VERSION 9 | spec.author = 'Robbert Klarenbeek' 10 | spec.email = 'robbertkl@renbeek.nl' 11 | spec.summary = 'Backup tool for Percona XtraDB Cluster' 12 | spec.description = spec.summary 13 | spec.homepage = 'https://github.com/robbertkl/pxcbackup' 14 | spec.license = 'MIT' 15 | 16 | spec.files = `git ls-files -z`.split("\x0") 17 | spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) } 18 | end 19 | --------------------------------------------------------------------------------