├── __init__.py ├── .gitignore ├── example.conf.json ├── example.backup ├── README.md ├── rotatebackups.py ├── incrbackup.py ├── pushbackup.py └── mysqlbackup.py /__init__.py: -------------------------------------------------------------------------------- 1 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | backup.* 3 | -------------------------------------------------------------------------------- /example.conf.json: -------------------------------------------------------------------------------- 1 | { 2 | "backup" : [ 3 | "/path/to/include" 4 | ], 5 | "exclude" : [ 6 | "*.extension.to.exclude", 7 | "*.another.extension", 8 | "/path/to/exclude" 9 | ] 10 | } 11 | -------------------------------------------------------------------------------- /example.backup: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | BACKUP_BASE=./backupscripts 4 | BACKUP_CONF=$BACKUP_BASE/backup.conf 5 | BACKUP_DIR=$BACKUP_BASE/backup-base 6 | BACKUP_CMD=$BACKUP_BASE/incrbackup.py 7 | BACKUP_SERVER=backup01 8 | BACKUP_USER=root 9 | BACKUP_NS=backup-name 10 | BACKUP_KEEP=90 11 | 12 | $BACKUP_CMD --server $BACKUP_SERVER -c $BACKUP_CONF -t $BACKUP_DIR -u $BACKUP_USER -n $BACKUP_NS -k $BACKUP_KEEP 13 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Python Backup Scripts 2 | ============= 3 | 4 | Python scripts for incremental backups of servers. This includes filesystem 5 | backups and mysql backups. 6 | 7 | Pulled Backups 8 | =========== 9 | Pulled filesystem backups are done through the incrbackup.py script. Backups 10 | are pulled down from remote systems to the backup server for security, the 11 | backup server can get to the remote systems being backed up but not vice versa. 12 | Hence if a remote system is compromised the backup server isn't. 13 | 14 | Pulled backups use rsync and hard links to keep multiple copies of one or 15 | more filesystems while using minimal space. If backing up remote 16 | servers this script assumes that the proper ssh keys have been setup from the 17 | backup server hosting this script to the servers being backed up. 18 | 19 | Backup paths can be either local or remote. The backup directory where 20 | the backups are stored must be local and must already exist. If a users isn't 21 | specified then the remote user used by ssh for rsync is considered to be a 22 | user named backup. 23 | 24 | Use the -h or the --help flag to get a listing of options. 25 | 26 | incrbackup.py [-hnksctu] 27 | [-h | --help] prints this help and usage message 28 | [-n | --name] backup namespace 29 | [-k | --keep] number of backups to keep before deleting 30 | [-s | --server] the server to backup, if remote 31 | [-c | --config] configuration file with backup paths 32 | [-t | --store] directory locally to store the backups 33 | [-u | --user] the remote username used to ssh for backups 34 | 35 | Backups read their include and exclude paths from a config file specified using 36 | the -f option. The config file looks like this. Exclude paths follow rsync 37 | exclude filter rules. 38 | 39 | { 40 | "backup" : [ 41 | "/path/to/include", 42 | "/another/path/to/include" 43 | ], 44 | "exclude" : [ 45 | "*.filetype.to.exclude", 46 | "*.another.to.exclude", 47 | "/path/to/exclude", 48 | "/another/path/to/exclude" 49 | ] 50 | } 51 | 52 | To use a non-standard port, include this to the config file along with the "backup" and "exclude" blocks. An exmple with port 2345 is shown. If no port block is given, port 22 is used by default. 53 | 54 | "port" : [ 55 | "ssh -p 2345" 56 | ] 57 | 58 | 59 | Usually the backup scripts are run from a remote, off-site, server pulling down 60 | content from the servers to backup. Scripts are usually setup to run from cron 61 | periodically. 62 | 63 | 64 | Pushed Filesystem Backups 65 | =========== 66 | Pushed filesystem backups are done through the pushbackup.py script. 67 | 68 | This is an incremental backup system that pushes to a remote server. Useful 69 | for remote systems that aren't always on (laptops). Backups use rsync and hard 70 | links to keep multiple full copies while using minimal space. It is assumed 71 | that the rotatebackups.py script exists on the remote backup server and that 72 | the proper ssh keys have been setup from the pushing server to the backup 73 | server. 74 | 75 | Use the -h or the --help flag to get a listing of options. 76 | 77 | pushbackup.py [-hnksctuxr] 78 | [-h | --help] prints this help and usage message 79 | [-n | --name] backup namespace 80 | [-k | --keep] number of backups to keep before deleting 81 | [-s | --server] the server to push to backup to 82 | [-c | --config] configuration file with backup paths 83 | [-t | --store] directory locally to store the backups 84 | [-u | --user] the remote username used to ssh for backups 85 | [-x | --ssh-key] the ssh key used to connect to the backup 86 | [-r | --rotate-script] the rotatebackups script remote location 87 | 88 | Pushed backup use the same config format as pulled backups. Pushed backups are 89 | usually run manually when needed. They should not be used to backup servers due 90 | to security reasons. If backing up server filesystem see pulled backups. 91 | 92 | MySQL Backups 93 | =========== 94 | MySQL backups are done through the mysqlbackup.py script. 95 | 96 | Use the -h or the --help flag to get a listing of options. 97 | 98 | mysqlbackup.py [-hkdbups] 99 | [-h | --help] prints this help and usage message 100 | [-k | --keep] number of days to keep backups before deleting 101 | [-d | --databases] a comma separated list of databases 102 | [-t | --store] directory locally to store the backups 103 | [-u | --user] the database user 104 | [-p | --password] the database password 105 | [-s | --host] the database server hostname 106 | [-o | --options] the json file to load the options from instead of using command line 107 | [-r | --restore] enables restore mode 108 | 109 | License and Bug Fixes 110 | =========== 111 | These works are public domain or licensed under the Apache Licene. You can do 112 | anything you want with them. Please feel free to send any improvements or 113 | bug fixes. 114 | -------------------------------------------------------------------------------- /rotatebackups.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | import sys 4 | import string 5 | import shutil 6 | import getopt 7 | import os 8 | import os.path 9 | import syslog 10 | import errno 11 | import logging 12 | import tempfile 13 | import datetime 14 | import subprocess 15 | import json 16 | 17 | from operator import itemgetter 18 | 19 | """ 20 | ----------------------------------------------------------------------------- 21 | Rotates backup folders, keeping a given number of backups and deleting older 22 | backups. 23 | 24 | Use the -h or the --help flag to get a listing of options. 25 | 26 | Program: Rotate Backups 27 | Author: Dennis E. Kubes 28 | Date: May 01, 2013 29 | Revision: 1.0 30 | 31 | Revision | Author | Comment 32 | ----------------------------------------------------------------------------- 33 | 20130501-1.0 Dennis E. Kubes Initial creation of script. 34 | ----------------------------------------------------------------------------- 35 | """ 36 | class RotateBackups: 37 | 38 | def __init__(self, keep=90, store=None, name=None): 39 | self.keep = keep 40 | self.store = store 41 | self.name = name 42 | 43 | def run_command(self, command=None, shell=False, ignore_errors=False, 44 | ignore_codes=None): 45 | result = subprocess.call(command, shell=False) 46 | if result and not ignore_errors and (not ignore_codes or result in set(ignore_codes)): 47 | raise BaseException(str(command) + " " + str(result)) 48 | 49 | def rotate_backups(self): 50 | 51 | padding = len(str(self.keep)) 52 | 53 | backups = [] 54 | final_backup_names = [] 55 | 56 | # add the backup directories to a list, dirs are the form num.prefix.date 57 | for backup_dir in os.listdir(self.store): 58 | bparts = backup_dir.split(".") 59 | if bparts[0].isdigit(): 60 | backups.append((backup_dir, bparts)) 61 | 62 | # only need to process backup directories if we have some 63 | if len(backups) > 0: 64 | 65 | # order the backups in the list by reverse number, highest first 66 | backups = sorted(backups, key=itemgetter(0), reverse=True) 67 | 68 | # perform shifting and processing on the backup directories 69 | for btup in backups: 70 | 71 | # unpack the original directory and backup parts 72 | origdir = btup[0] 73 | bparts = btup[1] 74 | 75 | # remove backups >= number of days to keep 76 | bnum = int(bparts[0]) 77 | if bnum >= self.keep: 78 | bpath = self.store + os.sep + origdir 79 | logging.debug(["rm", "-fr", bpath]) 80 | self.run_command(["rm", "-fr", bpath]) 81 | else: 82 | 83 | # above 0 gets shifted to one number higher and moved, 0 gets hardlink 84 | # copied to 1 if it is a directory. If 0 is file assumed that another 85 | # process will write out the new 0 file 86 | base_path = os.path.abspath(self.store) 87 | old_bpath = base_path + os.sep + origdir 88 | num_prefix = str(bnum + 1).zfill(padding) 89 | incr_name = num_prefix + "." + ".".join(bparts[1:]) 90 | new_bpath = base_path + os.sep + incr_name 91 | if bnum > 0: 92 | logging.debug([bnum, "mv", old_bpath, new_bpath]) 93 | self.run_command(["mv", old_bpath, new_bpath]) 94 | final_backup_names.append(new_bpath) 95 | 96 | elif bnum == 0: 97 | 98 | if os.path.isdir(old_bpath): 99 | logging.debug(["cp", "-al", old_bpath, new_bpath]) 100 | self.run_command(["cp", "-al", old_bpath, new_bpath]) 101 | final_backup_names.append(new_bpath) 102 | 103 | # get the current date and timestamp and create the zero backup path 104 | now = datetime.datetime.now() 105 | tstamp = now.strftime("%Y%m%d%H%M%S") 106 | zero_parts = ["".zfill(padding), tstamp] 107 | zero_parts.extend(bparts[2:]) 108 | zbackup_path = base_path + os.sep + ".".join(zero_parts) 109 | 110 | # move the zero directory to the new timestamp 111 | logging.debug([0, "mv", old_bpath, zbackup_path]) 112 | self.run_command(["mv", old_bpath, zbackup_path]) 113 | final_backup_names.append(zbackup_path) 114 | 115 | else: 116 | logging.debug(["mv", old_bpath, new_bpath]) 117 | self.run_command(["mv", old_bpath, new_bpath]) 118 | final_backup_names.append(new_bpath) 119 | 120 | # return the final backup file or directory names, most recent to least 121 | final_backup_names.reverse() 122 | return final_backup_names 123 | 124 | 125 | """ 126 | Prints out the usage for the command line. 127 | """ 128 | def usage(): 129 | usage = ["rotatebackups.py [-hkt]\n"] 130 | usage.append(" [-h | --help] prints this help and usage message\n") 131 | usage.append(" [-k | --keep] number of backups to keep before deleting\n") 132 | usage.append(" [-t | --store] directory locally to store the backups\n") 133 | message = "".join(usage) 134 | print(message) 135 | 136 | """ 137 | Main method that starts up the backup. 138 | """ 139 | def main(argv): 140 | 141 | # set the default values 142 | pid_file = tempfile.gettempdir() + os.sep + "rotbackup.pid" 143 | keep = 90 144 | store = None 145 | padding = 5 146 | 147 | try: 148 | 149 | # process the command line options 150 | opts, args = getopt.getopt(argv, "hk:t:p:", ["help", "keep=", "store="]) 151 | 152 | # if no arguments print usage 153 | if len(argv) == 0: 154 | usage() 155 | sys.exit() 156 | 157 | # loop through all of the command line options and set the appropriate 158 | # values, overriding defaults 159 | for opt, arg in opts: 160 | if opt in ("-h", "--help"): 161 | usage() 162 | sys.exit() 163 | elif opt in ("-k", "--keep"): 164 | keep = int(arg) 165 | elif opt in ("-t", "--store"): 166 | store = arg 167 | 168 | except(getopt.GetoptError, msg): 169 | # if an error happens print the usage and exit with an error 170 | usage() 171 | sys.exit(errno.EIO) 172 | 173 | # check options are set correctly 174 | if store == None: 175 | usage() 176 | sys.exit(errno.EPERM) 177 | 178 | # process, catch any errors, and perform cleanup 179 | try: 180 | 181 | # another rotate can't already be running 182 | if os.path.exists(pid_file): 183 | logging.warning("Rotate backups running, %s pid exists, exiting." % pid_file) 184 | sys.exit(errno.EBUSY) 185 | else: 186 | pid = str(os.getpid()) 187 | f = open(pid_file, "w") 188 | f.write("%s\n" % pid) 189 | f.close() 190 | 191 | # create the backup object and call its backup method 192 | rotback = RotateBackups(keep, store) 193 | rotated_names = rotback.rotate_backups() 194 | if (len(rotated_names) > 0): 195 | print("\n".join(rotated_names)) 196 | 197 | except(Exception): 198 | logging.exception("Rotate backups failed.") 199 | finally: 200 | os.remove(pid_file) 201 | 202 | # if we are running the script from the command line, run the main function 203 | if __name__ == "__main__": 204 | main(sys.argv[1:]) 205 | -------------------------------------------------------------------------------- /incrbackup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | import sys 4 | import string 5 | import shutil 6 | import getopt 7 | import os 8 | import os.path 9 | import syslog 10 | import errno 11 | import logging 12 | import tempfile 13 | import datetime 14 | import subprocess 15 | import json 16 | import rotatebackups 17 | 18 | from operator import itemgetter 19 | 20 | """ 21 | ----------------------------------------------------------------------------- 22 | An incremental backup system that uses rsync and hard links to keep multiple 23 | backups of one or more systems while using minimal space. If backing up remote 24 | servers this script assumes that the proper ssh keys have been setup from the 25 | backup server hosting this script to the servers being backed up. 26 | 27 | A pid file is placed into the system temp directory to prevent concurrent 28 | backups from running at once. The script provides options for the number of 29 | backups to keep. After the max number of backups is reached, backups are 30 | deleted starting with the oldest backup first. 31 | 32 | Backup paths can be either local or remote. The backup root directory where 33 | the backups are stored must be local and must already exist. If a users isn't 34 | specified then the remote user used by ssh for rsync is considered to be backup. 35 | 36 | Use the -h or the --help flag to get a listing of options. 37 | 38 | Program: Incremental Backups 39 | Author: Dennis E. Kubes 40 | Date: August 01, 2011 41 | Revision: 1.2 42 | 43 | Revision | Author | Comment 44 | ----------------------------------------------------------------------------- 45 | 20111122-1.0 Dennis E. Kubes Initial creation of script. 46 | 20131430-1.2 Dennis E. Kubes Added excludes logic, config json file. 47 | ----------------------------------------------------------------------------- 48 | """ 49 | class IncrementalBackup: 50 | 51 | def __init__(self, name="backup", server=None, keep=90, store=None, 52 | config_file=None, user="root"): 53 | self.name = name 54 | self.server = server 55 | self.keep = keep 56 | self.config_file = config_file 57 | self.store = store 58 | self.user = user 59 | 60 | def run_command(self, command=None, shell=False, ignore_errors=False, 61 | ignore_codes=None): 62 | result = subprocess.call(command, shell=False) 63 | if result and not ignore_errors and (not ignore_codes or result in set(ignore_codes)): 64 | raise BaseException(str(command) + " " + str(result)) 65 | 66 | def backup(self): 67 | 68 | # rotate the backups 69 | rotater = rotatebackups.RotateBackups(self.keep, self.store) 70 | rotated_names = rotater.rotate_backups() 71 | 72 | rsync_to = None 73 | if not rotated_names: 74 | # get the current date and timestamp and the zero backup name 75 | now = datetime.datetime.now() 76 | padding = len(str(self.keep)) 77 | tstamp = now.strftime("%Y%m%d%H%M%S") 78 | zbackup_name = ".".join(["".zfill(padding), tstamp, self.name]) 79 | rsync_to = self.store + os.sep + zbackup_name 80 | else: 81 | rsync_to = rotated_names[0] 82 | 83 | # create the base rsync command with excludes 84 | rsync_base = ["rsync", "-avR", "--ignore-errors", "--delete", "--delete-excluded"] 85 | 86 | # get the paths to backup either from the command line or from a paths file 87 | bpaths = [] 88 | expaths = [] 89 | if self.config_file: 90 | 91 | pf = open(self.config_file, "r") 92 | config = json.load(pf) 93 | pf.close() 94 | 95 | # add the paths to backup 96 | bpaths.extend(config["backup"]) 97 | 98 | # add and filter/exclude options 99 | if "exclude" in config: 100 | for exclude in config["exclude"]: 101 | rsync_base.extend(["--exclude", exclude]) 102 | 103 | if "port" in config: 104 | for thePort in config["port"]: 105 | rsync_base.extend(["-e", thePort]) 106 | 107 | # one rsync command per path, ignore files vanished errors 108 | for bpath in bpaths: 109 | bpath = bpath.strip() 110 | rsync_cmd = rsync_base[:] 111 | if self.server: 112 | bpath = self.user + "@" + self.server + ":" + bpath 113 | rsync_cmd.append(bpath) 114 | rsync_cmd.append(rsync_to) 115 | logging.debug(rsync_cmd) 116 | self.run_command(command=rsync_cmd, ignore_errors=True) 117 | 118 | """ 119 | Prints out the usage for the command line. 120 | """ 121 | def usage(): 122 | usage = ["incrbackup.py [-hnksctu]\n"] 123 | usage.append(" [-h | --help] prints this help and usage message\n") 124 | usage.append(" [-n | --name] backup namespace\n") 125 | usage.append(" [-k | --keep] number of backups to keep before deleting\n") 126 | usage.append(" [-s | --server] the server to backup, if remote\n") 127 | usage.append(" [-c | --config] configuration file with backup paths\n") 128 | usage.append(" [-t | --store] directory locally to store the backups\n") 129 | usage.append(" [-u | --user] the remote username used to ssh for backups\n") 130 | message = "".join(usage) 131 | print(message) 132 | 133 | """ 134 | Main method that starts up the backup. 135 | """ 136 | def main(argv): 137 | 138 | # set the default values 139 | pid_file = tempfile.gettempdir() + os.sep + "incrbackup.pid" 140 | name = "backup" 141 | keep = 90 142 | server = None 143 | config_file = None 144 | store = None 145 | user = "backup" 146 | 147 | try: 148 | 149 | # process the command line options 150 | opts, args = getopt.getopt(argv, "hn:k:s:c:t:u:", ["help", "name=", 151 | "keep=", "server=", "config=", "store=", "user="]) 152 | 153 | # if no arguments print usage 154 | if len(argv) == 0: 155 | usage() 156 | sys.exit() 157 | 158 | # loop through all of the command line options and set the appropriate 159 | # values, overriding defaults 160 | for opt, arg in opts: 161 | if opt in ("-h", "--help"): 162 | usage() 163 | sys.exit() 164 | elif opt in ("-n", "--name"): 165 | name = arg 166 | elif opt in ("-k", "--keep"): 167 | keep = int(arg) 168 | elif opt in ("-s", "--server"): 169 | server = arg 170 | elif opt in ("-c", "--config"): 171 | config_file = arg 172 | elif opt in ("-t", "--store"): 173 | store = arg 174 | elif opt in ("-u", "--user"): 175 | user = arg 176 | 177 | except(getopt.GetoptError, msg): 178 | # if an error happens print the usage and exit with an error 179 | usage() 180 | sys.exit(errno.EIO) 181 | 182 | # check options are set correctly 183 | if config_file == None or store == None: 184 | usage() 185 | sys.exit(errno.EPERM) 186 | 187 | # process backup, catch any errors, and perform cleanup 188 | try: 189 | 190 | # another backup can't already be running, if pid file doesn't exist, then 191 | # create it 192 | if os.path.exists(pid_file): 193 | logging.warning("Backup running, %s pid exists, exiting." % pid_file) 194 | sys.exit(errno.EBUSY) 195 | else: 196 | pid = str(os.getpid()) 197 | f = open(pid_file, "w") 198 | f.write("%s\n" % pid) 199 | f.close() 200 | 201 | # create the backup object and call its backup method 202 | ibackup = IncrementalBackup(name, server, keep, store, config_file, user) 203 | ibackup.backup() 204 | 205 | except(Exception): 206 | logging.exception("Incremental backup failed.") 207 | finally: 208 | os.remove(pid_file) 209 | 210 | # if we are running the script from the command line, run the main function 211 | if __name__ == "__main__": 212 | main(sys.argv[1:]) 213 | 214 | 215 | -------------------------------------------------------------------------------- /pushbackup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | import sys 4 | import string 5 | import shutil 6 | import getopt 7 | import os 8 | import os.path 9 | import syslog 10 | import errno 11 | import logging 12 | import tempfile 13 | import datetime 14 | import subprocess 15 | import json 16 | import paramiko 17 | 18 | from operator import itemgetter 19 | 20 | """ 21 | ----------------------------------------------------------------------------- 22 | An incremental backup system that pushes backups to a remote server. Useful 23 | for remote systems that aren't always on (laptops). Backups use rsync and hard 24 | links to keep multiple full copies while using minimal space. It is assumed 25 | that the rotatebackups.py script exists on the remote backup server and that 26 | the proper ssh keys have been setup from the pushing server to the backup 27 | server. 28 | 29 | A pid file is placed into the system temp directory to prevent concurrent 30 | backups from running at once. The script provides options for the number of 31 | backups to keep. After the max number of backups is reached, backups are 32 | deleted starting with the oldest backup first. 33 | 34 | Backup paths can be either local or remote. The backup root directory where 35 | the backups are stored must be local and must already exist. If a users isn't 36 | specified then the remote user used by ssh for rsync is considered to be backup. 37 | 38 | Use the -h or the --help flag to get a listing of options. 39 | 40 | Program: Push Backups 41 | Author: Dennis E. Kubes 42 | Date: May 01, 2013 43 | Revision: 1.0 44 | 45 | Revision | Author | Comment 46 | ----------------------------------------------------------------------------- 47 | 20131430-1.0 Dennis E. Kubes Initial creation of script. 48 | ----------------------------------------------------------------------------- 49 | """ 50 | class PushBackup: 51 | 52 | def __init__(self, name="backup", server=None, keep=90, store=None, 53 | config_file=None, user="root", ssh_key=None, rotate_script=None): 54 | self.name = name 55 | self.server = server 56 | self.keep = keep 57 | self.config_file = config_file 58 | self.store = store 59 | self.user = user 60 | self.ssh_key = ssh_key 61 | self.rotate_script = rotate_script 62 | 63 | def run_command(self, command=None, shell=False, ignore_errors=False, 64 | ignore_codes=None): 65 | result = subprocess.call(command, shell=False) 66 | if result and not ignore_errors and (not ignore_codes or result in set(ignore_codes)): 67 | raise BaseException(str(command) + " " + str(result)) 68 | 69 | def backup(self): 70 | 71 | # create the ssh client to run the remote rotate script 72 | client = paramiko.SSHClient() 73 | client.set_missing_host_key_policy(paramiko.AutoAddPolicy()) 74 | client.load_system_host_keys() 75 | client.connect(self.server, username=self.user, key_filename=self.ssh_key) 76 | 77 | # rotate the backups remotely by running the rotatebackups.py script on the 78 | # remote backup server 79 | rotate_cmd = [self.rotate_script, "-k", str(self.keep), "-t", self.store] 80 | stdin, stdout, stderr = client.exec_command(" ".join(rotate_cmd)) 81 | rotated_names = stdout.readlines() 82 | client.close() 83 | 84 | rsync_to = None 85 | if not rotated_names: 86 | # get the current date and timestamp and the zero backup name 87 | now = datetime.datetime.now() 88 | padding = len(str(self.keep)) 89 | tstamp = now.strftime("%Y%m%d%H%M%S") 90 | zbackup_name = ".".join(["".zfill(padding), tstamp, self.name]) 91 | rsync_to = self.store + os.sep + zbackup_name 92 | else: 93 | rsync_to = rotated_names[0] 94 | 95 | # create the base rsync command with excludes 96 | rsync_base = ["rsync", "-avR", "--ignore-errors", "--delete", "--delete-excluded"] 97 | 98 | # get the paths to backup either from the command line or from a paths file 99 | bpaths = [] 100 | expaths = [] 101 | if self.config_file: 102 | 103 | pf = open(self.config_file, "r") 104 | config = json.load(pf) 105 | pf.close() 106 | 107 | # add the paths to backup 108 | bpaths.extend(config["backup"]) 109 | 110 | # add and filter/exclude options 111 | if "exclude" in config: 112 | for exclude in config["exclude"]: 113 | rsync_base.extend(["--exclude", exclude]) 114 | 115 | # one rsync command per path, ignore files vanished errors 116 | for bpath in bpaths: 117 | bpath = bpath.strip() 118 | rsync_cmd = rsync_base[:] 119 | rsync_cmd.append(bpath) 120 | rsync_cmd.append(self.user + "@" + self.server + ":" + rsync_to) 121 | logging.debug(rsync_cmd) 122 | self.run_command(command=rsync_cmd, ignore_errors=True) 123 | 124 | """ 125 | Prints out the usage for the command line. 126 | """ 127 | def usage(): 128 | usage = ["pushbackup.py [-hnksctuxr]\n"] 129 | usage.append(" [-h | --help] prints this help and usage message\n") 130 | usage.append(" [-n | --name] backup namespace\n") 131 | usage.append(" [-k | --keep] number of backups to keep before deleting\n") 132 | usage.append(" [-s | --server] the server to push to backup to\n") 133 | usage.append(" [-c | --config] configuration file with backup paths\n") 134 | usage.append(" [-t | --store] directory locally to store the backups\n") 135 | usage.append(" [-u | --user] the remote username used to ssh for backups\n") 136 | usage.append(" [-x | --ssh-key] the ssh key used to connect to the backup\n") 137 | usage.append(" [-r | --rotate-script] the rotatebackups script remote location\n") 138 | message = "".join(usage) 139 | print(message) 140 | 141 | """ 142 | Main method that starts up the backup. 143 | """ 144 | def main(argv): 145 | 146 | # set the default values 147 | pid_file = tempfile.gettempdir() + os.sep + "pushbackup.pid" 148 | name = "backup" 149 | keep = 90 150 | server = None 151 | config_file = None 152 | store = None 153 | user = "backup" 154 | ssh_key = os.path.expanduser("~/.ssh/id_rsa") 155 | rotate_script = "rotatebackups.py" 156 | 157 | try: 158 | 159 | # process the command line options 160 | opts, args = getopt.getopt(argv, "hn:k:s:c:t:u:x:r:", ["help", "name=", 161 | "keep=", "server=", "config=", "store=", "user=", "ssh-key=", 162 | "rotate-script="]) 163 | 164 | # if no arguments print usage 165 | if len(argv) == 0: 166 | usage() 167 | sys.exit() 168 | 169 | # loop through all of the command line options and set the appropriate 170 | # values, overriding defaults 171 | for opt, arg in opts: 172 | if opt in ("-h", "--help"): 173 | usage() 174 | sys.exit() 175 | elif opt in ("-n", "--name"): 176 | name = arg 177 | elif opt in ("-k", "--keep"): 178 | keep = int(arg) 179 | elif opt in ("-s", "--server"): 180 | server = arg 181 | elif opt in ("-c", "--config"): 182 | config_file = arg 183 | elif opt in ("-t", "--store"): 184 | store = arg 185 | elif opt in ("-u", "--user"): 186 | user = arg 187 | elif opt in ("-x", "--ssh-key"): 188 | ssh_key = arg 189 | elif opt in ("-r", "--rotate-script"): 190 | rotate_script = arg 191 | 192 | except(getopt.GetoptError, msg): 193 | # if an error happens print the usage and exit with an error 194 | usage() 195 | sys.exit(errno.EIO) 196 | 197 | # check options are set correctly 198 | if config_file == None or store == None or server == None: 199 | usage() 200 | sys.exit(errno.EPERM) 201 | 202 | # process backup, catch any errors, and perform cleanup 203 | try: 204 | 205 | # another backup can't already be running, if pid file doesn't exist, then 206 | # create it 207 | if os.path.exists(pid_file): 208 | logging.warning("Backup running, %s pid exists, exiting." % pid_file) 209 | sys.exit(errno.EBUSY) 210 | else: 211 | pid = str(os.getpid()) 212 | f = open(pid_file, "w") 213 | f.write("%s\n" % pid) 214 | f.close() 215 | 216 | # create the backup object and call its backup method 217 | pbackup = PushBackup(name, server, keep, store, config_file, user, 218 | ssh_key, rotate_script) 219 | pbackup.backup() 220 | 221 | except(Exception): 222 | logging.exception("Incremental backup failed.") 223 | finally: 224 | os.remove(pid_file) 225 | 226 | # if we are running the script from the command line, run the main function 227 | if __name__ == "__main__": 228 | main(sys.argv[1:]) 229 | 230 | 231 | -------------------------------------------------------------------------------- /mysqlbackup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/python 2 | 3 | import sys 4 | import string 5 | import shutil 6 | import getopt 7 | import os 8 | import os.path 9 | import syslog 10 | import errno 11 | import logging 12 | import tempfile 13 | import datetime 14 | import subprocess 15 | import readline 16 | import json 17 | 18 | from operator import itemgetter 19 | 20 | """ 21 | ----------------------------------------------------------------------------- 22 | A script to backup mysql databases through the mysqldump utility. 23 | 24 | Use the -h or the --help flag to get a listing of options. 25 | 26 | Program: Mysql Database Backups 27 | Author: Dennis E. Kubes 28 | Date: April 28, 2013 29 | Revision: 1.0 30 | 31 | Revision | Author | Comment 32 | ----------------------------------------------------------------------------- 33 | 20130428-1.0 Dennis E. Kubes Initial creation of script. 34 | ----------------------------------------------------------------------------- 35 | """ 36 | 37 | def rlinput(prompt, prefill=''): 38 | readline.set_startup_hook(lambda: readline.insert_text(prefill)) 39 | try: 40 | return raw_input(prompt) 41 | finally: 42 | readline.set_startup_hook() 43 | 44 | def format_date(raw_date): 45 | return "%s-%s-%s %s:%s:%s" % (raw_date[0:4], raw_date[4:6], 46 | raw_date[6:8], raw_date[8:10], raw_date[10:12], raw_date[12:14]) 47 | 48 | class MysqlBackup: 49 | 50 | def __init__(self, keep=90, databases=None, store=None, user="root", 51 | password=None, host=None): 52 | self.host = host 53 | self.keep = keep 54 | self.databases = databases 55 | self.store = store 56 | self.user = user 57 | self.password = password 58 | self.host = host 59 | 60 | def run_command(self, command=None, shell=False, ignore_errors=False, 61 | ignore_codes=None, get_output=False, path="."): 62 | p = subprocess.Popen([command], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True, cwd=path) 63 | out, err = p.communicate() 64 | 65 | result = p.returncode 66 | if result and not ignore_errors and (not ignore_codes or result in set(ignore_codes)): 67 | raise BaseException(str(command) + " " + str(result)) 68 | 69 | def get_databases(self): 70 | 71 | if self.databases != None: 72 | return [s.strip() for s in self.databases.strip().split(",")] 73 | 74 | list_cmd = "mysql -u" + self.user 75 | if self.host != None: 76 | list_cmd += " -h " + self.host 77 | if self.password != None: 78 | list_cmd += " -p" + self.password 79 | list_cmd += " --silent -N -e 'show databases'" 80 | databases = os.popen(list_cmd).readlines() 81 | return [s.strip() for s in databases] 82 | 83 | def restore(self): 84 | dbbackup_path = self.store + os.sep 85 | backups = sorted(os.listdir(dbbackup_path), reverse=True) 86 | 87 | # show available options 88 | k = 1 89 | options = {} 90 | prev_date = "" 91 | databases = "" 92 | filenames = "" 93 | 94 | print("Available backups to restore:") 95 | for i in range(len(backups)): 96 | data = backups[i].split(".") 97 | date = data[0] 98 | 99 | if not prev_date: 100 | prev_date = date 101 | 102 | if (date != prev_date): 103 | print("["+str(k)+"]", "(%s) %s" % (format_date(prev_date), databases)) 104 | 105 | options[k] = { 106 | "date": prev_date, 107 | "databases": databases, 108 | "filenames": filenames 109 | } 110 | 111 | k += 1 112 | prev_date = date 113 | databases = "" 114 | filenames = "" 115 | 116 | databases += ("" if databases == "" else ",") + data[1] 117 | filenames += ("" if filenames == "" else ",") + backups[i] 118 | 119 | print("["+str(k)+"]", "(%s) %s" % (format_date(prev_date), databases)) 120 | options[k] = { 121 | "date": prev_date, 122 | "databases": databases, 123 | "filenames": filenames 124 | } 125 | 126 | # get the selection 127 | user_input = -1 128 | max_option = len(options.keys()) 129 | while True: 130 | user_input = int(raw_input("\nSelect backup: ")) 131 | if (user_input < 1) or (max_option < user_input): 132 | print("Error: The value should be between 1 and", max_option) 133 | else: 134 | break 135 | 136 | # get the databases to restore 137 | date = format_date(options[user_input]["date"]) 138 | filenames = options[user_input]["filenames"] 139 | selected_databases = rlinput("Databases to restore: ", options[user_input]["databases"]) 140 | databases = ",".join(filter(lambda db: db in selected_databases, self.get_databases())) 141 | if databases == "": 142 | print("Error: The selected databases doesn't match any created databases.") 143 | sys.exit() 144 | 145 | # ask for confirmation 146 | print("The databases \"%s\" are going to be restored using the version dated \"%s\"" % (databases, date)) 147 | confirmation = rlinput("Continue? [Y/n] ", "Y") 148 | if confirmation != "Y": 149 | print("Aborted.") 150 | sys.exit() 151 | 152 | # expand the filenames of the databases 153 | databases = databases.split(",") 154 | filenames = filter(lambda fln: reduce(lambda x,y: x or y, 155 | map(lambda dbn: dbn in fln, databases)), 156 | filenames.split(",")) 157 | 158 | # restore the databases 159 | for filename in filenames: 160 | db = filename.split(".")[1] 161 | restore_cmd = "gunzip < " + dbbackup_path + filename + \ 162 | " | mysql -u " + self.user 163 | if self.host != None: 164 | restore_cmd += " -h " + "'" + self.host + "'" 165 | if self.password != None: 166 | restore_cmd += " -p" + self.password 167 | restore_cmd += " " + db 168 | 169 | print("Restoring \"" + db + "\"...") 170 | sys.stdout.flush() 171 | logging.info("Restore db, %s from %s." % (db, dbbackup_path + filename)) 172 | self.run_command(restore_cmd) 173 | print("done") 174 | 175 | print("Restore complete!") 176 | 177 | def backup(self): 178 | 179 | padding = len(str(self.keep)) 180 | backups = [] 181 | 182 | # remove files older than keep days 183 | cutdate = datetime.datetime.now() - datetime.timedelta(days=self.keep) 184 | for backup_file in os.listdir(self.store): 185 | bparts = backup_file.split(".") 186 | if bparts[0].isdigit(): 187 | dumpdate = datetime.datetime.strptime(bparts[0], "%Y%m%d%H%M%S") 188 | if dumpdate < cutdate: 189 | os.remove(os.path.join(self.store, backup_file)) 190 | 191 | # get the current date and timestamp and the zero backup name 192 | tstamp = datetime.datetime.now().strftime("%Y%m%d%H%M%S") 193 | dbs = self.get_databases() 194 | skip = ["information_schema", "performance_schema", "test"] 195 | for db in dbs: 196 | if db in skip: 197 | continue 198 | 199 | dbbackup_name = ".".join([tstamp, db, "sql"]) 200 | dbbackup_path = self.store + os.sep + dbbackup_name 201 | 202 | dump_cmd = "mysqldump -u " + self.user 203 | if self.host != None: 204 | dump_cmd += " -h " + "'" + self.host + "'" 205 | if self.password != None: 206 | dump_cmd += " -p" + self.password 207 | dump_cmd += " -e --opt -c " + db + " | gzip > " + dbbackup_path + ".gz" 208 | logging.info("Dump db, %s to %s." % (db, dbbackup_path)) 209 | os.popen(dump_cmd) 210 | 211 | """ 212 | Prints out the usage for the command line. 213 | """ 214 | def usage(): 215 | usage = ["mysqlbackup.py [-hkdbups]\n"] 216 | usage.append(" [-h | --help] prints this help and usage message\n") 217 | usage.append(" [-k | --keep] number of days to keep backups before deleting\n") 218 | usage.append(" [-d | --databases] a comma separated list of databases\n") 219 | usage.append(" [-t | --store] directory locally to store the backups\n") 220 | usage.append(" [-u | --user] the database user\n") 221 | usage.append(" [-p | --password] the database password\n") 222 | usage.append(" [-s | --host] the database server hostname\n") 223 | usage.append(" [-o | --options] the json file to load the options from instead of using command line\n") 224 | usage.append(" [-r | --restore] enables restore mode\n") 225 | message = "".join(usage) 226 | print(message) 227 | 228 | """ 229 | Main method that starts up the backup. 230 | """ 231 | def main(argv): 232 | 233 | # set the default values 234 | pid_file = tempfile.gettempdir() + os.sep + "mysqlbackup.pid" 235 | keep = 90 236 | databases = None 237 | user = None 238 | password = None 239 | host = None 240 | store = None 241 | options = None 242 | restore = False 243 | 244 | try: 245 | 246 | # process the command line options 247 | st = "hn:k:d:t:u:p:s:o:r" 248 | lt = ["help", "keep=", "databases=", "store=", "user=", "password=", 249 | "host=", "options=", "restore"] 250 | opts, args = getopt.getopt(argv, st, lt) 251 | 252 | # if no arguments print usage 253 | if len(argv) == 0: 254 | usage() 255 | sys.exit() 256 | 257 | # detect if loading options from file and load the json 258 | vals = {} 259 | fopts = None 260 | for opt, arg in opts: 261 | vals[opt] = arg 262 | if ("-o" in vals.keys()) or ("--options" in vals.keys()): 263 | opt = "-o" if "-o" in vals.keys() else "--options" 264 | with open(vals[opt], 'r') as content_file: 265 | fopts = json.load(content_file) 266 | 267 | # merge with opts 268 | opts_keys = map(lambda val: val[0], opts) 269 | if fopts: 270 | for key in fopts.keys(): 271 | prefix = "" 272 | if key in st.split(":"): 273 | prefix = "-" 274 | elif key in map(lambda t: t[:-1] if t[-1] == "=" else t, lt): 275 | prefix = "--" 276 | else: 277 | continue 278 | if prefix+key not in opts_keys: 279 | opts.append((prefix+key, fopts[key])) 280 | 281 | # loop through all of the command line options and set the appropriate 282 | # values, overriding defaults 283 | for opt, arg in opts: 284 | if opt in ("-h", "--help"): 285 | usage() 286 | sys.exit() 287 | elif opt in ("-k", "--keep"): 288 | keep = int(arg) 289 | elif opt in ("-d", "--databases"): 290 | databases = arg 291 | elif opt in ("-t", "--store"): 292 | store = arg 293 | elif opt in ("-u", "--user"): 294 | user = arg 295 | elif opt in ("-p", "--password"): 296 | password = arg 297 | elif opt in ("-s", "--host"): 298 | host = arg 299 | elif opt in ("-r", "--restore"): 300 | restore = True 301 | 302 | except(getopt.GetoptError, msg): 303 | logging.warning(msg) 304 | # if an error happens print the usage and exit with an error 305 | usage() 306 | sys.exit(errno.EIO) 307 | 308 | # check options are set correctly 309 | if user == None or store == None: 310 | logging.warning("Backup store directory (-t) and user (-u) are required") 311 | usage() 312 | sys.exit(errno.EPERM) 313 | 314 | # process backup, catch any errors, and perform cleanup 315 | try: 316 | 317 | # another backup can't already be running, if pid file doesn't exist, then 318 | # create it 319 | if os.path.exists(pid_file): 320 | logging.warning("Backup running, %s pid exists, exiting." % pid_file) 321 | sys.exit(errno.EBUSY) 322 | else: 323 | pid = str(os.getpid()) 324 | f = open(pid_file, "w") 325 | f.write("%s\n" % pid) 326 | f.close() 327 | 328 | # create the backup object and call its backup method 329 | mysql_backup = MysqlBackup(keep, databases, store, user, password, host) 330 | if restore: 331 | mysql_backup.restore() 332 | else: 333 | mysql_backup.backup() 334 | 335 | except(Exception): 336 | logging.exception("Mysql backups failed.") 337 | finally: 338 | os.remove(pid_file) 339 | 340 | # if we are running the script from the command line, run the main function 341 | if __name__ == "__main__": 342 | main(sys.argv[1:]) 343 | --------------------------------------------------------------------------------