├── .travis.yml ├── LICENSE ├── README.markdown └── rotate-backups.py /.travis.yml: -------------------------------------------------------------------------------- 1 | language: python 2 | python: 3 | - "2.7" 4 | - "3.3" 5 | - "3.4" 6 | install: pip install pytest 7 | script: py.test rotate-backups.py 8 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | rotate-backups.py 2 | 3 | License: MIT 4 | 5 | Copyright (c) 2011 Adam Feuer 6 | 7 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 8 | 9 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 10 | 11 | -------------------------------------------------------------------------------- /README.markdown: -------------------------------------------------------------------------------- 1 | Rotate backups script 2 | ===================== 3 | 4 | This script is designed to be used by processes that create tarred and 5 | compressed backups every hour or every day. These backups accumulate, taking 6 | up disk space. 7 | 8 | By running this rotator script once per hour shortly *before* your hourly backup 9 | cron runs, you can save 24 houly backups, 7 daily backups and an arbitrary 10 | number of weekly backups (the default is 52). 11 | 12 | Here's what the script will do: 13 | 14 | 1. Rename new arrival tarballs to include tarball's mtime date, then move into /hourly/ dir. 15 | 2. For any hourly backups which are more than 24 hours old, either move them into daily, or delete 16 | 3. For any daily backups which are more than 7 days old, either move them into weekly, or delete 17 | 4. Delete excess backups from weekly dir (in excess of user setting: max_weekly_backups) 18 | 19 | This will effectively turn a user_backups dir like this: 20 | 21 | backups/ 22 | world.tar.bz2 23 | 24 | ...into this: 25 | 26 | user_backups_archive/ 27 | world/ 28 | hourly/ 29 | world-2008-01-01.tar.bz2 30 | 31 | Those hourly tarballs will continue to pile up for the first 24 hours, after 32 | which a daily directory will appear. After 7 days, another directory will 33 | appear for the weekly tarballs as well. 34 | 35 | Backups are moved from the incoming arrivals directory to the archives. If you 36 | do not produce hourly backups, but only produce daily backups, they system will 37 | only save the daily backups. 38 | 39 | 40 | How to install 41 | -------------- 42 | 43 | 1. Place this script somewhere on your server, for example: /usr/local/bin/rotate_backups.py 44 | 2. chmod a+x /usr/local/bin/rotate_backups.py 45 | 3. Add a cron like this --> 30 * * * * /usr/local/bin/rotate_backups.py > /dev/null 46 | 47 | In step three, we added a cronjob for 30 minutes after each hour. This would be 48 | a good setting if for example your backups cron runs every hour on the hour. 49 | It's best to do all your rotating shortly *before* your backups. 50 | 51 | 52 | How to configure 53 | ---------------- 54 | 55 | You can edit the defaults in the script below, or create a config file in /etc/default/rotate-backups or $HOME/.rotate-backupsrc 56 | 57 | The config file format follows the Python ConfigParser format (http://docs.python.org/library/configparser.html). Here is an example: 58 | 59 | ``` 60 | [Settings] 61 | backups_dir = /var/backups/latest/ 62 | archives_dir = /var/backups/archives/ 63 | hourly_backup_hour = 23 64 | weekly_backup_day = 6 65 | max_weekly_backups = 52 66 | backup_extensions = "tar.gz",".tar.bz2",".jar" 67 | log_level = ERROR 68 | ``` 69 | 70 | Requirements 71 | ------------ 72 | 73 | Python 2.7 74 | 75 | (I have not tested this with Python 3) 76 | 77 | Contact 78 | ------- 79 | 80 | If you have comments or improvements, let me know: 81 | 82 | Adam Feuer 83 | adamf@pobox.com 84 | http://adamfeuer.com 85 | 86 | Credits 87 | ------- 88 | 89 | This script is based on the DirectAdmin backup script written by Sean Schertell 90 | 91 | 92 | -------------------------------------------------------------------------------- /rotate-backups.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | """ 4 | Rotate backups script 5 | ===================== 6 | 7 | This script is designed to be used by processes that create tarred and 8 | compressed backups every hour or every day. These backups accumulate, taking 9 | up disk space. 10 | 11 | By running this rotator script once per hour shortly *before* your hourly backup 12 | cron runs, you can save 24 hourly backups, 7 daily backups and an arbitrary 13 | number of weekly backups (the default is 52). 14 | 15 | Here's what the script will do: 16 | 17 | 1. Rename new arrival tarballs to include tarball's mtime date, then move into /hourly/ dir. 18 | 2. For any hourly backups which are more than 24 hours old, either move them into daily, or delete. 19 | 3. For any daily backups which are more than 7 days old, either move them into weekly, or delete. 20 | 4. Delete excess backups from weekly dir (in excess of user setting: max_weekly_backups). 21 | 22 | This will effectively turn a user_backups dir like this: 23 | 24 | backups/ 25 | world.tar.bz2 26 | 27 | ...into this: 28 | 29 | user_backups_archive/ 30 | world/ 31 | hourly/ 32 | world-2008-01-01.tar.bz2 33 | 34 | Those hourly tarballs will continue to pile up for the first 24 hours, after 35 | which a daily directory will appear. After 7 days, another directory will 36 | appear for the weekly tarballs as well. 37 | 38 | Backups are moved from the incoming arrivals directory to the archives. If you 39 | do not produce hourly backups, but only produce daily backups, they system will 40 | only save the daily backups. 41 | 42 | 43 | How to install 44 | -------------- 45 | 46 | 1. Place this script somewhere on your server, for example: /usr/local/bin/rotate_backups.py 47 | 2. chmod a+x /usr/local/bin/rotate_backups.py 48 | 3. Add a cron like this --> 30 * * * * /usr/local/bin/rotate_backups.py > /dev/null 49 | 50 | In step three, we added a cronjob for 30 minutes after each hour. This would be 51 | a good setting if for example your backups cron runs every hour on the hour. 52 | It's best to do all your rotating shortly *before* your backups. 53 | 54 | 55 | How to configure 56 | ---------------- 57 | 58 | You can edit the defaults in the script below, or create a config file in /etc/default/rotate-backups or $HOME/.rotate-backupsrc 59 | 60 | The allowed log levels are INFO, WARNING, ERROR, and DEBUG. 61 | 62 | The config file format follows the Python ConfigParser format (http://docs.python.org/library/configparser.html). Here is an example: 63 | 64 | ``` 65 | [Settings] 66 | backups_dir = /var/backups/latest/ 67 | archives_dir = /var/backups/archives/ 68 | hourly_backup_hour = 23 69 | weekly_backup_day = 6 70 | max_weekly_backups = 52 71 | backup_extensions = "tar.gz",".tar.bz2",".jar" 72 | log_level = ERROR 73 | ``` 74 | 75 | Requirements 76 | ------------ 77 | 78 | Python 2.7 or Python 3.4 79 | 80 | Contact 81 | ------- 82 | 83 | If you have comments or improvements, let me know: 84 | 85 | Adam Feuer 86 | http://adamfeuer.com 87 | 88 | License 89 | ------- 90 | 91 | This script is based on the DirectAdmin backup script written by Sean Schertell. 92 | Modified by Adam Feuer 93 | http://adamfeuer.com 94 | 95 | License: MIT 96 | 97 | Copyright (c) 2011 Adam Feuer 98 | 99 | Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: 100 | 101 | The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. 102 | 103 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 104 | 105 | """ 106 | 107 | ############################################################################################# 108 | # Default Settings # 109 | # Note these can also be changed in /etc/default/rotate-backups or $HOME/.rotate-backupsrc # 110 | ############################################################################################# 111 | 112 | DEFAULTS = { 113 | 'backups_dir': '/var/backups/latest/', 114 | 'archives_dir': '/var/backups/archives/', 115 | 'hourly_backup_hour': 23, # 0-23 116 | 'weekly_backup_day': 6, # 0-6, Monday-Sunday 117 | 'max_weekly_backups': 52, 118 | 'backup_extensions': ['tar.gz', '.tar.bz2', '.jar'], # list of file extensions that will be backed up 119 | 'log_level': 'ERROR', 120 | } 121 | 122 | ############################################################################################# 123 | 124 | import os, sys, time, re, csv, traceback, logging, shutil 125 | 126 | PY3 = sys.version_info[0] == 3 127 | if not PY3: import StringIO # StringIO does not exist in python3 128 | 129 | 130 | try: 131 | # 3.x name 132 | import configparser 133 | except ImportError: 134 | # 2.x name 135 | import ConfigParser as configparser 136 | 137 | 138 | from datetime import datetime, timedelta 139 | 140 | allowed_log_levels = { 'INFO': logging.INFO, 'ERROR': logging.ERROR, 'WARNING': logging.WARNING, 'DEBUG': logging.DEBUG } 141 | 142 | LOGGER = logging.getLogger('rotate-backups') 143 | consoleHandler = logging.StreamHandler() 144 | consoleHandler.setLevel(allowed_log_levels[DEFAULTS["log_level"]]) 145 | formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s') 146 | consoleHandler.setFormatter(formatter) 147 | LOGGER.addHandler(consoleHandler) 148 | 149 | 150 | class SimpleConfig(object): 151 | def __init__(self): 152 | self.config = configparser.ConfigParser() 153 | global_configfile = '/etc/default/rotate-backups' 154 | local_configfile = os.path.join(os.getenv("HOME"), ".rotate-backupsrc") 155 | self.config.read([global_configfile, local_configfile]) 156 | log_level = self.config.get('Settings', 'log_level') if self.config.has_section('Settings') else None 157 | LOGGER.setLevel(allowed_log_levels.get(log_level, DEFAULTS["log_level"])) 158 | 159 | def __getattr__(self, setting): 160 | r = None 161 | 162 | if self.config.has_section('Settings'): 163 | if setting in ('hourly_backup_hour', 'weekly_backup_day', 'max_weekly_backups'): 164 | r = self.config.getint('Settings', setting) 165 | else: 166 | r = self.config.get('Settings', setting) 167 | 168 | if setting == 'backup_extensions': 169 | r = self.parse_extensions(r) 170 | 171 | return r or DEFAULTS.get(setting) 172 | 173 | def parse_extensions(self, extensions_string): 174 | if PY3: 175 | parser = csv.reader([extensions_string]) 176 | else: 177 | parser = csv.reader(StringIO.StringIO(extensions_string)) 178 | return list(parser)[0] 179 | 180 | 181 | def get_backups_in(account_name, directory, archives_dir): 182 | base_path = '%s/%s/' % (archives_dir, account_name) 183 | path_to_dir = '%s%s/' % (base_path, directory) 184 | backups = [] 185 | if os.path.isdir(path_to_dir): 186 | for filename in os.listdir(path_to_dir): 187 | path_to_file = os.path.join(path_to_dir, filename) 188 | backups.append(Backup(path_to_file)) 189 | backups.sort(key=lambda b: b.date) 190 | return backups 191 | 192 | 193 | class Backup(object): 194 | def __init__(self, path_to_file): 195 | """Instantiation also rewrites the filename if not already done, prepending the date.""" 196 | self.pattern = '(.*)(\-)([0-9]{4}\-[0-9]{2}\-[0-9]{2}\-[0-9]{4})' 197 | self.path_to_file = path_to_file 198 | self.filename = self.format_filename() 199 | self.set_account_and_date(self.filename) 200 | 201 | def set_account_and_date(self, filename): 202 | match_obj = re.match(self.pattern, filename) 203 | if match_obj is None: 204 | return filename 205 | self.account = match_obj.group(1) 206 | datestring = match_obj.group(3) 207 | time_struct = time.strptime(datestring, "%Y-%m-%d-%H%M") 208 | self.date = datetime(*time_struct[:5]) 209 | 210 | def move_to(self, directory, archives_dir): 211 | destination_dir = os.path.join(archives_dir, self.account, directory); 212 | new_filepath = os.path.join(archives_dir, self.account, directory, self.filename) 213 | try: 214 | LOGGER.info('Moving %s to %s.' % (self.path_to_file, new_filepath)) 215 | if not os.path.isdir(destination_dir): 216 | os.makedirs(destination_dir) 217 | shutil.move(self.path_to_file, new_filepath) 218 | except: 219 | LOGGER.error('Unable to move latest backups into %s/ directory.' % directory) 220 | LOGGER.error("Stacktrace: " + traceback.format_exc()) 221 | sys.exit(1) 222 | 223 | def remove(self): 224 | LOGGER.info('Removing %s' % self.path_to_file) 225 | os.remove(self.path_to_file) 226 | 227 | def format_filename(self): 228 | """If this filename hasn't yet been prepended with the date, do that now.""" 229 | # Does the filename include a date? 230 | path_parts = os.path.split(self.path_to_file) 231 | filename = path_parts[-1] 232 | parent_dir = os.sep + os.path.join(*path_parts[:-1]) 233 | if not re.match(self.pattern, filename.split('.')[0]): 234 | # No date, rename the file. 235 | self.mtime = time.localtime( os.path.getmtime(self.path_to_file) ) 236 | self.mtime_str = time.strftime('%Y-%m-%d-%H%M', self.mtime) 237 | account = filename.split('.')[0] 238 | extension = filename.split('.', 1)[1] 239 | filename = ('%s-%s.' + extension) % (account, self.mtime_str) 240 | new_filepath = os.path.join(parent_dir, filename) 241 | LOGGER.info('Renaming file to %s.' % new_filepath) 242 | shutil.move(self.path_to_file, new_filepath) 243 | self.path_to_file = new_filepath 244 | return filename 245 | 246 | def __cmp__(x, y): 247 | """For sorting by date.""" 248 | return cmp( x.date, y.date) 249 | 250 | 251 | def is_backup(filename, backup_extensions): 252 | for extension in backup_extensions: 253 | if filename.endswith(extension): 254 | return True 255 | return False 256 | 257 | def collect(archives_dir): 258 | """Return a collection of account objects for all accounts in backup directory.""" 259 | accounts = [] 260 | # Append all account names from archives_dir. 261 | for account_name in os.listdir(archives_dir): 262 | accounts.append(account_name) 263 | accounts = sorted(list(set(accounts))) # Uniquify. 264 | return accounts 265 | 266 | 267 | def check_dirs(backups_dir, archives_dir): 268 | # Make sure backups_dir actually exists. 269 | if not os.path.isdir(backups_dir): 270 | LOGGER.error("Unable to find backups directory: %s." % backups_dir) 271 | sys.exit(1) 272 | 273 | # Make sure archives_dir actually exists. 274 | if not os.path.isdir(archives_dir): 275 | try: 276 | os.mkdir(archives_dir) 277 | except: 278 | LOGGER.error("Unable to create archives directory: %s." % archives_dir) 279 | sys.exit(1) 280 | 281 | def rotate_new_arrivals(backups_dir, archives_dir, backup_extensions, period_name): 282 | for filename in os.listdir(backups_dir): 283 | if is_backup(filename, backup_extensions=backup_extensions): 284 | new_arrival = Backup(os.path.join(backups_dir, filename)) 285 | new_arrival.move_to(period_name, archives_dir) 286 | 287 | 288 | def is_rotation_time( 289 | date, 290 | period_name, 291 | hourly_backup_hour, 292 | weekly_backup_day, 293 | ): 294 | assert(period_name in ('hourly', 'daily', 'weekly')) 295 | 296 | if period_name == 'hourly': 297 | actual_time = date.hour 298 | config_time = hourly_backup_hour 299 | elif period_name == 'daily': 300 | actual_time = date.weekday() 301 | config_time = weekly_backup_day 302 | else: 303 | return False 304 | 305 | if actual_time == config_time: 306 | LOGGER.debug('%s equals %s.' % (actual_time, config_time)) 307 | return True 308 | else: 309 | LOGGER.debug('%s is not %s.' % (actual_time, config_time)) 310 | return False 311 | 312 | 313 | def rotate( 314 | account_name, 315 | period_name, 316 | next_period_name, 317 | max_age, 318 | archives_dir, 319 | **is_rotation_time_kw 320 | ): 321 | earliest_creation_date = datetime.now() - max_age 322 | for backup in get_backups_in( 323 | account_name=account_name, 324 | directory=period_name, 325 | archives_dir=archives_dir, 326 | ): 327 | if backup.date < earliest_creation_date: 328 | # This backup is too old, move to other backup directory or delete. 329 | if next_period_name and is_rotation_time( 330 | date=backup.date, 331 | period_name=period_name, 332 | **is_rotation_time_kw 333 | ): 334 | backup.move_to(next_period_name, archives_dir) 335 | else: 336 | backup.remove() 337 | 338 | 339 | try: 340 | import pytest 341 | 342 | import tempfile 343 | 344 | 345 | class TempDirContext(object): 346 | def __init__(self, **tempfilekwargs): 347 | self.tempfilekwargs = tempfilekwargs 348 | 349 | def __enter__(self): 350 | self.dir = tempfile.mkdtemp(**self.tempfilekwargs) 351 | print('Created {}'.format(self.dir)) 352 | return self.dir 353 | 354 | def __exit__(self, exc_type, exc_value, traceback): 355 | print('Removing {} and subfolders...'.format(self.dir)) 356 | shutil.rmtree(self.dir) 357 | print('Removed {}'.format(self.dir)) 358 | 359 | 360 | def create_basedirs(path): 361 | basedir = os.path.dirname(path) 362 | if not os.path.exists(basedir): 363 | os.makedirs(basedir) 364 | 365 | 366 | def create_empty_file(filename): 367 | assert not os.path.exists(filename) 368 | create_basedirs(path=filename) 369 | open(filename, 'a').close() 370 | 371 | 372 | def test_rotate_new_arrivals_moves_correctly(): 373 | with TempDirContext(prefix='rotate-backup-tmp') as tmpdir: 374 | create_empty_file(os.path.join(tmpdir, 'latest/dbdump.tar.bz2')) 375 | create_basedirs(os.path.join(tmpdir, 'archives')) 376 | 377 | backups_dir = os.path.join(tmpdir, 'latest') 378 | archives_dir = os.path.join(tmpdir, 'archives') 379 | 380 | assert len(os.listdir(backups_dir)) == 1 381 | assert not os.path.exists(archives_dir) 382 | 383 | rotate_new_arrivals( 384 | backups_dir=backups_dir, 385 | archives_dir=archives_dir, 386 | backup_extensions=DEFAULTS['backup_extensions'], 387 | period_name='hourly', 388 | ) 389 | 390 | assert len(os.listdir(backups_dir)) == 0 391 | assert os.listdir(archives_dir) == ['dbdump'] 392 | assert os.listdir(os.path.join(archives_dir, 'dbdump')) == ['hourly'] 393 | result_files = os.listdir(os.path.join(archives_dir, 'dbdump/hourly')) 394 | assert result_files[0].startswith('dbdump-') 395 | assert result_files[0].endswith('.tar.bz2') 396 | 397 | 398 | def test_rotate_new_arrivals_ignores_unmatched_files_and_does_not_create_archives_dir(): 399 | with TempDirContext(prefix='rotate-backup-tmp') as tmpdir: 400 | create_empty_file(os.path.join(tmpdir, 'latest/dbdump.tar.bz22')) 401 | create_basedirs(os.path.join(tmpdir, 'archives')) 402 | 403 | backups_dir = os.path.join(tmpdir, 'latest') 404 | archives_dir = os.path.join(tmpdir, 'archives') 405 | 406 | assert len(os.listdir(backups_dir)) == 1 407 | assert not os.path.exists(archives_dir) 408 | 409 | rotate_new_arrivals( 410 | backups_dir=backups_dir, 411 | archives_dir=archives_dir, 412 | backup_extensions=DEFAULTS['backup_extensions'], 413 | period_name='hourly' 414 | ) 415 | assert len(os.listdir(backups_dir)) == 1 416 | assert not os.path.exists(archives_dir) 417 | 418 | 419 | # def test_get_backups_in_returns_backup_objects(): 420 | # assert 0 421 | # ret = get_backups_in(account_name, directory, archives_dir) 422 | # assert isinstance(ret[0], Backup) 423 | 424 | 425 | # def test_rotate_removes_old_backups(): 426 | # assert 0 427 | 428 | 429 | # def test_rotate_moves_backups(): 430 | # assert 0 431 | 432 | 433 | except ImportError: 434 | pass 435 | 436 | ################################################### 437 | 438 | 439 | def do_move_to_archive_and_rotate( 440 | backups_dir, 441 | archives_dir, 442 | backup_extensions, 443 | max_weekly_backups, 444 | hourly_backup_hour, 445 | weekly_backup_day, 446 | **unused_config 447 | ): 448 | check_dirs(backups_dir=backups_dir, archives_dir=archives_dir) 449 | 450 | # For each account, rotate out new_arrivals, old dailies, old weeklies. 451 | rotate_new_arrivals( 452 | backups_dir=backups_dir, 453 | archives_dir=archives_dir, 454 | backup_extensions=backup_extensions, 455 | period_name='hourly', 456 | ) 457 | 458 | for account_name in collect(archives_dir=archives_dir): 459 | kw = dict( 460 | archives_dir=archives_dir, 461 | hourly_backup_hour=hourly_backup_hour, 462 | weekly_backup_day=weekly_backup_day, 463 | account_name=account_name, 464 | ) 465 | 466 | rotate( 467 | period_name='hourly', 468 | next_period_name='daily', 469 | max_age=timedelta(hours=24), 470 | **kw 471 | ) 472 | rotate( 473 | period_name='daily', 474 | next_period_name='weekly', 475 | max_age=timedelta(days=7), 476 | **kw 477 | ) 478 | rotate( 479 | period_name='weekly', 480 | next_period_name='', 481 | max_age=timedelta(days=7*max_weekly_backups), 482 | **kw 483 | ) 484 | 485 | 486 | if __name__ == '__main__': 487 | import argparse 488 | parser = argparse.ArgumentParser() 489 | parser.add_argument("--noconfig", help="don't look for config file", action="store_true") 490 | args = parser.parse_args() 491 | 492 | config = SimpleConfig().config.__dict__['_sections']['Settings'] 493 | config['max_weekly_backups'] = int(config['max_weekly_backups']) 494 | config['hourly_backup_hour'] = int(config['hourly_backup_hour']) 495 | config['weekly_backup_day'] = int(config['weekly_backup_day']) 496 | 497 | do_move_to_archive_and_rotate(**config) 498 | 499 | sys.exit(0) 500 | --------------------------------------------------------------------------------