├── .circleci └── config.yml ├── .gitignore ├── LICENSE ├── README.md ├── keeper.png ├── keeper.py └── test_keeper.py /.circleci/config.yml: -------------------------------------------------------------------------------- 1 | version: 2 2 | jobs: 3 | build: 4 | working_directory: ~/jonatanblue/rundeck-backup-restore 5 | parallelism: 1 6 | shell: /bin/bash --login 7 | docker: 8 | - image: circleci/python:3-stretch-browsers-legacy 9 | steps: 10 | # Check out the code 11 | - checkout 12 | # Run the tests 13 | - run: python3 -m unittest discover 14 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | *.pyc 2 | tmp/* 3 | .idea 4 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2017 Jonatan Bjork 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Rundeck Backup & Restore 2 | 3 | ![keeper logo](keeper.png) 4 | 5 | CLI tool for RunDeck backup and restore. 6 | 7 | [![CircleCI](https://circleci.com/gh/jonatanblue/rundeck-backup-restore/tree/master.svg?style=svg)](https://circleci.com/gh/jonatanblue/rundeck-backup-restore/tree/master) 8 | 9 | **NOTE:** This project is still in beta. Please proceed with caution. 10 | 11 | # How to use 12 | 13 | ## Run 14 | 15 | ### Backup 16 | 17 | Backup everything and save to a `.tar.gz` file with the current date and time. This will result in the file `/opt/rundeck-backup-2017-06-09--12-41-42.tar.gz` being created, where the timestamp is the time when the script started running. 18 | 19 | ./keeper.py backup --dest /opt 20 | 21 | Backup only database and storage, to the file `/opt/rundeck-backup-partial-2017-06-09--12-46-09.tar.gz` being created. `-partial-` indicates that only some directories were backed up. 22 | 23 | ./keeper.py --dirs=/var/lib/rundeck/data,/var/lib/rundeck/var/storage backup --dest /opt/ 24 | 25 | ### Restore 26 | 27 | Restore all directories into their absolute paths on the host machine. If any file already exists, the restore **should** refuse to do anything and exit with an error and show the offending file. 28 | 29 | ./keeper.py restore --file /opt/rundeck-backup-2017-55-09--08-06-19.tar.gz 30 | 31 | Restore only the directory `/var/lib/rundeck/data`. 32 | 33 | ./keeper.py --dirs=/var/lib/rundeck/data restore --file /opt/rundeck-backup-2017-55-09--08-06-19.tar.gz 34 | 35 | 36 | 37 | # Test 38 | 39 | python3 -m unittest discover 40 | 41 | # Contribute 42 | 43 | Contributions are welcome! If you spot any bugs, then please submit an issue with the steps to reproduce it. You can also create issues for general questions, or if you have a suggestion for a new feature. 44 | 45 | For bug fixes and new features, make sure you have added tests that cover the use cases before submitting your PR. 46 | 47 | # Similar tools 48 | 49 | * The [official docs](http://rundeck.org/2.6.11/administration/backup-and-recovery.html) describe two options: either manually copy all data, using a combination of tools, or export an archive file from the web. The latter can take several hours, and does not include all files you may need. 50 | * [ersiko/rundeck-backup](https://github.com/ersiko/rundeck-backup) last commit 31 Mar 2013. There's a [blog post](https://blog.tomas.cat/en/2013/03/27/tool-manage-rundeck-backups/) describing how to use it. 51 | 52 | # Credits 53 | 54 | Thanks to [nvtkaszpir](https://github.com/nvtkaszpir) for initial feedback and suggestions. 55 | -------------------------------------------------------------------------------- /keeper.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/jonatanblue/rundeck-backup-restore/6bb03318dc63f0c1d3efd3d50038a40d2e3d2bf2/keeper.png -------------------------------------------------------------------------------- /keeper.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import argparse 4 | import subprocess 5 | import os 6 | import sys 7 | import logging 8 | import tarfile 9 | from datetime import datetime 10 | 11 | 12 | class Keeper: 13 | 14 | def __init__(self, system_directories=None, ignore_running=False): 15 | if self._rundeck_is_running(): 16 | if not ignore_running: 17 | # Refuse to do anything if RunDeck is running 18 | # This is best practice according to the docs: 19 | # http://rundeck.org/2.6.11/administration/backup-and-recovery.html 20 | logging.error("rundeckd cannot be running while you take a backup" 21 | " or restore from backup") 22 | raise Exception("rundeckd is still running") 23 | else: 24 | logging.warning("rundeckd is running! Proceeding anyways due to --ignore-running flag") 25 | 26 | self.count = 0 27 | self.bar = None 28 | # Directories to include in backup and restore 29 | if system_directories is None: 30 | # Default is to backup and restore all directories 31 | self.system_directories = [ 32 | # ToDo: add /etc/rundeck/realm.properties for user auth? 33 | "/var/lib/rundeck/data", # database 34 | "/var/lib/rundeck/logs", # execution logs (biggest) 35 | "/var/lib/rundeck/.ssh", # ssh keys 36 | "/var/lib/rundeck/var/storage", # keystore files and metadata 37 | "/var/rundeck/projects" # project definitions 38 | ] 39 | else: 40 | self.system_directories = system_directories 41 | 42 | # Raise exception if duplicate or 43 | # overlapping directories are passed in 44 | if self._has_duplicate_or_overlap(self.system_directories): 45 | raise Exception("duplicate or overlapping directories detected") 46 | 47 | # Paths must be absolute 48 | for path in self.system_directories: 49 | if path[0] != "/": 50 | # Path is relative 51 | raise Exception( 52 | "relative paths not allowed, please fix {}".format(path) 53 | ) 54 | 55 | def _has_duplicate_or_overlap(self, paths): 56 | """Return true if list of paths has duplicate or overlapping paths""" 57 | if len(paths) > 1: 58 | # TODO: This seems inefficient; see if there's a simpler deduper possible 59 | first = paths[0] 60 | remaining = paths[1:] 61 | for item in remaining: 62 | if first in item or item in first: 63 | logging.error("found conflicting paths {},{}".format( 64 | first, 65 | item 66 | )) 67 | return True 68 | self._has_duplicate_or_overlap(remaining) 69 | return False 70 | 71 | def _rundeck_is_running(self): 72 | """Return True if rundeckd is running, False otherwise""" 73 | try: 74 | status = subprocess.check_output( 75 | ["service", "rundeckd", "status"], 76 | # Universal newlines ensures error.output is a string 77 | universal_newlines=True 78 | ) 79 | except subprocess.CalledProcessError as error: 80 | if "rundeckd" not in error.output: 81 | raise Exception( 82 | "error running service command, " 83 | "is rundeckd installed on this machine?" 84 | ) 85 | else: 86 | status = error.output 87 | if "rundeckd" in status and "running" in status: 88 | return True 89 | else: 90 | return False 91 | 92 | def backup(self, destination_path, filename): 93 | """Create a backup file""" 94 | # Start message 95 | logging.debug("starting backup") 96 | 97 | if not os.path.exists(destination_path): 98 | logging.debug("backup directory {} not found; creating now".format( 99 | destination_path)) 100 | os.makedirs(destination_path) 101 | 102 | file_path = os.path.join(destination_path, filename) 103 | logging.debug("using full backup path {}".format(file_path)) 104 | 105 | # Create tar file and save all directories to it 106 | with tarfile.open(file_path, mode='w:gz', dereference=True) as archive: 107 | for directory in self.system_directories: 108 | if os.path.isdir(directory): 109 | logging.info("adding directory {}".format(directory)) 110 | archive.add(directory) 111 | else: 112 | logging.warning("skipping missing directory {}".format( 113 | directory 114 | )) 115 | 116 | logging.info("backup complete") 117 | 118 | def restore(self, filepath, directories=None): 119 | """Restore files from a backup tar file""" 120 | def _check_paths_before_restore(pathlist): 121 | """Check all files and raise exceptions if any already exists""" 122 | for path in pathlist: 123 | full_path = os.path.join("/", path.name) 124 | if (os.path.isfile(full_path)): 125 | logging.error( 126 | "no action taken, refusing to restore when" 127 | " file already exists on file system: {}".format( 128 | full_path 129 | ) 130 | ) 131 | raise Exception( 132 | "refusing to overwrite existing file: {}".format( 133 | full_path 134 | ) 135 | ) 136 | logging.info("loading backup file...") 137 | with tarfile.open(filepath, 'r:gz') as archive: 138 | all_files = archive.getmembers() 139 | # All filenames go here 140 | files_to_restore = [] 141 | for tarinfo in all_files: 142 | # Check each directory against all files 143 | for path in self.system_directories: 144 | # Remove any '/' from the start of the path 145 | if path.startswith('/'): 146 | path = path[1:] 147 | # Save each individual file for later checking existence 148 | if tarinfo.name.startswith(path): 149 | files_to_restore.append(tarinfo) 150 | logging.debug("path: " + path) 151 | # Check that files don't already exist before restoring 152 | logging.info( 153 | "checking restore paths to avoid overwriting existing files..." 154 | ) 155 | _check_paths_before_restore(files_to_restore) 156 | logging.info("restoring files into directories {}".format( 157 | ",".join(self.system_directories) 158 | )) 159 | 160 | restore_members = files_to_restore 161 | logging.debug( 162 | "restoring files in {}".format(self.system_directories) 163 | ) 164 | archive.extractall( 165 | path="/", 166 | members=restore_members 167 | ) 168 | logging.info("restore complete: {} files".format( 169 | len(files_to_restore) 170 | )) 171 | 172 | 173 | def main(arguments): 174 | # Gather arguments 175 | parser_name = arguments.subparser_name 176 | debug_mode = arguments.debug 177 | 178 | # Enable debug logging if flag is set 179 | if debug_mode: 180 | log_level = logging.DEBUG 181 | else: 182 | log_level = logging.INFO 183 | # Set up logging 184 | logging.basicConfig( 185 | format='%(asctime)s %(levelname)s %(message)s', 186 | level=log_level) 187 | 188 | # Set backup directories 189 | if arguments.dirs: 190 | system_directories = arguments.dirs[0].split(",") 191 | if parser_name == "backup": 192 | # Validate that paths exist 193 | for directory in system_directories: 194 | if os.path.exists(directory) or os.access(directory, os.W_OK): 195 | pass 196 | else: 197 | raise Exception("directory is not a valid path " 198 | "or not writeable: {}".format(directory)) 199 | logging.warning("overriding default directories with {}".format( 200 | ",".join(system_directories) 201 | )) 202 | # Add "partial" to the name since we are overriding 203 | partial = "partial-" 204 | else: 205 | system_directories = None 206 | partial = "" 207 | 208 | # Restore does not have the option to ignore running, so default to False 209 | if "ignore_running" in arguments: 210 | ignore_running = arguments.ignore_running 211 | else: 212 | ignore_running = False 213 | keeper = Keeper( 214 | system_directories=system_directories, 215 | ignore_running=ignore_running 216 | ) 217 | 218 | if parser_name == "backup": 219 | # Set the name of the backup file to be created 220 | if arguments.filename: 221 | backup_filename = arguments.filename 222 | else: 223 | backup_filename = "rundeck-backup-" + partial + "{}.tar.gz".format( 224 | datetime.now().strftime('%Y-%m-%d--%H-%M-%S') 225 | ) 226 | keeper.backup( 227 | destination_path=arguments.dest, 228 | filename=backup_filename) 229 | elif parser_name == "restore": 230 | keeper.restore(filepath=arguments.file) 231 | 232 | 233 | def parse_args(args): 234 | parser = argparse.ArgumentParser( 235 | description='keeper: helper for backup and restore of RunDeck') 236 | parser.add_argument( 237 | '--debug', 238 | '-d', 239 | action='store_true', 240 | help='enable debug logging') 241 | parser.add_argument( 242 | '--dirs', 243 | type=str, 244 | nargs="*", 245 | help='comma-separated list that overrides the default list of system' 246 | 'directories to backup/restore') 247 | 248 | subparsers = parser.add_subparsers(help='command help', 249 | dest='subparser_name') 250 | 251 | # Backup options 252 | backup_parser = subparsers.add_parser('backup', help='create a backup') 253 | backup_parser.add_argument( 254 | '--dest', 255 | type=str, 256 | required=True, 257 | help='path to write backup file to') 258 | backup_parser.add_argument( 259 | '--filename', 260 | type=str, 261 | help='override the filename used the for backup file') 262 | backup_parser.add_argument( 263 | '--ignore-running', 264 | action='store_true', 265 | default=False, 266 | help='allow backup even if rundeckd is running' 267 | ) 268 | 269 | # Restore options 270 | restore_parser = subparsers.add_parser( 271 | 'restore', 272 | help='restore from a backup file') 273 | restore_parser.add_argument( 274 | '--file', 275 | type=str, 276 | help='path to backup file to restore from') 277 | 278 | return parser.parse_args(args) 279 | 280 | 281 | if __name__ == "__main__": 282 | parsed = parse_args(sys.argv[1:]) 283 | main(parsed) 284 | -------------------------------------------------------------------------------- /test_keeper.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | import logging 4 | import unittest 5 | import os 6 | import glob 7 | import shutil 8 | import tarfile 9 | import keeper 10 | from keeper import Keeper 11 | 12 | # Enable verbose logs for tests 13 | logger = logging.getLogger() 14 | logger.level = logging.DEBUG 15 | 16 | # Get current directory 17 | BASE_DIRECTORY = os.getcwd() 18 | 19 | 20 | # Override rundeck service check for unittests 21 | def rundeck_is_running(arg): 22 | return False 23 | 24 | 25 | Keeper._rundeck_is_running = rundeck_is_running 26 | 27 | 28 | class MockedKeeper(Keeper): 29 | def __init__(self, *args): 30 | pass 31 | 32 | 33 | class TestKeeper(unittest.TestCase): 34 | """Tests for `keeper.py`""" 35 | 36 | def setUp(self): 37 | """Set up defaults for all tests""" 38 | self.maxDiff = None 39 | 40 | def _create_dir(self, path): 41 | """Creates directory""" 42 | if not os.path.exists(path): 43 | os.makedirs(path) 44 | 45 | def _purge_directory(self, path): 46 | """Purges a directory and all its subdirectories 47 | 48 | WARNING: This will recursively delete the directory and all 49 | subdirectories forever. 50 | """ 51 | shutil.rmtree(path) 52 | 53 | def _list_files_in_tar(self, path): 54 | """Returns list of all file paths inside a tar file""" 55 | with tarfile.open(path, 'r:gz') as archive: 56 | return archive.getnames() 57 | 58 | def test_instantiating(self): 59 | """Test that Keeper class can be instantiated""" 60 | directories = [ 61 | "/var/lib/rundeck/data", # database 62 | "/var/lib/rundeck/logs", # execution logs (by far biggest) 63 | "/var/lib/rundeck/.ssh", # ssh keys 64 | "/var/lib/rundeck/var/storage", # key storage files and metadata 65 | "/var/rundeck/projects" # project definitions 66 | ] 67 | Keeper(system_directories=directories) 68 | 69 | def test_has_overlap(self): 70 | """Test that overlap check works""" 71 | overlapping_dirs = [ 72 | "/tmp/a/b", 73 | "/tmp/a" 74 | ] 75 | keeper = MockedKeeper() 76 | self.assertTrue(keeper._has_duplicate_or_overlap(overlapping_dirs)) 77 | 78 | def test_has_overlap_reverse(self): 79 | """Test that overlap check works""" 80 | overlapping_dirs = [ 81 | "/tmp/a", 82 | "/tmp/a/b" 83 | ] 84 | keeper = MockedKeeper() 85 | self.assertTrue(keeper._has_duplicate_or_overlap(overlapping_dirs)) 86 | 87 | def test_has_duplicate(self): 88 | """Test that duplicate check works""" 89 | duplicate_dirs = [ 90 | "/tmp/a/b", 91 | "/tmp/a/b" 92 | ] 93 | keeper = MockedKeeper() 94 | 95 | self.assertTrue(keeper._has_duplicate_or_overlap(duplicate_dirs)) 96 | 97 | def test_valid_path_list(self): 98 | """Test that a valid path list is valid according to check""" 99 | valid_dirs = [ 100 | "/tmp/a/b/c", 101 | "/tmp/a/b/d", 102 | "/tmp/q", 103 | "/var/troll" 104 | ] 105 | 106 | keeper = MockedKeeper() 107 | 108 | self.assertFalse(keeper._has_duplicate_or_overlap(valid_dirs)) 109 | 110 | def test_raises_exception_on_relative_paths(self): 111 | """Test that relative paths raise an exception""" 112 | contains_relative_paths = [ 113 | "some/path/here", 114 | "some/other/path", 115 | "/this/is/valid/though" 116 | ] 117 | with self.assertRaises(Exception): 118 | Keeper(system_directories=contains_relative_paths) 119 | 120 | def test_raises_exception_on_overlapping_dirs(self): 121 | """Test that exception is raised for overlapping dirs 122 | 123 | Passing overlapping directories should raise an exception. 124 | For example /tmp/a/b/c,/tmp/a/b should fail 125 | """ 126 | # Set bad directories 127 | bad_directories = [ 128 | "/tmp/keeper_python_unittest_raises/a/b/c", 129 | "/tmp/keeper_python_unittest_raises/a/b" 130 | ] 131 | # Set sails 132 | with self.assertRaises(Exception): 133 | Keeper(system_directories=bad_directories) 134 | 135 | def test_raises_exception_on_overlapping_dirs_reversed(self): 136 | """Test that exception is raised for overlapping dirs. 137 | 138 | For example /tmp/a/b,/tmp/a/b/c should fail 139 | """ 140 | # Set bad directories 141 | bad_directories = [ 142 | "/tmp/keeper_python_unittest_raises/a/b", 143 | "/tmp/keeper_python_unittest_raises/a/b/c" 144 | ] 145 | # Set sails 146 | with self.assertRaises(Exception): 147 | Keeper(system_directories=bad_directories) 148 | 149 | def test_backup(self): 150 | """Test creating a backup file from a set of directories""" 151 | cwd = os.getcwd() 152 | # Set paths 153 | file_paths = [ 154 | cwd + "/tmp/keeper_test_backup/house/room/file1.txt", 155 | cwd + "/tmp/keeper_test_backup/house/room/desk/file2.txt", 156 | cwd + "/tmp/keeper_test_backup/house/room/desk/file3.txt", 157 | cwd + "/tmp/keeper_test_backup/house/room/desk/drawer/file4", 158 | cwd + "/tmp/keeper_test_backup/house/room/locker/file5.txt" 159 | ] 160 | folder_paths_to_create = [ 161 | cwd + "/tmp/keeper_test_backup/house/room/desk/drawer", 162 | cwd + "/tmp/keeper_test_backup/house/room/locker" 163 | ] 164 | directories_to_backup = [ 165 | cwd + "/tmp/keeper_test_backup/house/room/desk/drawer", 166 | cwd + "/tmp/keeper_test_backup/house/room/locker" 167 | ] 168 | files_expected_in_tar = [ 169 | os.path.join( 170 | cwd.strip("/"), 171 | "tmp/keeper_test_backup/house/room/desk/drawer" 172 | ), 173 | os.path.join( 174 | cwd.strip("/"), 175 | "tmp/keeper_test_backup/house/room/desk/drawer/file4" 176 | ), 177 | os.path.join( 178 | cwd.strip("/"), 179 | "tmp/keeper_test_backup/house/room/locker" 180 | ), 181 | os.path.join( 182 | cwd.strip("/"), 183 | "tmp/keeper_test_backup/house/room/locker/file5.txt" 184 | ) 185 | ] 186 | 187 | keeper = Keeper(system_directories=directories_to_backup) 188 | 189 | # Create all directories 190 | for path in folder_paths_to_create: 191 | self._create_dir(path) 192 | 193 | # Create all files for backup test 194 | for path in file_paths: 195 | # Create file 196 | with open(path, "w") as file_handle: 197 | file_handle.write("lorem ipsum\n") 198 | 199 | # Create backup 200 | keeper.backup( 201 | destination_path=cwd + "/tmp/keeper_test_backup", 202 | filename="backup_test.tar.gz" 203 | ) 204 | 205 | # Get list of all file paths inside tar file 206 | files_in_tar = self._list_files_in_tar( 207 | cwd + "/tmp/keeper_test_backup/backup_test.tar.gz") 208 | 209 | # tar file can't be empty 210 | self.assertNotEqual(len(files_in_tar), 0) 211 | 212 | # Normpath the paths 213 | # NOTE: I don't know why this is necessary 214 | files_expected_in_tar = [ 215 | os.path.normpath(p) for p in files_expected_in_tar 216 | ] 217 | files_in_tar = [ 218 | os.path.normpath(p) for p in files_in_tar 219 | ] 220 | 221 | # Compare tar file and list of files 222 | self.assertEqual(set(files_expected_in_tar), set(files_in_tar)) 223 | 224 | # Recursively remove all directories and files used in test 225 | self._purge_directory(cwd + "/tmp/keeper_test_backup") 226 | 227 | def test_backup_skips_missing_dir(self): 228 | """Test that missing directory is skipped""" 229 | cwd = os.getcwd() 230 | # Set paths 231 | file_paths = [ 232 | cwd + "/tmp/keeper_test_backup/house/room/file1.txt", 233 | cwd + "/tmp/keeper_test_backup/house/room/desk/file2.txt", 234 | cwd + "/tmp/keeper_test_backup/house/room/desk/file3.txt", 235 | cwd + "/tmp/keeper_test_backup/house/room/desk/drawer/file4", 236 | cwd + "/tmp/keeper_test_backup/house/room/locker/file5.txt" 237 | ] 238 | folder_paths_to_create = [ 239 | cwd + "/tmp/keeper_test_backup/house/room/desk/drawer", 240 | cwd + "/tmp/keeper_test_backup/house/room/locker" 241 | ] 242 | directories_to_backup = [ 243 | cwd + "/tmp/keeper_test_backup/house/room/desk/drawer", 244 | cwd + "/tmp/keeper_test_backup/house/room/locker", 245 | cwd + "/tmp/keeper_test_backup/ghosthouse" # this does not exist 246 | ] 247 | files_expected_in_tar = [ 248 | os.path.join( 249 | cwd.strip("/"), 250 | "tmp/keeper_test_backup/house/room/desk/drawer" 251 | ), 252 | os.path.join( 253 | cwd.strip("/"), 254 | "tmp/keeper_test_backup/house/room/desk/drawer/file4" 255 | ), 256 | os.path.join( 257 | cwd.strip("/"), 258 | "tmp/keeper_test_backup/house/room/locker" 259 | ), 260 | os.path.join( 261 | cwd.strip("/"), 262 | "tmp/keeper_test_backup/house/room/locker/file5.txt" 263 | ) 264 | ] 265 | 266 | keeper = Keeper(system_directories=directories_to_backup) 267 | 268 | # Create all directories 269 | for path in folder_paths_to_create: 270 | self._create_dir(path) 271 | 272 | # Create all files for backup test 273 | for path in file_paths: 274 | # Create file 275 | with open(path, "w") as file_handle: 276 | file_handle.write("lorem ipsum\n") 277 | 278 | # Create backup 279 | keeper.backup( 280 | destination_path=cwd + "/tmp/keeper_test_backup", 281 | filename="backup_test.tar.gz" 282 | ) 283 | 284 | # Get list of all file paths inside tar file 285 | files_in_tar = self._list_files_in_tar( 286 | cwd + "/tmp/keeper_test_backup/backup_test.tar.gz") 287 | 288 | # tar file can't be empty 289 | self.assertNotEqual(len(files_in_tar), 0) 290 | 291 | # Normpath the paths 292 | # NOTE: I don't know why this is necessary 293 | files_expected_in_tar = [ 294 | os.path.normpath(p) for p in files_expected_in_tar 295 | ] 296 | files_in_tar = [ 297 | os.path.normpath(p) for p in files_in_tar 298 | ] 299 | 300 | # Compare tar file and list of files 301 | self.assertEqual(set(files_expected_in_tar), set(files_in_tar)) 302 | 303 | # Recursively remove all directories and files used in test 304 | self._purge_directory(cwd + "/tmp/keeper_test_backup") 305 | 306 | def test_restore(self): 307 | """Test restoring a set of directories and files from a backup file""" 308 | # Set paths 309 | cwd = os.getcwd() 310 | file_paths = [ 311 | cwd + "/tmp/keeper_test_restore/hotel/lobby/file1.txt", 312 | cwd + "/tmp/keeper_test_restore/hotel/lobby/desk/file2.txt", 313 | cwd + "/tmp/keeper_test_restore/hotel/lobby/desk/file3.txt", 314 | cwd + "/tmp/keeper_test_restore/hotel/lobby/desk/drawer/f4", 315 | cwd + "/tmp/keeper_test_restore/hotel/lobby/locker/file5.txt" 316 | ] 317 | folder_paths_to_create = [ 318 | cwd + "/tmp/keeper_test_restore/hotel/lobby/desk/drawer/", 319 | cwd + "/tmp/keeper_test_restore/hotel/lobby/locker" 320 | ] 321 | directories_to_backup = [ 322 | cwd + "/tmp/keeper_test_restore/hotel/lobby/desk/drawer/", 323 | cwd + "/tmp/keeper_test_restore/hotel/lobby/locker/" 324 | ] 325 | files_expected_in_restore = [ 326 | cwd + "/tmp/keeper_test_restore/hotel/lobby/locker/file5.txt", 327 | cwd + "/tmp/keeper_test_restore/hotel/lobby/desk/drawer/f4" 328 | ] 329 | 330 | keeper = Keeper(system_directories=directories_to_backup) 331 | 332 | # Create all directories 333 | for path in folder_paths_to_create: 334 | self._create_dir(path) 335 | 336 | # Create all files for backup 337 | for path in file_paths: 338 | # Create file 339 | with open(path, "w") as file_handle: 340 | file_handle.write("lorem ipsum\n") 341 | 342 | # Create backup 343 | keeper.backup( 344 | destination_path=cwd + "/tmp/keeper_test_restore", 345 | filename="restore_test.tar.gz" 346 | ) 347 | 348 | # Purge the source directory 349 | self._purge_directory(cwd + "/tmp/keeper_test_restore/hotel") 350 | 351 | # Restore 352 | keeper.restore( 353 | cwd + "/tmp/keeper_test_restore/restore_test.tar.gz") 354 | 355 | # List all directories 356 | restored = cwd + "/tmp/keeper_test_restore/hotel" 357 | files_found = [] 358 | for root, dirs, files in os.walk(restored): 359 | for f in files: 360 | files_found.append(os.path.join(root, f)) 361 | 362 | self.assertEqual(set(files_found), set(files_expected_in_restore)) 363 | 364 | # Clean up test files and directories 365 | self._purge_directory(cwd + "/tmp/keeper_test_restore") 366 | 367 | def test_restore_check_content(self): 368 | """Test restoring a file and check contents""" 369 | # Set paths 370 | cwd = os.getcwd() 371 | file_paths = [ 372 | cwd + "/tmp/keeper_test_r_check/a/b/file1.txt", 373 | cwd + "/tmp/keeper_test_r_check/a/b/c/file2.txt", 374 | cwd + "/tmp/keeper_test_r_check/a/b/c/file3.txt", 375 | cwd + "/tmp/keeper_test_r_check/a/b/c/e/f4", 376 | cwd + "/tmp/keeper_test_r_check/a/b/d/file5.txt" 377 | ] 378 | folder_paths_to_create = [ 379 | cwd + "/tmp/keeper_test_r_check/a/b/c/e/", 380 | cwd + "/tmp/keeper_test_r_check/a/b/d" 381 | ] 382 | directories_to_backup = [ 383 | cwd + "/tmp/keeper_test_r_check/a/b/d/" 384 | ] 385 | file_expected_in_restore = os.path.join( 386 | cwd + "/tmp/keeper_test_r_check/a/b/d/file5.txt" 387 | ) 388 | 389 | keeper = Keeper(system_directories=directories_to_backup) 390 | 391 | # Create all directories 392 | for path in folder_paths_to_create: 393 | self._create_dir(path) 394 | 395 | # Create all files for backup 396 | for path in file_paths: 397 | # Create file 398 | with open(path, "w") as file_handle: 399 | file_handle.write("lorem ipsum\n") 400 | 401 | # Create backup 402 | keeper.backup( 403 | destination_path=cwd + "/tmp/keeper_test_r_check", 404 | filename="restore_test.tar.gz" 405 | ) 406 | 407 | # Purge the source directory 408 | self._purge_directory(cwd + "/tmp/keeper_test_r_check/a") 409 | 410 | # Restore 411 | keeper.restore( 412 | cwd + "/tmp/keeper_test_r_check/restore_test.tar.gz") 413 | 414 | # Get file contents 415 | with open(file_expected_in_restore, 'r') as restored_file: 416 | content = restored_file.read() 417 | logging.debug("content " + content) 418 | 419 | self.assertEqual(content, "lorem ipsum\n") 420 | 421 | # Clean up test files and directories 422 | self._purge_directory(cwd + "/tmp/keeper_test_r_check") 423 | 424 | def test_restore_does_not_overwrite(self): 425 | """Test that existing files are not overwritten by restore""" 426 | cwd = os.getcwd() 427 | base = cwd + "/tmp/keeper_python_unittest_restore_no_overwrite" 428 | # Set paths 429 | file_paths = [ 430 | base + "/hotel/lobby/file1.txt", 431 | base + "/hotel/lobby/desk/file2.txt", 432 | base + "/hotel/lobby/desk/file3.txt", 433 | base + "/hotel/lobby/desk/drawer/f4", 434 | base + "/hotel/lobby/locker/file5.txt" 435 | ] 436 | folder_paths_to_create = [ 437 | base + "/hotel/lobby/desk/drawer/", 438 | base + "/hotel/lobby/locker" 439 | ] 440 | directories_to_backup = [ 441 | base + "/hotel/lobby/desk/drawer/", 442 | base + "/hotel/lobby/locker/" 443 | ] 444 | files_expected_in_restore = [ 445 | base + "/hotel/lobby/desk/drawer/f4", 446 | base + "/hotel/lobby/locker/file5.txt" 447 | ] 448 | 449 | keeper = Keeper(system_directories=directories_to_backup) 450 | 451 | # Create all directories 452 | for path in folder_paths_to_create: 453 | self._create_dir(path) 454 | 455 | # Create all files for backup 456 | for path in file_paths: 457 | # Create file 458 | with open(path, "w") as file_handle: 459 | file_handle.write("lorem ipsum\n") 460 | 461 | # Create backup 462 | keeper.backup( 463 | destination_path=base, 464 | filename="restore_test.tar.gz" 465 | ) 466 | 467 | # Write to files again 468 | for name in files_expected_in_restore: 469 | with open(name, "w") as file_handle: 470 | file_handle.write("new version\n") 471 | 472 | # Restore should raise exception on existing file 473 | with self.assertRaises(Exception): 474 | keeper.restore(base + "/restore_test.tar.gz") 475 | 476 | # Get file contents 477 | files_content = [] 478 | for name in files_expected_in_restore: 479 | with open(name, "r") as file_handle: 480 | content = file_handle.read() 481 | files_content.append(content) 482 | 483 | self.assertEqual( 484 | files_content, 485 | [ 486 | "new version\n", 487 | "new version\n" 488 | ] 489 | ) 490 | 491 | # Purge the test directory 492 | self._purge_directory(base) 493 | 494 | def test_backup_file_name_different_for_partial(self): 495 | """Test that partial backup file is named correctly 496 | 497 | If there is a directory override, the file should have 498 | "partial" in the name 499 | """ 500 | # Set paths 501 | cwd = os.getcwd() 502 | base = cwd + "/tmp/keeper_python_unittest_partial_name" 503 | file_paths = [ 504 | base + "/a/b/c.txt", 505 | base + "/q/r.txt" 506 | ] 507 | folder_paths_to_create = [ 508 | base + "/a/b", 509 | base + "/q" 510 | ] 511 | 512 | # Create all directories 513 | for path in folder_paths_to_create: 514 | self._create_dir(path) 515 | 516 | # Create all files for backup test 517 | for path in file_paths: 518 | # Create file 519 | with open(path, "w") as file_handle: 520 | file_handle.write("lorem ipsum\n") 521 | 522 | # Create backup 523 | args = keeper.parse_args([ 524 | '--dirs=' + cwd + '/tmp/keeper_python_unittest_partial_name/a/b', 525 | 'backup', 526 | '--dest', 'tmp/keeper_python_unittest_partial_name' 527 | ]) 528 | keeper.main(args) 529 | 530 | # Get filename 531 | archive_filename = glob.glob(base + "/*.tar.gz")[0] 532 | 533 | self.assertTrue("partial" in archive_filename) 534 | 535 | # Recursively remove all directories and files used in test 536 | self._purge_directory(cwd + "/tmp/keeper_python_unittest_partial_name") 537 | 538 | def test_restore_subset_directories(self): 539 | """Test restoring a subset of directories""" 540 | # Set paths 541 | cwd = os.getcwd() 542 | base = cwd + "/tmp/keeper_python_unittest_restore_subset" 543 | file_paths = [ 544 | base + "/a/b/file1.txt", 545 | base + "/a/b/c/file2.txt", 546 | base + "/a/b/c/file3.txt", 547 | base + "/a/b/c/e/file4.txt", 548 | base + "/a/b/d/file5.txt" 549 | ] 550 | folder_paths_to_create = [ 551 | base + "/a/b/c/e/", 552 | base + "/a/b/d" 553 | ] 554 | files_expected_in_restore = [ 555 | base + "/a/b/c/e/file4.txt" 556 | ] 557 | 558 | # Create all directories 559 | for path in folder_paths_to_create: 560 | self._create_dir(path) 561 | 562 | # Create all files for backup 563 | for path in file_paths: 564 | # Create file 565 | with open(path, "w") as file_handle: 566 | file_handle.write("lorem ipsum\n") 567 | 568 | # Create backup 569 | args = keeper.parse_args([ 570 | '--dirs=' + base + '/a/b', 571 | 'backup', 572 | '--dest', base, 573 | '--filename', "test.tar.gz" 574 | ]) 575 | keeper.main(args) 576 | 577 | # Purge the source directory 578 | self._purge_directory(base + "/a") 579 | 580 | # Restore 581 | args = keeper.parse_args([ 582 | '--dirs=' + base + '/a/b/c/e', 583 | 'restore', 584 | '--file=' + base + '/test.tar.gz' 585 | ]) 586 | keeper.main(args) 587 | 588 | # List all directories 589 | restored = base + "/a" 590 | files_found = [] 591 | for root, dirs, files in os.walk(restored): 592 | for f in files: 593 | files_found.append(os.path.join(root, f)) 594 | 595 | self.assertEqual(set(files_found), set(files_expected_in_restore)) 596 | 597 | # Clean up directory 598 | self._purge_directory(base) 599 | --------------------------------------------------------------------------------