├── CHANGELOG.md ├── LICENSE.md ├── README.md ├── plugin └── vifm │ └── saf-revisions │ ├── README.md │ └── init.lua └── saf /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Changelog 2 | 3 | ## [0.16] - 2023-08-24 4 | 5 | - Added optional [local-path] parameter at `saf backup`, to conveniently set directory before running backup, so it can be used without having to `cd` first. 6 | - Improved error output at `saf backup` if backup doesn't finish with success. 7 | - New `saf revisions [-h] [--quiet | --verbose {0,1,2} | --debug] [target-name] path` command to list all the changes in any local folder troughout backup target versions. 8 | - New [Vifm](https://vifm.info/) plugin, to provide full time-machine like restore! Starfield! Warp! In Vifm! 9 | 10 | ## [0.15] - 2023-08-19 11 | 12 | - Initial public fully working version. 13 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | MIT License 2 | 3 | Copyright (c) 2015-2023 Dusan Popovic 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | 23 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # saf 2 | 3 | **[Basic concepts](#basic-concepts)** | 4 | **[Installation](#installation)** | 5 | **[Usage](#tldr-usage)** | 6 | **[Commands](#commands)** | 7 | **[File manager integration](#file-manager-integration)** | 8 | **[Contributing](#contributing)** | 9 | **[Related projects](#related-projects)** | 10 | **[Changelog](#changelog)** | 11 | **[License](#license)** 12 | 13 | "one backup is saf, two are safe, three are safer" 14 | 15 | > Current saf version is 0.16, saf requires Python 3.10 or newer 16 | 17 | **saf** is a simple, reliable, rsync-based, battle tested, well rounded backup system, written in Python. 18 | 19 | It uses rsync to incrementally back up your data to a different directory, hard disk or remote server via SSH. All operations are incremental, atomic and optionally possible to resume. 20 | 21 | **saf** is using very limited set of commands in the background, so it plays well with storage shells with limited set of commands, like Hetzner Storage Box. 22 | 23 | Main backup code is used in production for many years and has proved its reliability, while recent **saf** updates only improve user facing commands. No guarantees but should be safe to use. 24 | 25 | ## Basic concepts 26 | 27 | In **saf** every local file system directory can become backup source location. Each backup source location can have one or more backup target locations, in practice same directory (and its sub-directories) can be backed up to several target locations. 28 | 29 | Each backup source location (initialised by `saf init`) contains `.saf.conf` file, which is a simple and easy to change configuration file in Python configparser format. It is easy to query backup source locations down the directory tree with `saf status` or `saf status --all`, or up the directory tree with `saf status --reverse`. 30 | 31 | Each `.saf.conf` can contain one or more `[.target]` sections, defining backup target locations. It is easy to add any number of different local or remote backup locations with `saf init --name `, where `target-location` can be local directory `/mnt/backup-disk/backup-path` or remote storage/server `my-server:/mnt/backup-disk/backup-path`. 32 | 33 | Every **saf** command is location relative. **saf** will find `.saf.conf` in current or any of the parent directories and perform requested command. First backup target is always considered default but all the commands can specify what target name they are referring to. 34 | 35 | > **_NOTE:_** Don't use relative paths to specify target location. Always specify full target location `/home/user/backup` instead of `./backup` or `~/backup`. Relative locations are user related and not suitable for backup target. 36 | 37 | > **_NOTE:_** Final slash `/` when specifying target location doesn't matter, **saf** will interpret location with or without slash correctly. 38 | 39 | Every `.saf.conf` contains `[exclude]` section that uses standard rsync exclude syntax, so we can specify exclusions for each backup source. 40 | 41 | ## Installation 42 | 43 | ### Clone from github 44 | 45 | ```bash 46 | git clone https://github.com/dusanx/saf 47 | ``` 48 | It is usually a good idea to put **saf** in path to be available on the command line, but it will work as good with specifying `/full/path/to/saf` on each invocation. Contibutors are welcome to create packages and send MR's for any specific OS or distribution. 49 | 50 | ### Packages for Arch or Debian or Ubuntu or Mint or Fedora or Gentoo or ... 51 | 52 | Contributors are welcome. 53 | 54 | ## TL;DR usage 55 | 56 | I want to create backup source point at my home folder, so I need to go there first: 57 | ```bash 58 | cd ~ 59 | ``` 60 | Initialize backup source and first local backup target (since we didn't specify `--name`, this target will be called B0): 61 | ```bash 62 | saf init /mnt/backup_ssd/saf-backup/home 63 | ``` 64 | > **_NOTE:_** Each **saf** command will require that `backup.marker` file exists at the destination and offer commands to create it if it is missing. This applies to any backup target, so **saf** can be sure that we can reach target location. In short just follow instructions when `backup.marker` related message appears. 65 | 66 | Check and modify `.saf.conf` responsible for current folder: 67 | ```bash 68 | saf status 69 | saf configedit 70 | ``` 71 | Run the first backup (`--name` can specify target if there is more than one): 72 | ```bash 73 | saf backup 74 | ``` 75 | Check the backups (`--name` can specify target if there is more than one): 76 | ```bash 77 | saf list 78 | saf realsizes 79 | ``` 80 | Add another target, this time over ssh on my-remote-server (specify target name too): 81 | ```bash 82 | saf init --name my-vpn my-remote-server:/backup/saf-will-use-this/home 83 | ``` 84 | Add another target, this time on my Hetzner Storage Box (replace u000000 with actual Hetzner user ID): 85 | ```bash 86 | saf init --name H0 u000000@u000000.your-storagebox.de:/home/backup/saf-backup 87 | ``` 88 | > **_NOTE:_** Make sure to check and edit `.saf.conf` to get all the parameters right, for instance remote storage box can be using different port than usual port 22. 89 | 90 | With single backup source and three different backup targets, we can issue any command without specifying `--name` (meaning **saf** will use first target in the list, in our case `B0`) or by specifying any of the `B0`, `my-vpn`, or `H0` names: 91 | ```bash 92 | saf list H0 93 | saf realsizes my-vpn 94 | saf backup my-vpn 95 | saf rmrf H0 ./bin/zoom 96 | saf backup # Note: this command will assume target B0 since it is the first target 97 | saf freespace H0 98 | ``` 99 | > **_NOTE:_** Every **saf** command is location relative. **saf** will find `.saf.conf` in current or any of the parent directories and perform command. First backup target is always considered default but all the commands can specify what target name they are referring to. 100 | 101 | ## Commands 102 | 103 | ### saf version [-h] 104 | 105 | Print **saf** version. 106 | 107 | ### saf init [-h] [--name taget-name] target 108 | 109 | Initialize **saf** backup source at current folder and add first backup target or add backup target to already existing `.saf.conf`. 110 | 111 | ### saf status [-h] [--all | --reverse] 112 | 113 | Check status of the current directory, is it covered by any `.saf.conf` backup source. 114 | Add `--all` to check all the locations up to root folder, since current folder can be covered by many different source locations. 115 | Add `--reverse` to find all `.saf.conf` files recursively in all the sub-directories. 116 | `--all` and `--reverse` can't be used at the same time, they are searching in different directions. 117 | 118 | ### saf configedit [-h] [--quiet | --verbose {0,1,2} | --debug] 119 | 120 | Find and edit `.saf.conf` closest to current directory. This `.saf.conf` will be used for all the **saf** commands issued in current directory. 121 | 122 | ### saf list [-h] [--quiet | --verbose {0,1,2} | --debug] [target-name] 123 | 124 | List all the backups in the specific [target-name] or using defult one. Sorted by time and date, with markers showing `keep-` difference, as specified in `.saf.conf` 125 | 126 | ### saf realsizes [-h] [--quiet | --verbose {0,1,2} | --debug] [target-name] 127 | 128 | Uses `du` to reach target location and check real folder sizes, since we are using rsync with hard links wherever possible. 129 | 130 | ### saf freespace [-h] [--quiet | --verbose {0,1,2} | --debug] [target-name] 131 | 132 | Checks free space left on the specified or default target location. 133 | 134 | ### saf prune [-h] [--quiet | --verbose {0,1,2} | --debug] [target-name] 135 | 136 | Prunes backups by criteria specifed in `.saf.conf`. Also executed automatically before each `saf backup`. 137 | 138 | ### saf backup [-h] [--quiet | --verbose {0,1,2} | --debug] [--resume] [target-name] [local-path] 139 | 140 | Runs the backup. It is probably a good idea to start it periodically for each backup target. Since **saf** is using current directory to determine what backup source and target to use, either cd to the desired directory before running backup or add desired local path as parameter so saf can cd there before backup. 141 | 142 | Last optional parameter can specify local path, so **saf** can cd there before executing backup. If local-path is specified there is no need to use `cd && saf backup`, `saf backup B0 /home/user` can be used instead. If local-path is specified, target-name must be specified too, even if it is the default target. 143 | 144 | Example usage for the local user home directory, for the default backup target: 145 | ```bash 146 | cd /home/user && saf backup 147 | ``` 148 | or 149 | ```bash 150 | saf backup B0 /home/user 151 | ``` 152 | 153 | ### saf rmrf [-h] [--quiet | --verbose {0,1,2} | --debug] [target-name] path 154 | 155 | It is not unusual that some big or unwanted files or folders slip into backups. To prevent their future appearance we can edit `.saf.conf` to exclude such files or directories. However, as they remain present and taking space in the existing backups, `rmrf` offers elegant way to remove them. For instance, if my `H0` backup contains unwanted `~/bin/zoom`, I can easily remove it everywhere with 156 | ```bash 157 | cd /home/user && saf rmrf H0 ./bin/zoom 158 | ``` 159 | 160 | > **_NOTE:_** Using relative paths in `rmrf` is going to work since all the paths are calculated from backup source location. 161 | 162 | ### saf difference [-h] [--quiet | --verbose {0,1,2} | --debug] [target-name] backup 163 | 164 | This command should compare backup with the previous one and detect differences. Currently works well with the local file system backup targets. Depending on the remote backup shell capabilities, maybe works with the remote targets if they have rsync command available. 165 | 166 | ### saf revisions [-h] [--quiet | --verbose {0,1,2} | --debug] [target-name] path 167 | 168 | Most useful for local backup storage use, `saf revisions` will list all the backup directories, scattered across all backups in local backup target, where specified path content differs. 169 | 170 | ```bash 171 | saf revisions ~/Desktop 172 | ``` 173 | 174 | For the remote backup targets, this command is much more limited, listing all the *possible* locations for the requested path. 175 | 176 | For the local backup targets, this command will list only those backups that contain requested folder content changes, so the list will both be short and only listing different/changed content. 177 | 178 | List of revisions can be used to manually `cd` to any of the backups and compare with or restore to a specified directory. 179 | 180 | ## File manager integration 181 | 182 | Using `saf revisions`, every file manager can have full time-machine like restore, assuming that somebody writes appropriate plugin. Contributors are welcome. 183 | 184 | ### Vifm 185 | 186 | [Vifm](https://vifm.info/) plugin is available at [plugin/vifm](plugin/vifm). 187 | 188 | Install Vifm plugin by copying or symlinking saf-revisions (plugin folder) to $VIFM/plugins/ (usually ~/.config/vifm/plugins/ or ~/.vifm/plugins/). This will make :Saf* commands available in Vifm: 189 | 190 | ``` 191 | :SafRevisionsToggle -- Activate or deactivate saf-revisions 192 | :SafRevisionsActivate -- Turn saf-revisions on 193 | :SafRevisionsDeactivate -- Turn saf-revisions off 194 | :SafPreviousRevision -- Starfield! Warp! Magic! Back in time. 195 | :SafNextRevision -- Starfield! Warp! Magic! Forward in time. 196 | ``` 197 | 198 | Map appropriate keys to saf-revisions plugin functions, pick any unused keys that work for you: 199 | ``` bash 200 | nnoremap @ :SafRevisionsToggle 201 | nnoremap < :SafPreviousRevision 202 | nnoremap > :SafNextRevision 203 | ``` 204 | Start Vifm, then in any folder backed up with saf to local file system backup target (reaching remote targets is not possible at the moment) execute `:SafRevisionsToggle` command, or with definitions like above press `@`. Navigate trough all the backups where folder content is changed with `:SafPreviousRevision` and `:SafNextRevision` functions or keys mapped to those functions. Switch back to normal with another `:SafRevisionsToggle`. 205 | 206 | > **_NOTE:_** Please keep in mind that you will be browsing actual backup folder content. Changing such content is possible but usually a bad idea. Limit interaction to review and copy content to backup source folder, not the other way around. 207 | 208 | ## Contributing 209 | 210 | Contributors are welcome. Code, documentation or packaging improvements are welcome. 211 | 212 | ## Related projects 213 | 214 | Thanks to [laurent22/rsync-time-backup](https://github.com/laurent22/rsync-time-backup) and [cytopia/linux-timemachine](https://github.com/cytopia/linux-timemachine) for the inspiration. 215 | 216 | ## Changelog 217 | 218 | **[Changelog](CHANGELOG.md)** 219 | 220 | ## License 221 | 222 | **[MIT License](LICENSE.md)** 223 | 224 | Copyright (c) 2015-2023 Dusan Popovic 225 | -------------------------------------------------------------------------------- /plugin/vifm/saf-revisions/README.md: -------------------------------------------------------------------------------- 1 | # saf-revisions Vifm plugin 2 | 3 | Install vifm plugin by copying or symlinking saf-revisions (plugin folder) to $VIFM/plugins/ (usually ~/.config/vifm/plugins/ or ~/.vifm/plugins/). This will make :Saf* commands available in Vifm: 4 | 5 | ``` 6 | :SafRevisionsToggle -- Activate or deactivate saf-revisions 7 | :SafRevisionsActivate -- Turn saf-revisions on 8 | :SafRevisionsDeactivate -- Turn saf-revisions off 9 | :SafPreviousRevision -- Starfield! Warp! Magic! Back in time. 10 | :SafNextRevision -- Starfield! Warp! Magic! Forward in time. 11 | ``` 12 | 13 | Map appropriate keys to saf-revisions plugin functions, pick any unused keys that work for you: 14 | ``` bash 15 | nnoremap @ :SafRevisionsToggle 16 | nnoremap < :SafPreviousRevision 17 | nnoremap > :SafNextRevision 18 | ``` 19 | Start Vifm, then in any folder backed up with saf to local file system backup target (reaching remote targets is not possible at the moment) execute `:SafRevisionsToggle` command, or with definitions like above press `@`. Navigate trough all the backups where folder content is changed with `:SafPreviousRevision` and `:SafNextRevision` functions or keys mapped to those functions. Switch back to normal with another `:SafRevisionsToggle`. 20 | 21 | > **_NOTE:_** Please keep in mind that you will be browsing actual backup folder content. Changing such content is possible but usually a bad idea. Limit interaction to review and copy content to backup source folder, not the other way around. 22 | -------------------------------------------------------------------------------- /plugin/vifm/saf-revisions/init.lua: -------------------------------------------------------------------------------- 1 | --[[ 2 | 3 | saf-revisions Vifm plugin is providing commands that use 4 | 5 | https://github.com/dusanx/saf 6 | 7 | to provide time-machine like restore functionality. 8 | 9 | Provides 10 | :SafRevisionsToggle -- Activate or deactivate saf-revisions 11 | :SafRevisionsActivate -- Turn saf-revisions on 12 | :SafRevisionsDeactivate -- Turn saf-revisions off 13 | :SafPreviousRevision -- Starfield! Warp! Magic! Back in time. 14 | :SafNextRevision -- Starfield! Warp! Magic! Forward in time. 15 | 16 | --]] 17 | 18 | local active = false 19 | local curr_path = nil 20 | local other_path = nil 21 | local revisions = {} 22 | local revisions_pos = nil 23 | 24 | local function show_revision() 25 | if revisions_pos == nil then return end 26 | local path = revisions[revisions_pos] 27 | if not vifm.currview():cd(path) then 28 | vifm.sb.error("Failed to navigate to: "..path..", possibly removed by prune.") 29 | return 30 | end 31 | vifm.sb.info("Saf revision: "..revisions_pos.." of "..#revisions.." in "..curr_path) 32 | end 33 | 34 | local function turn_on() 35 | if active then return end 36 | -- fetch revisions "saf revisions -q /path" 37 | revisions = {} 38 | vifm.sb.info("On big folders, saf revisions can take time, please wait...") 39 | local path = vifm.currview().cwd 40 | local job = vifm.startjob { cmd = 'saf revisions -q "'..path..'"'} 41 | for line in job:stdout():lines() do 42 | vifm.sb.info("On big folders, saf revisions can take time, please wait... "..line) 43 | table.insert(revisions, line) 44 | end 45 | job:wait() 46 | -- did we fetch anything? 47 | if #revisions <= 0 then 48 | vifm.sb.error("Got Nothing! Is saf in path? Is this dir in backup? Maybe this dir didn't change at all?") 49 | revisions = {} 50 | return 51 | end 52 | -- remote dir check, we can't handle remote 53 | revisions_pos = 1 54 | check = revisions[revisions_pos] 55 | if string.find(check, ":") ~= nil then 56 | vifm.sb.error("Looks like remote backup, we can't handle that.") 57 | revisions = {} 58 | return 59 | end 60 | -- we have the results 61 | active = true 62 | curr_path = vifm.currview().cwd 63 | other_path = vifm.otherview().cwd 64 | if not vifm.otherview():cd(path) then 65 | vifm.errordialog("saf", "Failed to navigate to: "..path) 66 | end 67 | show_revision() 68 | end 69 | 70 | local function turn_off() 71 | if not active then return end 72 | if not vifm.currview():cd(curr_path) then 73 | vifm.errordialog("saf", "Failed to navigate to: "..curr_path) 74 | end 75 | if not vifm.otherview():cd(other_path) then 76 | vifm.errordialog("saf", "Failed to navigate to: "..other_path) 77 | end 78 | active = false 79 | curr_path = nil 80 | other_path = nil 81 | revisions = {} 82 | revisions_pos = nil 83 | end 84 | 85 | local function SafRevisionsToggle(info) 86 | if not active then 87 | turn_on() 88 | else 89 | turn_off() 90 | end 91 | end 92 | 93 | local function SafRevisionsActivate(info) 94 | turn_on() 95 | end 96 | 97 | local function SafRevisionsDeactivate(info) 98 | turn_off() 99 | end 100 | 101 | local function SafPreviousRevision(info) 102 | if revisions_pos == nil then return end 103 | if revisions_pos < #revisions then revisions_pos = revisions_pos + 1 end 104 | show_revision() 105 | end 106 | 107 | local function SafNextRevision(info) 108 | if revisions_pos == nil then return end 109 | if revisions_pos > 1 then revisions_pos = revisions_pos - 1 end 110 | show_revision() 111 | end 112 | 113 | -- Add all commands 114 | 115 | if not vifm.cmds.add { 116 | name = "SafRevisionsToggle", 117 | description = "turn on or off saf revisions", 118 | handler = SafRevisionsToggle 119 | } then 120 | vifm.sb.error("Failed to register :SafRevisionsToggle") 121 | end 122 | 123 | if not vifm.cmds.add { 124 | name = "SafRevisionsActivate", 125 | description = "turn on saf revisions", 126 | handler = SafRevisionsActivate 127 | } then 128 | vifm.sb.error("Failed to register :SafRevisionsActivate") 129 | end 130 | 131 | if not vifm.cmds.add { 132 | name = "SafRevisionsDeactivate", 133 | description = "turn off saf revisions", 134 | handler = SafRevisionsDeactivate 135 | } then 136 | vifm.sb.error("Failed to register :SafRevisionsDeactivate") 137 | end 138 | 139 | if not vifm.cmds.add { 140 | name = "SafPreviousRevision", 141 | description = "navigate to previous revision", 142 | handler = SafPreviousRevision 143 | } then 144 | vifm.sb.error("Failed to register :SafPreviousRevision") 145 | end 146 | 147 | if not vifm.cmds.add { 148 | name = "SafNextRevision", 149 | description = "navigate to next revision", 150 | handler = SafNextRevision 151 | } then 152 | vifm.sb.error("Failed to register :SafNextRevision") 153 | end 154 | 155 | return {} 156 | -------------------------------------------------------------------------------- /saf: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python3 2 | 3 | import os 4 | import re 5 | import sys 6 | import argparse 7 | import tempfile 8 | import subprocess 9 | import configparser 10 | from enum import Enum 11 | from pathlib import Path 12 | from datetime import datetime, timedelta 13 | 14 | def errquit(*args): 15 | for ln in args: print(ln, file=sys.stderr) 16 | sys.exit(1) 17 | 18 | # used to enumerate fancy print modes 19 | class Mode(Enum): 20 | idle = 0 21 | hourly = 1 22 | daily = 2 23 | weekly = 3 24 | monthly = 4 25 | yearly = 5 26 | 27 | # dot.notation access to dictionary attributes 28 | class dotdict(dict): 29 | __getattr__ = dict.get 30 | __setattr__ = dict.__setitem__ 31 | __delattr__ = dict.__delitem__ 32 | 33 | # main class 34 | class saf: 35 | saf_version = '0.16' 36 | saf_verbose_level = 1 37 | 38 | ################### 39 | # Utility functions 40 | ################### 41 | 42 | def safely_execute(self, cmd, return_output = False, output_on_screen = False, stop_on_error = True): 43 | # uses bash to straighten up shell diferences 44 | final_cmd = ['bash', '-c'] 45 | final_cmd.append(' '.join(cmd)) 46 | if self.saf_verbose_level >= 2: 47 | print(f'Executing:\n{final_cmd}') 48 | # determine if we should catch output or not 49 | capture = False 50 | if return_output: capture = True 51 | if self.saf_verbose_level <= 1 and not output_on_screen: capture = True 52 | # determine if output should reach screen 53 | on_screen = False 54 | if output_on_screen and capture: 55 | on_screen = True 56 | # run 57 | try: 58 | if capture: 59 | res = subprocess.run(final_cmd, stdout = subprocess.PIPE, stderr = subprocess.STDOUT, text = True) 60 | else: 61 | res = subprocess.run(final_cmd) 62 | except Exception as e: 63 | errquit(f'Error executing {cmd}, exception {e}') 64 | if stop_on_error and res.returncode != 0: 65 | if capture: 66 | errquit(f'Error executing {cmd}, return code {res.returncode}', res.stdout) 67 | else: 68 | errquit(f'Error executing {cmd}, return code {res.returncode}') 69 | if on_screen: 70 | print(res.stdout) 71 | return res 72 | 73 | def split_server_location(self, location): 74 | # split and return [location, server], server is optional 75 | loc = location.split(':') 76 | if len(loc) == 1: 77 | ret = [loc[0], None] 78 | else: 79 | ret = [loc[1], loc[0]] 80 | if ret[0].endswith('/'): ret[0] = ret[0][:-1] 81 | return ret 82 | 83 | def get_backup_folders(self, target): 84 | # gather all backup folders 85 | if target.needs_ssh: 86 | cmd_prefix = f'ssh -q {target.ssh_options} {target.ssh_server} ' 87 | else: 88 | cmd_prefix = '' 89 | cmd = cmd_prefix + f'"ls" -1 "{target.path}"' 90 | res = self.safely_execute([cmd], return_output=True) 91 | # filter what looks like backup 92 | pattern = re.compile('^[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]-[0-9][0-9][0-9][0-9][0-9][0-9]$') #2023-08-04-233001 93 | backup_list = [x for x in res.stdout.split('\n') if pattern.match(x)] 94 | backup_list.sort() 95 | return backup_list 96 | 97 | def marker_check(self, target): 98 | # safely contruct command 99 | if target.needs_ssh: 100 | cmd_prefix = f'ssh -q {target.ssh_options} {target.ssh_server} ' 101 | else: 102 | cmd_prefix = '' 103 | cmd = cmd_prefix + f'"ls" -1 "{target.path}/backup.marker"' 104 | res = self.safely_execute([cmd], return_output=True, stop_on_error=False) 105 | line = res.stdout.split('\n')[0] 106 | if res.returncode != 0 or not line == f'{target.path}/backup.marker': 107 | errquit(f'Safety check failed - the destination does not appear to be a backup directory or drive (marker file not found)', 108 | f'If it is indeed a backup directory, you may add the marker file by running the following commands:', 109 | f'', 110 | f' {cmd_prefix}mkdir -p -- "{target.path}"', 111 | f' {cmd_prefix}touch "{target.path}/backup.marker"' 112 | ) 113 | 114 | def conf_or_err(self): 115 | # find first conf or err 116 | dir = Path.cwd().resolve() 117 | found = False 118 | while True: 119 | conf = dir / '.saf.conf' 120 | if conf.is_file(): 121 | found = True 122 | break 123 | if dir == Path('/'): 124 | break 125 | dir = dir.parent.resolve() 126 | if not found: 127 | errquit('Not a saf backup source (or any of the parent directories): .saf.conf not found') 128 | return conf 129 | 130 | def prune_sanity_check(self, target): 131 | t = target.short_name 132 | # check minimal values that make sense 133 | if target.keep_hourly < 1: 134 | errquit(f'.saf.conf [{t}] error: keep-hourly must be at least 1 day') 135 | if target.keep_daily < 1: 136 | errquit(f'.saf.conf [{t}] error: keep-daily must be at least 1 day') 137 | if target.keep_weekly < 7: 138 | errquit(f'.saf.conf [{t}] error: keep-weekly must be at least 7 days') 139 | if target.keep_monthly < 30: 140 | errquit(f'.saf.conf [{t}] error: keep-monthly must be at least 30 days') 141 | if target.keep_yearly < 365: 142 | errquit(f'.saf.conf [{t}] error: keep-yearly must be at least 365 days') 143 | # each next criteria must be at least one day longer than previous 144 | if target.keep_daily <= target.keep_hourly: 145 | errquit(f'.saf.conf [{t}] error: keep-daily must be at least one day longer than keep-hourly') 146 | if target.keep_weekly <= target.keep_daily: 147 | errquit(f'.saf.conf [{t}] error: keep-weekly must be at least one day longer than keep-daily') 148 | if target.keep_monthly <= target.keep_weekly: 149 | errquit(f'.saf.conf [{t}] error: keep-monthly must be at least one day longer than keep-weekly') 150 | if target.keep_yearly <= target.keep_monthly: 151 | errquit(f'.saf.conf [{t}] error: keep-yearly must be at least one day longer than keep-monthly') 152 | 153 | def backup_target_or_err(self, conf, want_target): 154 | c = configparser.ConfigParser(allow_no_value=True) 155 | c.read(str(conf)) 156 | # take first target or want_target 157 | target_name = None 158 | for s in c.sections(): 159 | if want_target is None and s.endswith('.target'): 160 | target_name = s 161 | break 162 | if s == f'{want_target}.target': 163 | target_name = f'{want_target}.target' 164 | break 165 | if target_name is None: 166 | if want_target is None: 167 | errquit(f'{conf} does not have any backup target defined') 168 | else: 169 | errquit(f'{conf} does not have backup target named {want_target}') 170 | # sort out and return target 171 | target = dotdict({}) 172 | target.name = target_name 173 | target.short_name = target_name.replace('.target', '') 174 | target.source_location = c['source']['location'] 175 | target.exclude = [item[0] for item in c.items('exclude')] 176 | target.full_path = c[target_name].get('location', None) 177 | if target.full_path is None: 178 | errquit(f'{conf} does not have location specified in {target_name}') 179 | if target.full_path.endswith('/'): target.full_path = target.full_path[:-1] 180 | split = self.split_server_location(target.full_path) 181 | target.ssh_server = split[1] 182 | target.path = split[0] 183 | target.needs_ssh = split[1] is not None 184 | target.keep_hourly = c[target_name].getint('keep-hourly', 2) 185 | target.keep_daily = c[target_name].getint('keep-daily', 30) 186 | target.keep_weekly = c[target_name].getint('keep-weekly', 60) 187 | target.keep_monthly = c[target_name].getint('keep-monthly', 730) 188 | target.keep_yearly = c[target_name].getint('keep-yearly', 3650) 189 | target.rsync_options = c[target_name].get('rsync-options', '') 190 | target.ssh_options = c[target_name].get('ssh-options', '') 191 | self.prune_sanity_check(target) 192 | return target 193 | 194 | ############## 195 | # Command line 196 | ############## 197 | 198 | def process_command_line(self): 199 | def add_verbose_options(p): 200 | p_verbose = p.add_mutually_exclusive_group() 201 | p_verbose.add_argument('--quiet', '-q', action='store_true', help='be very quiet') 202 | p_verbose.add_argument('--verbose', '-v', type=int, choices=range(0, 3), help='set verbose level 0..2 (default=1)') 203 | p_verbose.add_argument('--debug', '-d', action='store_true', help='be more verbose') 204 | parser = argparse.ArgumentParser() 205 | subcommands = parser.add_subparsers(help='sub-command', required=True) 206 | # saf version 207 | p_version = subcommands.add_parser('version', help='show saf version') 208 | p_version.set_defaults(cmd='version') 209 | # saf init 210 | p_init = subcommands.add_parser('init', help='initialize new backup source') 211 | p_init.set_defaults(cmd='init') 212 | p_init.add_argument('--name', '-n', metavar='target-name', help='set or add backup target name to the existing configuration') 213 | p_init.add_argument('target', metavar='target-path', help='local (/path) or remote (server:/path) location') 214 | # saf status 215 | p_status = subcommands.add_parser('status', help='query backup status for the location') 216 | p_status.set_defaults(cmd='status') 217 | p_status_reach = p_status.add_mutually_exclusive_group() 218 | p_status_reach.add_argument('--all', '-a', action='store_true', help='find all the saf backup locations for current directory') 219 | p_status_reach.add_argument('--reverse', '-r', action='store_true', help='find saf backup locations in any sub-directory') 220 | # saf configedit 221 | p_configedit = subcommands.add_parser('configedit', help='edit closest .saf.conf, responsible for this location') 222 | p_configedit.set_defaults(cmd='configedit') 223 | add_verbose_options(p_configedit) 224 | # saf list 225 | p_list = subcommands.add_parser('list', help='list backups at target location') 226 | p_list.set_defaults(cmd='list') 227 | add_verbose_options(p_list) 228 | p_list.add_argument('name', nargs='?', metavar='target-name', default=None, help='list specific backup target destination') 229 | # saf realsizes 230 | p_realsizes = subcommands.add_parser('realsizes', help='show content size on backup destination') 231 | p_realsizes.set_defaults(cmd='realsizes') 232 | add_verbose_options(p_realsizes) 233 | p_realsizes.add_argument('name', nargs='?', metavar='target-name', default=None, help='list specific backup target destination') 234 | # saf freespace 235 | p_freespace = subcommands.add_parser('freespace', help='show free space on backup target destination') 236 | p_freespace.set_defaults(cmd='freespace') 237 | add_verbose_options(p_freespace) 238 | p_freespace.add_argument('name', nargs='?', metavar='target-name', default=None, help='list specific backup target destination') 239 | # saf prune 240 | p_prune = subcommands.add_parser('prune', help='prune backups on target destination') 241 | p_prune.set_defaults(cmd='prune') 242 | add_verbose_options(p_prune) 243 | p_prune.add_argument('name', nargs='?', metavar='target-name', default=None, help='prune specific backup target destination') 244 | # saf backup 245 | p_backup = subcommands.add_parser('backup', help='run backup') 246 | p_backup.set_defaults(cmd='backup') 247 | add_verbose_options(p_backup) 248 | p_backup.add_argument('--resume', '-r', action='store_true', help='resume backup if partial backup exists, or start new if not (without this switch start new backup always)') 249 | p_backup.add_argument('name', nargs='?', metavar='target-name', default=None, help='backup specific backup target destination') 250 | p_backup.add_argument('cd', nargs='?', metavar='local-path', default=None, help='cd to local-path before backup') 251 | # saf rmrf 252 | p_rmrf = subcommands.add_parser('rmrf', help='run rmrf to remove same file/folder in all backups') 253 | p_rmrf.set_defaults(cmd='rmrf') 254 | add_verbose_options(p_rmrf) 255 | p_rmrf.add_argument('name', nargs='?', metavar='target-name', default=None, help='rmrf on specific backup target destination') 256 | p_rmrf.add_argument('path', help='what to rmrf') 257 | # saf difference 258 | p_difference = subcommands.add_parser('difference', help='compare any backup with previous') 259 | p_difference.set_defaults(cmd='difference') 260 | add_verbose_options(p_difference) 261 | p_difference.add_argument('name', nargs='?', metavar='target-name', default=None, help='difference on specific backup target destination') 262 | p_difference.add_argument('backup', help='what to difference (must be backup folder YYYY-MM-DD-HHMMSS)') 263 | # saf revisions 264 | p_revisions = subcommands.add_parser('revisions', help='list of all folder occurances in all backups') 265 | p_revisions.set_defaults(cmd='revisions') 266 | add_verbose_options(p_revisions) 267 | p_revisions.add_argument('name', nargs='?', metavar='target-name', default=None, help='revisions on specific backup target destination') 268 | p_revisions.add_argument('path', help='target path') 269 | # parse and process 270 | args = parser.parse_args() 271 | # verbose 272 | if 'verbose' in args: 273 | if args.verbose is not None: 274 | self.saf_verbose_level = args.verbose 275 | elif args.quiet: 276 | self.saf_verbose_level = 0 277 | elif args.debug: 278 | self.saf_verbose_level = 2 279 | # run command 280 | match args.cmd: 281 | case 'version': 282 | self.run_version(args) 283 | case 'init': 284 | self.run_init(args) 285 | case 'status': 286 | self.run_status(args) 287 | case 'configedit': 288 | self.run_configedit(args) 289 | case 'list': 290 | self.run_list(args) 291 | case 'realsizes': 292 | self.run_realsizes(args) 293 | case 'freespace': 294 | self.run_freespace(args) 295 | case 'prune': 296 | self.run_prune(args) 297 | case 'backup': 298 | self.run_backup(args) 299 | case 'rmrf': 300 | self.run_rmrf(args) 301 | case 'difference': 302 | self.run_difference(args) 303 | case 'revisions': 304 | self.run_revisions(args) 305 | case _: 306 | print('command not found?') 307 | 308 | ########## 309 | # Commands 310 | ########## 311 | 312 | def run_version(self, args): 313 | print(f'saf {self.saf_version}') 314 | 315 | def run_init(self, args): 316 | # prepare 317 | loc = self.split_server_location(args.target) 318 | local_template = loc[1] is None 319 | target_name = args.name if args.name is not None else 'B0' 320 | dir = Path.cwd().resolve() 321 | conf_file = dir / '.saf.conf' 322 | config_list = [] 323 | add_section = f'{target_name}.target' 324 | # read or create conf file 325 | if conf_file.is_file(): 326 | # can we add target_name? 327 | c = configparser.ConfigParser(allow_no_value=True) 328 | c.read(str(conf_file)) 329 | if c.has_section(add_section): 330 | errquit(f'Backup target "{target_name}" already exists, please use', 331 | f' saf configedit', 332 | f'or', 333 | f' saf init --name', 334 | f'to change or specify different target name.' 335 | ) 336 | with open(conf_file, 'r') as f: 337 | config_list = f.read().splitlines() 338 | else: 339 | config_list.append('# Moving .saf.conf to another location or changing') 340 | config_list.append('# [source] location= is usually a bad idea.') 341 | config_list.append('') 342 | config_list.append('[source]') 343 | config_list.append('') 344 | config_list.append(f"location = {dir}") 345 | config_list.append('') 346 | config_list.append('# [exclude] section follows rsync syntax:') 347 | config_list.append('# Exclude, "-", specifies an exclude pattern. Include, "+", specifies an include pattern.') 348 | config_list.append('# "- *.o" would exclude all names matching *.o') 349 | config_list.append('# "- /foo" would exclude a file (or directory) named foo in the transfer-root directory') 350 | config_list.append('# "- foo/" would exclude any directory named foo') 351 | config_list.append('# "- /foo/*/bar" would exclude any file named bar which is at two levels below a directory named foo in the transfer-root directory') 352 | config_list.append('# "- /foo/**/bar" would exclude any file named bar two or more levels below a directory named foo in the transfer-rootdirectory') 353 | config_list.append('') 354 | config_list.append('[exclude]') 355 | config_list.append('') 356 | config_list.append('- *.iso') 357 | config_list.append('- *.tmp') 358 | config_list.append('') 359 | config_list.append('# If there is more than one, first target location in the list') 360 | config_list.append('# will be default target location. Order of target locations can') 361 | config_list.append('# be changed without consequences, to determine which target to') 362 | config_list.append('# use as a default.') 363 | # add new backup target 364 | config_list.append('') 365 | config_list.append(f'[{add_section}]') 366 | config_list.append('') 367 | config_list.append(f'location = {args.target}') 368 | config_list.append('keep-hourly = 2') # keep all hourly backups for two days 369 | config_list.append('keep-daily = 30') # keep all daily backups for about a month 370 | config_list.append('keep-weekly = 60') # keep all weekly backups for about two months 371 | config_list.append('keep-monthly = 730') # keep all monthly backup for about two years 372 | config_list.append('keep-yearly = 3650') # keep all yearly backup for about ten years 373 | config_list.append('rsync-options = -D --compress --numeric-ids --links --hard-links --one-file-system --itemize-changes --times --recursive --perms --owner --group') 374 | if local_template: 375 | config_list.append('ssh-options =') 376 | else: 377 | config_list.append('ssh-options = -p 22') 378 | # save 379 | with open(conf_file, 'w') as f: 380 | f.writelines([s + '\n' for s in config_list]) 381 | print(f'Initialized .saf.conf in {dir}, with backup target location {target_name}.') 382 | print(f'You can edit manually or use "saf configedit" before first use.') 383 | 384 | def run_status(self, args): 385 | def conf_details(conf): 386 | print('') 387 | c = configparser.ConfigParser(allow_no_value=True) 388 | c.read(str(conf)) 389 | print(f'Source: {str(conf)} ==>') 390 | for s in c.sections(): 391 | if not s.endswith('.target'): 392 | continue 393 | target_name = s.replace('.target', '') 394 | target_location = c[s]['location'] 395 | print(f' ==> {target_name} ==> {target_location}') 396 | dir = Path.cwd().resolve() 397 | if args.reverse: 398 | print(f'Searching for all backup source locations starting from current directory ({dir}). Please wait.') 399 | for conf in dir.rglob('.saf.conf'): 400 | conf_details(conf) 401 | else: 402 | if args.all: 403 | print(f'Searching for all backup source locations that cover current directory ({dir}).') 404 | else: 405 | print(f'Searching for first backup source location that covers current directory ({dir}).') 406 | found = False 407 | while True: 408 | conf = dir / '.saf.conf' 409 | if conf.is_file(): 410 | conf_details(conf) 411 | found = True 412 | if not args.all: 413 | break 414 | if dir == Path('/'): 415 | break 416 | dir = dir.parent.resolve() 417 | if not found: 418 | errquit('', 'Not a saf backup source (or any of the parent directories): .saf.conf not found') 419 | 420 | def run_configedit(self, args): 421 | # must have $EDITOR 422 | ed = os.getenv('EDITOR') 423 | if ed is None: 424 | errquit('Shell environment variable $EDITOR is not defined.') 425 | # must be part of at least one backup source 426 | conf = self.conf_or_err() 427 | # edit 428 | self.safely_execute([ed, str(conf)]) 429 | 430 | def run_list(self, args): 431 | # must be part of at least one backup source 432 | conf = self.conf_or_err() 433 | # load specific conf backup target or err 434 | target = self.backup_target_or_err(conf, args.name) 435 | # make sure that destination exists 436 | self.marker_check(target) 437 | # some output 438 | if self.saf_verbose_level > 0: 439 | print(f'List of backups in {conf}, [{target.short_name}] ==> {target.full_path}') 440 | print('') 441 | # gather all backup folders 442 | backup_list = self.get_backup_folders(target) 443 | # print list with time difference 444 | now = datetime.now() 445 | maxl = len(str(len(backup_list))) 446 | mode = Mode.idle 447 | cut_hourly = now - timedelta(days = target.keep_hourly) 448 | cut_daily = now - timedelta(days = target.keep_daily) 449 | cut_weekly = now - timedelta(days = target.keep_weekly) 450 | cut_monthly = now - timedelta(days = target.keep_monthly) 451 | cut_yearly = now - timedelta(days = target.keep_yearly) 452 | for idx, backup_folder in enumerate(backup_list): 453 | backup_date = datetime.strptime(backup_folder, '%Y-%m-%d-%H%M%S') 454 | backup_diff = now - backup_date 455 | new_mode = mode 456 | if backup_date >= cut_monthly: new_mode = Mode.monthly 457 | if backup_date >= cut_weekly: new_mode = Mode.weekly 458 | if backup_date >= cut_daily: new_mode = Mode.daily 459 | if backup_date >= cut_hourly: new_mode = Mode.hourly 460 | if mode != new_mode: 461 | print('---- keep ' + str(new_mode).replace('Mode.', '') + ' ------------------------------------') 462 | mode = new_mode 463 | print('%*d/%*d' % (maxl, idx+1, maxl, len(backup_list)), end=' ') 464 | print (backup_folder, end=" ") 465 | # print time difference depending on mode 466 | match mode: 467 | case Mode.hourly: # 1+ hours ago 468 | h = int((backup_diff.days * 24) + (backup_diff.seconds / 3600)) 469 | interval = 'hours' if h != 1 else ' hour' 470 | print('%3d+ %s ago' % (h, interval)) 471 | case Mode.daily: # 5 days ago 472 | d = int(backup_diff.days) 473 | interval = 'days' if d != 1 else 'day' 474 | print('%3d+ %s ago' % (d, interval)) 475 | case Mode.weekly: # 5+ weeks ago 476 | w = int(backup_diff.days / 7) 477 | interval = 'weeks' if w != 1 else 'week' 478 | print('%3d+ %s ago' % (w, interval)) 479 | case Mode.monthly: # 22+ weeks ago, 2023 02 480 | w = int(backup_diff.days / 7) 481 | interval = 'weeks' if w != 1 else 'week' 482 | print('%3d+ %s ago, %s %d' % (w, interval, backup_date.strftime('%B'), backup_date.year)) 483 | case _: # multi year difference, print default 484 | print(now - backup_date) 485 | 486 | def run_realsizes(self, args): 487 | # must be part of at least one backup source 488 | conf = self.conf_or_err() 489 | # load specific conf backup target or err 490 | target = self.backup_target_or_err(conf, args.name) 491 | # make sure that destination exists 492 | self.marker_check(target) 493 | # some output 494 | if self.saf_verbose_level > 0: 495 | print(f'List of backup sizes in {conf}, [{target.short_name}] ==> {target.full_path}') 496 | print('') 497 | # just run magic command 498 | if target.needs_ssh: 499 | cmd_prefix = f'ssh -q {target.ssh_options} {target.ssh_server} ' 500 | else: 501 | cmd_prefix = '' 502 | cmd = cmd_prefix + f'du -h -d1 "{target.path}"' 503 | self.safely_execute([cmd], output_on_screen=True) 504 | 505 | def run_freespace(self, args): 506 | # must be part of at least one backup source 507 | conf = self.conf_or_err() 508 | # load specific conf backup target or err 509 | target = self.backup_target_or_err(conf, args.name) 510 | # make sure that destination exists 511 | self.marker_check(target) 512 | # some output 513 | if self.saf_verbose_level > 0: 514 | print(f'Free space in {conf}, [{target.short_name}] ==> {target.full_path}') 515 | print('') 516 | # just run magic command 517 | if target.needs_ssh: 518 | cmd_prefix = f'ssh -q {target.ssh_options} {target.ssh_server} ' 519 | else: 520 | cmd_prefix = '' 521 | cmd = cmd_prefix + f'df -h "{target.path}"' 522 | self.safely_execute([cmd], output_on_screen=True) 523 | 524 | def run_prune(self, args): 525 | # must be part of at least one backup source 526 | conf = self.conf_or_err() 527 | # load specific conf backup target or err 528 | target = self.backup_target_or_err(conf, args.name) 529 | # make sure that destination exists 530 | self.marker_check(target) 531 | # some output 532 | if self.saf_verbose_level > 0: 533 | print(f'Prune in {conf}, [{target.short_name}] ==> {target.full_path}') 534 | print('') 535 | # cutoff list [after, itemno] 536 | now = datetime.now() 537 | prune_criteria = [] 538 | prune_criteria.append(['hourly', now - timedelta(days = target.keep_hourly), 2]) 539 | prune_criteria.append(['daily', now - timedelta(days = target.keep_daily), 3]) 540 | prune_criteria.append(['weekly', now - timedelta(days = target.keep_weekly), 4]) 541 | prune_criteria.append(['monthly', now - timedelta(days = target.keep_monthly), 5]) 542 | prune_criteria.append(['yearly', now - timedelta(days = target.keep_yearly), -1]) 543 | # gather all backup folders, prepare expanded list 544 | backup_list = self.get_backup_folders(target) 545 | backup_list_full = [] 546 | for b in backup_list: 547 | d = datetime.strptime(b, '%Y-%m-%d-%H%M%S') 548 | b_record = dotdict({}) 549 | b_record.backup = b 550 | b_record.date = d 551 | b_record.b_day = int(int(d.strftime("%s")) / 86400) # days since epoch 552 | b_record.b_week = d.isocalendar().week 553 | b_record.b_month = d.month 554 | b_record.b_year = d.year 555 | backup_list_full.append([b, 556 | d, 557 | int(int(d.strftime("%s")) / 86400), # days since epoch 558 | d.isocalendar().week, 559 | d.month, 560 | d.year]) 561 | # create prune list 562 | for idx in range(0, len(backup_list_full) - 1): 563 | should_prune = None 564 | for check in prune_criteria: 565 | if backup_list_full[idx][1] < check[1]: 566 | if check[2] == -1 or backup_list_full[idx][check[2]] == backup_list_full[idx+1][check[2]]: 567 | should_prune = f'Because of {check[0]}, {check[2]}' 568 | break 569 | if should_prune is not None: 570 | if self.saf_verbose_level > 0: 571 | print(f'Prune {backup_list_full[idx][0]}') 572 | # actually delete backup 573 | if target.needs_ssh: 574 | cmd_prefix = f'ssh -q {target.ssh_options} {target.ssh_server} ' 575 | else: 576 | cmd_prefix = '' 577 | cmd = cmd_prefix + f'rm -rf "{target.path}/{backup_list_full[idx][0]}"' 578 | self.safely_execute([cmd]) 579 | 580 | def run_backup(self, args): 581 | # cd if cd is specified 582 | if args.cd is not None: 583 | cd = Path(args.cd).resolve() 584 | try: 585 | os.chdir(cd) 586 | except Exception as e: 587 | errquit(f'Error executing "os.chdir({cd})", exception {e}') 588 | # backup always prunes first 589 | self.run_prune(args) 590 | # must be part of at least one backup source 591 | conf = self.conf_or_err() 592 | # load specific conf backup target or err 593 | target = self.backup_target_or_err(conf, args.name) 594 | # make sure that destination exists 595 | self.marker_check(target) 596 | # some output 597 | if self.saf_verbose_level > 0: 598 | print(f'Backup in {conf}, [{target.short_name}] ==> {target.full_path}') 599 | print('') 600 | # resume? 601 | if not args.resume: 602 | if target.needs_ssh: 603 | cmd_prefix = f'ssh -q {target.ssh_options} {target.ssh_server} ' 604 | else: 605 | cmd_prefix = '' 606 | cmd = cmd_prefix + f'rm -rf "{target.path}/in-progress"' 607 | self.safely_execute([cmd]) 608 | # gather all needed to do actual backup 609 | now = datetime.now() 610 | finalfn = "%04d-%02d-%02d-%02d%02d%02d" % (now.year, now.month, now.day, now.hour, now.minute, now.second) 611 | backup_list = self.get_backup_folders(target) 612 | last_to_hardlink_with = None 613 | if len(backup_list) > 0: 614 | last_to_hardlink_with = Path(target.path + '/' + backup_list[-1]) 615 | # backup with temporary exclude file to in-progress 616 | tmp = tempfile.NamedTemporaryFile(mode = 'w', delete=False) 617 | try: 618 | tmp.writelines([str(s) + '\n' for s in target.exclude]) 619 | tmp.close() 620 | cmd = f'rsync {target.rsync_options}' 621 | cmd += f' --exclude-from="{tmp.name}"' 622 | if last_to_hardlink_with is not None: 623 | cmd += f' --link-dest="{last_to_hardlink_with}"' 624 | if target.needs_ssh: 625 | cmd += f' -e "ssh {target.ssh_options}"' 626 | cmd += f' -- "{target.source_location}/" "{target.full_path}/in-progress"' 627 | if self.saf_verbose_level == 0: 628 | self.safely_execute([cmd]) 629 | else: 630 | self.safely_execute([cmd], output_on_screen = True) 631 | finally: 632 | os.unlink(tmp.name) 633 | # finalize by renaming in-progress to actual backup name 634 | if target.needs_ssh: 635 | cmd_prefix = f'ssh -q {target.ssh_options} {target.ssh_server} ' 636 | else: 637 | cmd_prefix = '' 638 | cmd = cmd_prefix + f'mv "{target.path}/in-progress" "{target.path}/{finalfn}"' 639 | self.safely_execute([cmd]) 640 | 641 | def run_rmrf(self, args): 642 | # must be part of at least one backup source 643 | conf = self.conf_or_err() 644 | # load specific conf backup target or err 645 | target = self.backup_target_or_err(conf, args.name) 646 | # make sure that destination exists 647 | self.marker_check(target) 648 | # gather all needed to do actual rmrf 649 | rmrf_path = Path(args.path).resolve().relative_to(Path(target.source_location)) 650 | backup_list = self.get_backup_folders(target) 651 | # some output 652 | if self.saf_verbose_level > 0: 653 | print(f'Rmrf in {conf}, [{target.short_name}] ==> {target.full_path}') 654 | print(f'Removing {rmrf_path}') 655 | print('') 656 | # rmrf 657 | for b in backup_list: 658 | print(b) 659 | if target.needs_ssh: 660 | cmd_prefix = f'ssh -q {target.ssh_options} {target.ssh_server} ' 661 | else: 662 | cmd_prefix = '' 663 | cmd = cmd_prefix + f'rm -rf "{target.path}/{b}/{rmrf_path}"' 664 | self.safely_execute([cmd]) 665 | 666 | def run_difference(self, args): 667 | # must be part of at least one backup source 668 | conf = self.conf_or_err() 669 | # load specific conf backup target or err 670 | target = self.backup_target_or_err(conf, args.name) 671 | # make sure that destination exists 672 | self.marker_check(target) 673 | # gather all needed to do actual difference 674 | backup_list = self.get_backup_folders(target) 675 | if not args.backup in backup_list: 676 | errquit(f'Backup {args.backup} does not exist') 677 | if backup_list.index(args.backup) == 0: 678 | errquit(f"Backup {args.backup} is the first one so we can't compare with previous") 679 | compare_with = backup_list[backup_list.index(args.backup) - 1] 680 | # some output 681 | if self.saf_verbose_level > 0: 682 | print(f'Difference in {conf}, [{target.short_name}] ==> {target.full_path}') 683 | print(f'Comparing {args.backup} with {compare_with} /* experimental */') 684 | print("") 685 | # compare 686 | if target.needs_ssh: 687 | cmd_prefix = f'ssh -q {target.ssh_options} {target.ssh_server} ' 688 | else: 689 | cmd_prefix = '' 690 | cmd = f'{cmd_prefix}rsync --dry-run -arvc ' 691 | cmd += f' -- "{target.path}/{args.backup}/" "{target.path}/{compare_with}"' 692 | self.safely_execute([cmd], output_on_screen = True) 693 | 694 | def run_revisions(self, args): 695 | # must be part of at least one backup source 696 | conf = self.conf_or_err() 697 | # load specific conf backup target or err 698 | target = self.backup_target_or_err(conf, args.name) 699 | # make sure that destination exists 700 | self.marker_check(target) 701 | # must be a directory (we can't elegantly run revisions on files) 702 | if not Path(args.path).is_dir(): 703 | errquit(f"Revisions can only run on directory path, not on any specific file.") 704 | # can only be called on local file system backup target 705 | if target.needs_ssh and self.saf_verbose_level > 0: 706 | print(f'Warning: Revisions have very limited functionality with the remote target, only listing all possible changes.') 707 | print('') 708 | # gather all needed to do actual revisions 709 | source_path = Path(args.path).resolve() 710 | revisions_path = source_path.relative_to(Path(target.source_location)) 711 | backup_list = self.get_backup_folders(target) 712 | # some output 713 | if self.saf_verbose_level > 0: 714 | print(f'Revisions for {source_path} in {conf}, [{target.short_name}] ==> {target.full_path}') 715 | print('') 716 | # revisions 717 | candidate_list = [] 718 | for b in backup_list: 719 | candidate = Path(target.full_path) / b / revisions_path 720 | if target.needs_ssh or candidate.is_dir(): 721 | candidate_list.append(str(candidate)) 722 | candidate_list.reverse() 723 | final_list = [] 724 | last_added = source_path 725 | for idx, path in enumerate(candidate_list): 726 | # of very little use: on remote targets just add any candidate 727 | if target.needs_ssh: 728 | final_list.append(path) 729 | continue 730 | # on local targets we can really see differences, add only if differs from previous 731 | cmd = f'diff -q "{path}" "{last_added}" >/dev/null 2>&1 && echo "same" || echo "different"' 732 | res = self.safely_execute([cmd], return_output = True) 733 | if res.stdout.strip() == 'different': 734 | final_list.append(path) 735 | last_added = path 736 | continue 737 | # output 738 | for p in final_list: 739 | print(p) 740 | 741 | if __name__ == "__main__": 742 | backup = saf() 743 | backup.process_command_line() 744 | --------------------------------------------------------------------------------