├── .coveragerc ├── .travis.yml ├── CHANGELOG.rst ├── LICENSE.txt ├── MANIFEST.in ├── Makefile ├── README.rst ├── docs ├── api.rst ├── changelog.rst ├── conf.py ├── howto │ └── encrypted-usb-disk.rst ├── index.rst └── readme.rst ├── requirements-checks.txt ├── requirements-tests.txt ├── requirements-travis.txt ├── requirements.txt ├── rsync_system_backup ├── __init__.py ├── cli.py ├── destinations.py ├── exceptions.py └── tests.py ├── scripts └── install-on-travis.sh ├── setup.cfg ├── setup.py └── tox.ini /.coveragerc: -------------------------------------------------------------------------------- 1 | # .coveragerc: Configuration file for coverage.py. 2 | # http://nedbatchelder.com/code/coverage/ 3 | 4 | [run] 5 | source = rsync_system_backup 6 | omit = rsync_system_backup/tests.py 7 | 8 | # vim: ft=dosini 9 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | sudo: true 2 | language: python 3 | matrix: 4 | include: 5 | - python: 2.7 6 | - python: 3.4 7 | - python: 3.5 8 | - python: 3.6 9 | - python: 3.7 10 | dist: xenial 11 | - python: pypy 12 | install: 13 | - scripts/install-on-travis.sh 14 | script: 15 | - make check 16 | - make test 17 | after_success: 18 | - coveralls 19 | branches: 20 | except: 21 | - /^[0-9]/ 22 | -------------------------------------------------------------------------------- /CHANGELOG.rst: -------------------------------------------------------------------------------- 1 | Changelog 2 | ========= 3 | 4 | The purpose of this document is to list all of the notable changes to this 5 | project. The format was inspired by `Keep a Changelog`_. This project adheres 6 | to `semantic versioning`_. 7 | 8 | .. contents:: 9 | :local: 10 | 11 | .. _Keep a Changelog: http://keepachangelog.com/ 12 | .. _semantic versioning: http://semver.org/ 13 | 14 | `Release 1.1`_ (2019-08-02) 15 | --------------------------- 16 | 17 | - The ``--multi-fs`` option suggested in pull request `#3`_ was added. 18 | 19 | This option has the opposite effect of the rsync option ``--one-file-system`` 20 | because `rsync-system-backup` defaults to running rsync with 21 | ``--one-file-system`` and must be instructed not to using ``--multi-fs``." 22 | 23 | This change was merged by cherry picking the relevant commit from the remote 24 | branch, to avoid pulling in unrelated changes (see the pull request for 25 | details). 26 | 27 | - Stop testing on Python 2.6 (because Travis CI no longer supports it and 28 | working around this would cost an unreasonable amount of time) and start 29 | testing support for Python 3.7. 30 | 31 | .. _Release 1.1: https://github.com/xolox/python-rsync-system-backup/compare/1.0...1.1 32 | .. _#3: https://github.com/xolox/python-rsync-system-backup/pull/3 33 | 34 | `Release 1.0`_ (2018-05-04) 35 | --------------------------- 36 | 37 | **Bug fix: SSH tunnel support actually works now :-P (backwards incompatible).** 38 | 39 | This week I switched the backups of my VPS over to `rsync-system-backup`. The 40 | biggest hurdle was the fact that I never finished nor tested support for SSH 41 | tunnels in `rsync-system-backup` which I needed now: 42 | 43 | - The command line interface didn't expose the functionality. 44 | - Due to various bugs it wouldn't have worked even if the 45 | functionality had been exposed. 46 | 47 | The changes in this release: 48 | 49 | - Added the ``-t``, ``--tunnel`` command line option. 50 | - Integrated SSH tunnel support provided by `executor 19.3`_. 51 | - Removed the ``ForwardedDestination`` class (this functionality has been 52 | replaced by the new ``Destination.ssh_tunnel`` property). 53 | 54 | Because the removal of ``ForwardedDestination`` is backwards incompatible I've 55 | decided to bump the major version number. It's actually kind of fitting because 56 | I've been using `rsync-system-backup` for local backups for months now and 57 | that's working fine; the main thing missing was indeed SSH tunnel support :-). 58 | 59 | .. _Release 1.0: https://github.com/xolox/python-rsync-system-backup/compare/0.11...1.0 60 | .. _executor 19.3: http://executor.readthedocs.io/en/latest/changelog.html#release-19-3-2018-05-04 61 | 62 | `Release 0.11`_ (2018-04-30) 63 | ---------------------------- 64 | 65 | - Added support for the ``-x``, ``--exclude`` option. 66 | - Documented that ``--one-file-system`` is always used. 67 | 68 | .. _Release 0.11: https://github.com/xolox/python-rsync-system-backup/compare/0.10...0.11 69 | 70 | `Release 0.10`_ (2018-04-30) 71 | ---------------------------- 72 | 73 | - Switched to the more user friendly ``getopt.gnu_getopt()``. 74 | - Added this changelog, restructured the online documentation. 75 | - Documented the ``-f``, ``--force`` option in the readme. 76 | - Integrated the use of ``property_manager.sphinx``. 77 | 78 | .. _Release 0.10: https://github.com/xolox/python-rsync-system-backup/compare/0.9...0.10 79 | 80 | `Release 0.9`_ (2017-07-11) 81 | --------------------------- 82 | 83 | Explicitly handle unsupported platforms (by refusing to run without the 84 | ``-f``, ``--force`` option). Refer to issue `#1`_ for more information. 85 | 86 | .. _Release 0.9: https://github.com/xolox/python-rsync-system-backup/compare/0.8...0.9 87 | .. _#1: https://github.com/xolox/python-rsync-system-backup/issues/1 88 | 89 | `Release 0.8`_ (2017-06-24) 90 | --------------------------- 91 | 92 | Don't raise an exception when ``notify-send`` fails to deliver a desktop notification. 93 | 94 | .. _Release 0.8: https://github.com/xolox/python-rsync-system-backup/compare/0.7...0.8 95 | 96 | `Release 0.7`_ (2017-06-23) 97 | --------------------------- 98 | 99 | Ensure the destination directory is located under the expected mount point. 100 | 101 | .. _Release 0.7: https://github.com/xolox/python-rsync-system-backup/compare/0.6...0.7 102 | 103 | `Release 0.6`_ (2017-06-23) 104 | --------------------------- 105 | 106 | Incorporated the ``cryptdisks_start`` and ``cryptdisks_stop`` fallbacks into the how-to. 107 | 108 | .. _Release 0.6: https://github.com/xolox/python-rsync-system-backup/compare/0.5...0.6 109 | 110 | `Release 0.5`_ (2017-06-21) 111 | --------------------------- 112 | 113 | Gain independence from ``cryptdisks_start`` and ``cryptdisks_stop`` (a Debian-ism). 114 | 115 | .. _Release 0.5: https://github.com/xolox/python-rsync-system-backup/compare/0.4...0.5 116 | 117 | `Release 0.4`_ (2017-06-21) 118 | --------------------------- 119 | 120 | - Gracefully handle missing backup disk. 121 | - Added a how-to to the documentation. 122 | 123 | .. _Release 0.4: https://github.com/xolox/python-rsync-system-backup/compare/0.3...0.4 124 | 125 | `Release 0.3`_ (2017-06-06) 126 | --------------------------- 127 | 128 | Made it possible to disable desktop notifications. 129 | 130 | .. _Release 0.3: https://github.com/xolox/python-rsync-system-backup/compare/0.2...0.3 131 | 132 | `Release 0.2`_ (2017-05-06) 133 | --------------------------- 134 | 135 | - Don't render a traceback on known errors. 136 | - Fixed broken usage message formatting. 137 | - Document Python 3.6 compatibility. 138 | - Changed Sphinx theme. 139 | 140 | .. _Release 0.2: https://github.com/xolox/python-rsync-system-backup/compare/0.1.1...0.2 141 | 142 | `Release 0.1.1`_ (2017-04-17) 143 | ----------------------------- 144 | 145 | Changed system logging verbosity level from DEBUG to INFO. 146 | 147 | .. _Release 0.1.1: https://github.com/xolox/python-rsync-system-backup/compare/0.1...0.1.1 148 | 149 | `Release 0.1`_ (2017-04-14) 150 | --------------------------- 151 | 152 | Initial release (0.1, alpha). 153 | 154 | .. _Release 0.1: https://github.com/xolox/python-rsync-system-backup/tree/0.1 155 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Copyright (c) 2019 Peter Odding 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining 4 | a copy of this software and associated documentation files (the 5 | "Software"), to deal in the Software without restriction, including 6 | without limitation the rights to use, copy, modify, merge, publish, 7 | distribute, sublicense, and/or sell copies of the Software, and to 8 | permit persons to whom the Software is furnished to do so, subject to 9 | the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be 12 | included in all copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 15 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 16 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 17 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 18 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 19 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 20 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | graft docs 2 | include *.rst 3 | include *.txt 4 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Makefile for the `rsync-system-backup' package. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: August 2, 2019 5 | # URL: https://github.com/xolox/python-rsync-system-backup 6 | 7 | PACKAGE_NAME = rsync-system-backup 8 | WORKON_HOME ?= $(HOME)/.virtualenvs 9 | VIRTUAL_ENV ?= $(WORKON_HOME)/$(PACKAGE_NAME) 10 | PATH := $(VIRTUAL_ENV)/bin:$(PATH) 11 | MAKE := $(MAKE) --no-print-directory 12 | SHELL = bash 13 | 14 | default: 15 | @echo "Makefile for $(PACKAGE_NAME)" 16 | @echo 17 | @echo 'Usage:' 18 | @echo 19 | @echo ' make install install the package in a virtual environment' 20 | @echo ' make reset recreate the virtual environment' 21 | @echo ' make check check coding style (PEP-8, PEP-257)' 22 | @echo ' make test run the test suite, report coverage' 23 | @echo ' make tox run the tests on all Python versions' 24 | @echo ' make readme update usage in readme' 25 | @echo ' make docs update documentation using Sphinx' 26 | @echo ' make publish publish changes to GitHub/PyPI' 27 | @echo ' make clean cleanup all temporary files' 28 | @echo 29 | 30 | install: 31 | @test -d "$(VIRTUAL_ENV)" || mkdir -p "$(VIRTUAL_ENV)" 32 | @test -x "$(VIRTUAL_ENV)/bin/python" || virtualenv --python=python3 --quiet "$(VIRTUAL_ENV)" 33 | @test -x "$(VIRTUAL_ENV)/bin/pip" || easy_install pip 34 | @pip install --quiet --requirement=requirements.txt 35 | @pip uninstall --yes $(PACKAGE_NAME) &>/dev/null || true 36 | @pip install --quiet --no-deps --ignore-installed . 37 | 38 | reset: 39 | $(MAKE) clean 40 | rm -Rf "$(VIRTUAL_ENV)" 41 | $(MAKE) install 42 | 43 | check: install 44 | @pip install --upgrade --quiet --requirement=requirements-checks.txt 45 | @flake8 46 | 47 | test: install 48 | @pip install --quiet --requirement=requirements-tests.txt 49 | @py.test --cov 50 | @coverage html 51 | @coverage report --fail-under=90 &>/dev/null 52 | 53 | tox: install 54 | @pip install --quiet tox && tox 55 | 56 | readme: install 57 | @pip install --quiet cogapp && cog.py -r README.rst 58 | 59 | docs: readme 60 | @pip install --quiet sphinx 61 | @cd docs && sphinx-build -nb html -d build/doctrees . build/html 62 | 63 | publish: install 64 | git push origin && git push --tags origin 65 | $(MAKE) clean 66 | pip install --quiet twine wheel 67 | python setup.py sdist bdist_wheel 68 | twine upload dist/* 69 | $(MAKE) clean 70 | 71 | clean: 72 | @rm -Rf *.egg .cache .coverage .tox build dist docs/build htmlcov 73 | @find -depth -type d -name __pycache__ -exec rm -Rf {} \; 74 | @find -type f -name '*.pyc' -delete 75 | 76 | .PHONY: default install reset check test tox readme docs publish clean 77 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | rsync-system-backup: Linux system backups powered by rsync 2 | ========================================================== 3 | 4 | .. image:: https://travis-ci.org/xolox/python-rsync-system-backup.svg?branch=master 5 | :target: https://travis-ci.org/xolox/python-rsync-system-backup 6 | 7 | .. image:: https://coveralls.io/repos/xolox/python-rsync-system-backup/badge.svg?branch=master 8 | :target: https://coveralls.io/r/xolox/python-rsync-system-backup?branch=master 9 | 10 | The rsync-system-backup program uses rsync_ to create full system backups of 11 | Linux_ systems. Supported backup destinations include local disks (possibly 12 | encrypted using LUKS_) and remote systems that are running an SSH_ server or 13 | `rsync daemon`_. Each backup produces a timestamped snapshot and these 14 | snapshots are rotated according to a rotation scheme that you can configure. 15 | The package is currently tested on cPython 2.7, 3.4, 3.5, 3.6, 3.7 and PyPy 16 | (2.7). 17 | 18 | .. contents:: 19 | :depth: 3 20 | :local: 21 | 22 | Status 23 | ------ 24 | 25 | While this project brings together more than ten years of experience in 26 | creating (system) backups using rsync_, all of the actual Python code was 27 | written in the first few months of 2017 and has seen limited real world use. 28 | The project does however have an automated test suite with more than 90% test 29 | coverage and my intention is to extend the test coverage further. 30 | 31 | In May 2018 I changed the status from alpha to beta as part of release 1.0. The 32 | bump in major version number was triggered by a backwards incompatible code 33 | change, however at that point I had been using `rsync-system-backup` to make 34 | local backups of several of my Linux systems for the majority of a year. Also 35 | several colleagues of mine have used the how-to on setting up unattended 36 | backups to an encrypted USB disk. 37 | 38 | .. warning:: Please use the ``--dry-run`` option when you're getting familiar 39 | with how `rsync-system-backup` works and don't remove the option 40 | until you're confident that you have the right command line, 41 | because using `rsync-system-backup` in the wrong way can cause 42 | data loss (for example by accidentally swapping the ``SOURCE`` 43 | and ``DESTINATION`` arguments). 44 | 45 | Installation 46 | ------------ 47 | 48 | The `rsync-system-backup` package is available on PyPI_ which means 49 | installation should be as simple as: 50 | 51 | .. code-block:: sh 52 | 53 | $ pip install rsync-system-backup 54 | 55 | There's actually a multitude of ways to install Python packages (e.g. the `per 56 | user site-packages directory`_, `virtual environments`_ or just installing 57 | system wide) and I have no intention of getting into that discussion here, so 58 | if this intimidates you then read up on your options before returning to these 59 | instructions ;-). 60 | 61 | Usage 62 | ----- 63 | 64 | There are two ways to use the `rsync-system-backup` package: As the command 65 | line program ``rsync-system-backup`` and as a Python API. For details about the 66 | Python API please refer to the API documentation available on `Read the Docs`_. 67 | The command line interface is described below. 68 | 69 | Command line 70 | ~~~~~~~~~~~~ 71 | 72 | .. A DRY solution to avoid duplication of the `rsync-system-backup --help' text: 73 | .. 74 | .. [[[cog 75 | .. from humanfriendly.usage import inject_usage 76 | .. inject_usage('rsync_system_backup.cli') 77 | .. ]]] 78 | 79 | **Usage:** `rsync-system-backup [OPTIONS] [SOURCE] DESTINATION` 80 | 81 | Use rsync to create full system backups. 82 | 83 | The required DESTINATION argument specifies the (possibly remote) location 84 | where the backup is stored, in the syntax of rsync's command line interface. 85 | The optional SOURCE argument defaults to '/' which means the complete root 86 | filesystem will be included in the backup (other filesystems are excluded). 87 | 88 | Please use the ``--dry-run`` option when getting familiar with this program and 89 | don't remove it until you're confident that you have the right command line, 90 | because using this program in the wrong way can cause data loss (for example 91 | by accidentally swapping the SOURCE and DESTINATION arguments). 92 | 93 | Supported locations include: 94 | 95 | - Local disks (possibly encrypted using LUKS). 96 | - Remote systems that allow SSH connections. 97 | - Remote systems that are running an rsync daemon. 98 | - Connections to rsync daemons tunneled over SSH. 99 | 100 | The backup process consists of several steps: 101 | 102 | 1. First rsync is used to transfer all (relevant) files to a destination 103 | directory (whether on the local system or a remote system). Every time 104 | a backup is made, this same destination directory is updated. 105 | 106 | 2. After the files have been transferred a 'snapshot' of the destination 107 | directory is taken and stored in a directory with a timestamp in its 108 | name. These snapshots are created using 'cp ``--archive`` ``--link``'. 109 | 110 | 3. Finally the existing snapshots are rotated to purge old backups 111 | according to a rotation scheme that you can customize. 112 | 113 | **Supported options:** 114 | 115 | .. csv-table:: 116 | :header: Option, Description 117 | :widths: 30, 70 118 | 119 | 120 | "``-b``, ``--backup``","Create a backup using rsync but don't create a snapshot and don't rotate 121 | old snapshots unless the ``--snapshot`` and/or ``--rotate`` options are also given." 122 | "``-s``, ``--snapshot``","Create a snapshot of the destination directory but don't create a backup 123 | and don't rotate old snapshots unless the ``--backup`` and/or ``--rotate`` options 124 | are also given. 125 | 126 | This option can be used to create snapshots of an rsync daemon module using 127 | a 'post-xfer exec' command. If DESTINATION isn't given it defaults to the 128 | value of the environment variable ``$RSYNC_MODULE_PATH``." 129 | "``-r``, ``--rotate``","Rotate old snapshots but don't create a backup and snapshot unless the 130 | ``--backup`` and/or ``--snapshot`` options are also given. 131 | 132 | This option can be used to rotate old snapshots of an rsync daemon module 133 | using a 'post-xfer exec' command. If DESTINATION isn't given it defaults to 134 | the value of the environment variable ``$RSYNC_MODULE_PATH``." 135 | "``-m``, ``--mount=DIRECTORY``","Automatically mount the filesystem to which backups are written. 136 | 137 | When this option is given and ``DIRECTORY`` isn't already mounted, the 138 | 'mount' command is used to mount the filesystem to which backups are 139 | written before the backup starts. When 'mount' was called before the 140 | backup started, 'umount' will be called when the backup finishes. 141 | 142 | An entry for the mount point needs to be 143 | defined in /etc/fstab for this to work." 144 | "``-c``, ``--crypto=NAME``","Automatically unlock the encrypted filesystem to which backups are written. 145 | 146 | When this option is given and the ``NAME`` device isn't already unlocked, the 147 | cryptdisks_start command is used to unlock the encrypted filesystem to 148 | which backups are written before the backup starts. When cryptdisks_start 149 | was called before the backup started, cryptdisks_stop will be called 150 | when the backup finishes. 151 | 152 | An entry for the encrypted filesystem needs to be defined in /etc/crypttab 153 | for this to work. If the device of the encrypted filesystem is missing and 154 | rsync-system-backup is being run non-interactively, it will exit gracefully 155 | and not show any desktop notifications. 156 | 157 | If you want the backup process to run fully unattended you can configure a 158 | key file in /etc/crypttab, otherwise you will be asked for the password 159 | each time the encrypted filesystem is unlocked." 160 | "``-t``, ``--tunnel=TUNNEL_SPEC``","Connect to an rsync daemon through an SSH tunnel. This provides encryption 161 | for rsync client to daemon connections that are not otherwise encrypted. 162 | The value of ``TUNNEL_SPEC`` is expected to be an SSH alias, host name or IP 163 | address. Optionally a username can be prefixed (followed by '@') and/or a 164 | port number can be suffixed (preceded by ':')." 165 | "``-i``, ``--ionice=CLASS``","Use the 'ionice' program to set the I/O scheduling class and priority of 166 | the 'rm' invocations used to remove backups. ``CLASS`` is expected to be one of 167 | the values 'idle', 'best-effort' or 'realtime'. Refer to the man page of 168 | the 'ionice' program for details about these values." 169 | "``-u``, ``--no-sudo``","By default backup and snapshot creation is performed with superuser 170 | privileges, to ensure that all files are readable and filesystem 171 | metadata is preserved. The ``-u``, ``--no-sudo`` option disables 172 | the use of 'sudo' during these operations." 173 | "``-n``, ``--dry-run``","Don't make any changes, just report what would be done. This doesn't 174 | create a backup or snapshot but it does run rsync with the ``--dry-run`` 175 | option." 176 | ``--multi-fs``,"Allow rsync to cross filesystem boundaries. This option has the opposite 177 | effect of the rsync option ``--one-file-system`` because rsync-system-backup 178 | defaults to running rsync with ``--one-file-system`` and must be instructed 179 | not to using ``--multi-fs``." 180 | "``-x``, ``--exclude=PATTERN``","Selectively exclude certain files from being included in the backup. 181 | Refer to the rsync documentation for allowed ``PATTERN`` syntax. Note that 182 | rsync-system-backup always uses the 'rsync ``--one-file-system``' option." 183 | "``-f``, ``--force``","By default rsync-system-backup refuses to run on non-Linux systems because 184 | it was designed specifically for use on Linux. The use of the ``-f``, ``--force`` 185 | option sidesteps this sanity check. Please note that you are on your own if 186 | things break!" 187 | ``--disable-notifications``,"By default a desktop notification is shown (using notify-send) before the 188 | system backup starts and after the backup finishes. The use of this option 189 | disables the notifications (notify-send will not be called at all)." 190 | "``-v``, ``--verbose``",Make more noise (increase logging verbosity). Can be repeated. 191 | "``-q``, ``--quiet``",Make less noise (decrease logging verbosity). Can be repeated. 192 | "``-h``, ``--help``",Show this message and exit. 193 | 194 | .. [[[end]]] 195 | 196 | How it works 197 | ------------ 198 | 199 | I've been finetuning my approach to Linux system backups for years now and 200 | during that time rsync_ has become my swiss army knife of choice :-). I also 201 | believe that comprehensive documentation can be half the value of an open 202 | source project. The following sections attempt to provide a high level 203 | overview of my system backup strategy: 204 | 205 | .. contents:: 206 | :depth: 1 207 | :local: 208 | 209 | The (lack of) backup format 210 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ 211 | 212 | Each backup is a full copy of the filesystem tree, stored in the form of 213 | individual files and directories on the destination. This "backup format" makes 214 | it really easy to navigate through and recover from backups because you can use 215 | whatever method you are comfortable with, whether that is a file browser, 216 | terminal, Python_ script or even chroot_ :-). 217 | 218 | .. note:: You may want to configure updatedb_ to exclude the directory 219 | containing your system backups, otherwise the locate_ database 220 | will grow enormously. 221 | 222 | Snapshots and hard links 223 | ~~~~~~~~~~~~~~~~~~~~~~~~ 224 | 225 | Every time a backup is made the same destination directory is updated with 226 | additions, updates and deletions since the last backup. After the backup is 227 | done a snapshot of the destination directory is created using the command ``cp 228 | --archive --link`` with the current date and time encoded in the name. 229 | 230 | Due to the use of `hard links`_ each "version" of a file is only stored once. 231 | Because rsync_ by default doesn't modify files inplace it breaks `hard links`_ 232 | and thereby avoids modifying existing inodes_. This ensures that the contents 233 | of snapshots don't change when a new backup updates existing files. The 234 | combination of hard links and the avoidance of inplace modifications 235 | effectively provides a limited form of deduplication_. Each snapshot requires a 236 | couple of megabytes to store the directory names and hard links but the 237 | contents of files aren't duplicated. 238 | 239 | The article `Easy Automated Snapshot-Style Backups with Linux and Rsync`_ 240 | contains more details about this technique. 241 | 242 | Rotation of snapshots 243 | ~~~~~~~~~~~~~~~~~~~~~ 244 | 245 | Snapshots can be rotated according to a flexible rotation scheme, for example 246 | I've configured my laptop backup rotation to preserve the most recent 24 hourly 247 | backups, 30 daily backups and endless monthly backups. 248 | 249 | Backup destinations 250 | ~~~~~~~~~~~~~~~~~~~ 251 | 252 | While developing, maintaining and evolving backup scripts for various Linux 253 | laptops and servers I've learned that backups for different systems require 254 | different backup destinations and connection methods: 255 | 256 | .. contents:: 257 | :local: 258 | 259 | Encrypted USB disks 260 | +++++++++++++++++++ 261 | 262 | There's a LUKS_ encrypted USB disk on my desk at work that I use to keep 263 | hourly, daily and monthly backups of my work laptop. The disk is connected 264 | through the same USB hub that also connects my keyboard and mouse so I can't 265 | really forget about it :-). 266 | 267 | Automatic mounting 268 | ^^^^^^^^^^^^^^^^^^ 269 | 270 | Before the backup starts, the encrypted disk is automatically unlocked and 271 | mounted. The use of a key file enables this process to run unattended in the 272 | background. Once the backup is done the disk will be unmounted and locked 273 | again, so that it can be unplugged at any time (as long as a backup isn't 274 | running of course). 275 | 276 | Local server (rsync daemon) 277 | +++++++++++++++++++++++++++ 278 | 279 | My personal laptop transfers hourly backups to the `rsync daemon`_ running on 280 | the server in my home network using a direct TCP connection without SSH. Most 281 | of the time the laptop has an USB Ethernet adapter connected but the backup 282 | runs fine over a wireless connection as well. 283 | 284 | Remote server (rsync daemon over SSH tunnel) 285 | ++++++++++++++++++++++++++++++++++++++++++++ 286 | 287 | My VPS (virtual private server) transfers nightly backups to the `rsync 288 | daemon`_ running on the server in my home network over an `SSH tunnel`_ in 289 | order to encrypt the traffic and restrict access. The SSH account is configured 290 | to allow tunneling but disallow command execution. This setup enables the rsync 291 | client and server to run with root privileges without allowing the client to 292 | run arbitrary commands on the server. 293 | 294 | Alternative connection methods 295 | ------------------------------ 296 | 297 | Backing up to a local disk limits the effectiveness of backups but using SSH 298 | access between systems gives you more than you bargained for, because you're 299 | allowing arbitrary command execution. The `rsync daemon`_ provides an 300 | alternative that does not allow arbitrary command execution. The following 301 | sections discuss this option in more detail. 302 | 303 | Using rsync daemon 304 | ~~~~~~~~~~~~~~~~~~ 305 | 306 | To be able to write files as root and preserve all filesystem metadata, rsync 307 | must be running with root privileges. However most of my backups are stored on 308 | remote systems and opening up remote root access over SSH just to transfer 309 | backups feels like a very blunt way to solve the problem :-). 310 | 311 | Fortunately another solution is available: Configure an rsync daemon on the 312 | destination and instruct your rsync client to connect to the rsync daemon 313 | instead of connecting to the remote system over SSH. The rsync daemon 314 | configuration can restrict the access of the rsync client so that it can only 315 | write to the directory that contains the backup tree. 316 | 317 | In this setup no SSH connections are used and the traffic between the rsync 318 | client and server is not encrypted. If this is a problem for you then continue 319 | reading the next section. 320 | 321 | Enabling rsync daemon 322 | +++++++++++++++++++++ 323 | 324 | On Debian and derivatives like Ubuntu you can enable and configure an `rsync 325 | daemon`_ quite easily: 326 | 327 | 1. Make sure that rsync is installed: 328 | 329 | .. code-block:: sh 330 | 331 | $ sudo apt-get install rsync 332 | 333 | 2. Enable the rsync daemon by editing ``/etc/default/rsync`` and changing the 334 | line ``RSYNC_ENABLE=false`` to ``RSYNC_ENABLE=true``. Here's a one liner 335 | that accomplishes the task: 336 | 337 | .. code-block:: sh 338 | 339 | $ sudo sed -i 's/RSYNC_ENABLE=false/RSYNC_ENABLE=true/' /etc/default/rsync 340 | 341 | 3. Create the configuration file ``/etc/rsyncd.conf`` and define at least 342 | one module. Here's an example based on my rsync daemon configuration: 343 | 344 | .. code-block:: ini 345 | 346 | # Global settings. 347 | max connections = 4 348 | log file = /var/log/rsyncd.log 349 | 350 | # Defaults for modules. 351 | read only = no 352 | uid = 0 353 | gid = 0 354 | 355 | # Daily backups of my VPS. 356 | [vps_backups] 357 | path = /mnt/backups/vps/latest 358 | post-xfer exec = /usr/sbin/process-vps-backups 359 | 360 | # Hourly backups of my personal laptop. 361 | [laptop_backups] 362 | path = /mnt/backups/laptop/latest 363 | post-xfer exec = /usr/sbin/process-laptop-backups 364 | 365 | The ``post-xfer exec`` directives configure the rsync daemon to create a 366 | snapshot once the backup is done and rotate old snapshots afterwards. 367 | 368 | 4. Once you've created ``/etc/rsyncd.conf`` you can start the rsync daemon: 369 | 370 | .. code-block:: sh 371 | 372 | $ sudo service rsync start 373 | 374 | 5. If you're using a firewall you should make sure that the rsync daemon port 375 | is whitelisted to allow incoming connections. The rsync daemon port number 376 | defaults to 873. Here's an iptables command to accomplish this: 377 | 378 | .. code-block:: sh 379 | 380 | $ sudo iptables -A INPUT -p tcp -m tcp --dport 873 -m comment --comment "rsync daemon" -j ACCEPT 381 | 382 | Tunneling rsync daemon connections 383 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 384 | 385 | When your backups are transferred over the public internet you should 386 | definitely use SSH to encrypt the traffic, but if you're at all security 387 | conscious then you probably won't like having to open up remote root access 388 | over SSH just to transfer backups :-). 389 | 390 | The alternative is to use a non privileged SSH account to set up an `SSH 391 | tunnel`_ that redirects network traffic to the rsync daemon. The login shell of 392 | the SSH account can be set to ``/usr/sbin/nologin`` (or something similar like 393 | ``/bin/false``) to `disable command execution`_, in this case you need to pass 394 | ``-N`` to the SSH client. 395 | 396 | Contact 397 | ------- 398 | 399 | The latest version of `rsync-system-backup` is available on PyPI_ and GitHub_. 400 | The documentation is hosted on `Read the Docs`_ and includes a changelog_. For 401 | bug reports please create an issue on GitHub_. If you have questions, 402 | suggestions, etc. feel free to send me an e-mail at `peter@peterodding.com`_. 403 | 404 | License 405 | ------- 406 | 407 | This software is licensed under the `MIT license`_. 408 | 409 | © 2019 Peter Odding. 410 | 411 | .. External references: 412 | 413 | .. _changelog: https://rsync-system-backup.readthedocs.org/en/latest/changelog.html 414 | .. _chroot: https://manpages.debian.org/chroot 415 | .. _deduplication: https://en.wikipedia.org/wiki/Data_deduplication 416 | .. _disable command execution: https://unix.stackexchange.com/questions/155139/does-usr-sbin-nologin-as-a-login-shell-serve-a-security-purpose 417 | .. _Easy Automated Snapshot-Style Backups with Linux and Rsync: http://www.mikerubel.org/computers/rsync_snapshots/ 418 | .. _GitHub: https://github.com/xolox/python-rsync-system-backup 419 | .. _hard links: https://en.wikipedia.org/wiki/Hard_link 420 | .. _inodes: https://en.wikipedia.org/wiki/Inode 421 | .. _Linux: https://en.wikipedia.org/wiki/Linux 422 | .. _locate: https://manpages.debian.org/mlocate 423 | .. _LUKS: https://en.wikipedia.org/wiki/Linux_Unified_Key_Setup 424 | .. _MIT license: http://en.wikipedia.org/wiki/MIT_License 425 | .. _per user site-packages directory: https://www.python.org/dev/peps/pep-0370/ 426 | .. _peter@peterodding.com: peter@peterodding.com 427 | .. _PyPI: https://pypi.python.org/pypi/rsync-system-backup 428 | .. _Python Package Index: https://pypi.python.org/pypi/rsync-system-backup 429 | .. _Python: https://www.python.org/ 430 | .. _Read the Docs: https://rsync-system-backup.readthedocs.org 431 | .. _rsync daemon: https://manpages.debian.org/rsyncd.conf 432 | .. _rsync: http://en.wikipedia.org/wiki/rsync 433 | .. _SSH tunnel: https://en.wikipedia.org/wiki/Tunneling_protocol#Secure_Shell_tunneling 434 | .. _SSH: https://en.wikipedia.org/wiki/Secure_Shell 435 | .. _updatedb: https://manpages.debian.org/updatedb 436 | .. _virtual environments: http://docs.python-guide.org/en/latest/dev/virtualenvs/ 437 | -------------------------------------------------------------------------------- /docs/api.rst: -------------------------------------------------------------------------------- 1 | API documentation 2 | ================= 3 | 4 | The following documentation is based on the source code of version |release| of 5 | the `rsync-system-backup` package: 6 | 7 | .. contents:: 8 | :local: 9 | 10 | :mod:`rsync_system_backup` 11 | -------------------------- 12 | 13 | .. automodule:: rsync_system_backup 14 | :members: 15 | 16 | :mod:`rsync_system_backup.cli` 17 | ------------------------------ 18 | 19 | .. automodule:: rsync_system_backup.cli 20 | :members: 21 | 22 | :mod:`rsync_system_backup.destinations` 23 | --------------------------------------- 24 | 25 | .. automodule:: rsync_system_backup.destinations 26 | :members: 27 | 28 | :mod:`rsync_system_backup.exceptions` 29 | ------------------------------------- 30 | 31 | .. automodule:: rsync_system_backup.exceptions 32 | :members: 33 | -------------------------------------------------------------------------------- /docs/changelog.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../CHANGELOG.rst 2 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | """Documentation build configuration file for the `rsync-system-backup` package.""" 2 | 3 | import os 4 | import sys 5 | 6 | # Add the 'rsync-system-backup' source distribution's root directory to the module path. 7 | sys.path.insert(0, os.path.abspath(os.pardir)) 8 | 9 | # -- General configuration ----------------------------------------------------- 10 | 11 | # Sphinx extension module names. 12 | extensions = [ 13 | 'sphinx.ext.autodoc', 14 | 'sphinx.ext.doctest', 15 | 'sphinx.ext.intersphinx', 16 | 'sphinx.ext.viewcode', 17 | 'humanfriendly.sphinx', 18 | 'property_manager.sphinx', 19 | ] 20 | 21 | # Configuration for the `autodoc' extension. 22 | autodoc_member_order = 'bysource' 23 | 24 | # Paths that contain templates, relative to this directory. 25 | templates_path = ['templates'] 26 | 27 | # The suffix of source filenames. 28 | source_suffix = '.rst' 29 | 30 | # The master toctree document. 31 | master_doc = 'index' 32 | 33 | # General information about the project. 34 | project = 'rsync-system-backup' 35 | copyright = '2019, Peter Odding' 36 | 37 | # The version info for the project you're documenting, acts as replacement for 38 | # |version| and |release|, also used in various other places throughout the 39 | # built documents. 40 | 41 | # Find the package version and make it the release. 42 | from rsync_system_backup import __version__ as rsync_system_backup_version # noqa 43 | 44 | # The short X.Y version. 45 | version = '.'.join(rsync_system_backup_version.split('.')[:2]) 46 | 47 | # The full version, including alpha/beta/rc tags. 48 | release = rsync_system_backup_version 49 | 50 | # The language for content autogenerated by Sphinx. Refer to documentation 51 | # for a list of supported languages. 52 | language = 'en' 53 | 54 | # List of patterns, relative to source directory, that match files and 55 | # directories to ignore when looking for source files. 56 | exclude_patterns = ['build'] 57 | 58 | # If true, '()' will be appended to :func: etc. cross-reference text. 59 | add_function_parentheses = True 60 | 61 | # The name of the Pygments (syntax highlighting) style to use. 62 | pygments_style = 'sphinx' 63 | 64 | # Refer to the Python standard library. 65 | # From: http://twistedmatrix.com/trac/ticket/4582. 66 | intersphinx_mapping = { 67 | 'python': ('https://docs.python.org/2', None), 68 | 'executor': ('https://executor.readthedocs.io/en/latest/', None), 69 | 'linuxutils': ('https://linux-utils.readthedocs.io/en/latest/', None), 70 | 'propertymanager': ('https://property-manager.readthedocs.io/en/latest/', None), 71 | 'rotatebackups': ('https://rotate-backups.readthedocs.io/en/latest/', None), 72 | } 73 | 74 | # -- Options for HTML output --------------------------------------------------- 75 | 76 | # The theme to use for HTML and HTML Help pages. See the documentation for 77 | # a list of builtin themes. 78 | html_theme = 'nature' 79 | 80 | # Output file base name for HTML help builder. 81 | htmlhelp_basename = 'rsyncsystembackupdoc' 82 | -------------------------------------------------------------------------------- /docs/howto/encrypted-usb-disk.rst: -------------------------------------------------------------------------------- 1 | How to set up unattended backups to an encrypted USB disk 2 | ========================================================= 3 | 4 | This document explains how to set up unattended Linux system backups to an 5 | encrypted USB disk using LUKS_ filesystem encryption. These instructions are 6 | tested on Ubuntu_ (to be more specific I've used this process on 12.04, 14.04 7 | and 16.04) but I'd expect them to work just as well on Debian_ and Linux 8 | distributions based on Debian. 9 | 10 | Most of the steps in this how-to should in fact work fine on any Linux system 11 | (maybe with minor adjustments here and there) however there is one important 12 | thing to note: the configuration file `/etc/crypttab`_ and the commands 13 | cryptdisks_start_ and cryptdisks_stop_ are a Debian-ism that may not be 14 | available on other Linux distributions. The relevant sections explain 15 | how to tackle this (long story short: I wrote a fallback). 16 | 17 | .. contents:: 18 | :local: 19 | 20 | Install rsync-system-backup 21 | --------------------------- 22 | 23 | There are several ways to install `rsync-system-backup`, for example:: 24 | 25 | # Make sure pip (the Python package manager) and related packages are installed. 26 | sudo apt-get install python-{pip,pkg-resources,setuptools} 27 | 28 | # Use pip to install the Python package we need in /usr/local. The 29 | # executable will be available at /usr/local/bin/rsync-system-backup. 30 | sudo pip install --upgrade rsync-system-backup 31 | 32 | You can can also install the Python package and its dependencies in your home 33 | directory if you prefer that over "polluting" the system wide /usr/local 34 | directory:: 35 | 36 | # Use pip to install the Python package we need in ~/.local. The 37 | # executable will be available at ~/.local/bin/rsync-system-backup. 38 | pip install --upgrade --user rsync-system-backup 39 | 40 | If that is still "too global" for your tastes then feel free to set up a 41 | Python virtual environment ;-). 42 | 43 | Prepare the disk encryption 44 | --------------------------- 45 | 46 | .. contents:: 47 | :local: 48 | 49 | Create a key file 50 | ~~~~~~~~~~~~~~~~~ 51 | 52 | We will use a key file to enable `rsync-system-backup` to unlock the encrypted 53 | USB disk without requiring user interaction due to a password prompt. Basically 54 | the contents of the key file will serve as an alternate password that can be 55 | used in a noninteractive setting. 56 | 57 | .. code-block:: sh 58 | 59 | # Create a directory to store the key file. 60 | sudo mkdir -p /root/keys 61 | 62 | # Generate the key file from two kilobytes of pseudorandom data. 63 | sudo dd if=/dev/urandom of=/root/keys/backups.key bs=512 count=4 64 | 65 | # Make sure the directory and key file are private to the root user. 66 | sudo chown -R root:root /root/keys 67 | sudo chmod u=rwx,go= /root/keys 68 | sudo chmod u=r,go= /root/keys/backups.key 69 | 70 | .. warning:: I'm assuming here that the computer that you want to create 71 | backups of already has full disk encryption. If this is not the 72 | case it means that anyone with physical access to the computer can 73 | just power it off, rip out the hard disk and extract the contents 74 | of ``/root/keys/backups.key``, compromising the security of your 75 | backups! 76 | 77 | Enable encryption on the USB disk 78 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 79 | 80 | Enabling encryption on the USB disk will effectively wipe the existing contents 81 | of the disk, so you need to make sure of two things: 82 | 83 | 1. The existing contents of the disk have been backed up. 84 | 2. You're specifying the correct device file in the following command (please 85 | triple check or you might wipe the wrong disk). 86 | 87 | .. code-block:: sh 88 | 89 | # Enable LUKS disk encryption on the USB disk. 90 | sudo cryptsetup luksFormat /dev/sdx /root/keys/backups.key 91 | 92 | In the command above ``/dev/sdx`` is the device file (this is what you need to 93 | change, see `figuring out the correct device file`_ for hints) and 94 | ``/root/keys/backups.key`` is the name of the key file that we created in the 95 | previous step. 96 | 97 | Careful readers will notice that I'm not bothering to create a partition table 98 | on the USB disk, that's because we don't need it :-). 99 | 100 | Make sure the disk is not in use 101 | ++++++++++++++++++++++++++++++++ 102 | 103 | The ``luksFormat`` command above may give you an error like:: 104 | 105 | Cannot format device /dev/sdx which is still in use 106 | 107 | If this happens then most likely the USB disk that you attached already has a 108 | filesystem on it and your desktop environment automatically mounted that 109 | filesystem. You will need to unmount that filesystem before you can enable 110 | encryption on the disk. If you don't know how to do that: 111 | 112 | 1. Follow the steps in the section `figuring out the correct device file`_ and 113 | take note of the device file corresponding to the USB disk. 114 | 2. Run the ``mount`` command to get a list of mounted filesystems and look for 115 | lines that mention the relevant device file. Most likely a number will be 116 | appended at the end of the device file (this indicates a partition on the 117 | USB disk). 118 | 3. For each of the relevant entries in the ``mount`` output, run the following 119 | command:: 120 | 121 | sudo umount /dev/sdx1 122 | 123 | In the command above ``/dev/sdx1`` is the device file of a partition on the 124 | USB disk (this is what you need to change). 125 | 126 | .. _figuring out the correct device file: 127 | 128 | Figuring out the correct device file 129 | ++++++++++++++++++++++++++++++++++++ 130 | 131 | If you don't know or you're not sure what the device file for the 132 | ``luksFormat`` command above should be, here's one relatively 133 | foolproof way to figure it out: 134 | 135 | 1. Disconnect the USB disk from your computer. 136 | 137 | 2. Open a terminal and use the following command to observe 138 | new log entries being added to the system log:: 139 | 140 | # Follow the system log (watch for new entries). 141 | sudo tail -fn 0 /var/log/syslog 142 | 143 | 3. Connect the USB disk to your computer and give the disk a few seconds to 144 | spin up and properly establish a USB connection to your computer. 145 | 146 | 4. Observe the entries that just appeared in the system log. If you study them 147 | carefully you should be able to figure out the name of the device file. 148 | 149 | Configure a recovery password 150 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 151 | 152 | If your computer's hard disk breaks or your computer is stolen you will lose 153 | the key file required to unlock your encrypted backups, which would be rather 154 | ironic but not in a fun way :-P. To avoid this situation we can configure the 155 | disk encryption with a recovery password:: 156 | 157 | # Configure a recovery password. 158 | sudo cryptsetup --key-file=/root/keys/backups.key luksAddKey /dev/sdx 159 | 160 | In the command above ``/dev/sdx`` is the device file, this should be the same 161 | device file you used in the previous step. 162 | 163 | Configure the encrypted disk 164 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 165 | 166 | Once encryption has been enabled we can configure the encrypted disk 167 | in ``/etc/crypttab``. To do so we first need to determine the unique 168 | identifier of the encrypted disk:: 169 | 170 | # Determine the UUID of the encrypted disk. 171 | sudo blkid /dev/sdx 172 | 173 | In the command above ``/dev/sdx`` is the device file, this should be the same 174 | device file you used in the previous step. The ``blkid`` command will output a 175 | string called a UUID (a universally unique identifier), you need to copy this 176 | to your clipboard (or have photographic memory). Now that we know the UUID we 177 | can add the ``/etc/crypttab`` entry:: 178 | 179 | # Use a text editor to configure the encrypted disk. 180 | sudo nano /etc/crypttab 181 | 182 | If the file doesn't exist yet it implies that you're not using full disk 183 | encryption on your computer. Please reconsider! But I digress. Now you need to 184 | add a new line to the file, something like this:: 185 | 186 | backups UUID=13f6e17e-8c8b-4009-a7b3-356992415141 /root/keys/backups.key luks,discard,noauto 187 | 188 | Replace the part after ``UUID=`` with the output of ``blkid``. Everything else 189 | should be fine as is, unless you've chosen a different location for the key 190 | file. 191 | 192 | Unlock the encrypted disk 193 | ~~~~~~~~~~~~~~~~~~~~~~~~~ 194 | 195 | Thanks to the ``/etc/crypttab`` entry that we added in the previous step, 196 | unlocking the disk is very simple:: 197 | 198 | # Unlock the encrypted backups disk. 199 | sudo cryptdisks_start backups 200 | 201 | This won't ask for a password because we configured a key file. If you get a 202 | "command not found" error then here are two suggestions: 203 | 204 | 1. If you're running a Linux distribution based on Debian_ (like Ubuntu_) then 205 | you can install cryptdisks_start_ and cryptdisks_stop_ as follows:: 206 | 207 | # Make sure `cryptdisks_start' and `cryptdisks_stop' are installed. 208 | sudo apt-get install cryptsetup 209 | 210 | 2. If you're running a Linux distribution that's not based on Debian_ then the 211 | cryptdisks_start_ and cryptdisks_stop_ programs may not be available to you. 212 | Don't worry though, I've got your back! ;-) 213 | 214 | Because we've already installed `rsync-system-backup` its dependencies are 215 | also available. One of these dependencies installs the following two command 216 | line programs: 217 | 218 | - ``cryptdisks-start-fallback`` 219 | - ``cryptdisks-stop-fallback`` 220 | 221 | These programs are not as full featured as their "official" counterparts but 222 | they should work fine for the purposes of this how-to. Instead of the 223 | command given at the start of this section, please use the following 224 | command:: 225 | 226 | sudo cryptdisks-start-fallback backups 227 | 228 | Prepare the encrypted filesystem 229 | -------------------------------- 230 | 231 | .. contents:: 232 | :local: 233 | 234 | Format the encrypted filesystem 235 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 236 | 237 | After the encrypted disk is unlocked using ``cryptdisks_start`` the device file 238 | ``/dev/mapper/backups`` provides access to the unlocked data. Encrypting the 239 | disk hasn't created an actual filesystem yet so that's what we'll do next:: 240 | 241 | sudo mkfs.ext4 /dev/mapper/backups 242 | 243 | Configure the encrypted filesystem 244 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 245 | 246 | We'll add an entry to ``/etc/fstab`` so that it's as easy to mount the 247 | filesystem as it was easy to unlock the disk:: 248 | 249 | # Use a text editor to configure the encrypted filesystem. 250 | sudo nano /etc/fstab 251 | 252 | Add a new line to the file, something like this:: 253 | 254 | /dev/mapper/backups /mnt/backups ext4 noauto,errors=remount-ro 0 0 255 | 256 | Also make sure the mount point exists:: 257 | 258 | sudo mkdir -p /mnt/backups 259 | 260 | Mount the encrypted filesystem 261 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 262 | 263 | This should be familiar to most of you:: 264 | 265 | sudo mount /mnt/backups 266 | 267 | Decide on a directory layout 268 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 269 | 270 | On my backup disks I am using a directory layout of multiple levels because my 271 | backups and I go way back :-). The first level consists of the names I chose to 272 | describe the laptops I've had over the years: 273 | 274 | - /mnt/backups 275 | 276 | - zenbook 277 | - hp-probook 278 | - macbook-pro 279 | 280 | Each of these directories has subdirectories with the names of the Ubuntu 281 | releases that were installed on those laptops over the years: 282 | 283 | - /mnt/backups 284 | 285 | - zenbook 286 | 287 | - lucid 288 | - precise 289 | 290 | - hp-probook 291 | 292 | - precise 293 | - trusty 294 | 295 | - macbook-pro 296 | 297 | - xenial 298 | 299 | Each of the directories named after an Ubuntu release stores a collection of 300 | timestamped system backups, something like this: 301 | 302 | - /mnt/backups 303 | 304 | - zenbook 305 | 306 | - lucid 307 | 308 | - 2011-02-05 15:30 309 | - 2011-03-19 11:45 310 | 311 | - precise 312 | 313 | - 2013-04-10 14:00 314 | - 2013-05-10 14:00 315 | 316 | - hp-probook 317 | 318 | - precise 319 | 320 | - 2014-03-12 16:15 321 | 322 | - trusty 323 | 324 | - 2016-06-15 12:00 325 | 326 | - macbook-pro 327 | 328 | - xenial 329 | 330 | - 2017-03-19 23:15 331 | - 2017-04-01 12:34 332 | - 2017-05-02 17:00 333 | - latest 334 | 335 | The dates were made up and in reality I have hundreds of timestamped system 336 | backups, but you get the idea :-). 337 | 338 | Whether you use the same directory layout or something simpler is up to you. 339 | 340 | Create your first backup 341 | ------------------------ 342 | 343 | Here's an example of how you can create a system backup:: 344 | 345 | sudo rsync-system-backup -c backups -m /mnt/backups /mnt/backups/latest 346 | 347 | That last directory must be a subdirectory of ``/mnt/backups``, if you want to 348 | keep things simple then just use ``/mnt/backups/latest`` (whatever you do, 349 | don't just pass it ``/mnt/backups``). 350 | 351 | If you get a "command not found" error from ``sudo`` try the following instead:: 352 | 353 | sudo $(which rsync-system-backup) -c backups -m /mnt/backups /mnt/backups/latest 354 | 355 | Configure unattended backups 356 | ---------------------------- 357 | 358 | The final part of this how-to configures your system to automatically run 359 | `rsync-system-backup` at an interval of your choosing, for example once every 360 | four hours. The easiest way to accomplish this is using cron. To do so we'll 361 | create a new configuration file:: 362 | 363 | # Use a text editor to configure unattended backups. 364 | sudo nano /etc/cron.d/rsync-system-backup 365 | 366 | Create the file with the following contents:: 367 | 368 | # Cron by default starts subcommands in a very sparse environment where the 369 | # $PATH contains just /usr/bin and /bin. Since we expect a reasonably sane 370 | # $PATH we have to set it ourselves: 371 | PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin 372 | 373 | # Create a full system backup every four hours. 374 | 0 */4 * * * root rsync-system-backup -c backups -m /mnt/backups /mnt/backups/latest 375 | 376 | Depending on how you installed `rsync-system-backup` you may need to adjust the 377 | ``PATH`` variable or change the program name into an absolute pathname. 378 | 379 | Silencing desktop notifications 380 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 381 | 382 | When `rsync-system-backup` is running non-interactively and it finds that the 383 | device of the encrypted filesystem is missing, it will exit gracefully and not 384 | show any desktop notifications. This is intended to avoid noise when the backup 385 | disk isn't connected. 386 | 387 | If the desktop notifications announcing the start and completion of a system 388 | backup drive you bonkers, add the ``--disable-notifications`` option to the 389 | `rsync-system-backup` command line to silence desktop notifications. 390 | 391 | 392 | .. External references: 393 | 394 | .. _/etc/crypttab: https://manpages.debian.org/crypttab 395 | .. _cryptdisks_start: https://manpages.debian.org/cryptdisks_start 396 | .. _cryptdisks_stop: https://manpages.debian.org/cryptdisks_stop 397 | .. _Debian: https://en.wikipedia.org/wiki/Debian 398 | .. _LUKS: https://en.wikipedia.org/wiki/Linux_Unified_Key_Setup 399 | .. _Ubuntu: https://en.wikipedia.org/wiki/Ubuntu_(operating_system) 400 | -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | rsync-system-backup: Linux system backups powered by rsync 2 | ========================================================== 3 | 4 | Welcome to the documentation of `rsync-system-backup` version |release|! 5 | 6 | .. contents:: 7 | :local: 8 | 9 | User documentation 10 | ------------------ 11 | 12 | The readme explains the high level concepts and is mostly targeted at users of 13 | the command line interface. It's probably the best place to start reading: 14 | 15 | .. toctree:: 16 | readme.rst 17 | 18 | The following instructions are also intended for users of the command line 19 | interface but are too detailed to be included in the readme: 20 | 21 | .. toctree:: 22 | howto/encrypted-usb-disk.rst 23 | 24 | API documentation 25 | ----------------- 26 | 27 | The following documentation is targeted at people who are interested in using 28 | the Python API: 29 | 30 | .. toctree:: 31 | api.rst 32 | 33 | Change log 34 | ---------- 35 | 36 | The change log lists notable changes to the project: 37 | 38 | .. toctree:: 39 | changelog.rst 40 | -------------------------------------------------------------------------------- /docs/readme.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../README.rst 2 | -------------------------------------------------------------------------------- /requirements-checks.txt: -------------------------------------------------------------------------------- 1 | # Python packages required to run `make check'. 2 | flake8 >= 2.6.0 3 | flake8-docstrings >= 0.2.8 4 | pyflakes >= 1.2.3 5 | 6 | # https://gitlab.com/pycqa/flake8-docstrings/issues/36 7 | pydocstyle < 4 8 | -------------------------------------------------------------------------------- /requirements-tests.txt: -------------------------------------------------------------------------------- 1 | mock >= 2.0.0 2 | pytest >= 2.6.1 3 | pytest-cov >= 2.2.1 4 | -------------------------------------------------------------------------------- /requirements-travis.txt: -------------------------------------------------------------------------------- 1 | --requirement=requirements-checks.txt 2 | --requirement=requirements-tests.txt 3 | --requirement=requirements.txt 4 | coveralls 5 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | coloredlogs >= 6.1 2 | executor >= 19.3 3 | humanfriendly >= 3.4 4 | linux-utils >= 0.4 5 | proc >= 0.14 6 | property-manager >= 2.3 7 | rotate-backups >= 4.4 8 | -------------------------------------------------------------------------------- /rsync_system_backup/__init__.py: -------------------------------------------------------------------------------- 1 | # rsync-system-backup: Linux system backups powered by rsync. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: August 2, 2019 5 | # URL: https://github.com/xolox/python-rsync-system-backup 6 | 7 | """ 8 | Simple to use Python API for Linux system backups powered by rsync. 9 | 10 | The :mod:`rsync_system_backup` module contains the Python API of the 11 | `rsync-system-backup` package. The core logic of the package is contained in 12 | the :class:`RsyncSystemBackup` class. 13 | """ 14 | 15 | # Standard library modules. 16 | import logging 17 | import os 18 | import time 19 | 20 | # External dependencies. 21 | from executor import quote 22 | from executor.contexts import LocalContext, create_context 23 | from humanfriendly import Timer, compact, concatenate 24 | from linux_utils.crypttab import parse_crypttab 25 | from linux_utils.luks import cryptdisks_start, cryptdisks_stop 26 | from proc.notify import notify_desktop 27 | from property_manager import ( 28 | PropertyManager, 29 | cached_property, 30 | clear_property, 31 | lazy_property, 32 | mutable_property, 33 | required_property, 34 | set_property, 35 | ) 36 | from rotate_backups import Location, RotateBackups 37 | 38 | # Modules included in our package. 39 | from rsync_system_backup.destinations import Destination 40 | from rsync_system_backup.exceptions import ( 41 | DestinationContextUnavailable, 42 | FailedToMountError, 43 | FailedToUnlockError, 44 | InvalidDestinationDirectory, 45 | MissingBackupDiskError, 46 | UnsupportedPlatformError, 47 | ) 48 | 49 | # Semi-standard module versioning. 50 | __version__ = '1.1' 51 | 52 | # Public identifiers that require documentation. 53 | __all__ = ( 54 | 'DEFAULT_ROTATION_SCHEME', 55 | 'RsyncSystemBackup', 56 | '__version__', 57 | 'ensure_trailing_slash', 58 | 'logger', 59 | ) 60 | 61 | # Initialize a logger for this module. 62 | logger = logging.getLogger(__name__) 63 | 64 | DEFAULT_ROTATION_SCHEME = dict(hourly=24, daily=7, weekly=4, monthly='always') 65 | """The default rotation scheme for system backup snapshots (a dictionary).""" 66 | 67 | 68 | class RsyncSystemBackup(PropertyManager): 69 | 70 | """ 71 | Python API for the ``rsync-system-backup`` program. 72 | 73 | The :func:`execute()` method is the main entry point. 74 | If you're looking for finer grained control refer to 75 | :func:`unlock_device()`, :func:`mount_filesystem()`, 76 | :func:`transfer_changes()`, :func:`create_snapshot()` 77 | and :func:`rotate_snapshots()`. 78 | """ 79 | 80 | @mutable_property 81 | def backup_enabled(self): 82 | """:data:`True` to enable :func:`transfer_changes()`, :data:`False` otherwise.""" 83 | return True 84 | 85 | @mutable_property 86 | def crypto_device(self): 87 | """The name of the encrypted filesystem to use (a string or :data:`None`).""" 88 | 89 | @property 90 | def crypto_device_available(self): 91 | """ 92 | :data:`True` if the encrypted filesystem is available, :data:`False` otherwise. 93 | 94 | This property is an alias for the 95 | :attr:`~linux_utils.crypttab.EncryptedFileSystemEntry.is_available` 96 | property of :attr:`crypttab_entry`. 97 | """ 98 | return self.crypttab_entry.is_available if self.crypttab_entry else False 99 | 100 | @property 101 | def crypto_device_unlocked(self): 102 | """ 103 | :data:`True` if the encrypted filesystem is unlocked, :data:`False` otherwise. 104 | 105 | This property is an alias for the 106 | :attr:`~linux_utils.crypttab.EncryptedFileSystemEntry.is_unlocked` 107 | property of :attr:`crypttab_entry`. 108 | """ 109 | return self.crypttab_entry.is_unlocked if self.crypttab_entry else False 110 | 111 | @cached_property 112 | def crypttab_entry(self): 113 | """ 114 | The entry in ``/etc/crypttab`` corresponding to :attr:`crypto_device`. 115 | 116 | The value of this property is computed automatically by parsing 117 | ``/etc/crypttab`` and looking for an entry whose `target` (the 118 | first of the four fields) matches :attr:`crypto_device`. 119 | 120 | When an entry is found an 121 | :class:`~linux_utils.crypttab.EncryptedFileSystemEntry` object is 122 | constructed, otherwise the result is :data:`None`. 123 | """ 124 | if self.crypto_device: 125 | logger.debug("Parsing /etc/crypttab to determine device file of encrypted filesystem %r ..", 126 | self.crypto_device) 127 | for entry in parse_crypttab(context=self.destination_context): 128 | if entry.target == self.crypto_device: 129 | return entry 130 | 131 | @required_property 132 | def destination(self): 133 | """ 134 | The destination where backups are stored (a :class:`.Destination` object). 135 | 136 | The value of :attr:`destination` defaults to the value of the 137 | environment variable ``$RSYNC_MODULE_PATH`` which is set by the `rsync 138 | daemon`_ before it runs the ``post-xfer exec`` command. 139 | """ 140 | rsync_module_path = os.environ.get('RSYNC_MODULE_PATH') 141 | return (Destination(expression=rsync_module_path) 142 | if rsync_module_path else None) 143 | 144 | @destination.setter 145 | def destination(self, value): 146 | """Automatically coerce strings to :class:`.Destination` objects.""" 147 | if not isinstance(value, Destination): 148 | value = Destination(expression=value) 149 | set_property(self, 'destination', value) 150 | clear_property(self, 'destination_context') 151 | 152 | @cached_property 153 | def destination_context(self): 154 | """ 155 | The execution context of the system that stores the backup (the destination). 156 | 157 | This is an execution context created by :mod:`executor.contexts`. 158 | 159 | :raises: :exc:`.DestinationContextUnavailable` when the destination is 160 | an rsync daemon module (which doesn't allow arbitrary command 161 | execution). 162 | """ 163 | if self.destination.module: 164 | raise DestinationContextUnavailable(compact(""" 165 | Error: The execution context of the backup destination isn't 166 | available because the destination ({dest}) is an rsync daemon 167 | module! (tip: reconsider your command line options) 168 | """, dest=self.destination.expression)) 169 | else: 170 | context_opts = dict(sudo=self.sudo_enabled) 171 | if self.destination.hostname: 172 | context_opts['ssh_alias'] = self.destination.hostname 173 | context_opts['ssh_user'] = self.destination.username 174 | return create_context(**context_opts) 175 | 176 | @mutable_property 177 | def dry_run(self): 178 | """:data:`True` to simulate the backup without writing any files, :data:`False` otherwise.""" 179 | return False 180 | 181 | @mutable_property 182 | def multi_fs(self): 183 | """ 184 | :data:`True` to allow rsync to cross filesystem boundaries, :data:`False` otherwise. 185 | 186 | This property has the opposite effect of the rsync command line 187 | option ``--one-file-system`` because :attr:`multi_fs` defaults to 188 | :data:`False` which means rsync is run with ``--one-file-system``. 189 | You can set :attr:`multi_fs` to :data:`True` to omit 190 | ``--one-file-system`` from the rsync command line. 191 | """ 192 | return False 193 | 194 | @lazy_property(writable=True) 195 | def exclude_list(self): 196 | """ 197 | A list of patterns (strings) that are excluded from the system backup. 198 | 199 | The patterns in :attr:`exclude_list` are passed on to rsync using 200 | the ``--exclude`` option. 201 | """ 202 | return [] 203 | 204 | @lazy_property(writable=True) 205 | def excluded_roots(self): 206 | """ 207 | A list of patterns (strings) that are excluded from the system backup. 208 | 209 | All of the patterns in this list will be rooted to the top of the 210 | filesystem hierarchy when they're given the rsync, to avoid 211 | unintentionally excluding deeply nested directories that happen to 212 | match names in this list. This is done using the ``--filter=-/ 213 | PATTERN`` option. 214 | """ 215 | return [ 216 | '/dev/', 217 | '/home/*/.cache/', 218 | '/media/', 219 | '/mnt/', 220 | '/proc/', 221 | '/run/', 222 | '/sys/', 223 | '/tmp/', 224 | '/var/cache/', 225 | '/var/tmp/', 226 | ] 227 | 228 | @mutable_property 229 | def force(self): 230 | """:data:`True` to run `rsync-system-backup` on unsupported platforms, :data:`False` otherwise.""" 231 | return False 232 | 233 | @mutable_property 234 | def ionice(self): 235 | """ 236 | The I/O scheduling class for rsync (a string or :data:`None`). 237 | 238 | When this property is set ionice_ will be used to set the I/O 239 | scheduling class for rsync. This can be useful to reduce the 240 | impact of backups on the rest of the system. 241 | 242 | The value of this property is expected to be one of 243 | the strings 'idle', 'best-effort' or 'realtime'. 244 | 245 | .. _ionice: https://manpages.debian.org/ionice 246 | """ 247 | 248 | @mutable_property 249 | def mount_point(self): 250 | """The pathname of the mount point to use (a string or :data:`None`).""" 251 | 252 | @property 253 | def mount_point_active(self): 254 | """:data:`True` if :attr:`mount_point` is mounted already, :data:`False` otherwise.""" 255 | return (self.destination_context.test('mountpoint', self.mount_point) 256 | if self.mount_point else False) 257 | 258 | @mutable_property 259 | def notifications_enabled(self): 260 | """ 261 | Whether desktop notifications are used (a boolean). 262 | 263 | By default desktop notifications are enabled when a real backup is 264 | being made but disabled during dry runs. 265 | """ 266 | return not self.dry_run 267 | 268 | @mutable_property 269 | def rotation_scheme(self): 270 | """The rotation scheme for snapshots (a dictionary, defaults to :data:`DEFAULT_ROTATION_SCHEME`).""" 271 | return DEFAULT_ROTATION_SCHEME 272 | 273 | @mutable_property 274 | def snapshot_enabled(self): 275 | """:data:`True` to enable :func:`create_snapshot()`, :data:`False` otherwise.""" 276 | return True 277 | 278 | @mutable_property 279 | def source(self): 280 | """The pathname of the directory to backup (a string, defaults to '/').""" 281 | return '/' 282 | 283 | @lazy_property(writable=True) 284 | def source_context(self): 285 | """ 286 | The execution context of the system that is being backed up (the source). 287 | 288 | This is expected to be an execution context created by 289 | :mod:`executor.contexts`. It defaults to 290 | :class:`executor.contexts.LocalContext`. 291 | """ 292 | return LocalContext() 293 | 294 | @mutable_property 295 | def rotate_enabled(self): 296 | """:data:`True` to enable :func:`rotate_snapshots()`, :data:`False` otherwise.""" 297 | return True 298 | 299 | @mutable_property 300 | def sudo_enabled(self): 301 | """:data:`True` to run ``rsync`` and snapshot creation with superuser privileges, :data:`False` otherwise.""" 302 | return True 303 | 304 | def execute(self): 305 | """ 306 | Execute the requested actions (backup, snapshot and/or rotate). 307 | 308 | The :func:`execute()` method defines the high level control flow 309 | of the backup / snapshot / rotation process according to 310 | the caller's requested configuration: 311 | 312 | 1. When :attr:`backup_enabled` is set :func:`notify_starting()` shows a 313 | desktop notification to give the user a heads up that a system 314 | backup is about to start (because the backup may have a noticeable 315 | impact on system performance). 316 | 317 | 2. When :attr:`crypto_device` is set :func:`unlock_device()` ensures 318 | that the configured encrypted device is unlocked. 319 | 320 | 3. When :attr:`mount_point` is set :func:`mount_filesystem()` ensures 321 | that the configured filesystem is mounted. 322 | 323 | 4. When :attr:`backup_enabled` is set :func:`transfer_changes()` 324 | creates or updates the system backup on :attr:`destination` 325 | using rsync. 326 | 327 | 5. When :attr:`snapshot_enabled` is set :func:`create_snapshot()` 328 | creates a snapshot of the :attr:`destination` directory. 329 | 330 | 6. When :attr:`rotate_enabled` is set :func:`rotate_snapshots()` 331 | rotates snapshots. 332 | 333 | 7. When :attr:`backup_enabled` is set :func:`notify_finished()` shows 334 | a desktop notification to give the user a heads up that the 335 | system backup has finished (or failed). 336 | """ 337 | self.ensure_supported_platform() 338 | try: 339 | # We use a `with' statement to enable cleanup commands that 340 | # are run before this method returns. The unlock_device() 341 | # and mount_filesystem() methods depend on this. 342 | with self.destination_context: 343 | self.execute_helper() 344 | except DestinationContextUnavailable: 345 | # When the destination is an rsync daemon module we can't just 346 | # assume that the same server is also accessible over SSH, so in 347 | # this case no destination context is available. 348 | self.execute_helper() 349 | 350 | def ensure_supported_platform(self): 351 | """ 352 | Make sure we're running on a supported platform. 353 | 354 | :raises: :exc:`.UnsupportedPlatformError` when the output of the 355 | ``uname`` command doesn't include the word 'Linux' and 356 | :attr:`force` is :data:`False`. 357 | 358 | When :attr:`force` is :data:`True` this method logs a warning message 359 | instead of raising an exception. 360 | """ 361 | uname_output = self.source_context.capture('uname', capture=True, check=False, shell=False) 362 | if 'linux' not in uname_output.lower(): 363 | if self.force: 364 | logger.warning(compact(""" 365 | It looks like you aren't running Linux (which is the only 366 | platform supported by rsync-system-backup) however the -f, 367 | --force option was given so I will continue anyway. Please 368 | note that you are on your own if things break! 369 | """)) 370 | else: 371 | raise UnsupportedPlatformError(compact(""" 372 | It looks like you aren't running Linux, which is the only 373 | platform supported by rsync-system-backup! You can use the 374 | -f, --force option to override this sanity check. Please 375 | note that you are on your own if things break. 376 | """)) 377 | 378 | def execute_helper(self): 379 | """Helper for :func:`execute()`.""" 380 | timer = Timer() 381 | actions = [] 382 | if self.crypto_device and not self.crypto_device_available: 383 | msg = "Encrypted filesystem %s isn't available! (the device file %s doesn't exist)" 384 | raise MissingBackupDiskError(msg % (self.crypto_device, self.crypttab_entry.source_device)) 385 | if self.backup_enabled: 386 | self.notify_starting() 387 | self.unlock_device() 388 | try: 389 | self.mount_filesystem() 390 | if self.backup_enabled: 391 | self.transfer_changes() 392 | actions.append('create backup') 393 | if self.snapshot_enabled: 394 | self.create_snapshot() 395 | actions.append('create snapshot') 396 | if self.rotate_enabled: 397 | self.rotate_snapshots() 398 | actions.append('rotate old snapshots') 399 | except Exception: 400 | self.notify_failed(timer) 401 | raise 402 | else: 403 | if self.backup_enabled: 404 | self.notify_finished(timer) 405 | if actions: 406 | logger.info("Took %s to %s.", timer, concatenate(actions)) 407 | 408 | def notify_starting(self): 409 | """Notify the desktop environment that a system backup is starting.""" 410 | if self.notifications_enabled: 411 | body = "Starting dry-run" if self.dry_run else "Starting backup" 412 | notify_desktop(summary="System backups", body=body) 413 | 414 | def notify_finished(self, timer): 415 | """Notify the desktop environment that a system backup has finished.""" 416 | if self.notifications_enabled: 417 | body = "Finished backup in %s." % timer 418 | notify_desktop(summary="System backups", body=body) 419 | 420 | def notify_failed(self, timer): 421 | """Notify the desktop environment that a system backup has failed.""" 422 | if self.notifications_enabled: 423 | body = "Backup failed after %s! Review the system logs for details." % timer 424 | notify_desktop(summary="System backups", body=body, urgency='critical') 425 | 426 | def unlock_device(self): 427 | """ 428 | Automatically unlock the encrypted filesystem to which backups are written. 429 | 430 | :raises: The following exceptions can be raised: 431 | 432 | - :exc:`.DestinationContextUnavailable`, refer 433 | to :attr:`destination_context` for details. 434 | - :exc:`~executor.ExternalCommandFailed` when the 435 | cryptdisks_start_ command reports an error. 436 | 437 | When :attr:`crypto_device` is set this method uses 438 | :func:`~linux_utils.luks.cryptdisks_start()` to unlock the encrypted 439 | filesystem to which backups are written before the backup starts. When 440 | :func:`~linux_utils.luks.cryptdisks_start()` was called before the 441 | backup started, :func:`~linux_utils.luks.cryptdisks_stop()` will be 442 | called when the backup finishes. 443 | 444 | To enable the use of :func:`~linux_utils.luks.cryptdisks_start()` and 445 | :func:`~linux_utils.luks.cryptdisks_stop()` you need to create an 446 | `/etc/crypttab`_ entry that maps your physical device to a symbolic 447 | name. If you want this process to run fully unattended you can 448 | configure a key file in `/etc/crypttab`_, otherwise you will be asked 449 | for the password when the encrypted filesystem is unlocked. 450 | 451 | .. _/etc/crypttab: https://manpages.debian.org/crypttab 452 | .. _cryptdisks_start: https://manpages.debian.org/cryptdisks_start 453 | """ 454 | if self.crypto_device: 455 | if self.crypto_device_unlocked: 456 | logger.info("Encrypted filesystem is already unlocked (%s) ..", self.crypto_device) 457 | else: 458 | cryptdisks_start( 459 | context=self.destination_context, 460 | target=self.crypto_device, 461 | ) 462 | if not self.crypto_device_unlocked: 463 | msg = "Failed to unlock encrypted filesystem! (%s)" 464 | raise FailedToUnlockError(msg % self.crypto_device) 465 | self.destination_context.cleanup( 466 | cryptdisks_stop, 467 | context=self.destination_context, 468 | target=self.crypto_device, 469 | ) 470 | 471 | def mount_filesystem(self): 472 | """ 473 | Automatically mount the filesystem to which backups are written. 474 | 475 | :raises: The following exceptions can be raised: 476 | 477 | - :exc:`.DestinationContextUnavailable`, refer 478 | to :attr:`destination_context` for details. 479 | - :exc:`~executor.ExternalCommandFailed` when 480 | the mount_ command reports an error. 481 | 482 | When :attr:`mount_point` is set this method uses the mount_ command to 483 | mount the filesystem to which backups are written before the backup 484 | starts. When mount_ was called before the backup started, umount_ will 485 | be called when the backup finishes. An entry for the mount point needs 486 | to be defined in `/etc/fstab`_. 487 | 488 | .. _mount: https://manpages.debian.org/mount 489 | .. _umount: https://manpages.debian.org/umount 490 | .. _/etc/fstab: https://manpages.debian.org/fstab 491 | """ 492 | if self.mount_point: 493 | if self.mount_point_active: 494 | logger.info("Filesystem is already mounted (%s) ..", self.mount_point) 495 | else: 496 | logger.info("Mounting filesystem (%s) ..", self.mount_point) 497 | self.destination_context.execute('mount', self.mount_point, sudo=True) 498 | if not self.mount_point_active: 499 | msg = "Failed to mount filesystem! (%s)" 500 | raise FailedToMountError(msg % self.crypto_device) 501 | self.destination_context.cleanup('umount', self.mount_point, sudo=True) 502 | 503 | def transfer_changes(self): 504 | """ 505 | Use rsync to synchronize the files on the local system to the backup destination. 506 | 507 | :raises: :exc:`.InvalidDestinationDirectory` when :attr:`mount_point` 508 | is set and :attr:`destination` is a local directory that is 509 | not located under :attr:`mount_point`. 510 | """ 511 | # Attempt to ensure that the destination directory is located under the 512 | # mount point to prevent the user from shooting themselves in the foot. 513 | if self.mount_point and not self.destination.hostname: 514 | mount_point = os.path.abspath(self.mount_point) 515 | destination = os.path.abspath(self.destination.directory) 516 | common_prefix = os.path.commonprefix([mount_point, destination]) 517 | if os.path.abspath(common_prefix) != mount_point: 518 | msg = "Destination directory (%s) not located under mount point (%s)!" 519 | raise InvalidDestinationDirectory(msg % (destination, mount_point)) 520 | # The following `with' statement enables rsync daemon connections 521 | # tunneled over SSH. For this use case we spawn a local SSH client with 522 | # port forwarding configured, wait for the forwarded port to become 523 | # connected, have rsync connect through the tunnel and shut down the 524 | # SSH client after rsync is finished. 525 | with self.destination: 526 | rsync_command = ['rsync'] 527 | if self.dry_run: 528 | rsync_command.append('--dry-run') 529 | rsync_command.append('--verbose') 530 | # The following rsync options delete files in the backup 531 | # destination that no longer exist on the local system. 532 | # Due to snapshotting this won't cause data loss. 533 | rsync_command.append('--delete') 534 | rsync_command.append('--delete-excluded') 535 | # The following rsync options are intended to preserve 536 | # as much filesystem metadata as possible. 537 | rsync_command.append('--acls') 538 | rsync_command.append('--archive') 539 | rsync_command.append('--hard-links') 540 | rsync_command.append('--numeric-ids') 541 | rsync_command.append('--xattrs') 542 | # The following rsync option avoids including mounted external 543 | # drives like USB sticks in system backups. 544 | if not self.multi_fs: 545 | rsync_command.append('--one-file-system') 546 | # The following rsync options exclude irrelevant directories (to my 547 | # subjective mind) from the system backup. 548 | for pattern in self.excluded_roots: 549 | rsync_command.append('--filter=-/ %s' % pattern) 550 | # The following rsync options allow user defined exclusion. 551 | for pattern in self.exclude_list: 552 | rsync_command.append('--exclude=%s' % pattern) 553 | # Source the backup from the root of the local filesystem 554 | # and make sure the pathname ends in a trailing slash. 555 | rsync_command.append(ensure_trailing_slash(self.source)) 556 | # Target the backup at the configured destination. 557 | rsync_command.append(ensure_trailing_slash(self.destination.expression)) 558 | # Automatically create missing destination directories. 559 | try: 560 | if not self.destination_context.is_directory(self.destination.directory): 561 | logger.info("Creating missing destination directory: %s", self.destination.directory) 562 | self.destination_context.execute('mkdir', '-p', self.destination.directory, tty=False) 563 | except DestinationContextUnavailable: 564 | # Don't fail when the destination doesn't allow for this 565 | # (because its an rsync daemon module). 566 | pass 567 | # Execute the rsync command. 568 | timer = Timer() 569 | logger.info("Creating system backup using rsync ..") 570 | cmd = self.source_context.execute(*rsync_command, **dict( 571 | # Don't raise an exception when rsync exits with 572 | # a nonzero status code. From `man rsync': 573 | # - 23: Partial transfer due to error. 574 | # - 24: Partial transfer due to vanished source files. 575 | # This can be expected on a running system 576 | # without proper filesystem snapshots :-). 577 | check=False, 578 | # Clear $HOME so that rsync ignores ~/.cvsignore. 579 | environment=dict(HOME=''), 580 | # Run rsync under ionice. 581 | ionice=self.ionice, 582 | # Run rsync with superuser privileges so that it has read 583 | # access to all files on the local filesystem? 584 | sudo=self.sudo_enabled, 585 | )) 586 | if cmd.returncode in (0, 23, 24): 587 | logger.info("Took %s to create backup.", timer) 588 | if cmd.returncode != 0: 589 | logger.warning( 590 | "Ignoring `partial transfer' warnings (rsync exited with %i).", 591 | cmd.returncode, 592 | ) 593 | else: 594 | logger.error("Backup failed after %s! (rsync exited with %i)", 595 | timer, cmd.returncode) 596 | raise cmd.error_type(cmd) 597 | 598 | def create_snapshot(self): 599 | """ 600 | Create a snapshot of the destination directory. 601 | 602 | :raises: The following exceptions can be raised: 603 | 604 | - :exc:`.DestinationContextUnavailable`, refer 605 | to :attr:`destination_context` for details. 606 | - :exc:`.ParentDirectoryUnavailable`, refer 607 | to :attr:`.parent_directory` for details. 608 | - :exc:`~executor.ExternalCommandFailed` when 609 | the ``cp`` command reports an error. 610 | """ 611 | # Compose the `cp' command needed to create a snapshot. 612 | snapshot = os.path.join(self.destination.parent_directory, 613 | time.strftime('%Y-%m-%d %H:%M:%S')) 614 | cp_command = [ 615 | 'cp', '--archive', '--link', 616 | self.destination.directory, 617 | snapshot, 618 | ] 619 | # Execute the `cp' command? 620 | if self.dry_run: 621 | logger.info("Snapshot command: %s", quote(cp_command)) 622 | else: 623 | timer = Timer() 624 | logger.info("Creating snapshot: %s", snapshot) 625 | self.destination_context.execute(*cp_command, ionice=self.ionice) 626 | logger.info("Took %s to create snapshot.", timer) 627 | 628 | def rotate_snapshots(self): 629 | """ 630 | Rotate system backup snapshots using :mod:`.rotate_backups`. 631 | 632 | :raises: The following exceptions can be raised: 633 | 634 | - :exc:`.DestinationContextUnavailable`, refer 635 | to :attr:`destination_context` for details. 636 | - :exc:`.ParentDirectoryUnavailable`, refer 637 | to :attr:`.parent_directory` for details. 638 | - Any exceptions raised by :mod:`.rotate_backups`. 639 | 640 | The values of the :attr:`dry_run`, :attr:`ionice` and 641 | :attr:`rotation_scheme` properties are passed on to the 642 | :class:`~rotate_backups.RotateBackups` class. 643 | """ 644 | helper = RotateBackups( 645 | dry_run=self.dry_run, 646 | io_scheduling_class=self.ionice, 647 | rotation_scheme=self.rotation_scheme, 648 | ) 649 | helper.rotate_backups(Location( 650 | context=self.destination_context, 651 | directory=self.destination.parent_directory, 652 | )) 653 | 654 | 655 | def ensure_trailing_slash(expression): 656 | """ 657 | Add a trailing slash to rsync source/destination locations. 658 | 659 | :param expression: The rsync source/destination expression (a string). 660 | :returns: The same expression with exactly one trailing slash. 661 | """ 662 | if expression: 663 | # Strip any existing trailing slashes. 664 | expression = expression.rstrip('/') 665 | # Add exactly one trailing slash. 666 | expression += '/' 667 | return expression 668 | -------------------------------------------------------------------------------- /rsync_system_backup/cli.py: -------------------------------------------------------------------------------- 1 | # rsync-system-backup: Linux system backups powered by rsync. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: August 2, 2019 5 | # URL: https://github.com/xolox/python-rsync-system-backup 6 | 7 | """ 8 | Usage: rsync-system-backup [OPTIONS] [SOURCE] DESTINATION 9 | 10 | Use rsync to create full system backups. 11 | 12 | The required DESTINATION argument specifies the (possibly remote) location 13 | where the backup is stored, in the syntax of rsync's command line interface. 14 | The optional SOURCE argument defaults to '/' which means the complete root 15 | filesystem will be included in the backup (other filesystems are excluded). 16 | 17 | Please use the --dry-run option when getting familiar with this program and 18 | don't remove it until you're confident that you have the right command line, 19 | because using this program in the wrong way can cause data loss (for example 20 | by accidentally swapping the SOURCE and DESTINATION arguments). 21 | 22 | Supported locations include: 23 | 24 | - Local disks (possibly encrypted using LUKS). 25 | - Remote systems that allow SSH connections. 26 | - Remote systems that are running an rsync daemon. 27 | - Connections to rsync daemons tunneled over SSH. 28 | 29 | The backup process consists of several steps: 30 | 31 | 1. First rsync is used to transfer all (relevant) files to a destination 32 | directory (whether on the local system or a remote system). Every time 33 | a backup is made, this same destination directory is updated. 34 | 35 | 2. After the files have been transferred a 'snapshot' of the destination 36 | directory is taken and stored in a directory with a timestamp in its 37 | name. These snapshots are created using 'cp --archive --link'. 38 | 39 | 3. Finally the existing snapshots are rotated to purge old backups 40 | according to a rotation scheme that you can customize. 41 | 42 | Supported options: 43 | 44 | -b, --backup 45 | 46 | Create a backup using rsync but don't create a snapshot and don't rotate 47 | old snapshots unless the --snapshot and/or --rotate options are also given. 48 | 49 | -s, --snapshot 50 | 51 | Create a snapshot of the destination directory but don't create a backup 52 | and don't rotate old snapshots unless the --backup and/or --rotate options 53 | are also given. 54 | 55 | This option can be used to create snapshots of an rsync daemon module using 56 | a 'post-xfer exec' command. If DESTINATION isn't given it defaults to the 57 | value of the environment variable $RSYNC_MODULE_PATH. 58 | 59 | -r, --rotate 60 | 61 | Rotate old snapshots but don't create a backup and snapshot unless the 62 | --backup and/or --snapshot options are also given. 63 | 64 | This option can be used to rotate old snapshots of an rsync daemon module 65 | using a 'post-xfer exec' command. If DESTINATION isn't given it defaults to 66 | the value of the environment variable $RSYNC_MODULE_PATH. 67 | 68 | -m, --mount=DIRECTORY 69 | 70 | Automatically mount the filesystem to which backups are written. 71 | 72 | When this option is given and DIRECTORY isn't already mounted, the 73 | 'mount' command is used to mount the filesystem to which backups are 74 | written before the backup starts. When 'mount' was called before the 75 | backup started, 'umount' will be called when the backup finishes. 76 | 77 | An entry for the mount point needs to be 78 | defined in /etc/fstab for this to work. 79 | 80 | -c, --crypto=NAME 81 | 82 | Automatically unlock the encrypted filesystem to which backups are written. 83 | 84 | When this option is given and the NAME device isn't already unlocked, the 85 | cryptdisks_start command is used to unlock the encrypted filesystem to 86 | which backups are written before the backup starts. When cryptdisks_start 87 | was called before the backup started, cryptdisks_stop will be called 88 | when the backup finishes. 89 | 90 | An entry for the encrypted filesystem needs to be defined in /etc/crypttab 91 | for this to work. If the device of the encrypted filesystem is missing and 92 | rsync-system-backup is being run non-interactively, it will exit gracefully 93 | and not show any desktop notifications. 94 | 95 | If you want the backup process to run fully unattended you can configure a 96 | key file in /etc/crypttab, otherwise you will be asked for the password 97 | each time the encrypted filesystem is unlocked. 98 | 99 | -t, --tunnel=TUNNEL_SPEC 100 | 101 | Connect to an rsync daemon through an SSH tunnel. This provides encryption 102 | for rsync client to daemon connections that are not otherwise encrypted. 103 | The value of TUNNEL_SPEC is expected to be an SSH alias, host name or IP 104 | address. Optionally a username can be prefixed (followed by '@') and/or a 105 | port number can be suffixed (preceded by ':'). 106 | 107 | -i, --ionice=CLASS 108 | 109 | Use the 'ionice' program to set the I/O scheduling class and priority of 110 | the 'rm' invocations used to remove backups. CLASS is expected to be one of 111 | the values 'idle', 'best-effort' or 'realtime'. Refer to the man page of 112 | the 'ionice' program for details about these values. 113 | 114 | -u, --no-sudo 115 | 116 | By default backup and snapshot creation is performed with superuser 117 | privileges, to ensure that all files are readable and filesystem 118 | metadata is preserved. The -u, --no-sudo option disables 119 | the use of 'sudo' during these operations. 120 | 121 | -n, --dry-run 122 | 123 | Don't make any changes, just report what would be done. This doesn't 124 | create a backup or snapshot but it does run rsync with the --dry-run 125 | option. 126 | 127 | --multi-fs 128 | 129 | Allow rsync to cross filesystem boundaries. This option has the opposite 130 | effect of the rsync option --one-file-system because rsync-system-backup 131 | defaults to running rsync with --one-file-system and must be instructed 132 | not to using --multi-fs. 133 | 134 | -x, --exclude=PATTERN 135 | 136 | Selectively exclude certain files from being included in the backup. 137 | Refer to the rsync documentation for allowed PATTERN syntax. Note that 138 | rsync-system-backup always uses the 'rsync --one-file-system' option. 139 | 140 | -f, --force 141 | 142 | By default rsync-system-backup refuses to run on non-Linux systems because 143 | it was designed specifically for use on Linux. The use of the -f, --force 144 | option sidesteps this sanity check. Please note that you are on your own if 145 | things break! 146 | 147 | --disable-notifications 148 | 149 | By default a desktop notification is shown (using notify-send) before the 150 | system backup starts and after the backup finishes. The use of this option 151 | disables the notifications (notify-send will not be called at all). 152 | 153 | -v, --verbose 154 | 155 | Make more noise (increase logging verbosity). Can be repeated. 156 | 157 | -q, --quiet 158 | 159 | Make less noise (decrease logging verbosity). Can be repeated. 160 | 161 | -h, --help 162 | 163 | Show this message and exit. 164 | """ 165 | 166 | # Standard library modules. 167 | import getopt 168 | import logging 169 | import os 170 | import sys 171 | 172 | # External dependencies. 173 | import coloredlogs 174 | from executor import validate_ionice_class 175 | from executor.contexts import create_context 176 | from executor.ssh.client import SecureTunnel 177 | from humanfriendly.terminal import connected_to_terminal, usage, warning 178 | 179 | # Modules included in our package. 180 | from rsync_system_backup import RsyncSystemBackup 181 | from rsync_system_backup.destinations import Destination, RSYNCD_PORT 182 | from rsync_system_backup.exceptions import MissingBackupDiskError, RsyncSystemBackupError 183 | 184 | # Public identifiers that require documentation. 185 | __all__ = ( 186 | 'enable_explicit_action', 187 | 'logger', 188 | 'main', 189 | ) 190 | 191 | # Initialize a logger. 192 | logger = logging.getLogger(__name__) 193 | 194 | 195 | def main(): 196 | """Command line interface for the ``rsync-system-backup`` program.""" 197 | # Initialize logging to the terminal and system log. 198 | coloredlogs.install(syslog=True) 199 | # Parse the command line arguments. 200 | context_opts = dict() 201 | program_opts = dict() 202 | dest_opts = dict() 203 | try: 204 | options, arguments = getopt.gnu_getopt(sys.argv[1:], 'bsrm:c:t:i:unx:fvqh', [ 205 | 'backup', 'snapshot', 'rotate', 'mount=', 'crypto=', 'tunnel=', 206 | 'ionice=', 'no-sudo', 'dry-run', 'multi-fs', 'exclude=', 'force', 207 | 'disable-notifications', 'verbose', 'quiet', 'help', 208 | ]) 209 | for option, value in options: 210 | if option in ('-b', '--backup'): 211 | enable_explicit_action(program_opts, 'backup_enabled') 212 | elif option in ('-s', '--snapshot'): 213 | enable_explicit_action(program_opts, 'snapshot_enabled') 214 | elif option in ('-r', '--rotate'): 215 | enable_explicit_action(program_opts, 'rotate_enabled') 216 | elif option in ('-m', '--mount'): 217 | program_opts['mount_point'] = value 218 | elif option in ('-c', '--crypto'): 219 | program_opts['crypto_device'] = value 220 | elif option in ('-t', '--tunnel'): 221 | ssh_user, _, value = value.rpartition('@') 222 | ssh_alias, _, port_number = value.partition(':') 223 | tunnel_opts = dict( 224 | ssh_alias=ssh_alias, 225 | ssh_user=ssh_user, 226 | # The port number of the rsync daemon. 227 | remote_port=RSYNCD_PORT, 228 | ) 229 | if port_number: 230 | # The port number of the SSH server. 231 | tunnel_opts['port'] = int(port_number) 232 | dest_opts['ssh_tunnel'] = SecureTunnel(**tunnel_opts) 233 | elif option in ('-i', '--ionice'): 234 | value = value.lower().strip() 235 | validate_ionice_class(value) 236 | program_opts['ionice'] = value 237 | elif option in ('-u', '--no-sudo'): 238 | program_opts['sudo_enabled'] = False 239 | elif option in ('-n', '--dry-run'): 240 | logger.info("Performing a dry run (because of %s option) ..", option) 241 | program_opts['dry_run'] = True 242 | elif option in ('-f', '--force'): 243 | program_opts['force'] = True 244 | elif option in ('-x', '--exclude'): 245 | program_opts.setdefault('exclude_list', []) 246 | program_opts['exclude_list'].append(value) 247 | elif option == '--multi-fs': 248 | program_opts['multi_fs'] = True 249 | elif option == '--disable-notifications': 250 | program_opts['notifications_enabled'] = False 251 | elif option in ('-v', '--verbose'): 252 | coloredlogs.increase_verbosity() 253 | elif option in ('-q', '--quiet'): 254 | coloredlogs.decrease_verbosity() 255 | elif option in ('-h', '--help'): 256 | usage(__doc__) 257 | return 258 | else: 259 | raise Exception("Unhandled option! (programming error)") 260 | if len(arguments) > 2: 261 | msg = "Expected one or two positional arguments! (got %i)" 262 | raise Exception(msg % len(arguments)) 263 | if len(arguments) == 2: 264 | # Get the source from the first of two arguments. 265 | program_opts['source'] = arguments.pop(0) 266 | if arguments: 267 | # Get the destination from the second (or only) argument. 268 | dest_opts['expression'] = arguments[0] 269 | program_opts['destination'] = Destination(**dest_opts) 270 | elif not os.environ.get('RSYNC_MODULE_PATH'): 271 | # Show a usage message when no destination is given. 272 | usage(__doc__) 273 | return 274 | except Exception as e: 275 | warning("Error: %s", e) 276 | sys.exit(1) 277 | try: 278 | # Inject the source context into the program options. 279 | program_opts['source_context'] = create_context(**context_opts) 280 | # Initialize the program with the command line 281 | # options and execute the requested action(s). 282 | RsyncSystemBackup(**program_opts).execute() 283 | except Exception as e: 284 | if isinstance(e, RsyncSystemBackupError): 285 | # Special handling when the backup disk isn't available. 286 | if isinstance(e, MissingBackupDiskError): 287 | # Check if we're connected to a terminal to decide whether the 288 | # error should be propagated or silenced, the idea being that 289 | # rsync-system-backup should keep quiet when it's being run 290 | # from cron and the backup disk isn't available. 291 | if not connected_to_terminal(): 292 | logger.info("Skipping backup: %s", e) 293 | sys.exit(0) 294 | # Known problems shouldn't produce 295 | # an intimidating traceback to users. 296 | logger.error("Aborting due to error: %s", e) 297 | else: 298 | # Unhandled exceptions do get a traceback, 299 | # because it may help fix programming errors. 300 | logger.exception("Aborting due to unhandled exception!") 301 | sys.exit(1) 302 | 303 | 304 | def enable_explicit_action(options, explicit_action): 305 | """ 306 | Explicitly enable an action and disable other implicit actions. 307 | 308 | :param options: A dictionary of options. 309 | :param explicit_action: The action to enable (one of the strings 310 | 'backup_enabled', 'snapshot_enabled', 311 | 'rotate_enabled'). 312 | """ 313 | options[explicit_action] = True 314 | for implicit_action in 'backup_enabled', 'snapshot_enabled', 'rotate_enabled': 315 | if implicit_action != explicit_action: 316 | options.setdefault(implicit_action, False) 317 | -------------------------------------------------------------------------------- /rsync_system_backup/destinations.py: -------------------------------------------------------------------------------- 1 | # rsync-system-backup: Linux system backups powered by rsync. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: August 2, 2019 5 | # URL: https://github.com/xolox/python-rsync-system-backup 6 | 7 | """Parsing of rsync destination syntax (and then some).""" 8 | 9 | # Standard library modules. 10 | import logging 11 | import os 12 | import re 13 | 14 | # External dependencies. 15 | from humanfriendly import compact 16 | from property_manager import ( 17 | PropertyManager, 18 | mutable_property, 19 | required_property, 20 | set_property, 21 | ) 22 | 23 | # Modules included in our package. 24 | from rsync_system_backup.exceptions import ( 25 | InvalidDestinationError, 26 | ParentDirectoryUnavailable, 27 | ) 28 | 29 | RSYNCD_PORT = 873 30 | """ 31 | The default port of the `rsync daemon`_ (an integer). 32 | 33 | .. _rsync daemon: https://manpages.debian.org/rsyncd.conf 34 | """ 35 | 36 | LOCAL_DESTINATION = re.compile('^(?P.+)$') 37 | """ 38 | A compiled regular expression pattern to parse local destinations, 39 | used as a fall back because it matches any nonempty string. 40 | """ 41 | 42 | SSH_DESTINATION = re.compile(''' 43 | ^ ( (?P [^@]+ ) @ )? # optional username 44 | (?P [^:]+ ) : # mandatory host name 45 | (?P .* ) # optional pathname 46 | ''', re.VERBOSE) 47 | """ 48 | A compiled regular expression pattern to parse remote destinations 49 | of the form ``[USER@]HOST:DEST`` (using an SSH connection). 50 | """ 51 | 52 | SIMPLE_DAEMON_DESTINATION = re.compile(''' 53 | ^ ( (?P [^@]+ ) @ )? # optional username 54 | (?P [^:]+ ) :: # mandatory host name 55 | (?P [^/]+ ) # mandatory module name 56 | ( / (?P .* ) )? $ # optional pathname (without leading slash) 57 | ''', re.VERBOSE) 58 | """ 59 | A compiled regular expression pattern to parse remote destinations of the 60 | form ``[USER@]HOST::MODULE[/DIRECTORY]`` (using an rsync daemon connection). 61 | """ 62 | 63 | ADVANCED_DAEMON_DESTINATION = re.compile(r''' 64 | ^ rsync:// # static prefix 65 | ( (?P[^@]+) @ )? # optional username 66 | (?P [^:/]+ ) # mandatory host name 67 | ( : (?P \d+ ) )? # optional port number 68 | / (?P [^/]+ ) # mandatory module name 69 | ( / (?P .* ) )? $ # optional pathname (without leading slash) 70 | ''', re.VERBOSE) 71 | """ 72 | A compiled regular expression pattern to parse remote destinations of the form 73 | ``rsync://[USER@]HOST[:PORT]/MODULE[/DIRECTORY]`` (using an rsync daemon 74 | connection). 75 | """ 76 | 77 | DESTINATION_PATTERNS = [ 78 | ADVANCED_DAEMON_DESTINATION, 79 | SIMPLE_DAEMON_DESTINATION, 80 | SSH_DESTINATION, 81 | LOCAL_DESTINATION, 82 | ] 83 | """ 84 | A list of compiled regular expression patterns to match destination 85 | expressions. The patterns are ordered by decreasing specificity. 86 | """ 87 | 88 | 89 | # Public identifiers that require documentation. 90 | __all__ = ( 91 | 'logger', 92 | 'RSYNCD_PORT', 93 | 'LOCAL_DESTINATION', 94 | 'SSH_DESTINATION', 95 | 'SIMPLE_DAEMON_DESTINATION', 96 | 'ADVANCED_DAEMON_DESTINATION', 97 | 'DESTINATION_PATTERNS', 98 | 'Destination', 99 | ) 100 | 101 | # Initialize a logger for this module. 102 | logger = logging.getLogger(__name__) 103 | 104 | 105 | class Destination(PropertyManager): 106 | 107 | """ 108 | The :class:`Destination` class represents a location where backups are stored. 109 | 110 | The :attr:`expression` property is a required property whose value is 111 | parsed to populate the values of the :attr:`username`, :attr:`hostname`, 112 | :attr:`port_number`, :attr:`module` and :attr:`directory` properties. 113 | 114 | When you read the value of the :attr:`expression` property you get back a 115 | computed value based on the values of the previously mentioned properties. 116 | This makes it possible to manipulate the destination before passing it on 117 | to rsync. 118 | """ 119 | 120 | @required_property 121 | def expression(self): 122 | """ 123 | The destination in rsync's command line syntax (a string). 124 | 125 | :raises: :exc:`.InvalidDestinationError` when you try to set 126 | this property to a value that cannot be parsed. 127 | """ 128 | if not (self.hostname or self.directory): 129 | # This is a bit tricky: Returning None here ensures that a 130 | # TypeError will be raised when a Destination object is 131 | # created without specifying a value for `expression'. 132 | return None 133 | value = 'rsync://' if self.module else '' 134 | if self.hostname: 135 | if self.username: 136 | value += self.username + '@' 137 | value += self.hostname 138 | if self.module: 139 | if self.port_number: 140 | value += ':%s' % self.port_number 141 | value += '/' + self.module 142 | else: 143 | value += ':' 144 | if self.directory: 145 | value += self.directory 146 | return value 147 | 148 | @expression.setter 149 | def expression(self, value): 150 | """Automatically parse expression strings.""" 151 | for pattern in DESTINATION_PATTERNS: 152 | match = pattern.match(value) 153 | if match: 154 | captures = match.groupdict() 155 | non_empty = dict((n, c) for n, c in captures.items() if c) 156 | self.set_properties(**non_empty) 157 | break 158 | else: 159 | msg = "Failed to parse expression! (%s)" 160 | raise InvalidDestinationError(msg % value) 161 | 162 | @mutable_property 163 | def directory(self): 164 | """The pathname of the directory where the backup should be written (a string).""" 165 | return '' 166 | 167 | @mutable_property 168 | def hostname(self): 169 | """The host name or IP address of a remote system (a string).""" 170 | return '' 171 | 172 | @mutable_property 173 | def module(self): 174 | """The name of a module exported by an `rsync daemon`_ (a string).""" 175 | return '' 176 | 177 | @mutable_property 178 | def parent_directory(self): 179 | """ 180 | The pathname of the parent directory of the backup directory (a string). 181 | 182 | :raises: :exc:`.ParentDirectoryUnavailable` when the parent directory 183 | can't be determined because :attr:`directory` is empty or '/'. 184 | """ 185 | directory = os.path.dirname(self.directory.rstrip('/')) 186 | if not directory: 187 | raise ParentDirectoryUnavailable(compact(""" 188 | Failed to determine the parent directory of the destination 189 | directory! This makes it impossible to create and rotate 190 | snapshots for the destination {dest}. 191 | """, dest=self.expression)) 192 | return directory 193 | 194 | @mutable_property 195 | def port_number(self): 196 | """ 197 | The port number of a remote `rsync daemon`_ (a number). 198 | 199 | When :attr:`ssh_tunnel` is set the value of :attr:`port_number` 200 | defaults to :attr:`executor.ssh.client.SecureTunnel.local_port`, 201 | otherwise it defaults to :data:`RSYNCD_PORT`. 202 | """ 203 | return self.ssh_tunnel.local_port if self.ssh_tunnel is not None else RSYNCD_PORT 204 | 205 | @port_number.setter 206 | def port_number(self, value): 207 | """Automatically coerce port numbers to integers.""" 208 | set_property(self, 'port_number', int(value)) 209 | 210 | @mutable_property 211 | def ssh_tunnel(self): 212 | """A :class:`~executor.ssh.client.SecureTunnel` object or :data:`None` (defaults to :data:`None`).""" 213 | 214 | @mutable_property 215 | def username(self): 216 | """The username for connecting to a remote system (a string).""" 217 | return '' 218 | 219 | def __enter__(self): 220 | """Automatically open :attr:`ssh_tunnel` when required.""" 221 | if self.ssh_tunnel: 222 | self.ssh_tunnel.__enter__() 223 | return self 224 | 225 | def __exit__(self, exc_type=None, exc_value=None, traceback=None): 226 | """Automatically close :attr:`ssh_tunnel` when required""" 227 | if self.ssh_tunnel: 228 | self.ssh_tunnel.__exit__(exc_type, exc_value, traceback) 229 | -------------------------------------------------------------------------------- /rsync_system_backup/exceptions.py: -------------------------------------------------------------------------------- 1 | # rsync-system-backup: Linux system backups powered by rsync. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: July 11, 2017 5 | # URL: https://github.com/xolox/python-rsync-system-backup 6 | 7 | """Custom exceptions used by rsync-system-backup.""" 8 | 9 | 10 | class RsyncSystemBackupError(Exception): 11 | 12 | """Base exception for custom exceptions raised by rsync-system-backup.""" 13 | 14 | 15 | class UnsupportedPlatformError(RsyncSystemBackupError): 16 | 17 | """Raised when an unsupported (non-Linux) platform is detected.""" 18 | 19 | 20 | class InvalidDestinationError(RsyncSystemBackupError): 21 | 22 | """Raised when the given destination expression can't be parsed.""" 23 | 24 | 25 | class MissingBackupDiskError(RsyncSystemBackupError): 26 | 27 | """Raised when the encrypted filesystem isn't available.""" 28 | 29 | 30 | class FailedToUnlockError(RsyncSystemBackupError): 31 | 32 | """Raised when cryptdisks_start_ fails to unlock the encrypted device.""" 33 | 34 | 35 | class FailedToMountError(RsyncSystemBackupError): 36 | 37 | """Raised when mount_ fails to mount the backup destination.""" 38 | 39 | 40 | class DestinationContextUnavailable(RsyncSystemBackupError): 41 | 42 | """Raised when snapshot creation and rotation are disabled because we're connected to an rsync daemon.""" 43 | 44 | 45 | class ParentDirectoryUnavailable(RsyncSystemBackupError): 46 | 47 | """Raised when the parent directory of the backup directory cannot be determined.""" 48 | 49 | 50 | class InvalidDestinationDirectory(RsyncSystemBackupError): 51 | 52 | """Raised when the backup directory isn't located inside the given mount point.""" 53 | -------------------------------------------------------------------------------- /rsync_system_backup/tests.py: -------------------------------------------------------------------------------- 1 | # Test suite for the `rsync-system-backup' Python package. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: April 30, 2018 5 | # URL: https://github.com/xolox/python-rsync-system-backup 6 | 7 | """Test suite for the `rsync-system-backup` package.""" 8 | 9 | # Standard library modules. 10 | import contextlib 11 | import logging 12 | import os 13 | 14 | # External dependencies. 15 | from executor import ExternalCommandFailed, execute 16 | from humanfriendly import Timer 17 | from humanfriendly.testing import MockedProgram, TemporaryDirectory, TestCase, run_cli 18 | from linux_utils.luks import ( 19 | create_encrypted_filesystem, 20 | create_image_file, 21 | cryptdisks_start, 22 | cryptdisks_stop, 23 | TemporaryKeyFile, 24 | ) 25 | from mock import MagicMock 26 | from rotate_backups import RotateBackups 27 | 28 | # The module we're testing. 29 | from rsync_system_backup import DEFAULT_ROTATION_SCHEME, RsyncSystemBackup 30 | from rsync_system_backup.cli import main 31 | from rsync_system_backup.destinations import Destination 32 | from rsync_system_backup.exceptions import ( 33 | DestinationContextUnavailable, 34 | FailedToMountError, 35 | InvalidDestinationDirectory, 36 | InvalidDestinationError, 37 | MissingBackupDiskError, 38 | UnsupportedPlatformError, 39 | ) 40 | 41 | # Initialize a logger for this module. 42 | logger = logging.getLogger(__name__) 43 | 44 | # Configuration defaults. 45 | CRYPTO_NAME = 'rsync-system-backup' 46 | FILESYSTEM_DEVICE = '/dev/mapper/%s' % CRYPTO_NAME 47 | IMAGE_FILE = '/tmp/rsync-system-backup.img' 48 | KEY_FILE = '/tmp/rsync-system-backup.key' 49 | MOUNT_POINT = '/mnt/rsync-system-backup' 50 | 51 | 52 | class RsyncSystemBackupsTestCase(TestCase): 53 | 54 | """:mod:`unittest` compatible container for `rsync-system-backup` tests.""" 55 | 56 | def test_usage(self): 57 | """Test the usage message.""" 58 | # Make sure the usage message is shown when no arguments 59 | # are given and when the -h or --help option is given. 60 | for options in [], ['-h'], ['--help']: 61 | exit_code, output = run_cli(main, *options) 62 | assert "Usage:" in output 63 | 64 | def test_invalid_arguments(self): 65 | """Test the handling of incorrect command line arguments.""" 66 | # More than two arguments should report an error. 67 | exit_code, output = run_cli(main, 'a', 'b', 'c', merged=True) 68 | assert exit_code != 0 69 | assert "Error" in output 70 | # Invalid `ionice' values should report an error. 71 | exit_code, output = run_cli(main, '--ionice=foo', merged=True) 72 | assert exit_code != 0 73 | assert "Error" in output 74 | 75 | def test_destination_parsing(self): 76 | """Test the parsing of rsync destinations.""" 77 | # Our first test case is trivial: The pathname of a local directory. 78 | dest = Destination(expression='/mnt/backups/laptop') 79 | assert dest.directory == '/mnt/backups/laptop' 80 | assert not dest.hostname 81 | assert not dest.username 82 | assert not dest.module 83 | # Our second test case involves an SSH connection. 84 | dest = Destination(expression='backup-server:/backups/laptop') 85 | assert dest.hostname == 'backup-server' 86 | assert dest.directory == '/backups/laptop' 87 | assert not dest.username 88 | assert not dest.module 89 | # Our third test case specifies the remote username for SSH. 90 | dest = Destination(expression='backup-user@backup-server:/backups/laptop') 91 | assert dest.hostname == 'backup-server' 92 | assert dest.username == 'backup-user' 93 | assert dest.directory == '/backups/laptop' 94 | assert not dest.module 95 | # Our fourth test case involves the root of an rsync daemon module. 96 | dest = Destination(expression='backup-user@backup-server::laptop_backups') 97 | assert dest.hostname == 'backup-server' 98 | assert dest.username == 'backup-user' 99 | assert dest.module == 'laptop_backups' 100 | assert not dest.directory 101 | # Our fourth test case concerns the alternative syntax for rsync daemon modules. 102 | dest = Destination(expression='rsync://backup-user@backup-server:12345/laptop_backups/some-directory') 103 | assert dest.hostname == 'backup-server' 104 | assert dest.port_number == 12345 105 | assert dest.username == 'backup-user' 106 | assert dest.module == 'laptop_backups' 107 | assert dest.directory == 'some-directory' 108 | # Finally we will also check that the intended exception types are 109 | # raised when no valid destination is given. 110 | self.assertRaises(TypeError, Destination) 111 | self.assertRaises(InvalidDestinationError, Destination, expression='') 112 | 113 | def test_rsync_module_path_as_destination(self): 114 | """Test that destination defaults to ``$RSYNC_MODULE_PATH``.""" 115 | with TemporaryDirectory() as temporary_directory: 116 | try: 117 | os.environ['RSYNC_MODULE_PATH'] = temporary_directory 118 | program = RsyncSystemBackup() 119 | assert program.destination.directory == temporary_directory 120 | assert not program.destination.hostname 121 | assert not program.destination.username 122 | assert not program.destination.module 123 | finally: 124 | os.environ.pop('RSYNC_MODULE_PATH') 125 | 126 | def test_destination_context(self): 127 | """Test destination context creation.""" 128 | # Make sure DestinationContextUnavailable is raised when the 129 | # destination is an rsync daemon module. 130 | program = RsyncSystemBackup(destination='server::backups/system') 131 | self.assertRaises(DestinationContextUnavailable, lambda: program.destination_context) 132 | # Make sure the SSH alias and user are copied from the destination 133 | # expression to the destination context. 134 | program = RsyncSystemBackup(destination='backup-user@backup-server:backups/system') 135 | assert program.destination_context.ssh_alias == 'backup-server' 136 | assert program.destination_context.ssh_user == 'backup-user' 137 | 138 | def test_notifications(self): 139 | """Test the desktop notification functionality.""" 140 | timer = Timer() 141 | program = RsyncSystemBackup(destination='/backups/system') 142 | # The happy path. 143 | with MockedProgram('notify-send', returncode=0): 144 | program.notify_starting() 145 | program.notify_finished(timer) 146 | program.notify_failed(timer) 147 | # The sad path (should not raise exceptions). 148 | with MockedProgram('notify-send', returncode=1): 149 | program.notify_starting() 150 | program.notify_finished(timer) 151 | program.notify_failed(timer) 152 | 153 | def test_simple_backup(self): 154 | """Test a backup of an alternative source directory to a local destination.""" 155 | with TemporaryDirectory() as temporary_directory: 156 | source = os.path.join(temporary_directory, 'source') 157 | destination = os.path.join(temporary_directory, 'destination') 158 | latest_directory = os.path.join(destination, 'latest') 159 | # Create a source for testing. 160 | self.create_source(source) 161 | # Run the program through the command line interface. 162 | exit_code, output = run_cli( 163 | main, '--no-sudo', '--ionice=idle', '--disable-notifications', 164 | source, latest_directory, 165 | ) 166 | assert exit_code == 0 167 | # Make sure the backup was created. 168 | self.verify_destination(latest_directory) 169 | # Make sure a snapshot was created. 170 | assert len(find_snapshots(destination)) == 1 171 | 172 | def test_dry_run(self): 173 | """Test that ``rsync-system-backup --dry-run ...`` works as intended.""" 174 | with TemporaryDirectory() as temporary_directory: 175 | source = os.path.join(temporary_directory, 'source') 176 | destination = os.path.join(temporary_directory, 'destination') 177 | latest_directory = os.path.join(destination, 'latest') 178 | os.makedirs(latest_directory) 179 | # Create a source for testing. 180 | self.create_source(source) 181 | # Run the program through the command line interface. 182 | exit_code, output = run_cli( 183 | main, '--dry-run', '--no-sudo', 184 | source, latest_directory, 185 | ) 186 | assert exit_code == 0 187 | # Make sure no backup was created. 188 | assert len(os.listdir(latest_directory)) == 0 189 | # Make sure no snapshot was created. 190 | assert len(find_snapshots(destination)) == 0 191 | 192 | def test_backup_only(self): 193 | """Test that ``rsync-system-backup --backup`` works as intended.""" 194 | # Check that by default a backup is performed and a snapshot is created. 195 | with TemporaryDirectory() as temporary_directory: 196 | source = os.path.join(temporary_directory, 'source') 197 | destination = os.path.join(temporary_directory, 'destination') 198 | latest_directory = os.path.join(destination, 'latest') 199 | # Create a source for testing. 200 | self.create_source(source) 201 | # Run the program through the command line interface. 202 | exit_code, output = run_cli( 203 | main, '--backup', '--no-sudo', '--disable-notifications', 204 | source, latest_directory, 205 | ) 206 | assert exit_code == 0 207 | # Make sure the backup was created. 208 | self.verify_destination(latest_directory) 209 | # Make sure no snapshot was created. 210 | assert len(find_snapshots(destination)) == 0 211 | 212 | def test_encrypted_backup(self): 213 | """ 214 | Test a backup to an encrypted filesystem. 215 | 216 | To make this test work you need to make the following additions 217 | to system files (and create ``/mnt/rsync-system-backup``): 218 | 219 | .. code-block:: sh 220 | 221 | $ grep rsync-system-backup /etc/fstab 222 | /dev/mapper/rsync-system-backup /mnt/rsync-system-backup ext4 noauto 0 0 223 | 224 | $ grep rsync-system-backup /etc/crypttab 225 | rsync-system-backup /tmp/rsync-system-backup.img /tmp/rsync-system-backup.key luks,noauto 226 | 227 | $ sudo cat /etc/sudoers.d/rsync-system-backup 228 | peter ALL=NOPASSWD:/usr/sbin/cryptdisks_start rsync-system-backup 229 | peter ALL=NOPASSWD:/usr/sbin/cryptdisks_stop rsync-system-backup 230 | peter ALL=NOPASSWD:/bin/mount /mnt/rsync-system-backup 231 | peter ALL=NOPASSWD:/bin/umount /mnt/rsync-system-backup 232 | peter ALL=NOPASSWD:/sbin/mkfs.ext4 /dev/mapper/rsync-system-backup 233 | peter ALL=NOPASSWD:/usr/bin/test -e /dev/mapper/rsync-system-backup 234 | peter ALL=NOPASSWD:/bin/mountpoint /mnt/rsync-system-backup 235 | peter ALL=NOPASSWD:/usr/bin/test -d /mnt/rsync-system-backup/latest 236 | peter ALL=NOPASSWD:/bin/mkdir -p /mnt/rsync-system-backup/latest 237 | peter ALL=NOPASSWD:/usr/bin/rsync * /tmp/* /mnt/rsync-system-backup/latest/ 238 | peter ALL=NOPASSWD:/bin/cp --archive --link /mnt/rsync-system-backup/latest /mnt/rsync-system-backup/* 239 | peter ALL=NOPASSWD:/usr/bin/test -d /mnt/rsync-system-backup 240 | peter ALL=NOPASSWD:/usr/bin/test -r /mnt/rsync-system-backup 241 | peter ALL=NOPASSWD:/usr/bin/find /mnt/rsync-system-backup * 242 | peter ALL=NOPASSWD:/usr/bin/test -w /mnt/rsync-system-backup 243 | peter ALL=NOPASSWD:/bin/rm --recursive /mnt/rsync-system-backup/latest 244 | 245 | Of course you should change ``/etc/sudoers.d/rsync-system-backup`` to 246 | replace ``peter`` with your actual username :-). 247 | """ 248 | if not os.path.isdir(MOUNT_POINT): 249 | return self.skipTest("Skipping test because %s doesn't exist!", MOUNT_POINT) 250 | with TemporaryDirectory() as source: 251 | destination = os.path.join(MOUNT_POINT, 'latest') 252 | with prepared_image_file(): 253 | # Create a source for testing. 254 | self.create_source(source) 255 | # Run the program through the command line interface. 256 | self.create_encrypted_backup(source, destination) 257 | # Unlock the encrypted image file. 258 | with unlocked_device(CRYPTO_NAME): 259 | # Mount the encrypted filesystem. 260 | with active_mountpoint(MOUNT_POINT): 261 | # Verify that the backup was successful. 262 | self.verify_destination(destination) 263 | # Invoke rsync-system-backup using the same command line 264 | # arguments, but this time while the encrypted device is 265 | # already unlocked and the filesystem is already mounted. 266 | self.create_encrypted_backup(source, destination) 267 | # Verify that the backup was successful. 268 | self.verify_destination(destination) 269 | # Invoke rsync-system-backup using the same command line 270 | # arguments, but this time while the encrypted device is 271 | # already unlocked although the filesystem isn't mounted. 272 | self.create_encrypted_backup(source, destination) 273 | # Verify that the backup was successful. 274 | with active_mountpoint(MOUNT_POINT): 275 | self.verify_destination(destination) 276 | 277 | def create_encrypted_backup(self, source, destination): 278 | """Create a backup to an encrypted device using the command line interface.""" 279 | # Wipe an existing backup (if any). 280 | if os.path.isdir(destination): 281 | execute('rm', '--recursive', destination, sudo=True) 282 | # Create a new backup. 283 | exit_code, output = run_cli( 284 | main, '--crypto=%s' % CRYPTO_NAME, 285 | '--mount=%s' % MOUNT_POINT, 286 | '--disable-notifications', 287 | # We skip snapshot creation and rotation to minimize the number 288 | # of commands required in /etc/sudoers.d/rsync-system-backup. 289 | '--backup', 290 | source, destination, 291 | ) 292 | assert exit_code == 0 293 | 294 | def test_missing_crypto_device(self): 295 | """Test that MissingBackupDiskError is raised as expected.""" 296 | if not os.path.isdir(MOUNT_POINT): 297 | return self.skipTest("Skipping test because %s doesn't exist!", MOUNT_POINT) 298 | # Make sure the image file doesn't exist. 299 | if os.path.exists(IMAGE_FILE): 300 | os.unlink(IMAGE_FILE) 301 | # Ask rsync-system-backup to use the encrypted filesystem on the image 302 | # file anyway, because we know it will fail and that's exactly what 303 | # we're interested in :-). 304 | program = RsyncSystemBackup( 305 | crypto_device=CRYPTO_NAME, 306 | destination=os.path.join(MOUNT_POINT, 'latest'), 307 | mount_point=MOUNT_POINT, 308 | notifications_enabled=False, 309 | ) 310 | self.assertRaises(MissingBackupDiskError, program.execute) 311 | 312 | def test_mount_failure(self): 313 | """Test that FailedToMountError is raised as expected.""" 314 | if not os.path.isdir(MOUNT_POINT): 315 | return self.skipTest("Skipping test because %s doesn't exist!", MOUNT_POINT) 316 | with prepared_image_file(create_filesystem=False): 317 | program = RsyncSystemBackup( 318 | crypto_device=CRYPTO_NAME, 319 | destination=os.path.join(MOUNT_POINT, 'latest'), 320 | mount_point=MOUNT_POINT, 321 | notifications_enabled=False, 322 | ) 323 | # When `mount' fails it should exit with a nonzero exit code, 324 | # thereby causing executor to raise an ExternalCommandFailed 325 | # exception that obscures the FailedToMountError exception that 326 | # we're interested in. The check=False option enables our 327 | # `last resort error handling' code path to be reached. 328 | program.destination_context.options['check'] = False 329 | self.assertRaises(FailedToMountError, program.execute) 330 | 331 | def test_invalid_destination_directory(self): 332 | """Test that InvalidDestinationDirectory is raised as expected.""" 333 | if not os.path.isdir(MOUNT_POINT): 334 | return self.skipTest("Skipping test because %s doesn't exist!", MOUNT_POINT) 335 | with prepared_image_file(): 336 | program = RsyncSystemBackup( 337 | crypto_device=CRYPTO_NAME, 338 | destination='/some/random/directory', 339 | mount_point=MOUNT_POINT, 340 | notifications_enabled=False, 341 | ) 342 | self.assertRaises(InvalidDestinationDirectory, program.transfer_changes) 343 | 344 | def test_unsupported_platform_error(self): 345 | """Test that UnsupportedPlatformError is raised as expected.""" 346 | with MockedProgram('uname'): 347 | program = RsyncSystemBackup(destination='/some/random/directory') 348 | self.assertRaises(UnsupportedPlatformError, program.execute) 349 | 350 | def test_unsupported_platform_with_force(self): 351 | """Test that UnsupportedPlatformError is raised as expected.""" 352 | with MockedProgram('uname'): 353 | program = RsyncSystemBackup(destination='/some/random/directory', force=True) 354 | # Avoid making an actual backup. 355 | program.execute_helper = MagicMock() 356 | program.execute() 357 | assert program.execute_helper.called 358 | 359 | def test_backup_failure(self): 360 | """Test that an exception is raised when ``rsync`` fails.""" 361 | program = RsyncSystemBackup( 362 | destination='0.0.0.0::module/directory', 363 | notifications_enabled=False, 364 | sudo_enabled=False, 365 | ) 366 | self.assertRaises(ExternalCommandFailed, program.execute) 367 | 368 | def test_exclude_list(self): 369 | """Test that ``rsync-system-backup --exclude`` works as intended.""" 370 | with TemporaryDirectory() as temporary_directory: 371 | source = os.path.join(temporary_directory, 'source') 372 | destination = os.path.join(temporary_directory, 'destination') 373 | latest_directory = os.path.join(destination, 'latest') 374 | # Create a source directory with two files. 375 | os.makedirs(source) 376 | with open(os.path.join(source, 'included.txt'), 'w') as handle: 377 | handle.write("This file should be included.\n") 378 | with open(os.path.join(source, 'excluded.txt'), 'w') as handle: 379 | handle.write("This file should be excluded.\n") 380 | # Run the program through the command line interface. 381 | exit_code, output = run_cli( 382 | main, '--backup', '--exclude=excluded.txt', '--no-sudo', '--disable-notifications', 383 | source, latest_directory, 384 | ) 385 | assert exit_code == 0 386 | # Make sure one of the files was copied and the other wasn't. 387 | assert os.path.isfile(os.path.join(latest_directory, 'included.txt')) 388 | assert not os.path.exists(os.path.join(latest_directory, 'excluded.txt')) 389 | 390 | def create_source(self, source): 391 | """Create a source directory for testing backups.""" 392 | if not os.path.isdir(source): 393 | os.makedirs(source) 394 | # Create a text file in the source directory. 395 | text_file = os.path.join(source, 'notes.txt') 396 | with open(text_file, 'w') as handle: 397 | handle.write("This file should be included in the backup.\n") 398 | # Create a subdirectory in the source directory. 399 | subdirectory = os.path.join(source, 'subdirectory') 400 | os.mkdir(subdirectory) 401 | # Create a symbolic link in the subdirectory. 402 | symlink = os.path.join(subdirectory, 'symbolic-link') 403 | os.symlink('../include-me.txt', symlink) 404 | 405 | def verify_destination(self, destination): 406 | """Verify the contents of a destination directory.""" 407 | # Make sure the text file was copied to the destination. 408 | text_file = os.path.join(destination, 'notes.txt') 409 | assert os.path.isfile(text_file) 410 | with open(text_file) as handle: 411 | assert handle.read() == "This file should be included in the backup.\n" 412 | # Make sure the subdirectory was copied to the destination. 413 | subdirectory = os.path.join(destination, 'subdirectory') 414 | assert os.path.isdir(subdirectory) 415 | # Make sure the symbolic link was copied to the destination. 416 | symlink = os.path.join(subdirectory, 'symbolic-link') 417 | assert os.path.islink(symlink) 418 | 419 | 420 | @contextlib.contextmanager 421 | def prepared_image_file(create_filesystem=True): 422 | """Prepare an image file containing an encrypted filesystem (ext4 on top of LUKS).""" 423 | with TemporaryKeyFile(filename=KEY_FILE): 424 | create_image_file(filename=IMAGE_FILE, size='10M') 425 | create_encrypted_filesystem(device_file=IMAGE_FILE, key_file=KEY_FILE) 426 | # Create a filesystem on the encrypted image file? 427 | if create_filesystem: 428 | with unlocked_device(CRYPTO_NAME): 429 | execute('mkfs.ext4', FILESYSTEM_DEVICE, sudo=True) 430 | yield 431 | os.unlink(IMAGE_FILE) 432 | 433 | 434 | @contextlib.contextmanager 435 | def unlocked_device(crypto_device): 436 | """Context manager that runs ``cryptdisks_start`` and ``cryptdisks_stop``.""" 437 | cryptdisks_start(target=crypto_device) 438 | yield 439 | cryptdisks_stop(target=crypto_device) 440 | 441 | 442 | @contextlib.contextmanager 443 | def active_mountpoint(mount_point): 444 | """Context manager that runs ``mount`` and ``umount``.""" 445 | execute('mount', mount_point, sudo=True) 446 | yield 447 | execute('umount', mount_point, sudo=True) 448 | 449 | 450 | def find_snapshots(directory): 451 | """Abuse :mod:`rotate_backups` to scan a directory for snapshots.""" 452 | helper = RotateBackups(DEFAULT_ROTATION_SCHEME) 453 | return helper.collect_backups(directory) 454 | -------------------------------------------------------------------------------- /scripts/install-on-travis.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -e 2 | 3 | # Install the required Python packages. 4 | pip install --requirement=requirements-travis.txt 5 | 6 | # Install the project itself, making sure that potential character encoding 7 | # and/or decoding errors in the setup script are caught as soon as possible. 8 | LC_ALL=C pip install . 9 | 10 | # Let apt-get, dpkg and related tools know that we want the following 11 | # commands to be 100% automated (no interactive prompts). 12 | export DEBIAN_FRONTEND=noninteractive 13 | 14 | # Update apt-get's package lists. 15 | sudo -E apt-get update -qq 16 | 17 | # Use apt-get to install cryptdisks_start, cryptdisks_stop, cryptsetup, 18 | # mkfs.ext4 and rsync. 19 | sudo -E apt-get install --yes cryptsetup cryptsetup-bin e2fsprogs rsync 20 | 21 | if [ "$TRAVIS" != true ]; then 22 | cat >&2 << EOF 23 | 24 | Error: I'm refusing to touch /etc/fstab and /etc/crypttab because this 25 | shell script was written specifically for Travis CI where each build 26 | starts from a clean virtual machine image or snapshot! 27 | 28 | If you really know what you're getting yourself into you can set the 29 | environment variable TRAVIS=true to bypass this sanity check ... 30 | 31 | EOF 32 | exit 1 33 | fi 34 | 35 | # Append our mount point to /etc/fstab. 36 | sudo tee -a /etc/fstab >/dev/null << EOF 37 | /dev/mapper/rsync-system-backup /mnt/rsync-system-backup ext4 noauto 0 0 38 | EOF 39 | 40 | # Append our crypto device to /etc/crypttab. 41 | sudo tee -a /etc/crypttab >/dev/null << EOF 42 | rsync-system-backup /tmp/rsync-system-backup.img /tmp/rsync-system-backup.key luks,noauto 43 | EOF 44 | 45 | # Make sure the mount point exists (this enables the test). 46 | sudo mkdir -p /mnt/rsync-system-backup 47 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | # Enable universal wheels because `rsync-system-backup' is 2 | # pure Python and works on Python 2 and 3 alike. 3 | 4 | [wheel] 5 | universal=1 6 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Setup script for the `rsync-system-backup' package. 4 | # 5 | # Author: Peter Odding 6 | # Last Change: August 2, 2019 7 | # URL: https://github.com/xolox/python-rsync-system-backup 8 | 9 | """ 10 | Setup script for the ``rsync-system-backup`` package. 11 | 12 | **python setup.py install** 13 | Install from the working directory into the current Python environment. 14 | 15 | **python setup.py sdist** 16 | Build a source distribution archive. 17 | 18 | **python setup.py bdist_wheel** 19 | Build a wheel distribution archive. 20 | """ 21 | 22 | # Standard library modules. 23 | import codecs 24 | import os 25 | import re 26 | 27 | # De-facto standard solution for Python packaging. 28 | from setuptools import find_packages, setup 29 | 30 | 31 | def get_contents(*args): 32 | """Get the contents of a file relative to the source distribution directory.""" 33 | with codecs.open(get_absolute_path(*args), 'r', 'UTF-8') as handle: 34 | return handle.read() 35 | 36 | 37 | def get_version(*args): 38 | """Extract the version number from a Python module.""" 39 | contents = get_contents(*args) 40 | metadata = dict(re.findall('__([a-z]+)__ = [\'"]([^\'"]+)', contents)) 41 | return metadata['version'] 42 | 43 | 44 | def get_requirements(*args): 45 | """Get requirements from pip requirement files.""" 46 | requirements = set() 47 | with open(get_absolute_path(*args)) as handle: 48 | for line in handle: 49 | # Strip comments. 50 | line = re.sub(r'^#.*|\s#.*', '', line) 51 | # Ignore empty lines 52 | if line and not line.isspace(): 53 | requirements.add(re.sub(r'\s+', '', line)) 54 | return sorted(requirements) 55 | 56 | 57 | def get_absolute_path(*args): 58 | """Transform relative pathnames into absolute pathnames.""" 59 | return os.path.join(os.path.dirname(os.path.abspath(__file__)), *args) 60 | 61 | 62 | setup(name="rsync-system-backup", 63 | version=get_version('rsync_system_backup', '__init__.py'), 64 | description="Linux system backups powered by rsync", 65 | long_description=get_contents('README.rst'), 66 | url='https://github.com/xolox/python-rsync-system-backup', 67 | author="Peter Odding", 68 | author_email='peter@peterodding.com', 69 | license='MIT', 70 | packages=find_packages(), 71 | entry_points=dict(console_scripts=[ 72 | 'rsync-system-backup = rsync_system_backup.cli:main', 73 | ]), 74 | install_requires=get_requirements('requirements.txt'), 75 | classifiers=[ 76 | 'Development Status :: 4 - Beta', 77 | 'Environment :: Console', 78 | 'Intended Audience :: Developers', 79 | 'Intended Audience :: System Administrators', 80 | 'License :: OSI Approved :: MIT License', 81 | 'Operating System :: POSIX :: Linux', 82 | 'Programming Language :: Python', 83 | 'Programming Language :: Python :: 2', 84 | 'Programming Language :: Python :: 2.7', 85 | 'Programming Language :: Python :: 3', 86 | 'Programming Language :: Python :: 3.4', 87 | 'Programming Language :: Python :: 3.5', 88 | 'Programming Language :: Python :: 3.6', 89 | 'Programming Language :: Python :: 3.7', 90 | 'Programming Language :: Python :: Implementation :: CPython', 91 | 'Programming Language :: Python :: Implementation :: PyPy', 92 | 'Topic :: Software Development', 93 | 'Topic :: Software Development :: Libraries :: Python Modules', 94 | 'Topic :: System :: Archiving :: Backup', 95 | 'Topic :: System :: Systems Administration', 96 | ]) 97 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | [tox] 2 | envlist = py27, py34, py35, py36, py37, pypy 3 | 4 | [testenv] 5 | deps = -rrequirements-tests.txt 6 | commands = py.test {posargs} 7 | 8 | [pytest] 9 | addopts = --verbose 10 | python_files = rsync_system_backup/tests.py 11 | 12 | [flake8] 13 | exclude = .tox 14 | ignore = D211,D400,D401 15 | max-line-length = 120 16 | --------------------------------------------------------------------------------