├── .coveragerc ├── .travis.yml ├── CHANGELOG.rst ├── LICENSE.txt ├── MANIFEST.in ├── Makefile ├── README.rst ├── docs ├── api.rst ├── changelog.rst ├── conf.py ├── index.rst └── readme.rst ├── requirements-checks.txt ├── requirements-tests.txt ├── requirements-travis.txt ├── requirements.txt ├── rotate_backups ├── __init__.py ├── cli.py └── tests.py ├── scripts ├── install-on-travis.sh └── run-on-travis.sh ├── setup.cfg ├── setup.py └── tox.ini /.coveragerc: -------------------------------------------------------------------------------- 1 | [run] 2 | source = rotate_backups 3 | omit = rotate_backups/tests.py 4 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | sudo: false 2 | language: python 3 | matrix: 4 | - os: osx 5 | language: generic 6 | - python: pypy 7 | - python: 2.7 8 | - python: 3.5 9 | - python: 3.6 10 | - python: 3.7 11 | - python: 3.8 12 | install: 13 | - scripts/install-on-travis.sh 14 | script: 15 | - scripts/run-on-travis.sh make check 16 | - scripts/run-on-travis.sh make test 17 | after_success: 18 | - scripts/run-on-travis.sh coveralls 19 | branches: 20 | except: 21 | - /^[0-9]/ 22 | -------------------------------------------------------------------------------- /CHANGELOG.rst: -------------------------------------------------------------------------------- 1 | Changelog 2 | ========= 3 | 4 | The purpose of this document is to list all of the notable changes to this 5 | project. The format was inspired by `Keep a Changelog`_. This project adheres 6 | to `semantic versioning`_. 7 | 8 | .. contents:: 9 | :local: 10 | 11 | .. _Keep a Changelog: http://keepachangelog.com/ 12 | .. _semantic versioning: http://semver.org/ 13 | 14 | `Release 8.1`_ (2020-05-17) 15 | --------------------------- 16 | 17 | - Bug fix to really make the 'hour', 'minute' and 'second' capture groups in 18 | user defined timestamp patterns optional (this fixes issue `#26`_). 19 | 20 | - Fixed :man:`humanfriendly` 8 deprecation warnings. 21 | 22 | .. _Release 8.1: https://github.com/xolox/python-rotate-backups/compare/8.0...8.1 23 | .. _#26: https://github.com/xolox/python-rotate-backups/issues/26 24 | 25 | `Release 8.0`_ (2020-02-18) 26 | --------------------------- 27 | 28 | This is a bit of an awkward release: 29 | 30 | - An :exc:`~exceptions.ImportError` was reported in issue `#24`_ caused by a 31 | backwards incompatible change in :pypi:`humanfriendly` concerning an 32 | undocumented module level variable (shouldn't have used that). 33 | 34 | - I've now updated :pypi:`rotate-backups` to be compatible with the newest 35 | release of :pypi:`humanfriendly` however in the mean time that package 36 | dropped support for Python 3.4. 37 | 38 | - This explains how a simple bug fix release concerning two lines in the code 39 | base triggered a major version bump because compatibility is changed. 40 | 41 | - While I was at it I set up Python 3.8 testing on Travis CI which seems to 42 | work fine, so I've documented Python 3.8 as compatible. Python 3.9 seems 43 | to be a whole other story, I'll get to that soon. 44 | 45 | .. _Release 8.0: https://github.com/xolox/python-rotate-backups/compare/7.2...8.0 46 | .. _#24: https://github.com/xolox/python-rotate-backups/issues/24 47 | 48 | `Release 7.2`_ (2020-02-14) 49 | --------------------------- 50 | 51 | Merged pull request `#23`_ which makes it possible to customize the regular 52 | expression that's used to match timestamps in filenames using a new command 53 | line option ``rotate-backups --timestamp-pattern``. 54 | 55 | The pull request wasn't exactly complete (the code couldn't have run as 56 | written, although it showed the general idea clear enough) so I decided to 57 | treat `#23`_ as more of a feature suggestion. However there was no reason no to 58 | merge the pull request and use it as a base for my changes, hence why I decided 59 | to do so despite rewriting the code. 60 | 61 | Changes from the pull request: 62 | 63 | - Renamed ``timestamp`` to :attr:`~rotate_backups.RotateBackups.timestamp_pattern` 64 | to make it less ambiguous. 65 | 66 | - Added validation that custom patterns provided by callers define 67 | named capture groups corresponding to the required date components 68 | (year, month and day). 69 | 70 | - Rewrote the mapping from capture groups to :class:`datetime.datetime` 71 | arguments as follows: 72 | 73 | - Previously positional :class:`datetime.datetime` arguments were used 74 | which depended on the order of capture groups in the hard coded 75 | regular expression pattern to function correctly. 76 | 77 | - Now that users can define their own patterns, this is no longer a 78 | reasonable approach. As such the code now constructs and passes a 79 | dictionary of keyword arguments to :class:`datetime.datetime`. 80 | 81 | - Updated the documentation and the command line interface usage message to 82 | describe the new command line option and configuration file option. 83 | 84 | - Added tests for the new behavior. 85 | 86 | .. _Release 7.2: https://github.com/xolox/python-rotate-backups/compare/7.1...7.2 87 | .. _#23: https://github.com/xolox/python-rotate-backups/pull/23 88 | 89 | `Release 7.1`_ (2020-02-13) 90 | --------------------------- 91 | 92 | - Make it possibly to disable system logging using ``rotate-backups 93 | --syslog=false`` (fixes `#20`_). 94 | 95 | - Explicitly support numeric :man:`ionice` classes (as required by 96 | :man:`busybox` and suggested in `#14`_): 97 | 98 | - This follows up on a pull request to :pypi:`executor` (a dependency of 99 | :pypi:`rotate-backups`) that was merged in 2018. 100 | 101 | - Since that pull request was merged this new "feature" has been implicitly 102 | supported by :pypi:`rotate-backups` by upgrading the installed version of 103 | the :pypi:`executor` package, however this probably wasn't clear to anyone 104 | who's not a Python developer 😇. 105 | 106 | - I've now merged pull request `#14`_ which adds a test to confirm that 107 | numeric :man:`ionice` classes are supported. 108 | 109 | - I also bumped the :pypi:`executor` requirement and updated the usage 110 | instructions to point out that numeric :man:`ionice` classes are now 111 | supported. 112 | 113 | .. _Release 7.1: https://github.com/xolox/python-rotate-backups/compare/7.0...7.1 114 | .. _#20: https://github.com/xolox/python-rotate-backups/issues/20 115 | .. _#14: https://github.com/xolox/python-rotate-backups/pull/14 116 | 117 | `Release 7.0`_ (2020-02-12) 118 | --------------------------- 119 | 120 | **Significant changes:** 121 | 122 | - Sanity checks are done to ensure the directory with backups exists, is 123 | readable and is writable. However `#18`_ made it clear that such sanity 124 | checks can misjudge the situation, which made me realize an escape hatch 125 | should be provided. The new ``--force`` option makes ``rotate-backups`` 126 | continue even if sanity checks fail. 127 | 128 | - Skip the sanity check that the directory with backups is writable when the 129 | ``--removal-command`` option is given (because custom removal commands imply 130 | custom semantics, see `#18`_ for an example). 131 | 132 | **Miscellaneous changes:** 133 | 134 | - Start testing on Python 3.7 and document compatibility. 135 | - Dropped Python 2.6 (I don't think anyone still cares about this 😉). 136 | - Copied Travis CI workarounds for MacOS from :pypi:`humanfriendly`. 137 | - Updated ``Makefile`` to use Python 3 for local development. 138 | - Bumped copyright to 2020. 139 | 140 | .. _Release 7.0: https://github.com/xolox/python-rotate-backups/compare/6.0...7.0 141 | .. _#18: https://github.com/xolox/python-rotate-backups/issues/18 142 | 143 | `Release 6.0`_ (2018-08-03) 144 | --------------------------- 145 | 146 | This is a bug fix release that changes the behavior of the program, and because 147 | `rotate-backups` involves the deletion of important files I'm considering this 148 | a significant change in behavior that deserves a major version bump... 149 | 150 | It was reported in issue `#12`_ that filenames that match the filename pattern 151 | but contain digits with invalid values for the year/month/day/etc fields would 152 | cause a ``ValueError`` exception to be raised. 153 | 154 | Starting from this release these filenames are ignored instead, although a 155 | warning is logged to make sure the operator understands what's going on. 156 | 157 | .. _Release 6.0: https://github.com/xolox/python-rotate-backups/compare/5.3...6.0 158 | .. _#12: https://github.com/xolox/python-rotate-backups/issues/12 159 | 160 | `Release 5.3`_ (2018-08-03) 161 | --------------------------- 162 | 163 | - Merged pull request `#11`_ which introduces the ``--use-rmdir`` option with 164 | the suggested use case of removing CephFS snapshots. 165 | - Replaced ``--use-rmdir`` with ``--removal-command=rmdir`` (more general). 166 | 167 | .. _Release 5.3: https://github.com/xolox/python-rotate-backups/compare/5.2...5.3 168 | .. _#11: https://github.com/xolox/python-rotate-backups/pull/11 169 | 170 | `Release 5.2`_ (2018-04-27) 171 | --------------------------- 172 | 173 | - Added support for filename patterns in configuration files (`#10`_). 174 | - Bug fix: Skip human friendly pathname formatting for remote backups. 175 | - Improved documentation using ``property_manager.sphinx`` module. 176 | 177 | .. _Release 5.2: https://github.com/xolox/python-rotate-backups/compare/5.1...5.2 178 | .. _#10: https://github.com/xolox/python-rotate-backups/issues/10 179 | 180 | `Release 5.1`_ (2018-04-27) 181 | --------------------------- 182 | 183 | - Properly document supported configuration options (`#7`_, `#8`_). 184 | - Properly document backup collection strategy (`#8`_). 185 | - Avoid ``u''`` prefixes in log output of include/exclude list processing. 186 | - Added this changelog, restructured the online documentation. 187 | - Added ``license`` key to ``setup.py`` script. 188 | 189 | .. _Release 5.1: https://github.com/xolox/python-rotate-backups/compare/5.0...5.1 190 | .. _#7: https://github.com/xolox/python-rotate-backups/issues/7 191 | .. _#8: https://github.com/xolox/python-rotate-backups/issues/8 192 | 193 | `Release 5.0`_ (2018-03-29) 194 | --------------------------- 195 | 196 | The focus of this release is improved configuration file handling: 197 | 198 | - Refactor configuration file handling (backwards incompatible). These changes 199 | are backwards incompatible because of the following change in semantics 200 | between the logic that was previously in `rotate-backups` and has since been 201 | moved to update-dotdee_: 202 | 203 | - Previously only the first configuration file that was found in a default 204 | location was loaded (there was a 'break' in the loop). 205 | 206 | - Now all configuration files in default locations will be loaded. 207 | 208 | My impression is that this won't bite any unsuspecting users, at least not in 209 | a destructive way, but I guess only time and a lack of negative feedback will 210 | tell :-p. 211 | 212 | - Added Python 3.6 to supported versions. 213 | - Include documentation in source distributions. 214 | - Change theme of Sphinx documentation. 215 | - Moved test helpers to ``humanfriendly.testing``. 216 | 217 | .. _Release 5.0: https://github.com/xolox/python-rotate-backups/compare/4.4...5.0 218 | .. _update-dotdee: https://update-dotdee.readthedocs.io/en/latest/ 219 | 220 | `Release 4.4`_ (2017-04-13) 221 | --------------------------- 222 | 223 | Moved ``ionice`` support to executor_. 224 | 225 | .. _Release 4.4: https://github.com/xolox/python-rotate-backups/compare/4.3.1...4.4 226 | .. _executor: https://executor.readthedocs.io/en/latest/ 227 | 228 | `Release 4.3.1`_ (2017-04-13) 229 | ----------------------------- 230 | 231 | Restore Python 2.6 compatibility by pinning `simpleeval` dependency. 232 | 233 | While working on an unreleased Python project that uses `rotate-backups` I 234 | noticed that the tox build for Python 2.6 was broken. Whether it's worth it for 235 | me to keep supporting Python 2.6 is a valid question, but right now the readme 236 | and setup script imply compatibility with Python 2.6 so I feel half obliged to 237 | 'fix this issue' :-). 238 | 239 | .. _Release 4.3.1: https://github.com/xolox/python-rotate-backups/compare/4.3...4.3.1 240 | 241 | `Release 4.3`_ (2016-10-31) 242 | --------------------------- 243 | 244 | Added MacOS compatibility (`#6`_): 245 | 246 | - Ignore ``stat --format=%m`` failures. 247 | - Don't use ``ionice`` when not available. 248 | 249 | .. _Release 4.3: https://github.com/xolox/python-rotate-backups/compare/4.2...4.3 250 | .. _#6: https://github.com/xolox/python-rotate-backups/issues/6 251 | 252 | `Release 4.2`_ (2016-08-05) 253 | --------------------------- 254 | 255 | - Document default / alternative rotation algorithms (`#2`_, `#3`_, `#5`_). 256 | - Implement 'minutely' option (`#5`_). 257 | 258 | .. _Release 4.2: https://github.com/xolox/python-rotate-backups/compare/4.1...4.2 259 | .. _#2: https://github.com/xolox/python-rotate-backups/issues/2 260 | .. _#3: https://github.com/xolox/python-rotate-backups/issues/3 261 | .. _#5: https://github.com/xolox/python-rotate-backups/issues/5 262 | 263 | `Release 4.1`_ (2016-08-05) 264 | --------------------------- 265 | 266 | - Enable choice for newest backup per time slot (`#5`_). 267 | - Converted ``RotateBackups`` attributes to properties (I ❤ documentability :-). 268 | - Renamed 'constructor' to 'initializer' where applicable. 269 | - Simplified the ``rotate_backups.cli`` module a bit. 270 | 271 | .. _Release 4.1: https://github.com/xolox/python-rotate-backups/compare/4.0...4.1 272 | .. _#5: https://github.com/xolox/python-rotate-backups/issues/5 273 | 274 | `Release 4.0`_ (2016-07-09) 275 | --------------------------- 276 | 277 | Added support for concurrent backup rotation. 278 | 279 | .. _Release 4.0: https://github.com/xolox/python-rotate-backups/compare/3.5...4.0 280 | 281 | `Release 3.5`_ (2016-07-09) 282 | --------------------------- 283 | 284 | - Use key properties on ``Location`` objects. 285 | - Bring test coverage back up to >= 90%. 286 | 287 | .. _Release 3.5: https://github.com/xolox/python-rotate-backups/compare/3.4...3.5 288 | 289 | `Release 3.4`_ (2016-07-09) 290 | --------------------------- 291 | 292 | Added support for expression evaluation for retention periods. 293 | 294 | .. _Release 3.4: https://github.com/xolox/python-rotate-backups/compare/3.3...3.4 295 | 296 | `Release 3.3`_ (2016-07-09) 297 | --------------------------- 298 | 299 | Started using verboselogs_. 300 | 301 | .. _Release 3.3: https://github.com/xolox/python-rotate-backups/compare/3.2...3.3 302 | .. _verboselogs: https://verboselogs.readthedocs.io/ 303 | 304 | `Release 3.2`_ (2016-07-08) 305 | --------------------------- 306 | 307 | - Added support for Python 2.6 :-P. 308 | 309 | By switching to the ``key_property`` support added in `property-manager` 2.0 310 | I was able to reduce code duplication and improve compatibility:: 311 | 312 | 6 files changed, 20 insertions(+), 23 deletions(-) 313 | 314 | This removes the dependency on ``functools.total_ordering`` and to the best 315 | of my knowledge this was the only Python >= 2.7 feature that I was using so 316 | out of curiosity I changed ``tox.ini`` to run the tests on Python 2.6 and 317 | indeed everything worked fine! :-) 318 | 319 | - Refactored the makefile and ``setup.py`` script (checkers, docs, wheels, 320 | twine, etc). 321 | 322 | .. _Release 3.2: https://github.com/xolox/python-rotate-backups/compare/3.1...3.2 323 | 324 | `Release 3.1`_ (2016-04-13) 325 | --------------------------- 326 | 327 | Implement relaxed rotation mode, adding a ``--relaxed`` option (`#2`_, `#3`_). 328 | 329 | .. _Release 3.1: https://github.com/xolox/python-rotate-backups/compare/3.0...3.1 330 | .. _#2: https://github.com/xolox/python-rotate-backups/issues/2 331 | .. _#3: https://github.com/xolox/python-rotate-backups/issues/3 332 | 333 | `Release 3.0`_ (2016-04-13) 334 | --------------------------- 335 | 336 | - Support for backup rotation on remote systems. 337 | - Added Python 3.5 to supported versions. 338 | - Added support for ``-q``, ``--quiet`` command line option. 339 | - Delegate system logging to coloredlogs. 340 | - Improved ``rotate_backups.load_config_file()`` documentation. 341 | - Use ``humanfriendly.sphinx`` module to generate documentation. 342 | - Configured autodoc to order members based on source order. 343 | 344 | Some backwards incompatible changes slipped in here, e.g. removing 345 | ``Backup.__init__()`` and renaming ``Backup.datetime`` to ``Backup.timestamp``. 346 | 347 | In fact the refactoring that I've started here isn't finished yet, because the 348 | separation of concerns between the ``RotateBackups``, ``Location`` and 349 | ``Backup`` classes doesn't make a lot of sense at the moment and I'd like to 350 | improve on this. Rewriting projects takes time though :-(. 351 | 352 | .. _Release 3.0: https://github.com/xolox/python-rotate-backups/compare/2.3...3.0 353 | 354 | `Release 2.3`_ (2015-08-30) 355 | --------------------------- 356 | 357 | Add/restore Python 3.4 compatibility. 358 | 359 | It was always the intention to support Python 3 but a couple of setbacks made 360 | it harder than just "flipping the switch" before now :-). This issue was 361 | reported here: https://github.com/xolox/python-naturalsort/issues/2. 362 | 363 | .. _Release 2.3: https://github.com/xolox/python-rotate-backups/compare/2.2...2.3 364 | 365 | `Release 2.2`_ (2015-07-19) 366 | --------------------------- 367 | 368 | Added support for configuration files. 369 | 370 | .. _Release 2.2: https://github.com/xolox/python-rotate-backups/compare/2.1...2.2 371 | 372 | `Release 2.1`_ (2015-07-19) 373 | --------------------------- 374 | 375 | Bug fix: Guard against empty rotation schemes. 376 | 377 | .. _Release 2.1: https://github.com/xolox/python-rotate-backups/compare/2.0...2.1 378 | 379 | `Release 2.0`_ (2015-07-19) 380 | --------------------------- 381 | 382 | Backwards incompatible: Implement a new Python API. 383 | 384 | The idea is that this restructuring will make it easier to re-use (parts of) 385 | the `rotate-backups` package in my other Python projects.. 386 | 387 | .. _Release 2.0: https://github.com/xolox/python-rotate-backups/compare/1.1...2.0 388 | 389 | `Release 1.1`_ (2015-07-19) 390 | --------------------------- 391 | 392 | Merged pull request `#1`_: Add include/exclude filters. 393 | 394 | I made significant changes while merging this (e.g. the short option for 395 | the include list and the use of shell patterns using the fnmatch module) 396 | and I added tests to verify the behavior of the include/exclude logic. 397 | 398 | .. _Release 1.1: https://github.com/xolox/python-rotate-backups/compare/1.0...1.1 399 | .. _#1: https://github.com/xolox/python-rotate-backups/pull/1 400 | 401 | `Release 1.0`_ (2015-07-19) 402 | --------------------------- 403 | 404 | - Started working on a proper test suite. 405 | - Split the command line interface from the Python API. 406 | - Prepare for API documentation on Read The Docs. 407 | - Switch from ``py_modules=[...]`` to ``packages=find_packages()`` in ``setup.py``. 408 | 409 | .. _Release 1.0: https://github.com/xolox/python-rotate-backups/compare/0.1.2...1.0 410 | 411 | `Release 0.1.2`_ (2015-07-15) 412 | ----------------------------- 413 | 414 | - Bug fix for ``-y``, ``--yearly`` command line option mapping. 415 | - Fixed some typos (in the README and a comment in ``setup.py``). 416 | 417 | .. _Release 0.1.2: https://github.com/xolox/python-rotate-backups/compare/0.1.1...0.1.2 418 | 419 | `Release 0.1.1`_ (2014-07-03) 420 | ----------------------------- 421 | 422 | - Added missing dependency. 423 | - Removed Sphinx-isms from README (PyPI doesn't like it, falls back to plain text). 424 | 425 | .. _Release 0.1.1: https://github.com/xolox/python-rotate-backups/compare/0.1...0.1.1 426 | 427 | `Release 0.1`_ (2014-07-03) 428 | --------------------------- 429 | 430 | Initial commit (not very well tested yet). 431 | 432 | .. _Release 0.1: https://github.com/xolox/python-rotate-backups/tree/0.1 433 | -------------------------------------------------------------------------------- /LICENSE.txt: -------------------------------------------------------------------------------- 1 | Copyright (c) 2020 Peter Odding 2 | 3 | Permission is hereby granted, free of charge, to any person obtaining 4 | a copy of this software and associated documentation files (the 5 | "Software"), to deal in the Software without restriction, including 6 | without limitation the rights to use, copy, modify, merge, publish, 7 | distribute, sublicense, and/or sell copies of the Software, and to 8 | permit persons to whom the Software is furnished to do so, subject to 9 | the following conditions: 10 | 11 | The above copyright notice and this permission notice shall be 12 | included in all copies or substantial portions of the Software. 13 | 14 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 15 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 16 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND 17 | NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE 18 | LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 19 | OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION 20 | WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. 21 | -------------------------------------------------------------------------------- /MANIFEST.in: -------------------------------------------------------------------------------- 1 | include *.rst 2 | include *.txt 3 | graft docs 4 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | # Makefile for the `rotate-backups' package. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: February 11, 2020 5 | # URL: https://github.com/xolox/python-rotate-backups 6 | 7 | PACKAGE_NAME = rotate-backups 8 | WORKON_HOME ?= $(HOME)/.virtualenvs 9 | VIRTUAL_ENV ?= $(WORKON_HOME)/$(PACKAGE_NAME) 10 | PYTHON ?= python3 11 | PATH := $(VIRTUAL_ENV)/bin:$(PATH) 12 | MAKE := $(MAKE) --no-print-directory 13 | SHELL = bash 14 | 15 | default: 16 | @echo "Makefile for $(PACKAGE_NAME)" 17 | @echo 18 | @echo 'Usage:' 19 | @echo 20 | @echo ' make install install the package in a virtual environment' 21 | @echo ' make reset recreate the virtual environment' 22 | @echo ' make check check coding style (PEP-8, PEP-257)' 23 | @echo ' make test run the test suite, report coverage' 24 | @echo ' make tox run the tests on all Python versions' 25 | @echo ' make readme update usage in readme' 26 | @echo ' make docs update documentation using Sphinx' 27 | @echo ' make publish publish changes to GitHub/PyPI' 28 | @echo ' make clean cleanup all temporary files' 29 | @echo 30 | 31 | install: 32 | @test -d "$(VIRTUAL_ENV)" || mkdir -p "$(VIRTUAL_ENV)" 33 | @test -x "$(VIRTUAL_ENV)/bin/python" || virtualenv --python=$(PYTHON) --quiet "$(VIRTUAL_ENV)" 34 | @pip install --quiet --requirement=requirements.txt 35 | @pip uninstall --yes $(PACKAGE_NAME) &>/dev/null || true 36 | @pip install --quiet --no-deps --ignore-installed . 37 | 38 | reset: 39 | @$(MAKE) clean 40 | @rm -Rf "$(VIRTUAL_ENV)" 41 | @$(MAKE) install 42 | 43 | check: install 44 | @pip install --upgrade --quiet --requirement=requirements-checks.txt 45 | @flake8 46 | 47 | test: install 48 | @pip install --quiet --requirement=requirements-tests.txt 49 | @py.test --cov 50 | @coverage html 51 | @coverage report --fail-under=90 &>/dev/null 52 | 53 | tox: install 54 | @pip install --quiet tox 55 | @tox 56 | 57 | readme: install 58 | @pip install --quiet cogapp 59 | @cog.py -r README.rst 60 | 61 | docs: readme 62 | @pip install --quiet sphinx 63 | @cd docs && sphinx-build -nb html -d build/doctrees . build/html 64 | 65 | publish: install 66 | @git push origin && git push --tags origin 67 | @$(MAKE) clean 68 | @pip install --quiet twine wheel 69 | @python setup.py sdist bdist_wheel 70 | @twine upload dist/* 71 | @$(MAKE) clean 72 | 73 | clean: 74 | @rm -Rf *.egg .cache .coverage .tox build dist docs/build htmlcov 75 | @find -depth -type d -name __pycache__ -exec rm -Rf {} \; 76 | @find -type f -name '*.pyc' -delete 77 | 78 | .PHONY: default install reset check test tox readme docs publish clean 79 | -------------------------------------------------------------------------------- /README.rst: -------------------------------------------------------------------------------- 1 | rotate-backups: Simple command line interface for backup rotation 2 | ================================================================= 3 | 4 | .. image:: https://travis-ci.org/xolox/python-rotate-backups.svg?branch=master 5 | :target: https://travis-ci.org/xolox/python-rotate-backups 6 | 7 | .. image:: https://coveralls.io/repos/xolox/python-rotate-backups/badge.svg?branch=master 8 | :target: https://coveralls.io/r/xolox/python-rotate-backups?branch=master 9 | 10 | Backups are good for you. Most people learn this the hard way (including me). 11 | Nowadays my Linux laptop automatically creates a full system snapshot every 12 | four hours by pushing changed files to an `rsync`_ daemon running on the server 13 | in my home network and creating a snapshot afterwards using the ``cp -al`` 14 | command (the article `Easy Automated Snapshot-Style Backups with Linux and 15 | Rsync`_ explains the basic technique). The server has a second disk attached 16 | which asynchronously copies from the main disk so that a single disk failure 17 | doesn't wipe all of my backups (the "time delayed replication" aspect has also 18 | proven to be very useful). 19 | 20 | Okay, cool, now I have backups of everything, up to date and going back in 21 | time! But I'm running through disk space like crazy... A proper deduplicating 22 | filesystem would be awesome but I'm running crappy consumer grade hardware and 23 | e.g. ZFS has not been a good experience in the past. So I'm going to have to 24 | delete backups... 25 | 26 | Deleting backups is never nice, but an easy and proper rotation scheme can help 27 | a lot. I wanted to keep things manageable so I wrote a Python script to do it 28 | for me. Over the years I actually wrote several variants. Because I kept 29 | copy/pasting these scripts around I decided to bring the main features together 30 | in a properly documented Python package and upload it to the `Python Package 31 | Index`_. 32 | 33 | The `rotate-backups` package is currently tested on cPython 2.7, 3.5+ and PyPy 34 | (2.7). It's tested on Linux and Mac OS X and may work on other unixes but 35 | definitely won't work on Windows right now. 36 | 37 | .. contents:: 38 | :local: 39 | 40 | Features 41 | -------- 42 | 43 | **Dry run mode** 44 | **Use it.** I'm serious. If you don't and `rotate-backups` eats more backups 45 | than intended you have no right to complain ;-) 46 | 47 | **Flexible rotation** 48 | Rotation with any combination of hourly, daily, weekly, monthly and yearly 49 | retention periods. 50 | 51 | **Fuzzy timestamp matching in filenames** 52 | The modification times of the files and/or directories are not relevant. If 53 | you speak Python regular expressions, here is how the fuzzy matching 54 | works:: 55 | 56 | # Required components. 57 | (?P\d{4}) \D? 58 | (?P\d{2}) \D? 59 | (?P\d{2}) \D? 60 | ( 61 | # Optional components. 62 | (?P\d{2}) \D? 63 | (?P\d{2}) \D? 64 | (?P\d{2})? 65 | )? 66 | 67 | **All actions are logged** 68 | Log messages are saved to the system log (e.g. ``/var/log/syslog``) so you 69 | can retrace what happened when something seems to have gone wrong. 70 | 71 | Installation 72 | ------------ 73 | 74 | The `rotate-backups` package is available on PyPI_ which means installation 75 | should be as simple as: 76 | 77 | .. code-block:: sh 78 | 79 | $ pip install rotate-backups 80 | 81 | There's actually a multitude of ways to install Python packages (e.g. the `per 82 | user site-packages directory`_, `virtual environments`_ or just installing 83 | system wide) and I have no intention of getting into that discussion here, so 84 | if this intimidates you then read up on your options before returning to these 85 | instructions ;-). 86 | 87 | Usage 88 | ----- 89 | 90 | There are two ways to use the `rotate-backups` package: As the command line 91 | program ``rotate-backups`` and as a Python API. For details about the Python 92 | API please refer to the API documentation available on `Read the Docs`_. The 93 | command line interface is described below. 94 | 95 | Command line 96 | ~~~~~~~~~~~~ 97 | 98 | .. A DRY solution to avoid duplication of the `rotate-backups --help' text: 99 | .. 100 | .. [[[cog 101 | .. from humanfriendly.usage import inject_usage 102 | .. inject_usage('rotate_backups.cli') 103 | .. ]]] 104 | 105 | **Usage:** `rotate-backups [OPTIONS] [DIRECTORY, ..]` 106 | 107 | Easy rotation of backups based on the Python package by the same name. 108 | 109 | To use this program you specify a rotation scheme via (a combination of) the 110 | ``--hourly``, ``--daily``, ``--weekly``, ``--monthly`` and/or ``--yearly`` options and the 111 | directory (or directories) containing backups to rotate as one or more 112 | positional arguments. 113 | 114 | You can rotate backups on a remote system over SSH by prefixing a DIRECTORY 115 | with an SSH alias and separating the two with a colon (similar to how rsync 116 | accepts remote locations). 117 | 118 | Instead of specifying directories and a rotation scheme on the command line you 119 | can also add them to a configuration file. For more details refer to the online 120 | documentation (see also the ``--config`` option). 121 | 122 | Please use the ``--dry-run`` option to test the effect of the specified rotation 123 | scheme before letting this program loose on your precious backups! If you don't 124 | test the results using the dry run mode and this program eats more backups than 125 | intended you have no right to complain ;-). 126 | 127 | **Supported options:** 128 | 129 | .. csv-table:: 130 | :header: Option, Description 131 | :widths: 30, 70 132 | 133 | 134 | "``-M``, ``--minutely=COUNT``","In a literal sense this option sets the number of ""backups per minute"" to 135 | preserve during rotation. For most use cases that doesn't make a lot of 136 | sense :-) but you can combine the ``--minutely`` and ``--relaxed`` options to 137 | preserve more than one backup per hour. Refer to the usage of the ``-H``, 138 | ``--hourly`` option for details about ``COUNT``." 139 | "``-H``, ``--hourly=COUNT``","Set the number of hourly backups to preserve during rotation: 140 | 141 | - If ``COUNT`` is a number it gives the number of hourly backups to preserve, 142 | starting from the most recent hourly backup and counting back in time. 143 | - Alternatively you can provide an expression that will be evaluated to get 144 | a number (e.g. if ``COUNT`` is ""7 \* 2"" the result would be 14). 145 | - You can also pass ""always"" for ``COUNT``, in this case all hourly backups are 146 | preserved. 147 | - By default no hourly backups are preserved." 148 | "``-d``, ``--daily=COUNT``","Set the number of daily backups to preserve during rotation. Refer to the 149 | usage of the ``-H``, ``--hourly`` option for details about ``COUNT``." 150 | "``-w``, ``--weekly=COUNT``","Set the number of weekly backups to preserve during rotation. Refer to the 151 | usage of the ``-H``, ``--hourly`` option for details about ``COUNT``." 152 | "``-m``, ``--monthly=COUNT``","Set the number of monthly backups to preserve during rotation. Refer to the 153 | usage of the ``-H``, ``--hourly`` option for details about ``COUNT``." 154 | "``-y``, ``--yearly=COUNT``","Set the number of yearly backups to preserve during rotation. Refer to the 155 | usage of the ``-H``, ``--hourly`` option for details about ``COUNT``." 156 | "``-t``, ``--timestamp-pattern=PATTERN``","Customize the regular expression pattern that is used to match and extract 157 | timestamps from filenames. ``PATTERN`` is expected to be a Python compatible 158 | regular expression that must define the named capture groups 'year', 159 | 'month' and 'day' and may define 'hour', 'minute' and 'second'." 160 | "``-I``, ``--include=PATTERN``","Only process backups that match the shell pattern given by ``PATTERN``. This 161 | argument can be repeated. Make sure to quote ``PATTERN`` so the shell doesn't 162 | expand the pattern before it's received by rotate-backups." 163 | "``-x``, ``--exclude=PATTERN``","Don't process backups that match the shell pattern given by ``PATTERN``. This 164 | argument can be repeated. Make sure to quote ``PATTERN`` so the shell doesn't 165 | expand the pattern before it's received by rotate-backups." 166 | "``-j``, ``--parallel``","Remove backups in parallel, one backup per mount point at a time. The idea 167 | behind this approach is that parallel rotation is most useful when the 168 | files to be removed are on different disks and so multiple devices can be 169 | utilized at the same time. 170 | 171 | Because mount points are per system the ``-j``, ``--parallel`` option will also 172 | parallelize over backups located on multiple remote systems." 173 | "``-p``, ``--prefer-recent``","By default the first (oldest) backup in each time slot is preserved. If 174 | you'd prefer to keep the most recent backup in each time slot instead then 175 | this option is for you." 176 | "``-r``, ``--relaxed``","By default the time window for each rotation scheme is enforced (this is 177 | referred to as strict rotation) but the ``-r``, ``--relaxed`` option can be used 178 | to alter this behavior. The easiest way to explain the difference between 179 | strict and relaxed rotation is using an example: 180 | 181 | - When using strict rotation and the number of hourly backups to preserve 182 | is three, only backups created in the relevant time window (the hour of 183 | the most recent backup and the two hours leading up to that) will match 184 | the hourly frequency. 185 | 186 | - When using relaxed rotation the three most recent backups will all match 187 | the hourly frequency (and thus be preserved), regardless of the 188 | calculated time window. 189 | 190 | If the explanation above is not clear enough, here's a simple way to decide 191 | whether you want to customize this behavior or not: 192 | 193 | - If your backups are created at regular intervals and you never miss an 194 | interval then strict rotation (the default) is probably the best choice. 195 | 196 | - If your backups are created at irregular intervals then you may want to 197 | use the ``-r``, ``--relaxed`` option in order to preserve more backups." 198 | "``-i``, ``--ionice=CLASS``","Use the ""ionice"" program to set the I/O scheduling class and priority of 199 | the ""rm"" invocations used to remove backups. ``CLASS`` is expected to be one of 200 | the values ""idle"" (3), ""best-effort"" (2) or ""realtime"" (1). Refer to the 201 | man page of the ""ionice"" program for details about these values. The 202 | numeric values are required by the 'busybox' implementation of 'ionice'." 203 | "``-c``, ``--config=FILENAME``","Load configuration from ``FILENAME``. If this option isn't given the following 204 | default locations are searched for configuration files: 205 | 206 | - /etc/rotate-backups.ini and /etc/rotate-backups.d/\*.ini 207 | - ~/.rotate-backups.ini and ~/.rotate-backups.d/\*.ini 208 | - ~/.config/rotate-backups.ini and ~/.config/rotate-backups.d/\*.ini 209 | 210 | Any available configuration files are loaded in the order given above, so 211 | that sections in user-specific configuration files override sections by the 212 | same name in system-wide configuration files. For more details refer to the 213 | online documentation." 214 | "``-C``, ``--removal-command=CMD``","Change the command used to remove backups. The value of ``CMD`` defaults to 215 | ``rm ``-f``R``. This choice was made because it works regardless of whether 216 | ""backups to be rotated"" are files or directories or a mixture of both. 217 | 218 | As an example of why you might want to change this, CephFS snapshots are 219 | represented as regular directory trees that can be deleted at once with a 220 | single 'rmdir' command (even though according to POSIX semantics this 221 | command should refuse to remove nonempty directories, but I digress)." 222 | "``-u``, ``--use-sudo``","Enable the use of ""sudo"" to rotate backups in directories that are not 223 | readable and/or writable for the current user (or the user logged in to a 224 | remote system over SSH)." 225 | "``-S``, ``--syslog=CHOICE``","Explicitly enable or disable system logging instead of letting the program 226 | figure out what to do. The values '1', 'yes', 'true' and 'on' enable system 227 | logging whereas the values '0', 'no', 'false' and 'off' disable it." 228 | "``-f``, ``--force``","If a sanity check fails an error is reported and the program aborts. You 229 | can use ``--force`` to continue with backup rotation instead. Sanity checks 230 | are done to ensure that the given DIRECTORY exists, is readable and is 231 | writable. If the ``--removal-command`` option is given then the last sanity 232 | check (that the given location is writable) is skipped (because custom 233 | removal commands imply custom semantics)." 234 | "``-n``, ``--dry-run``","Don't make any changes, just print what would be done. This makes it easy 235 | to evaluate the impact of a rotation scheme without losing any backups." 236 | "``-v``, ``--verbose``",Increase logging verbosity (can be repeated). 237 | "``-q``, ``--quiet``",Decrease logging verbosity (can be repeated). 238 | "``-h``, ``--help``",Show this message and exit. 239 | 240 | .. [[[end]]] 241 | 242 | Configuration files 243 | ~~~~~~~~~~~~~~~~~~~ 244 | 245 | Instead of specifying directories and rotation schemes on the command line you 246 | can also add them to a configuration file. 247 | 248 | .. [[[cog 249 | .. from update_dotdee import inject_documentation 250 | .. inject_documentation(program_name='rotate-backups') 251 | .. ]]] 252 | 253 | Configuration files are text files in the subset of `ini syntax`_ supported by 254 | Python's configparser_ module. They can be located in the following places: 255 | 256 | ========= ============================ ================================= 257 | Directory Main configuration file Modular configuration files 258 | ========= ============================ ================================= 259 | /etc /etc/rotate-backups.ini /etc/rotate-backups.d/\*.ini 260 | ~ ~/.rotate-backups.ini ~/.rotate-backups.d/\*.ini 261 | ~/.config ~/.config/rotate-backups.ini ~/.config/rotate-backups.d/\*.ini 262 | ========= ============================ ================================= 263 | 264 | The available configuration files are loaded in the order given above, so that 265 | user specific configuration files override system wide configuration files. 266 | 267 | .. _configparser: https://docs.python.org/3/library/configparser.html 268 | .. _ini syntax: https://en.wikipedia.org/wiki/INI_file 269 | 270 | .. [[[end]]] 271 | 272 | You can load a configuration file in a nonstandard location using the command 273 | line option ``--config``, in this case the default locations mentioned above 274 | are ignored. 275 | 276 | Each section in the configuration defines a directory that contains backups to 277 | be rotated. The options in each section define the rotation scheme and other 278 | options. Here's an example based on how I use `rotate-backups` to rotate the 279 | backups of the Linux installations that I make regular backups of: 280 | 281 | .. code-block:: ini 282 | 283 | # /etc/rotate-backups.ini: 284 | # Configuration file for the rotate-backups program that specifies 285 | # directories containing backups to be rotated according to specific 286 | # rotation schemes. 287 | 288 | [/backups/laptop] 289 | hourly = 24 290 | daily = 7 291 | weekly = 4 292 | monthly = 12 293 | yearly = always 294 | ionice = idle 295 | 296 | [/backups/server] 297 | daily = 7 * 2 298 | weekly = 4 * 2 299 | monthly = 12 * 4 300 | yearly = always 301 | ionice = idle 302 | 303 | [/backups/mopidy] 304 | daily = 7 305 | weekly = 4 306 | monthly = 2 307 | ionice = idle 308 | 309 | [/backups/xbmc] 310 | daily = 7 311 | weekly = 4 312 | monthly = 2 313 | ionice = idle 314 | 315 | As you can see in the retention periods of the directory ``/backups/server`` in 316 | the example above you are allowed to use expressions that evaluate to a number 317 | (instead of having to write out the literal number). 318 | 319 | Here's an example of a configuration for two remote directories: 320 | 321 | .. code-block:: ini 322 | 323 | # SSH as a regular user and use `sudo' to elevate privileges. 324 | [server:/backups/laptop] 325 | use-sudo = yes 326 | hourly = 24 327 | daily = 7 328 | weekly = 4 329 | monthly = 12 330 | yearly = always 331 | ionice = idle 332 | 333 | # SSH as the root user (avoids sudo passwords). 334 | [server:/backups/server] 335 | ssh-user = root 336 | hourly = 24 337 | daily = 7 338 | weekly = 4 339 | monthly = 12 340 | yearly = always 341 | ionice = idle 342 | 343 | As this example shows you have the option to connect as the root user or to 344 | connect as a regular user and use ``sudo`` to elevate privileges. 345 | 346 | Customizing the rotation algorithm 347 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 348 | 349 | Since publishing `rotate-backups` I've found that the default rotation 350 | algorithm is not to everyone's satisfaction and because the suggested 351 | alternatives were just as valid as the choices that I initially made, 352 | options were added to expose the alternative behaviors: 353 | 354 | +-------------------------------------+-------------------------------------+ 355 | | Default | Alternative | 356 | +=====================================+=====================================+ 357 | | Strict rotation (the time window | Relaxed rotation (time windows are | 358 | | for each rotation frequency is | not enforced). Enabled by the | 359 | | enforced). | ``-r``, ``--relaxed`` option. | 360 | +-------------------------------------+-------------------------------------+ 361 | | The oldest backup in each time slot | The newest backup in each time slot | 362 | | is preserved and newer backups in | is preserved and older backups in | 363 | | the time slot are removed. | the time slot are removed. Enabled | 364 | | | by the ``-p``, ``--prefer-recent`` | 365 | | | option. | 366 | +-------------------------------------+-------------------------------------+ 367 | 368 | Supported configuration options 369 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 370 | 371 | - Rotation schemes are defined using the ``minutely``, ``hourly``, ``daily``, 372 | ``weekly``, ``monthly`` and ``yearly`` options, these options support the 373 | same values as documented for the command line interface. 374 | 375 | - The ``timestamp-pattern`` option can be used to customize the regular 376 | expression that's used to extract timestamps from filenames. The value is 377 | expected to be a Python compatible regular expression that must contain the 378 | named capture groups 'year', 'month' and 'day' and may contain the groups 379 | 'hour', 'minute' and 'second'. As an example here is the default regular 380 | expression:: 381 | 382 | # Required components. 383 | (?P\d{4} ) \D? 384 | (?P\d{2}) \D? 385 | (?P\d{2} ) \D? 386 | (?: 387 | # Optional components. 388 | (?P\d{2} ) \D? 389 | (?P\d{2}) \D? 390 | (?P\d{2})? 391 | )? 392 | 393 | Note how this pattern spans multiple lines: Regular expressions are compiled 394 | using the `re.VERBOSE`_ flag which means whitespace (including newlines) is 395 | ignored. 396 | 397 | - The ``include-list`` and ``exclude-list`` options define a comma separated 398 | list of filename patterns to include or exclude, respectively: 399 | 400 | - Make sure *not* to quote the patterns in the configuration file, 401 | just provide them literally. 402 | 403 | - If an include or exclude list is defined in the configuration file it 404 | overrides the include or exclude list given on the command line. 405 | 406 | - The ``prefer-recent``, ``strict`` and ``use-sudo`` options expect a boolean 407 | value (``yes``, ``no``, ``true``, ``false``, ``1`` or ``0``). 408 | 409 | - The ``removal-command`` option can be used to customize the command that is 410 | used to remove backups. 411 | 412 | - The ``ionice`` option expects one of the I/O scheduling class names ``idle``, 413 | ``best-effort`` or ``realtime`` (or the corresponding numbers). 414 | 415 | - The ``ssh-user`` option can be used to override the name of the remote SSH 416 | account that's used to connect to a remote system. 417 | 418 | How it works 419 | ------------ 420 | 421 | The basic premise of `rotate-backups` is fairly simple: 422 | 423 | 1. You point `rotate-backups` at a directory containing timestamped backups. 424 | 425 | 2. It will scan the directory for entries (it doesn't matter whether they are 426 | files or directories) with a recognizable timestamp in the name. 427 | 428 | .. note:: All of the matched directory entries are considered to be backups 429 | of the same data source, i.e. there's no filename similarity logic 430 | to distinguish unrelated backups that are located in the same 431 | directory. If this presents a problem consider using the 432 | ``--include`` and/or ``--exclude`` options. 433 | 434 | 3. The user defined rotation scheme is applied to the entries. If this doesn't 435 | do what you'd expect it to you can try the ``--relaxed`` and/or 436 | ``--prefer-recent`` options. 437 | 438 | 4. The entries to rotate are removed (or printed in dry run). 439 | 440 | Contact 441 | ------- 442 | 443 | The latest version of `rotate-backups` is available on PyPI_ and GitHub_. The 444 | documentation is hosted on `Read the Docs`_ and includes a changelog_. For bug 445 | reports please create an issue on GitHub_. If you have questions, suggestions, 446 | etc. feel free to send me an e-mail at `peter@peterodding.com`_. 447 | 448 | License 449 | ------- 450 | 451 | This software is licensed under the `MIT license`_. 452 | 453 | © 2020 Peter Odding. 454 | 455 | .. External references: 456 | 457 | .. _changelog: https://rotate-backups.readthedocs.org/en/latest/changelog.html 458 | .. _Easy Automated Snapshot-Style Backups with Linux and Rsync: http://www.mikerubel.org/computers/rsync_snapshots/ 459 | .. _GitHub: https://github.com/xolox/python-rotate-backups 460 | .. _MIT license: http://en.wikipedia.org/wiki/MIT_License 461 | .. _per user site-packages directory: https://www.python.org/dev/peps/pep-0370/ 462 | .. _peter@peterodding.com: peter@peterodding.com 463 | .. _PyPI: https://pypi.python.org/pypi/rotate-backups 464 | .. _Python Package Index: https://pypi.python.org/pypi/rotate-backups 465 | .. _re.VERBOSE: https://docs.python.org/3/library/re.html#re.VERBOSE 466 | .. _Read the Docs: https://rotate-backups.readthedocs.org 467 | .. _rsync: http://en.wikipedia.org/wiki/rsync 468 | .. _virtual environments: http://docs.python-guide.org/en/latest/dev/virtualenvs/ 469 | -------------------------------------------------------------------------------- /docs/api.rst: -------------------------------------------------------------------------------- 1 | API documentation 2 | ================= 3 | 4 | This documentation is based on the source code of version |release| of the 5 | `rotate-backups` package. The following modules are available: 6 | 7 | .. contents:: 8 | :local: 9 | 10 | :mod:`rotate_backups` 11 | --------------------- 12 | 13 | .. automodule:: rotate_backups 14 | :members: 15 | 16 | :mod:`rotate_backups.cli` 17 | ------------------------- 18 | 19 | .. automodule:: rotate_backups.cli 20 | :members: 21 | -------------------------------------------------------------------------------- /docs/changelog.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../CHANGELOG.rst 2 | -------------------------------------------------------------------------------- /docs/conf.py: -------------------------------------------------------------------------------- 1 | """ 2 | Documentation build configuration file for the `rotate-backups` package. 3 | 4 | This Python script contains the Sphinx configuration for building the 5 | documentation of the `rotate-backups` project. This file is execfile()d 6 | with the current directory set to its containing dir. 7 | """ 8 | 9 | import os 10 | import sys 11 | 12 | # Add the 'rotate-backups' source distribution's root directory to the module path. 13 | sys.path.insert(0, os.path.abspath(os.pardir)) 14 | 15 | # -- General configuration ----------------------------------------------------- 16 | 17 | # Sphinx extension module names. 18 | extensions = [ 19 | 'sphinx.ext.autodoc', 20 | 'sphinx.ext.doctest', 21 | 'sphinx.ext.intersphinx', 22 | 'sphinx.ext.viewcode', 23 | 'humanfriendly.sphinx', 24 | 'property_manager.sphinx', 25 | ] 26 | 27 | # Configuration for the `autodoc' extension. 28 | autodoc_member_order = 'bysource' 29 | 30 | # Paths that contain templates, relative to this directory. 31 | templates_path = ['templates'] 32 | 33 | # The suffix of source filenames. 34 | source_suffix = '.rst' 35 | 36 | # The master toctree document. 37 | master_doc = 'index' 38 | 39 | # General information about the project. 40 | project = 'rotate-backups' 41 | copyright = '2020, Peter Odding' 42 | 43 | # The version info for the project you're documenting, acts as replacement for 44 | # |version| and |release|, also used in various other places throughout the 45 | # built documents. 46 | 47 | # Find the package version and make it the release. 48 | from rotate_backups import __version__ as rotate_backups_version # noqa 49 | 50 | # The short X.Y version. 51 | version = '.'.join(rotate_backups_version.split('.')[:2]) 52 | 53 | # The full version, including alpha/beta/rc tags. 54 | release = rotate_backups_version 55 | 56 | # The language for content autogenerated by Sphinx. Refer to documentation 57 | # for a list of supported languages. 58 | language = 'en' 59 | 60 | # List of patterns, relative to source directory, that match files and 61 | # directories to ignore when looking for source files. 62 | exclude_patterns = ['build'] 63 | 64 | # If true, '()' will be appended to :func: etc. cross-reference text. 65 | add_function_parentheses = True 66 | 67 | # The name of the Pygments (syntax highlighting) style to use. 68 | pygments_style = 'sphinx' 69 | 70 | # Refer to the Python standard library. 71 | # From: http://twistedmatrix.com/trac/ticket/4582. 72 | intersphinx_mapping = { 73 | 'python2': ('https://docs.python.org/2/', None), 74 | 'python3': ('https://docs.python.org/3/', None), 75 | 'dateutil': ('https://dateutil.readthedocs.io/en/latest/', None), 76 | 'executor': ('https://executor.readthedocs.io/en/latest/', None), 77 | 'humanfriendly': ('https://humanfriendly.readthedocs.io/en/latest/', None), 78 | 'propertymanager': ('https://property-manager.readthedocs.io/en/latest/', None), 79 | 'updatedotdee': ('https://update-dotdee.readthedocs.io/en/latest/', None), 80 | } 81 | 82 | # -- Options for HTML output --------------------------------------------------- 83 | 84 | # The theme to use for HTML and HTML Help pages. See the documentation for 85 | # a list of builtin themes. 86 | html_theme = 'nature' 87 | -------------------------------------------------------------------------------- /docs/index.rst: -------------------------------------------------------------------------------- 1 | rotate-backups: Simple command line interface for backup rotation 2 | ================================================================= 3 | 4 | Welcome to the documentation of `rotate-backups` version |release|! 5 | The following sections are available: 6 | 7 | .. contents:: 8 | :local: 9 | 10 | User documentation 11 | ------------------ 12 | 13 | The readme is the best place to start reading, it's targeted at all users and 14 | documents the command line interface: 15 | 16 | .. toctree:: 17 | readme.rst 18 | 19 | API documentation 20 | ----------------- 21 | 22 | The following API documentation is automatically generated from the source code: 23 | 24 | .. toctree:: 25 | api.rst 26 | 27 | Change log 28 | ---------- 29 | 30 | The change log lists notable changes to the project: 31 | 32 | .. toctree:: 33 | changelog.rst 34 | 35 | -------------------------------------------------------------------------------- /docs/readme.rst: -------------------------------------------------------------------------------- 1 | .. include:: ../README.rst 2 | -------------------------------------------------------------------------------- /requirements-checks.txt: -------------------------------------------------------------------------------- 1 | # Python packages required to run `make check'. 2 | 3 | flake8 >= 2.6.0 4 | flake8-docstrings >= 0.2.8 5 | pyflakes >= 1.2.3 6 | -------------------------------------------------------------------------------- /requirements-tests.txt: -------------------------------------------------------------------------------- 1 | # Python packages required to run `make test'. 2 | 3 | pytest >= 3.0.7 4 | pytest-cov >= 2.2.1 5 | -------------------------------------------------------------------------------- /requirements-travis.txt: -------------------------------------------------------------------------------- 1 | # Python packages required on Travis CI. 2 | 3 | # Pull in the requirements for code style checks, 4 | # the test suite and installation requirements. 5 | --requirement=requirements-checks.txt 6 | --requirement=requirements-tests.txt 7 | --requirement=requirements.txt 8 | 9 | # Prepare to submit coverage data. 10 | coveralls 11 | -------------------------------------------------------------------------------- /requirements.txt: -------------------------------------------------------------------------------- 1 | # Installation requirements. 2 | 3 | coloredlogs >= 5.1 4 | executor >= 23.1 5 | humanfriendly >= 8.0 6 | naturalsort >= 1.4 7 | property-manager >= 3.0 8 | python-dateutil >= 2.2 9 | simpleeval >= 0.8.7 10 | six >= 1.9.0 11 | update-dotdee >= 6.0 12 | verboselogs >= 1.5 13 | -------------------------------------------------------------------------------- /rotate_backups/__init__.py: -------------------------------------------------------------------------------- 1 | # rotate-backups: Simple command line interface for backup rotation. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: May 17, 2020 5 | # URL: https://github.com/xolox/python-rotate-backups 6 | 7 | """ 8 | Simple to use Python API for rotation of backups. 9 | 10 | The :mod:`rotate_backups` module contains the Python API of the 11 | `rotate-backups` package. The core logic of the package is contained in the 12 | :class:`RotateBackups` class. 13 | """ 14 | 15 | # Standard library modules. 16 | import collections 17 | import datetime 18 | import fnmatch 19 | import numbers 20 | import os 21 | import re 22 | import shlex 23 | 24 | # External dependencies. 25 | from dateutil.relativedelta import relativedelta 26 | from executor import ExternalCommandFailed 27 | from executor.concurrent import CommandPool 28 | from executor.contexts import RemoteContext, create_context 29 | from humanfriendly import Timer, coerce_boolean, coerce_pattern, format_path, parse_path 30 | from humanfriendly.text import concatenate, pluralize, split 31 | from natsort import natsort 32 | from property_manager import ( 33 | PropertyManager, 34 | cached_property, 35 | key_property, 36 | lazy_property, 37 | mutable_property, 38 | required_property, 39 | set_property, 40 | ) 41 | from simpleeval import simple_eval 42 | from six import string_types 43 | from update_dotdee import ConfigLoader 44 | from verboselogs import VerboseLogger 45 | 46 | # Semi-standard module versioning. 47 | __version__ = '8.1' 48 | 49 | # Initialize a logger for this module. 50 | logger = VerboseLogger(__name__) 51 | 52 | DEFAULT_REMOVAL_COMMAND = ['rm', '-fR'] 53 | """The default removal command (a list of strings).""" 54 | 55 | ORDERED_FREQUENCIES = ( 56 | ('minutely', relativedelta(minutes=1)), 57 | ('hourly', relativedelta(hours=1)), 58 | ('daily', relativedelta(days=1)), 59 | ('weekly', relativedelta(weeks=1)), 60 | ('monthly', relativedelta(months=1)), 61 | ('yearly', relativedelta(years=1)), 62 | ) 63 | """ 64 | An iterable of tuples with two values each: 65 | 66 | - The name of a rotation frequency (a string like 'hourly', 'daily', etc.). 67 | - A :class:`~dateutil.relativedelta.relativedelta` object. 68 | 69 | The tuples are sorted by increasing delta (intentionally). 70 | """ 71 | 72 | SUPPORTED_DATE_COMPONENTS = ( 73 | ('year', True), 74 | ('month', True), 75 | ('day', True), 76 | ('hour', False), 77 | ('minute', False), 78 | ('second', False), 79 | ) 80 | """ 81 | An iterable of tuples with two values each: 82 | 83 | - The name of a date component (a string). 84 | - :data:`True` for required components, :data:`False` for optional components. 85 | """ 86 | 87 | SUPPORTED_FREQUENCIES = dict(ORDERED_FREQUENCIES) 88 | """ 89 | A dictionary with rotation frequency names (strings) as keys and 90 | :class:`~dateutil.relativedelta.relativedelta` objects as values. This 91 | dictionary is generated based on the tuples in :data:`ORDERED_FREQUENCIES`. 92 | """ 93 | 94 | TIMESTAMP_PATTERN = re.compile(r''' 95 | # Required components. 96 | (?P\d{4} ) \D? 97 | (?P\d{2}) \D? 98 | (?P\d{2} ) \D? 99 | (?: 100 | # Optional components. 101 | (?P\d{2} ) \D? 102 | (?P\d{2}) \D? 103 | (?P\d{2})? 104 | )? 105 | ''', re.VERBOSE) 106 | """ 107 | A compiled regular expression object used to match timestamps encoded in 108 | filenames. 109 | """ 110 | 111 | 112 | def coerce_location(value, **options): 113 | """ 114 | Coerce a string to a :class:`Location` object. 115 | 116 | :param value: The value to coerce (a string or :class:`Location` object). 117 | :param options: Any keyword arguments are passed on to 118 | :func:`~executor.contexts.create_context()`. 119 | :returns: A :class:`Location` object. 120 | """ 121 | # Location objects pass through untouched. 122 | if not isinstance(value, Location): 123 | # Other values are expected to be strings. 124 | if not isinstance(value, string_types): 125 | msg = "Expected Location object or string, got %s instead!" 126 | raise ValueError(msg % type(value)) 127 | # Try to parse a remote location. 128 | ssh_alias, _, directory = value.partition(':') 129 | if ssh_alias and directory and '/' not in ssh_alias: 130 | options['ssh_alias'] = ssh_alias 131 | else: 132 | directory = value 133 | # Create the location object. 134 | value = Location( 135 | context=create_context(**options), 136 | directory=parse_path(directory), 137 | ) 138 | return value 139 | 140 | 141 | def coerce_retention_period(value): 142 | """ 143 | Coerce a retention period to a Python value. 144 | 145 | :param value: A string containing the text 'always', a number or 146 | an expression that can be evaluated to a number. 147 | :returns: A number or the string 'always'. 148 | :raises: :exc:`~exceptions.ValueError` when the string can't be coerced. 149 | """ 150 | # Numbers pass through untouched. 151 | if not isinstance(value, numbers.Number): 152 | # Other values are expected to be strings. 153 | if not isinstance(value, string_types): 154 | msg = "Expected string, got %s instead!" 155 | raise ValueError(msg % type(value)) 156 | # Check for the literal string `always'. 157 | value = value.strip() 158 | if value.lower() == 'always': 159 | value = 'always' 160 | else: 161 | # Evaluate other strings as expressions. 162 | value = simple_eval(value) 163 | if not isinstance(value, numbers.Number): 164 | msg = "Expected numeric result, got %s instead!" 165 | raise ValueError(msg % type(value)) 166 | return value 167 | 168 | 169 | def load_config_file(configuration_file=None, expand=True): 170 | """ 171 | Load a configuration file with backup directories and rotation schemes. 172 | 173 | :param configuration_file: Override the pathname of the configuration file 174 | to load (a string or :data:`None`). 175 | :param expand: :data:`True` to expand filename patterns to their matches, 176 | :data:`False` otherwise. 177 | :returns: A generator of tuples with four values each: 178 | 179 | 1. An execution context created using :mod:`executor.contexts`. 180 | 2. The pathname of a directory with backups (a string). 181 | 3. A dictionary with the rotation scheme. 182 | 4. A dictionary with additional options. 183 | :raises: :exc:`~exceptions.ValueError` when `configuration_file` is given 184 | but doesn't exist or can't be loaded. 185 | 186 | This function is used by :class:`RotateBackups` to discover user defined 187 | rotation schemes and by :mod:`rotate_backups.cli` to discover directories 188 | for which backup rotation is configured. When `configuration_file` isn't 189 | given :class:`~update_dotdee.ConfigLoader` is used to search for 190 | configuration files in the following locations: 191 | 192 | - ``/etc/rotate-backups.ini`` and ``/etc/rotate-backups.d/*.ini`` 193 | - ``~/.rotate-backups.ini`` and ``~/.rotate-backups.d/*.ini`` 194 | - ``~/.config/rotate-backups.ini`` and ``~/.config/rotate-backups.d/*.ini`` 195 | 196 | All of the available configuration files are loaded in the order given 197 | above, so that sections in user-specific configuration files override 198 | sections by the same name in system-wide configuration files. 199 | """ 200 | expand_notice_given = False 201 | if configuration_file: 202 | loader = ConfigLoader(available_files=[configuration_file], strict=True) 203 | else: 204 | loader = ConfigLoader(program_name='rotate-backups', strict=False) 205 | for section in loader.section_names: 206 | items = dict(loader.get_options(section)) 207 | context_options = {} 208 | if coerce_boolean(items.get('use-sudo')): 209 | context_options['sudo'] = True 210 | if items.get('ssh-user'): 211 | context_options['ssh_user'] = items['ssh-user'] 212 | location = coerce_location(section, **context_options) 213 | rotation_scheme = dict((name, coerce_retention_period(items[name])) 214 | for name in SUPPORTED_FREQUENCIES 215 | if name in items) 216 | options = dict( 217 | exclude_list=split(items.get('exclude-list', '')), 218 | include_list=split(items.get('include-list', '')), 219 | io_scheduling_class=items.get('ionice'), 220 | prefer_recent=coerce_boolean(items.get('prefer-recent', 'no')), 221 | strict=coerce_boolean(items.get('strict', 'yes')), 222 | ) 223 | # Don't override the value of the 'removal_command' property unless the 224 | # 'removal-command' configuration file option has a value set. 225 | if items.get('removal-command'): 226 | options['removal_command'] = shlex.split(items['removal-command']) 227 | # Don't override the value of the 'timestamp_pattern' property unless the 228 | # 'timestamp-pattern' configuration file option has a value set. 229 | if items.get('timestamp-pattern'): 230 | options['timestamp_pattern'] = items['timestamp-pattern'] 231 | # Expand filename patterns? 232 | if expand and location.have_wildcards: 233 | logger.verbose("Expanding filename pattern %s on %s ..", location.directory, location.context) 234 | if location.is_remote and not expand_notice_given: 235 | logger.notice("Expanding remote filename patterns (may be slow) ..") 236 | expand_notice_given = True 237 | for match in sorted(location.context.glob(location.directory)): 238 | if location.context.is_directory(match): 239 | logger.verbose("Matched directory: %s", match) 240 | expanded = Location(context=location.context, directory=match) 241 | yield expanded, rotation_scheme, options 242 | else: 243 | logger.verbose("Ignoring match (not a directory): %s", match) 244 | else: 245 | yield location, rotation_scheme, options 246 | 247 | 248 | def rotate_backups(directory, rotation_scheme, **options): 249 | """ 250 | Rotate the backups in a directory according to a flexible rotation scheme. 251 | 252 | .. note:: This function exists to preserve backwards compatibility with 253 | older versions of the `rotate-backups` package where all of the 254 | logic was exposed as a single function. Please refer to the 255 | documentation of the :class:`RotateBackups` initializer and the 256 | :func:`~RotateBackups.rotate_backups()` method for an explanation 257 | of this function's parameters. 258 | """ 259 | program = RotateBackups(rotation_scheme=rotation_scheme, **options) 260 | program.rotate_backups(directory) 261 | 262 | 263 | class RotateBackups(PropertyManager): 264 | 265 | """Python API for the ``rotate-backups`` program.""" 266 | 267 | def __init__(self, rotation_scheme, **options): 268 | """ 269 | Initialize a :class:`RotateBackups` object. 270 | 271 | :param rotation_scheme: Used to set :attr:`rotation_scheme`. 272 | :param options: Any keyword arguments are used to set the values of 273 | instance properties that support assignment 274 | (:attr:`config_file`, :attr:`dry_run`, 275 | :attr:`exclude_list`, :attr:`include_list`, 276 | :attr:`io_scheduling_class`, :attr:`removal_command` 277 | and :attr:`strict`). 278 | """ 279 | options.update(rotation_scheme=rotation_scheme) 280 | super(RotateBackups, self).__init__(**options) 281 | 282 | @mutable_property 283 | def config_file(self): 284 | """ 285 | The pathname of a configuration file (a string or :data:`None`). 286 | 287 | When this property is set :func:`rotate_backups()` will use 288 | :func:`load_config_file()` to give the user (operator) a chance to set 289 | the rotation scheme and other options via a configuration file. 290 | """ 291 | 292 | @mutable_property 293 | def dry_run(self): 294 | """ 295 | :data:`True` to simulate rotation, :data:`False` to actually remove backups (defaults to :data:`False`). 296 | 297 | If this is :data:`True` then :func:`rotate_backups()` won't make any 298 | actual changes, which provides a 'preview' of the effect of the 299 | rotation scheme. Right now this is only useful in the command line 300 | interface because there's no return value. 301 | """ 302 | return False 303 | 304 | @cached_property(writable=True) 305 | def exclude_list(self): 306 | """ 307 | Filename patterns to exclude specific backups (a list of strings). 308 | 309 | This is a list of strings with :mod:`fnmatch` patterns. When 310 | :func:`collect_backups()` encounters a backup whose name matches any of 311 | the patterns in this list the backup will be ignored, *even if it also 312 | matches the include list* (it's the only logical way to combine both 313 | lists). 314 | 315 | :see also: :attr:`include_list` 316 | """ 317 | return [] 318 | 319 | @mutable_property 320 | def force(self): 321 | """ 322 | :data:`True` to continue if sanity checks fail, :data:`False` to raise an exception. 323 | 324 | Sanity checks are performed before backup rotation starts to ensure 325 | that the given location exists, is readable and is writable. If 326 | :attr:`removal_command` is customized then the last sanity check (that 327 | the given location is writable) is skipped (because custom removal 328 | commands imply custom semantics, see also `#18`_). If a sanity check 329 | fails an exception is raised, but you can set :attr:`force` to 330 | :data:`True` to continue with backup rotation instead (the default is 331 | obviously :data:`False`). 332 | 333 | .. seealso:: :func:`Location.ensure_exists()`, 334 | :func:`Location.ensure_readable()` and 335 | :func:`Location.ensure_writable()` 336 | 337 | .. _#18: https://github.com/xolox/python-rotate-backups/issues/18 338 | """ 339 | return False 340 | 341 | @cached_property(writable=True) 342 | def include_list(self): 343 | """ 344 | Filename patterns to select specific backups (a list of strings). 345 | 346 | This is a list of strings with :mod:`fnmatch` patterns. When it's not 347 | empty :func:`collect_backups()` will only collect backups whose name 348 | matches a pattern in the list. 349 | 350 | :see also: :attr:`exclude_list` 351 | """ 352 | return [] 353 | 354 | @mutable_property 355 | def io_scheduling_class(self): 356 | """ 357 | The I/O scheduling class for backup rotation (a string or :data:`None`). 358 | 359 | When this property is set (and :attr:`~Location.have_ionice` is 360 | :data:`True`) then ionice_ will be used to set the I/O scheduling class 361 | for backup rotation. This can be useful to reduce the impact of backup 362 | rotation on the rest of the system. 363 | 364 | The value of this property is expected to be one of the strings 'idle', 365 | 'best-effort' or 'realtime'. 366 | 367 | .. _ionice: https://linux.die.net/man/1/ionice 368 | """ 369 | 370 | @mutable_property 371 | def prefer_recent(self): 372 | """ 373 | Whether to prefer older or newer backups in each time slot (a boolean). 374 | 375 | Defaults to :data:`False` which means the oldest backup in each time 376 | slot (an hour, a day, etc.) is preserved while newer backups in the 377 | time slot are removed. You can set this to :data:`True` if you would 378 | like to preserve the newest backup in each time slot instead. 379 | """ 380 | return False 381 | 382 | @mutable_property 383 | def removal_command(self): 384 | """ 385 | The command used to remove backups (a list of strings). 386 | 387 | By default the command ``rm -fR`` is used. This choice was made because 388 | it works regardless of whether the user's "backups to be rotated" are 389 | files or directories or a mixture of both. 390 | 391 | .. versionadded: 5.3 392 | This option was added as a generalization of the idea suggested in 393 | `pull request 11`_, which made it clear to me that being able to 394 | customize the removal command has its uses. 395 | 396 | .. _pull request 11: https://github.com/xolox/python-rotate-backups/pull/11 397 | """ 398 | return DEFAULT_REMOVAL_COMMAND 399 | 400 | @required_property 401 | def rotation_scheme(self): 402 | """ 403 | The rotation scheme to apply to backups (a dictionary). 404 | 405 | Each key in this dictionary defines a rotation frequency (one of the 406 | strings 'minutely', 'hourly', 'daily', 'weekly', 'monthly' and 407 | 'yearly') and each value defines a retention count: 408 | 409 | - An integer value represents the number of backups to preserve in the 410 | given rotation frequency, starting from the most recent backup and 411 | counting back in time. 412 | 413 | - The string 'always' means all backups in the given rotation frequency 414 | are preserved (this is intended to be used with the biggest frequency 415 | in the rotation scheme, e.g. yearly). 416 | 417 | No backups are preserved for rotation frequencies that are not present 418 | in the dictionary. 419 | """ 420 | 421 | @mutable_property 422 | def strict(self): 423 | """ 424 | Whether to enforce the time window for each rotation frequency (a boolean, defaults to :data:`True`). 425 | 426 | The easiest way to explain the difference between strict and relaxed 427 | rotation is using an example: 428 | 429 | - If :attr:`strict` is :data:`True` and the number of hourly backups to 430 | preserve is three, only backups created in the relevant time window 431 | (the hour of the most recent backup and the two hours leading up to 432 | that) will match the hourly frequency. 433 | 434 | - If :attr:`strict` is :data:`False` then the three most recent backups 435 | will all match the hourly frequency (and thus be preserved), 436 | regardless of the calculated time window. 437 | 438 | If the explanation above is not clear enough, here's a simple way to 439 | decide whether you want to customize this behavior: 440 | 441 | - If your backups are created at regular intervals and you never miss 442 | an interval then the default (:data:`True`) is most likely fine. 443 | 444 | - If your backups are created at irregular intervals then you may want 445 | to set :attr:`strict` to :data:`False` to convince 446 | :class:`RotateBackups` to preserve more backups. 447 | """ 448 | return True 449 | 450 | @mutable_property 451 | def timestamp_pattern(self): 452 | """ 453 | The pattern used to extract timestamps from filenames (defaults to :data:`TIMESTAMP_PATTERN`). 454 | 455 | The value of this property is a compiled regular expression object. 456 | Callers can provide their own compiled regular expression which 457 | makes it possible to customize the compilation flags (see the 458 | :func:`re.compile()` documentation for details). 459 | 460 | The regular expression pattern is expected to be a Python compatible 461 | regular expression that defines the named capture groups 'year', 462 | 'month' and 'day' and optionally 'hour', 'minute' and 'second'. 463 | 464 | String values are automatically coerced to compiled regular expressions 465 | by calling :func:`~humanfriendly.coerce_pattern()`, in this case only 466 | the :data:`re.VERBOSE` flag is used. 467 | 468 | If the caller provides a custom pattern it will be validated 469 | to confirm that the pattern contains named capture groups 470 | corresponding to each of the required date components 471 | defined by :data:`SUPPORTED_DATE_COMPONENTS`. 472 | """ 473 | return TIMESTAMP_PATTERN 474 | 475 | @timestamp_pattern.setter 476 | def timestamp_pattern(self, value): 477 | """Coerce the value of :attr:`timestamp_pattern` to a compiled regular expression.""" 478 | pattern = coerce_pattern(value, re.VERBOSE) 479 | for component, required in SUPPORTED_DATE_COMPONENTS: 480 | if component not in pattern.groupindex and required: 481 | raise ValueError("Pattern is missing required capture group! (%s)" % component) 482 | set_property(self, 'timestamp_pattern', pattern) 483 | 484 | def rotate_concurrent(self, *locations, **kw): 485 | """ 486 | Rotate the backups in the given locations concurrently. 487 | 488 | :param locations: One or more values accepted by :func:`coerce_location()`. 489 | :param kw: Any keyword arguments are passed on to :func:`rotate_backups()`. 490 | 491 | This function uses :func:`rotate_backups()` to prepare rotation 492 | commands for the given locations and then it removes backups in 493 | parallel, one backup per mount point at a time. 494 | 495 | The idea behind this approach is that parallel rotation is most useful 496 | when the files to be removed are on different disks and so multiple 497 | devices can be utilized at the same time. 498 | 499 | Because mount points are per system :func:`rotate_concurrent()` will 500 | also parallelize over backups located on multiple remote systems. 501 | """ 502 | timer = Timer() 503 | pool = CommandPool(concurrency=10) 504 | logger.info("Scanning %s ..", pluralize(len(locations), "backup location")) 505 | for location in locations: 506 | for cmd in self.rotate_backups(location, prepare=True, **kw): 507 | pool.add(cmd) 508 | if pool.num_commands > 0: 509 | backups = pluralize(pool.num_commands, "backup") 510 | logger.info("Preparing to rotate %s (in parallel) ..", backups) 511 | pool.run() 512 | logger.info("Successfully rotated %s in %s.", backups, timer) 513 | 514 | def rotate_backups(self, location, load_config=True, prepare=False): 515 | """ 516 | Rotate the backups in a directory according to a flexible rotation scheme. 517 | 518 | :param location: Any value accepted by :func:`coerce_location()`. 519 | :param load_config: If :data:`True` (so by default) the rotation scheme 520 | and other options can be customized by the user in 521 | a configuration file. In this case the caller's 522 | arguments are only used when the configuration file 523 | doesn't define a configuration for the location. 524 | :param prepare: If this is :data:`True` (not the default) then 525 | :func:`rotate_backups()` will prepare the required 526 | rotation commands without running them. 527 | :returns: A list with the rotation commands 528 | (:class:`~executor.ExternalCommand` objects). 529 | :raises: :exc:`~exceptions.ValueError` when the given location doesn't 530 | exist, isn't readable or isn't writable. The third check is 531 | only performed when dry run isn't enabled. 532 | 533 | This function binds the main methods of the :class:`RotateBackups` 534 | class together to implement backup rotation with an easy to use Python 535 | API. If you're using `rotate-backups` as a Python API and the default 536 | behavior is not satisfactory, consider writing your own 537 | :func:`rotate_backups()` function based on the underlying 538 | :func:`collect_backups()`, :func:`group_backups()`, 539 | :func:`apply_rotation_scheme()` and 540 | :func:`find_preservation_criteria()` methods. 541 | """ 542 | rotation_commands = [] 543 | location = coerce_location(location) 544 | # Load configuration overrides by user? 545 | if load_config: 546 | location = self.load_config_file(location) 547 | # Collect the backups in the given directory. 548 | sorted_backups = self.collect_backups(location) 549 | if not sorted_backups: 550 | logger.info("No backups found in %s.", location) 551 | return 552 | # Make sure the directory is writable, but only when the default 553 | # removal command is being used (because custom removal commands 554 | # imply custom semantics that we shouldn't get in the way of, see 555 | # https://github.com/xolox/python-rotate-backups/issues/18 for 556 | # more details about one such use case). 557 | if not self.dry_run and (self.removal_command == DEFAULT_REMOVAL_COMMAND): 558 | location.ensure_writable(self.force) 559 | most_recent_backup = sorted_backups[-1] 560 | # Group the backups by the rotation frequencies. 561 | backups_by_frequency = self.group_backups(sorted_backups) 562 | # Apply the user defined rotation scheme. 563 | self.apply_rotation_scheme(backups_by_frequency, most_recent_backup.timestamp) 564 | # Find which backups to preserve and why. 565 | backups_to_preserve = self.find_preservation_criteria(backups_by_frequency) 566 | # Apply the calculated rotation scheme. 567 | for backup in sorted_backups: 568 | friendly_name = backup.pathname 569 | if not location.is_remote: 570 | # Use human friendly pathname formatting for local backups. 571 | friendly_name = format_path(backup.pathname) 572 | if backup in backups_to_preserve: 573 | matching_periods = backups_to_preserve[backup] 574 | logger.info("Preserving %s (matches %s retention %s) ..", 575 | friendly_name, concatenate(map(repr, matching_periods)), 576 | "period" if len(matching_periods) == 1 else "periods") 577 | else: 578 | logger.info("Deleting %s ..", friendly_name) 579 | if not self.dry_run: 580 | # Copy the list with the (possibly user defined) removal command. 581 | removal_command = list(self.removal_command) 582 | # Add the pathname of the backup as the final argument. 583 | removal_command.append(backup.pathname) 584 | # Construct the command object. 585 | command = location.context.prepare( 586 | command=removal_command, 587 | group_by=(location.ssh_alias, location.mount_point), 588 | ionice=self.io_scheduling_class, 589 | ) 590 | rotation_commands.append(command) 591 | if not prepare: 592 | timer = Timer() 593 | command.wait() 594 | logger.verbose("Deleted %s in %s.", friendly_name, timer) 595 | if len(backups_to_preserve) == len(sorted_backups): 596 | logger.info("Nothing to do! (all backups preserved)") 597 | return rotation_commands 598 | 599 | def load_config_file(self, location): 600 | """ 601 | Load a rotation scheme and other options from a configuration file. 602 | 603 | :param location: Any value accepted by :func:`coerce_location()`. 604 | :returns: The configured or given :class:`Location` object. 605 | """ 606 | location = coerce_location(location) 607 | for configured_location, rotation_scheme, options in load_config_file(self.config_file, expand=False): 608 | if configured_location.match(location): 609 | logger.verbose("Loading configuration for %s ..", location) 610 | if rotation_scheme: 611 | self.rotation_scheme = rotation_scheme 612 | for name, value in options.items(): 613 | if value: 614 | setattr(self, name, value) 615 | # Create a new Location object based on the directory of the 616 | # given location and the execution context of the configured 617 | # location, because: 618 | # 619 | # 1. The directory of the configured location may be a filename 620 | # pattern whereas we are interested in the expanded name. 621 | # 622 | # 2. The execution context of the given location may lack some 623 | # details of the configured location. 624 | return Location( 625 | context=configured_location.context, 626 | directory=location.directory, 627 | ) 628 | logger.verbose("No configuration found for %s.", location) 629 | return location 630 | 631 | def collect_backups(self, location): 632 | """ 633 | Collect the backups at the given location. 634 | 635 | :param location: Any value accepted by :func:`coerce_location()`. 636 | :returns: A sorted :class:`list` of :class:`Backup` objects (the 637 | backups are sorted by their date). 638 | :raises: :exc:`~exceptions.ValueError` when the given directory doesn't 639 | exist or isn't readable. 640 | """ 641 | backups = [] 642 | location = coerce_location(location) 643 | logger.info("Scanning %s for backups ..", location) 644 | location.ensure_readable(self.force) 645 | for entry in natsort(location.context.list_entries(location.directory)): 646 | match = self.timestamp_pattern.search(entry) 647 | if match: 648 | if self.exclude_list and any(fnmatch.fnmatch(entry, p) for p in self.exclude_list): 649 | logger.verbose("Excluded %s (it matched the exclude list).", entry) 650 | elif self.include_list and not any(fnmatch.fnmatch(entry, p) for p in self.include_list): 651 | logger.verbose("Excluded %s (it didn't match the include list).", entry) 652 | else: 653 | try: 654 | backups.append(Backup( 655 | pathname=os.path.join(location.directory, entry), 656 | timestamp=self.match_to_datetime(match), 657 | )) 658 | except ValueError as e: 659 | logger.notice("Ignoring %s due to invalid date (%s).", entry, e) 660 | else: 661 | logger.debug("Failed to match time stamp in filename: %s", entry) 662 | if backups: 663 | logger.info("Found %i timestamped backups in %s.", len(backups), location) 664 | return sorted(backups) 665 | 666 | def match_to_datetime(self, match): 667 | """ 668 | Convert a regular expression match to a :class:`~datetime.datetime` value. 669 | 670 | :param match: A regular expression match object. 671 | :returns: A :class:`~datetime.datetime` value. 672 | :raises: :exc:`exceptions.ValueError` when a required date component is 673 | not captured by the pattern, the captured value is an empty 674 | string or the captured value cannot be interpreted as a 675 | base-10 integer. 676 | 677 | .. seealso:: :data:`SUPPORTED_DATE_COMPONENTS` 678 | """ 679 | kw = {} 680 | captures = match.groupdict() 681 | for component, required in SUPPORTED_DATE_COMPONENTS: 682 | value = captures.get(component) 683 | if value: 684 | kw[component] = int(value, 10) 685 | elif required: 686 | raise ValueError("Missing required date component! (%s)" % component) 687 | else: 688 | kw[component] = 0 689 | return datetime.datetime(**kw) 690 | 691 | def group_backups(self, backups): 692 | """ 693 | Group backups collected by :func:`collect_backups()` by rotation frequencies. 694 | 695 | :param backups: A :class:`set` of :class:`Backup` objects. 696 | :returns: A :class:`dict` whose keys are the names of rotation 697 | frequencies ('hourly', 'daily', etc.) and whose values are 698 | dictionaries. Each nested dictionary contains lists of 699 | :class:`Backup` objects that are grouped together because 700 | they belong into the same time unit for the corresponding 701 | rotation frequency. 702 | """ 703 | backups_by_frequency = dict((frequency, collections.defaultdict(list)) for frequency in SUPPORTED_FREQUENCIES) 704 | for b in backups: 705 | backups_by_frequency['minutely'][(b.year, b.month, b.day, b.hour, b.minute)].append(b) 706 | backups_by_frequency['hourly'][(b.year, b.month, b.day, b.hour)].append(b) 707 | backups_by_frequency['daily'][(b.year, b.month, b.day)].append(b) 708 | backups_by_frequency['weekly'][(b.year, b.week)].append(b) 709 | backups_by_frequency['monthly'][(b.year, b.month)].append(b) 710 | backups_by_frequency['yearly'][b.year].append(b) 711 | return backups_by_frequency 712 | 713 | def apply_rotation_scheme(self, backups_by_frequency, most_recent_backup): 714 | """ 715 | Apply the user defined rotation scheme to the result of :func:`group_backups()`. 716 | 717 | :param backups_by_frequency: A :class:`dict` in the format generated by 718 | :func:`group_backups()`. 719 | :param most_recent_backup: The :class:`~datetime.datetime` of the most 720 | recent backup. 721 | :raises: :exc:`~exceptions.ValueError` when the rotation scheme 722 | dictionary is empty (this would cause all backups to be 723 | deleted). 724 | 725 | .. note:: This method mutates the given data structure by removing all 726 | backups that should be removed to apply the user defined 727 | rotation scheme. 728 | """ 729 | if not self.rotation_scheme: 730 | raise ValueError("Refusing to use empty rotation scheme! (all backups would be deleted)") 731 | for frequency, backups in backups_by_frequency.items(): 732 | # Ignore frequencies not specified by the user. 733 | if frequency not in self.rotation_scheme: 734 | backups.clear() 735 | else: 736 | # Reduce the number of backups in each time slot of this 737 | # rotation frequency to a single backup (the oldest one or the 738 | # newest one). 739 | for period, backups_in_period in backups.items(): 740 | index = -1 if self.prefer_recent else 0 741 | selected_backup = sorted(backups_in_period)[index] 742 | backups[period] = [selected_backup] 743 | # Check if we need to rotate away backups in old periods. 744 | retention_period = self.rotation_scheme[frequency] 745 | if retention_period != 'always': 746 | # Remove backups created before the minimum date of this 747 | # rotation frequency? (relative to the most recent backup) 748 | if self.strict: 749 | minimum_date = most_recent_backup - SUPPORTED_FREQUENCIES[frequency] * retention_period 750 | for period, backups_in_period in list(backups.items()): 751 | for backup in backups_in_period: 752 | if backup.timestamp < minimum_date: 753 | backups_in_period.remove(backup) 754 | if not backups_in_period: 755 | backups.pop(period) 756 | # If there are more periods remaining than the user 757 | # requested to be preserved we delete the oldest one(s). 758 | items_to_preserve = sorted(backups.items())[-retention_period:] 759 | backups_by_frequency[frequency] = dict(items_to_preserve) 760 | 761 | def find_preservation_criteria(self, backups_by_frequency): 762 | """ 763 | Collect the criteria used to decide which backups to preserve. 764 | 765 | :param backups_by_frequency: A :class:`dict` in the format generated by 766 | :func:`group_backups()` which has been 767 | processed by :func:`apply_rotation_scheme()`. 768 | :returns: A :class:`dict` with :class:`Backup` objects as keys and 769 | :class:`list` objects containing strings (rotation 770 | frequencies) as values. 771 | """ 772 | backups_to_preserve = collections.defaultdict(list) 773 | for frequency, delta in ORDERED_FREQUENCIES: 774 | for period in backups_by_frequency[frequency].values(): 775 | for backup in period: 776 | backups_to_preserve[backup].append(frequency) 777 | return backups_to_preserve 778 | 779 | 780 | class Location(PropertyManager): 781 | 782 | """:class:`Location` objects represent a root directory containing backups.""" 783 | 784 | @required_property 785 | def context(self): 786 | """An execution context created using :mod:`executor.contexts`.""" 787 | 788 | @required_property 789 | def directory(self): 790 | """The pathname of a directory containing backups (a string).""" 791 | 792 | @lazy_property 793 | def have_ionice(self): 794 | """:data:`True` when ionice_ is available, :data:`False` otherwise.""" 795 | return self.context.have_ionice 796 | 797 | @lazy_property 798 | def have_wildcards(self): 799 | """:data:`True` if :attr:`directory` is a filename pattern, :data:`False` otherwise.""" 800 | return '*' in self.directory 801 | 802 | @lazy_property 803 | def mount_point(self): 804 | """ 805 | The pathname of the mount point of :attr:`directory` (a string or :data:`None`). 806 | 807 | If the ``stat --format=%m ...`` command that is used to determine the 808 | mount point fails, the value of this property defaults to :data:`None`. 809 | This enables graceful degradation on e.g. Mac OS X whose ``stat`` 810 | implementation is rather bare bones compared to GNU/Linux. 811 | """ 812 | try: 813 | return self.context.capture('stat', '--format=%m', self.directory, silent=True) 814 | except ExternalCommandFailed: 815 | return None 816 | 817 | @lazy_property 818 | def is_remote(self): 819 | """:data:`True` if the location is remote, :data:`False` otherwise.""" 820 | return isinstance(self.context, RemoteContext) 821 | 822 | @lazy_property 823 | def ssh_alias(self): 824 | """The SSH alias of a remote location (a string or :data:`None`).""" 825 | return self.context.ssh_alias if self.is_remote else None 826 | 827 | @property 828 | def key_properties(self): 829 | """ 830 | A list of strings with the names of the :attr:`~custom_property.key` properties. 831 | 832 | Overrides :attr:`~property_manager.PropertyManager.key_properties` to 833 | customize the ordering of :class:`Location` objects so that they are 834 | ordered first by their :attr:`ssh_alias` and second by their 835 | :attr:`directory`. 836 | """ 837 | return ['ssh_alias', 'directory'] if self.is_remote else ['directory'] 838 | 839 | def ensure_exists(self, override=False): 840 | """ 841 | Sanity check that the location exists. 842 | 843 | :param override: :data:`True` to log a message, :data:`False` to raise 844 | an exception (when the sanity check fails). 845 | :returns: :data:`True` if the sanity check succeeds, 846 | :data:`False` if it fails (and `override` is :data:`True`). 847 | :raises: :exc:`~exceptions.ValueError` when the sanity 848 | check fails and `override` is :data:`False`. 849 | 850 | .. seealso:: :func:`ensure_readable()`, :func:`ensure_writable()` and :func:`add_hints()` 851 | """ 852 | if self.context.is_directory(self.directory): 853 | logger.verbose("Confirmed that location exists: %s", self) 854 | return True 855 | elif override: 856 | logger.notice("It seems %s doesn't exist but --force was given so continuing anyway ..", self) 857 | return False 858 | else: 859 | message = "It seems %s doesn't exist or isn't accessible due to filesystem permissions!" 860 | raise ValueError(self.add_hints(message % self)) 861 | 862 | def ensure_readable(self, override=False): 863 | """ 864 | Sanity check that the location exists and is readable. 865 | 866 | :param override: :data:`True` to log a message, :data:`False` to raise 867 | an exception (when the sanity check fails). 868 | :returns: :data:`True` if the sanity check succeeds, 869 | :data:`False` if it fails (and `override` is :data:`True`). 870 | :raises: :exc:`~exceptions.ValueError` when the sanity 871 | check fails and `override` is :data:`False`. 872 | 873 | .. seealso:: :func:`ensure_exists()`, :func:`ensure_writable()` and :func:`add_hints()` 874 | """ 875 | # Only sanity check that the location is readable when its 876 | # existence has been confirmed, to avoid multiple notices 877 | # about the same underlying problem. 878 | if self.ensure_exists(override): 879 | if self.context.is_readable(self.directory): 880 | logger.verbose("Confirmed that location is readable: %s", self) 881 | return True 882 | elif override: 883 | logger.notice("It seems %s isn't readable but --force was given so continuing anyway ..", self) 884 | else: 885 | message = "It seems %s isn't readable!" 886 | raise ValueError(self.add_hints(message % self)) 887 | return False 888 | 889 | def ensure_writable(self, override=False): 890 | """ 891 | Sanity check that the directory exists and is writable. 892 | 893 | :param override: :data:`True` to log a message, :data:`False` to raise 894 | an exception (when the sanity check fails). 895 | :returns: :data:`True` if the sanity check succeeds, 896 | :data:`False` if it fails (and `override` is :data:`True`). 897 | :raises: :exc:`~exceptions.ValueError` when the sanity 898 | check fails and `override` is :data:`False`. 899 | 900 | .. seealso:: :func:`ensure_exists()`, :func:`ensure_readable()` and :func:`add_hints()` 901 | """ 902 | # Only sanity check that the location is readable when its 903 | # existence has been confirmed, to avoid multiple notices 904 | # about the same underlying problem. 905 | if self.ensure_exists(override): 906 | if self.context.is_writable(self.directory): 907 | logger.verbose("Confirmed that location is writable: %s", self) 908 | return True 909 | elif override: 910 | logger.notice("It seems %s isn't writable but --force was given so continuing anyway ..", self) 911 | else: 912 | message = "It seems %s isn't writable!" 913 | raise ValueError(self.add_hints(message % self)) 914 | return False 915 | 916 | def add_hints(self, message): 917 | """ 918 | Provide hints about failing sanity checks. 919 | 920 | :param message: The message to the user (a string). 921 | :returns: The message including hints (a string). 922 | 923 | When superuser privileges aren't being used a hint about the 924 | ``--use-sudo`` option will be added (in case a sanity check failed 925 | because we don't have permission to one of the parent directories). 926 | 927 | In all cases a hint about the ``--force`` option is added (in case the 928 | sanity checks themselves are considered the problem, which is obviously 929 | up to the operator to decide). 930 | 931 | .. seealso:: :func:`ensure_exists()`, :func:`ensure_readable()` and :func:`ensure_writable()` 932 | """ 933 | sentences = [message] 934 | if not self.context.have_superuser_privileges: 935 | sentences.append("If filesystem permissions are the problem consider using the --use-sudo option.") 936 | sentences.append("To continue despite this failing sanity check you can use --force.") 937 | return " ".join(sentences) 938 | 939 | def match(self, location): 940 | """ 941 | Check if the given location "matches". 942 | 943 | :param location: The :class:`Location` object to try to match. 944 | :returns: :data:`True` if the two locations are on the same system and 945 | the :attr:`directory` can be matched as a filename pattern or 946 | a literal match on the normalized pathname. 947 | """ 948 | if self.ssh_alias != location.ssh_alias: 949 | # Never match locations on other systems. 950 | return False 951 | elif self.have_wildcards: 952 | # Match filename patterns using fnmatch(). 953 | return fnmatch.fnmatch(location.directory, self.directory) 954 | else: 955 | # Compare normalized directory pathnames. 956 | self = os.path.normpath(self.directory) 957 | other = os.path.normpath(location.directory) 958 | return self == other 959 | 960 | def __str__(self): 961 | """Render a simple human readable representation of a location.""" 962 | return '%s:%s' % (self.ssh_alias, self.directory) if self.ssh_alias else self.directory 963 | 964 | 965 | class Backup(PropertyManager): 966 | 967 | """:class:`Backup` objects represent a rotation subject.""" 968 | 969 | key_properties = 'timestamp', 'pathname' 970 | """ 971 | Customize the ordering of :class:`Backup` objects. 972 | 973 | :class:`Backup` objects are ordered first by their :attr:`timestamp` and 974 | second by their :attr:`pathname`. This class variable overrides 975 | :attr:`~property_manager.PropertyManager.key_properties`. 976 | """ 977 | 978 | @key_property 979 | def pathname(self): 980 | """The pathname of the backup (a string).""" 981 | 982 | @key_property 983 | def timestamp(self): 984 | """The date and time when the backup was created (a :class:`~datetime.datetime` object).""" 985 | 986 | @property 987 | def week(self): 988 | """The ISO week number of :attr:`timestamp` (a number).""" 989 | return self.timestamp.isocalendar()[1] 990 | 991 | def __getattr__(self, name): 992 | """Defer attribute access to :attr:`timestamp`.""" 993 | return getattr(self.timestamp, name) 994 | -------------------------------------------------------------------------------- /rotate_backups/cli.py: -------------------------------------------------------------------------------- 1 | # rotate-backups: Simple command line interface for backup rotation. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: May 17, 2020 5 | # URL: https://github.com/xolox/python-rotate-backups 6 | 7 | """ 8 | Usage: rotate-backups [OPTIONS] [DIRECTORY, ..] 9 | 10 | Easy rotation of backups based on the Python package by the same name. 11 | 12 | To use this program you specify a rotation scheme via (a combination of) the 13 | --hourly, --daily, --weekly, --monthly and/or --yearly options and the 14 | directory (or directories) containing backups to rotate as one or more 15 | positional arguments. 16 | 17 | You can rotate backups on a remote system over SSH by prefixing a DIRECTORY 18 | with an SSH alias and separating the two with a colon (similar to how rsync 19 | accepts remote locations). 20 | 21 | Instead of specifying directories and a rotation scheme on the command line you 22 | can also add them to a configuration file. For more details refer to the online 23 | documentation (see also the --config option). 24 | 25 | Please use the --dry-run option to test the effect of the specified rotation 26 | scheme before letting this program loose on your precious backups! If you don't 27 | test the results using the dry run mode and this program eats more backups than 28 | intended you have no right to complain ;-). 29 | 30 | Supported options: 31 | 32 | -M, --minutely=COUNT 33 | 34 | In a literal sense this option sets the number of "backups per minute" to 35 | preserve during rotation. For most use cases that doesn't make a lot of 36 | sense :-) but you can combine the --minutely and --relaxed options to 37 | preserve more than one backup per hour. Refer to the usage of the -H, 38 | --hourly option for details about COUNT. 39 | 40 | -H, --hourly=COUNT 41 | 42 | Set the number of hourly backups to preserve during rotation: 43 | 44 | - If COUNT is a number it gives the number of hourly backups to preserve, 45 | starting from the most recent hourly backup and counting back in time. 46 | - Alternatively you can provide an expression that will be evaluated to get 47 | a number (e.g. if COUNT is `7 * 2' the result would be 14). 48 | - You can also pass `always' for COUNT, in this case all hourly backups are 49 | preserved. 50 | - By default no hourly backups are preserved. 51 | 52 | -d, --daily=COUNT 53 | 54 | Set the number of daily backups to preserve during rotation. Refer to the 55 | usage of the -H, --hourly option for details about COUNT. 56 | 57 | -w, --weekly=COUNT 58 | 59 | Set the number of weekly backups to preserve during rotation. Refer to the 60 | usage of the -H, --hourly option for details about COUNT. 61 | 62 | -m, --monthly=COUNT 63 | 64 | Set the number of monthly backups to preserve during rotation. Refer to the 65 | usage of the -H, --hourly option for details about COUNT. 66 | 67 | -y, --yearly=COUNT 68 | 69 | Set the number of yearly backups to preserve during rotation. Refer to the 70 | usage of the -H, --hourly option for details about COUNT. 71 | 72 | -t, --timestamp-pattern=PATTERN 73 | 74 | Customize the regular expression pattern that is used to match and extract 75 | timestamps from filenames. PATTERN is expected to be a Python compatible 76 | regular expression that must define the named capture groups 'year', 77 | 'month' and 'day' and may define 'hour', 'minute' and 'second'. 78 | 79 | -I, --include=PATTERN 80 | 81 | Only process backups that match the shell pattern given by PATTERN. This 82 | argument can be repeated. Make sure to quote PATTERN so the shell doesn't 83 | expand the pattern before it's received by rotate-backups. 84 | 85 | -x, --exclude=PATTERN 86 | 87 | Don't process backups that match the shell pattern given by PATTERN. This 88 | argument can be repeated. Make sure to quote PATTERN so the shell doesn't 89 | expand the pattern before it's received by rotate-backups. 90 | 91 | -j, --parallel 92 | 93 | Remove backups in parallel, one backup per mount point at a time. The idea 94 | behind this approach is that parallel rotation is most useful when the 95 | files to be removed are on different disks and so multiple devices can be 96 | utilized at the same time. 97 | 98 | Because mount points are per system the -j, --parallel option will also 99 | parallelize over backups located on multiple remote systems. 100 | 101 | -p, --prefer-recent 102 | 103 | By default the first (oldest) backup in each time slot is preserved. If 104 | you'd prefer to keep the most recent backup in each time slot instead then 105 | this option is for you. 106 | 107 | -r, --relaxed 108 | 109 | By default the time window for each rotation scheme is enforced (this is 110 | referred to as strict rotation) but the -r, --relaxed option can be used 111 | to alter this behavior. The easiest way to explain the difference between 112 | strict and relaxed rotation is using an example: 113 | 114 | - When using strict rotation and the number of hourly backups to preserve 115 | is three, only backups created in the relevant time window (the hour of 116 | the most recent backup and the two hours leading up to that) will match 117 | the hourly frequency. 118 | 119 | - When using relaxed rotation the three most recent backups will all match 120 | the hourly frequency (and thus be preserved), regardless of the 121 | calculated time window. 122 | 123 | If the explanation above is not clear enough, here's a simple way to decide 124 | whether you want to customize this behavior or not: 125 | 126 | - If your backups are created at regular intervals and you never miss an 127 | interval then strict rotation (the default) is probably the best choice. 128 | 129 | - If your backups are created at irregular intervals then you may want to 130 | use the -r, --relaxed option in order to preserve more backups. 131 | 132 | -i, --ionice=CLASS 133 | 134 | Use the `ionice' program to set the I/O scheduling class and priority of 135 | the `rm' invocations used to remove backups. CLASS is expected to be one of 136 | the values `idle' (3), `best-effort' (2) or `realtime' (1). Refer to the 137 | man page of the `ionice' program for details about these values. The 138 | numeric values are required by the 'busybox' implementation of 'ionice'. 139 | 140 | -c, --config=FILENAME 141 | 142 | Load configuration from FILENAME. If this option isn't given the following 143 | default locations are searched for configuration files: 144 | 145 | - /etc/rotate-backups.ini and /etc/rotate-backups.d/*.ini 146 | - ~/.rotate-backups.ini and ~/.rotate-backups.d/*.ini 147 | - ~/.config/rotate-backups.ini and ~/.config/rotate-backups.d/*.ini 148 | 149 | Any available configuration files are loaded in the order given above, so 150 | that sections in user-specific configuration files override sections by the 151 | same name in system-wide configuration files. For more details refer to the 152 | online documentation. 153 | 154 | -C, --removal-command=CMD 155 | 156 | Change the command used to remove backups. The value of CMD defaults to 157 | ``rm -fR``. This choice was made because it works regardless of whether 158 | "backups to be rotated" are files or directories or a mixture of both. 159 | 160 | As an example of why you might want to change this, CephFS snapshots are 161 | represented as regular directory trees that can be deleted at once with a 162 | single 'rmdir' command (even though according to POSIX semantics this 163 | command should refuse to remove nonempty directories, but I digress). 164 | 165 | -u, --use-sudo 166 | 167 | Enable the use of `sudo' to rotate backups in directories that are not 168 | readable and/or writable for the current user (or the user logged in to a 169 | remote system over SSH). 170 | 171 | -S, --syslog=CHOICE 172 | 173 | Explicitly enable or disable system logging instead of letting the program 174 | figure out what to do. The values '1', 'yes', 'true' and 'on' enable system 175 | logging whereas the values '0', 'no', 'false' and 'off' disable it. 176 | 177 | -f, --force 178 | 179 | If a sanity check fails an error is reported and the program aborts. You 180 | can use --force to continue with backup rotation instead. Sanity checks 181 | are done to ensure that the given DIRECTORY exists, is readable and is 182 | writable. If the --removal-command option is given then the last sanity 183 | check (that the given location is writable) is skipped (because custom 184 | removal commands imply custom semantics). 185 | 186 | -n, --dry-run 187 | 188 | Don't make any changes, just print what would be done. This makes it easy 189 | to evaluate the impact of a rotation scheme without losing any backups. 190 | 191 | -v, --verbose 192 | 193 | Increase logging verbosity (can be repeated). 194 | 195 | -q, --quiet 196 | 197 | Decrease logging verbosity (can be repeated). 198 | 199 | -h, --help 200 | 201 | Show this message and exit. 202 | """ 203 | 204 | # Standard library modules. 205 | import getopt 206 | import shlex 207 | import sys 208 | 209 | # External dependencies. 210 | import coloredlogs 211 | from coloredlogs.syslog import enable_system_logging 212 | from executor import validate_ionice_class 213 | from humanfriendly import coerce_boolean, parse_path 214 | from humanfriendly.compat import on_windows 215 | from humanfriendly.terminal import usage 216 | from humanfriendly.text import pluralize 217 | from verboselogs import VerboseLogger 218 | 219 | # Modules included in our package. 220 | from rotate_backups import ( 221 | RotateBackups, 222 | coerce_location, 223 | coerce_retention_period, 224 | load_config_file, 225 | ) 226 | 227 | # Initialize a logger. 228 | logger = VerboseLogger(__name__) 229 | 230 | 231 | def main(): 232 | """Command line interface for the ``rotate-backups`` program.""" 233 | coloredlogs.install() 234 | # Command line option defaults. 235 | rotation_scheme = {} 236 | kw = dict(include_list=[], exclude_list=[]) 237 | parallel = False 238 | use_sudo = False 239 | use_syslog = (not on_windows()) 240 | # Internal state. 241 | selected_locations = [] 242 | # Parse the command line arguments. 243 | try: 244 | options, arguments = getopt.getopt(sys.argv[1:], 'M:H:d:w:m:y:t:I:x:jpri:c:C:uS:fnvqh', [ 245 | 'minutely=', 'hourly=', 'daily=', 'weekly=', 'monthly=', 'yearly=', 246 | 'timestamp-pattern=', 'include=', 'exclude=', 'parallel', 247 | 'prefer-recent', 'relaxed', 'ionice=', 'config=', 248 | 'removal-command=', 'use-sudo', 'syslog=', 'force', 249 | 'dry-run', 'verbose', 'quiet', 'help', 250 | ]) 251 | for option, value in options: 252 | if option in ('-M', '--minutely'): 253 | rotation_scheme['minutely'] = coerce_retention_period(value) 254 | elif option in ('-H', '--hourly'): 255 | rotation_scheme['hourly'] = coerce_retention_period(value) 256 | elif option in ('-d', '--daily'): 257 | rotation_scheme['daily'] = coerce_retention_period(value) 258 | elif option in ('-w', '--weekly'): 259 | rotation_scheme['weekly'] = coerce_retention_period(value) 260 | elif option in ('-m', '--monthly'): 261 | rotation_scheme['monthly'] = coerce_retention_period(value) 262 | elif option in ('-y', '--yearly'): 263 | rotation_scheme['yearly'] = coerce_retention_period(value) 264 | elif option in ('-t', '--timestamp-pattern'): 265 | kw['timestamp_pattern'] = value 266 | elif option in ('-I', '--include'): 267 | kw['include_list'].append(value) 268 | elif option in ('-x', '--exclude'): 269 | kw['exclude_list'].append(value) 270 | elif option in ('-j', '--parallel'): 271 | parallel = True 272 | elif option in ('-p', '--prefer-recent'): 273 | kw['prefer_recent'] = True 274 | elif option in ('-r', '--relaxed'): 275 | kw['strict'] = False 276 | elif option in ('-i', '--ionice'): 277 | value = validate_ionice_class(value.lower().strip()) 278 | kw['io_scheduling_class'] = value 279 | elif option in ('-c', '--config'): 280 | kw['config_file'] = parse_path(value) 281 | elif option in ('-C', '--removal-command'): 282 | removal_command = shlex.split(value) 283 | logger.info("Using custom removal command: %s", removal_command) 284 | kw['removal_command'] = removal_command 285 | elif option in ('-u', '--use-sudo'): 286 | use_sudo = True 287 | elif option in ('-S', '--syslog'): 288 | use_syslog = coerce_boolean(value) 289 | elif option in ('-f', '--force'): 290 | kw['force'] = True 291 | elif option in ('-n', '--dry-run'): 292 | logger.info("Performing a dry run (because of %s option) ..", option) 293 | kw['dry_run'] = True 294 | elif option in ('-v', '--verbose'): 295 | coloredlogs.increase_verbosity() 296 | elif option in ('-q', '--quiet'): 297 | coloredlogs.decrease_verbosity() 298 | elif option in ('-h', '--help'): 299 | usage(__doc__) 300 | return 301 | else: 302 | assert False, "Unhandled option! (programming error)" 303 | if use_syslog: 304 | enable_system_logging() 305 | if rotation_scheme: 306 | logger.verbose("Rotation scheme defined on command line: %s", rotation_scheme) 307 | if arguments: 308 | # Rotation of the locations given on the command line. 309 | location_source = 'command line arguments' 310 | selected_locations.extend(coerce_location(value, sudo=use_sudo) for value in arguments) 311 | else: 312 | # Rotation of all configured locations. 313 | location_source = 'configuration file' 314 | selected_locations.extend( 315 | location for location, rotation_scheme, options 316 | in load_config_file(configuration_file=kw.get('config_file'), expand=True) 317 | ) 318 | # Inform the user which location(s) will be rotated. 319 | if selected_locations: 320 | logger.verbose("Selected %s based on %s:", 321 | pluralize(len(selected_locations), "location"), 322 | location_source) 323 | for number, location in enumerate(selected_locations, start=1): 324 | logger.verbose(" %i. %s", number, location) 325 | else: 326 | # Show the usage message when no directories are given nor configured. 327 | logger.verbose("No location(s) to rotate selected.") 328 | usage(__doc__) 329 | return 330 | except Exception as e: 331 | logger.error("%s", e) 332 | sys.exit(1) 333 | # Rotate the backups in the selected directories. 334 | program = RotateBackups(rotation_scheme, **kw) 335 | if parallel: 336 | program.rotate_concurrent(*selected_locations) 337 | else: 338 | for location in selected_locations: 339 | program.rotate_backups(location) 340 | -------------------------------------------------------------------------------- /rotate_backups/tests.py: -------------------------------------------------------------------------------- 1 | # Test suite for the `rotate-backups' Python package. 2 | # 3 | # Author: Peter Odding 4 | # Last Change: May 17, 2020 5 | # URL: https://github.com/xolox/python-rotate-backups 6 | 7 | """Test suite for the `rotate-backups` package.""" 8 | 9 | # Standard library modules. 10 | import contextlib 11 | import datetime 12 | import logging 13 | import os 14 | 15 | # External dependencies. 16 | from executor import ExternalCommandFailed 17 | from executor.contexts import RemoteContext 18 | from humanfriendly.testing import TemporaryDirectory, TestCase, run_cli, touch 19 | from six.moves import configparser 20 | 21 | # The module we're testing. 22 | from rotate_backups import ( 23 | RotateBackups, 24 | coerce_location, 25 | coerce_retention_period, 26 | load_config_file, 27 | ) 28 | from rotate_backups.cli import main 29 | 30 | # Initialize a logger for this module. 31 | logger = logging.getLogger(__name__) 32 | 33 | SAMPLE_BACKUP_SET = set([ 34 | '2013-10-10@20:07', '2013-10-11@20:06', '2013-10-12@20:06', '2013-10-13@20:07', '2013-10-14@20:06', 35 | '2013-10-15@20:06', '2013-10-16@20:06', '2013-10-17@20:07', '2013-10-18@20:06', '2013-10-19@20:06', 36 | '2013-10-20@20:05', '2013-10-21@20:07', '2013-10-22@20:06', '2013-10-23@20:06', '2013-10-24@20:06', 37 | '2013-10-25@20:06', '2013-10-26@20:06', '2013-10-27@20:06', '2013-10-28@20:07', '2013-10-29@20:06', 38 | '2013-10-30@20:07', '2013-10-31@20:07', '2013-11-01@20:06', '2013-11-02@20:06', '2013-11-03@20:05', 39 | '2013-11-04@20:07', '2013-11-05@20:06', '2013-11-06@20:07', '2013-11-07@20:07', '2013-11-08@20:07', 40 | '2013-11-09@20:06', '2013-11-10@20:06', '2013-11-11@20:07', '2013-11-12@20:06', '2013-11-13@20:07', 41 | '2013-11-14@20:06', '2013-11-15@20:07', '2013-11-16@20:06', '2013-11-17@20:07', '2013-11-18@20:07', 42 | '2013-11-19@20:06', '2013-11-20@20:07', '2013-11-21@20:06', '2013-11-22@20:06', '2013-11-23@20:07', 43 | '2013-11-24@20:06', '2013-11-25@20:07', '2013-11-26@20:06', '2013-11-27@20:07', '2013-11-28@20:06', 44 | '2013-11-29@20:07', '2013-11-30@20:06', '2013-12-01@20:07', '2013-12-02@20:06', '2013-12-03@20:07', 45 | '2013-12-04@20:07', '2013-12-05@20:06', '2013-12-06@20:07', '2013-12-07@20:06', '2013-12-08@20:06', 46 | '2013-12-09@20:07', '2013-12-10@20:06', '2013-12-11@20:07', '2013-12-12@20:07', '2013-12-13@20:07', 47 | '2013-12-14@20:06', '2013-12-15@20:06', '2013-12-16@20:07', '2013-12-17@20:06', '2013-12-18@20:07', 48 | '2013-12-19@20:07', '2013-12-20@20:08', '2013-12-21@20:06', '2013-12-22@20:07', '2013-12-23@20:08', 49 | '2013-12-24@20:07', '2013-12-25@20:07', '2013-12-26@20:06', '2013-12-27@20:07', '2013-12-28@20:06', 50 | '2013-12-29@20:07', '2013-12-30@20:07', '2013-12-31@20:06', '2014-01-01@20:07', '2014-01-02@20:07', 51 | '2014-01-03@20:08', '2014-01-04@20:06', '2014-01-05@20:07', '2014-01-06@20:07', '2014-01-07@20:06', 52 | '2014-01-08@20:09', '2014-01-09@20:07', '2014-01-10@20:07', '2014-01-11@20:06', '2014-01-12@20:07', 53 | '2014-01-13@20:07', '2014-01-14@20:07', '2014-01-15@20:06', '2014-01-16@20:06', '2014-01-17@20:04', 54 | '2014-01-18@20:02', '2014-01-19@20:02', '2014-01-20@20:04', '2014-01-21@20:04', '2014-01-22@20:04', 55 | '2014-01-23@20:05', '2014-01-24@20:08', '2014-01-25@20:03', '2014-01-26@20:02', '2014-01-27@20:08', 56 | '2014-01-28@20:07', '2014-01-29@20:07', '2014-01-30@20:08', '2014-01-31@20:04', '2014-02-01@20:05', 57 | '2014-02-02@20:03', '2014-02-03@20:05', '2014-02-04@20:06', '2014-02-05@20:07', '2014-02-06@20:06', 58 | '2014-02-07@20:05', '2014-02-08@20:06', '2014-02-09@20:04', '2014-02-10@20:07', '2014-02-11@20:07', 59 | '2014-02-12@20:07', '2014-02-13@20:06', '2014-02-14@20:06', '2014-02-15@20:05', '2014-02-16@20:04', 60 | '2014-02-17@20:06', '2014-02-18@20:04', '2014-02-19@20:08', '2014-02-20@20:06', '2014-02-21@20:07', 61 | '2014-02-22@20:05', '2014-02-23@20:06', '2014-02-24@20:05', '2014-02-25@20:06', '2014-02-26@20:04', 62 | '2014-02-27@20:05', '2014-02-28@20:03', '2014-03-01@20:04', '2014-03-02@20:01', '2014-03-03@20:05', 63 | '2014-03-04@20:06', '2014-03-05@20:05', '2014-03-06@20:24', '2014-03-07@20:03', '2014-03-08@20:04', 64 | '2014-03-09@20:01', '2014-03-10@20:05', '2014-03-11@20:05', '2014-03-12@20:05', '2014-03-13@20:05', 65 | '2014-03-14@20:04', '2014-03-15@20:04', '2014-03-16@20:02', '2014-03-17@20:04', '2014-03-18@20:06', 66 | '2014-03-19@20:06', '2014-03-20@20:06', '2014-03-21@20:04', '2014-03-22@20:03', '2014-03-23@20:01', 67 | '2014-03-24@20:03', '2014-03-25@20:05', '2014-03-26@20:03', '2014-03-27@20:04', '2014-03-28@20:03', 68 | '2014-03-29@20:03', '2014-03-30@20:01', '2014-03-31@20:04', '2014-04-01@20:03', '2014-04-02@20:05', 69 | '2014-04-03@20:03', '2014-04-04@20:04', '2014-04-05@20:02', '2014-04-06@20:02', '2014-04-07@20:02', 70 | '2014-04-08@20:04', '2014-04-09@20:04', '2014-04-10@20:04', '2014-04-11@20:04', '2014-04-12@20:03', 71 | '2014-04-13@20:01', '2014-04-14@20:05', '2014-04-15@20:05', '2014-04-16@20:06', '2014-04-17@20:05', 72 | '2014-04-18@20:06', '2014-04-19@20:02', '2014-04-20@20:01', '2014-04-21@20:01', '2014-04-22@20:06', 73 | '2014-04-23@20:06', '2014-04-24@20:05', '2014-04-25@20:04', '2014-04-26@20:02', '2014-04-27@20:02', 74 | '2014-04-28@20:05', '2014-04-29@20:05', '2014-04-30@20:05', '2014-05-01@20:06', '2014-05-02@20:05', 75 | '2014-05-03@20:03', '2014-05-04@20:01', '2014-05-05@20:06', '2014-05-06@20:06', '2014-05-07@20:05', 76 | '2014-05-08@20:03', '2014-05-09@20:01', '2014-05-10@20:01', '2014-05-11@20:01', '2014-05-12@20:05', 77 | '2014-05-13@20:06', '2014-05-14@20:04', '2014-05-15@20:06', '2014-05-16@20:05', '2014-05-17@20:02', 78 | '2014-05-18@20:01', '2014-05-19@20:02', '2014-05-20@20:04', '2014-05-21@20:03', '2014-05-22@20:02', 79 | '2014-05-23@20:02', '2014-05-24@20:01', '2014-05-25@20:01', '2014-05-26@20:05', '2014-05-27@20:03', 80 | '2014-05-28@20:03', '2014-05-29@20:01', '2014-05-30@20:02', '2014-05-31@20:02', '2014-06-01@20:01', 81 | '2014-06-02@20:05', '2014-06-03@20:02', '2014-06-04@20:03', '2014-06-05@20:03', '2014-06-06@20:02', 82 | '2014-06-07@20:01', '2014-06-08@20:01', '2014-06-09@20:01', '2014-06-10@20:02', '2014-06-11@20:02', 83 | '2014-06-12@20:03', '2014-06-13@20:05', '2014-06-14@20:01', '2014-06-15@20:01', '2014-06-16@20:02', 84 | '2014-06-17@20:01', '2014-06-18@20:01', '2014-06-19@20:04', '2014-06-20@20:02', '2014-06-21@20:02', 85 | '2014-06-22@20:01', '2014-06-23@20:04', '2014-06-24@20:06', '2014-06-25@20:03', '2014-06-26@20:04', 86 | '2014-06-27@20:02', '2014-06-28@20:02', '2014-06-29@20:01', '2014-06-30@20:03', '2014-07-01@20:02', 87 | '2014-07-02@20:03', 'some-random-directory', 88 | ]) 89 | 90 | 91 | class RotateBackupsTestCase(TestCase): 92 | 93 | """:mod:`unittest` compatible container for `rotate-backups` tests.""" 94 | 95 | def test_retention_period_coercion(self): 96 | """Test coercion of retention period expressions.""" 97 | # Test that invalid values are refused. 98 | self.assertRaises(ValueError, coerce_retention_period, ['not', 'a', 'string']) 99 | # Test that invalid evaluation results are refused. 100 | self.assertRaises(ValueError, coerce_retention_period, 'None') 101 | # Check that the string `always' makes it through alive :-). 102 | assert coerce_retention_period('always') == 'always' 103 | assert coerce_retention_period('Always') == 'always' 104 | # Check that anything else properly evaluates to a number. 105 | assert coerce_retention_period(42) == 42 106 | assert coerce_retention_period('42') == 42 107 | assert coerce_retention_period('21 * 2') == 42 108 | 109 | def test_location_coercion(self): 110 | """Test coercion of locations.""" 111 | # Test that invalid values are refused. 112 | self.assertRaises(ValueError, lambda: coerce_location(['not', 'a', 'string'])) 113 | # Test that remote locations are properly parsed. 114 | location = coerce_location('some-host:/some/directory') 115 | assert isinstance(location.context, RemoteContext) 116 | assert location.directory == '/some/directory' 117 | 118 | def test_argument_validation(self): 119 | """Test argument validation.""" 120 | # Test that an invalid ionice scheduling class causes an error to be reported. 121 | returncode, output = run_cli(main, '--ionice=unsupported-class') 122 | assert returncode != 0 123 | # Test that a numbered ionice scheduling class does not causes an error. 124 | returncode, output = run_cli(main, '--ionice=3') 125 | assert returncode == 0 126 | # Test that an invalid rotation scheme causes an error to be reported. 127 | returncode, output = run_cli(main, '--hourly=not-a-number') 128 | assert returncode != 0 129 | # Argument validation tests that require an empty directory. 130 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 131 | # Test that non-existing directories cause an error to be reported. 132 | returncode, output = run_cli(main, os.path.join(root, 'does-not-exist')) 133 | assert returncode != 0 134 | # Test that loading of a custom configuration file raises an 135 | # exception when the configuration file cannot be loaded. 136 | self.assertRaises(ValueError, lambda: list(load_config_file(os.path.join(root, 'rotate-backups.ini')))) 137 | # Test that an empty rotation scheme raises an exception. 138 | self.create_sample_backup_set(root) 139 | self.assertRaises(ValueError, lambda: RotateBackups(rotation_scheme={}).rotate_backups(root)) 140 | # Argument validation tests that assume the current user isn't root. 141 | if os.getuid() != 0: 142 | # I'm being lazy and will assume that this test suite will only be 143 | # run on systems where users other than root do not have access to 144 | # /root. 145 | returncode, output = run_cli(main, '-n', '/root') 146 | assert returncode != 0 147 | 148 | def test_invalid_dates(self): 149 | """Make sure filenames with invalid dates don't cause an exception.""" 150 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 151 | file_with_valid_date = os.path.join(root, 'snapshot-201808030034.tar.gz') 152 | file_with_invalid_date = os.path.join(root, 'snapshot-180731150101.tar.gz') 153 | for filename in file_with_valid_date, file_with_invalid_date: 154 | touch(filename) 155 | program = RotateBackups(rotation_scheme=dict(monthly='always')) 156 | backups = program.collect_backups(root) 157 | assert len(backups) == 1 158 | assert backups[0].pathname == file_with_valid_date 159 | 160 | def test_dry_run(self): 161 | """Make sure dry run doesn't remove any backups.""" 162 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 163 | self.create_sample_backup_set(root) 164 | run_cli( 165 | main, '--dry-run', '--verbose', 166 | '--daily=7', '--weekly=7', 167 | '--monthly=12', '--yearly=always', 168 | root, 169 | ) 170 | backups_that_were_preserved = set(os.listdir(root)) 171 | assert backups_that_were_preserved == SAMPLE_BACKUP_SET 172 | 173 | def test_rotate_backups(self): 174 | """Test the :func:`.rotate_backups()` function.""" 175 | # These are the backups expected to be preserved. After each backup 176 | # I've noted which rotation scheme it falls in and the number of 177 | # preserved backups within that rotation scheme (counting up as we 178 | # progress through the backups sorted by date). 179 | expected_to_be_preserved = set([ 180 | '2013-10-10@20:07', # monthly (1), yearly (1) 181 | '2013-11-01@20:06', # monthly (2) 182 | '2013-12-01@20:07', # monthly (3) 183 | '2014-01-01@20:07', # monthly (4), yearly (2) 184 | '2014-02-01@20:05', # monthly (5) 185 | '2014-03-01@20:04', # monthly (6) 186 | '2014-04-01@20:03', # monthly (7) 187 | '2014-05-01@20:06', # monthly (8) 188 | '2014-06-01@20:01', # monthly (9) 189 | '2014-06-09@20:01', # weekly (1) 190 | '2014-06-16@20:02', # weekly (2) 191 | '2014-06-23@20:04', # weekly (3) 192 | '2014-06-26@20:04', # daily (1) 193 | '2014-06-27@20:02', # daily (2) 194 | '2014-06-28@20:02', # daily (3) 195 | '2014-06-29@20:01', # daily (4) 196 | '2014-06-30@20:03', # daily (5), weekly (4) 197 | '2014-07-01@20:02', # daily (6), monthly (10) 198 | '2014-07-02@20:03', # hourly (1), daily (7) 199 | 'some-random-directory', # no recognizable time stamp, should definitely be preserved 200 | 'rotate-backups.ini', # no recognizable time stamp, should definitely be preserved 201 | ]) 202 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 203 | # Specify the rotation scheme and options through a configuration file. 204 | config_file = os.path.join(root, 'rotate-backups.ini') 205 | parser = configparser.RawConfigParser() 206 | parser.add_section(root) 207 | parser.set(root, 'hourly', '24') 208 | parser.set(root, 'daily', '7') 209 | parser.set(root, 'weekly', '4') 210 | parser.set(root, 'monthly', '12') 211 | parser.set(root, 'yearly', 'always') 212 | parser.set(root, 'ionice', 'idle') 213 | with open(config_file, 'w') as handle: 214 | parser.write(handle) 215 | self.create_sample_backup_set(root) 216 | run_cli(main, '--verbose', '--config=%s' % config_file) 217 | backups_that_were_preserved = set(os.listdir(root)) 218 | assert backups_that_were_preserved == expected_to_be_preserved 219 | 220 | def test_rotate_concurrent(self): 221 | """Test the :func:`.rotate_concurrent()` function.""" 222 | # These are the backups expected to be preserved 223 | # (the same as in test_rotate_backups). 224 | expected_to_be_preserved = set([ 225 | '2013-10-10@20:07', # monthly, yearly (1) 226 | '2013-11-01@20:06', # monthly (2) 227 | '2013-12-01@20:07', # monthly (3) 228 | '2014-01-01@20:07', # monthly (4), yearly (2) 229 | '2014-02-01@20:05', # monthly (5) 230 | '2014-03-01@20:04', # monthly (6) 231 | '2014-04-01@20:03', # monthly (7) 232 | '2014-05-01@20:06', # monthly (8) 233 | '2014-06-01@20:01', # monthly (9) 234 | '2014-06-09@20:01', # weekly (1) 235 | '2014-06-16@20:02', # weekly (2) 236 | '2014-06-23@20:04', # weekly (3) 237 | '2014-06-26@20:04', # daily (1) 238 | '2014-06-27@20:02', # daily (2) 239 | '2014-06-28@20:02', # daily (3) 240 | '2014-06-29@20:01', # daily (4) 241 | '2014-06-30@20:03', # daily (5), weekly (4) 242 | '2014-07-01@20:02', # daily (6), monthly (10) 243 | '2014-07-02@20:03', # hourly (1), daily (7) 244 | 'some-random-directory', # no recognizable time stamp, should definitely be preserved 245 | ]) 246 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 247 | self.create_sample_backup_set(root) 248 | run_cli( 249 | main, '--verbose', '--hourly=24', '--daily=7', '--weekly=4', 250 | '--monthly=12', '--yearly=always', '--parallel', root, 251 | ) 252 | backups_that_were_preserved = set(os.listdir(root)) 253 | assert backups_that_were_preserved == expected_to_be_preserved 254 | 255 | def test_include_list(self): 256 | """Test include list logic.""" 257 | # These are the backups expected to be preserved within the year 2014 258 | # (other years are excluded and so should all be preserved, see below). 259 | # After each backup I've noted which rotation scheme it falls in. 260 | expected_to_be_preserved = set([ 261 | '2014-01-01@20:07', # monthly, yearly 262 | '2014-02-01@20:05', # monthly 263 | '2014-03-01@20:04', # monthly 264 | '2014-04-01@20:03', # monthly 265 | '2014-05-01@20:06', # monthly 266 | '2014-06-01@20:01', # monthly 267 | '2014-06-09@20:01', # weekly 268 | '2014-06-16@20:02', # weekly 269 | '2014-06-23@20:04', # weekly 270 | '2014-06-26@20:04', # daily 271 | '2014-06-27@20:02', # daily 272 | '2014-06-28@20:02', # daily 273 | '2014-06-29@20:01', # daily 274 | '2014-06-30@20:03', # daily, weekly 275 | '2014-07-01@20:02', # daily, monthly 276 | '2014-07-02@20:03', # hourly, daily 277 | 'some-random-directory', # no recognizable time stamp, should definitely be preserved 278 | ]) 279 | for name in SAMPLE_BACKUP_SET: 280 | if not name.startswith('2014-'): 281 | expected_to_be_preserved.add(name) 282 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 283 | self.create_sample_backup_set(root) 284 | run_cli( 285 | main, '--verbose', '--ionice=idle', '--hourly=24', '--daily=7', 286 | '--weekly=4', '--monthly=12', '--yearly=always', 287 | '--include=2014-*', root, 288 | ) 289 | backups_that_were_preserved = set(os.listdir(root)) 290 | assert backups_that_were_preserved == expected_to_be_preserved 291 | 292 | def test_exclude_list(self): 293 | """Test exclude list logic.""" 294 | # These are the backups expected to be preserved. After each backup 295 | # I've noted which rotation scheme it falls in and the number of 296 | # preserved backups within that rotation scheme (counting up as we 297 | # progress through the backups sorted by date). 298 | expected_to_be_preserved = set([ 299 | '2013-10-10@20:07', # monthly (1), yearly (1) 300 | '2013-11-01@20:06', # monthly (2) 301 | '2013-12-01@20:07', # monthly (3) 302 | '2014-01-01@20:07', # monthly (4), yearly (2) 303 | '2014-02-01@20:05', # monthly (5) 304 | '2014-03-01@20:04', # monthly (6) 305 | '2014-04-01@20:03', # monthly (7) 306 | '2014-05-01@20:06', # monthly (8) 307 | '2014-05-19@20:02', # weekly (1) 308 | '2014-05-26@20:05', # weekly (2) 309 | '2014-06-01@20:01', # monthly (9) 310 | '2014-06-09@20:01', # weekly (3) 311 | '2014-06-16@20:02', # weekly (4) 312 | '2014-06-23@20:04', # weekly (5) 313 | '2014-06-26@20:04', # daily (1) 314 | '2014-06-27@20:02', # daily (2) 315 | '2014-06-28@20:02', # daily (3) 316 | '2014-06-29@20:01', # daily (4) 317 | '2014-06-30@20:03', # daily (5), weekly (6) 318 | '2014-07-01@20:02', # daily (6), monthly (10) 319 | '2014-07-02@20:03', # hourly (1), daily (7) 320 | 'some-random-directory', # no recognizable time stamp, should definitely be preserved 321 | ]) 322 | for name in SAMPLE_BACKUP_SET: 323 | if name.startswith('2014-05-'): 324 | expected_to_be_preserved.add(name) 325 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 326 | self.create_sample_backup_set(root) 327 | run_cli( 328 | main, '--verbose', '--ionice=idle', '--hourly=24', '--daily=7', 329 | '--weekly=4', '--monthly=12', '--yearly=always', 330 | '--exclude=2014-05-*', root, 331 | ) 332 | backups_that_were_preserved = set(os.listdir(root)) 333 | assert backups_that_were_preserved == expected_to_be_preserved 334 | 335 | def test_strict_rotation(self): 336 | """Test strict rotation.""" 337 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 338 | os.mkdir(os.path.join(root, 'galera_backup_db4.sl.example.lab_2016-03-17_10-00')) 339 | os.mkdir(os.path.join(root, 'galera_backup_db4.sl.example.lab_2016-03-17_12-00')) 340 | os.mkdir(os.path.join(root, 'galera_backup_db4.sl.example.lab_2016-03-17_16-00')) 341 | run_cli(main, '--hourly=3', '--daily=1', root) 342 | assert os.path.exists(os.path.join(root, 'galera_backup_db4.sl.example.lab_2016-03-17_10-00')) 343 | assert os.path.exists(os.path.join(root, 'galera_backup_db4.sl.example.lab_2016-03-17_12-00')) is False 344 | assert os.path.exists(os.path.join(root, 'galera_backup_db4.sl.example.lab_2016-03-17_16-00')) 345 | 346 | def test_relaxed_rotation(self): 347 | """Test relaxed rotation.""" 348 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 349 | os.mkdir(os.path.join(root, 'galera_backup_db4.sl.example.lab_2016-03-17_10-00')) 350 | os.mkdir(os.path.join(root, 'galera_backup_db4.sl.example.lab_2016-03-17_12-00')) 351 | os.mkdir(os.path.join(root, 'galera_backup_db4.sl.example.lab_2016-03-17_16-00')) 352 | run_cli(main, '--hourly=3', '--daily=1', '--relaxed', root) 353 | assert os.path.exists(os.path.join(root, 'galera_backup_db4.sl.example.lab_2016-03-17_10-00')) 354 | assert os.path.exists(os.path.join(root, 'galera_backup_db4.sl.example.lab_2016-03-17_12-00')) 355 | assert os.path.exists(os.path.join(root, 'galera_backup_db4.sl.example.lab_2016-03-17_16-00')) 356 | 357 | def test_prefer_old(self): 358 | """Test the default preference for the oldest backup in each time slot.""" 359 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 360 | os.mkdir(os.path.join(root, 'backup-2016-01-10_21-15-00')) 361 | os.mkdir(os.path.join(root, 'backup-2016-01-10_21-30-00')) 362 | os.mkdir(os.path.join(root, 'backup-2016-01-10_21-45-00')) 363 | run_cli(main, '--hourly=1', root) 364 | assert os.path.exists(os.path.join(root, 'backup-2016-01-10_21-15-00')) 365 | assert not os.path.exists(os.path.join(root, 'backup-2016-01-10_21-30-00')) 366 | assert not os.path.exists(os.path.join(root, 'backup-2016-01-10_21-45-00')) 367 | 368 | def test_prefer_new(self): 369 | """Test the alternative preference for the newest backup in each time slot.""" 370 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 371 | os.mkdir(os.path.join(root, 'backup-2016-01-10_21-15-00')) 372 | os.mkdir(os.path.join(root, 'backup-2016-01-10_21-30-00')) 373 | os.mkdir(os.path.join(root, 'backup-2016-01-10_21-45-00')) 374 | run_cli(main, '--hourly=1', '--prefer-recent', root) 375 | assert not os.path.exists(os.path.join(root, 'backup-2016-01-10_21-15-00')) 376 | assert not os.path.exists(os.path.join(root, 'backup-2016-01-10_21-30-00')) 377 | assert os.path.exists(os.path.join(root, 'backup-2016-01-10_21-45-00')) 378 | 379 | def test_minutely_rotation(self): 380 | """Test rotation with multiple backups per hour.""" 381 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 382 | os.mkdir(os.path.join(root, 'backup-2016-01-10_21-15-00')) 383 | os.mkdir(os.path.join(root, 'backup-2016-01-10_21-30-00')) 384 | os.mkdir(os.path.join(root, 'backup-2016-01-10_21-45-00')) 385 | run_cli(main, '--prefer-recent', '--relaxed', '--minutely=2', root) 386 | assert not os.path.exists(os.path.join(root, 'backup-2016-01-10_21-15-00')) 387 | assert os.path.exists(os.path.join(root, 'backup-2016-01-10_21-30-00')) 388 | assert os.path.exists(os.path.join(root, 'backup-2016-01-10_21-45-00')) 389 | 390 | def test_removal_command(self): 391 | """Test that the removal command can be customized.""" 392 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 393 | today = datetime.datetime.now() 394 | for date in today, (today - datetime.timedelta(hours=24)): 395 | os.mkdir(os.path.join(root, date.strftime('%Y-%m-%d'))) 396 | program = RotateBackups(removal_command=['rmdir'], rotation_scheme=dict(monthly='always')) 397 | commands = program.rotate_backups(root, prepare=True) 398 | assert any(cmd.command_line[0] == 'rmdir' for cmd in commands) 399 | 400 | def test_force(self): 401 | """Test that sanity checks can be overridden.""" 402 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 403 | for date in '2019-03-05', '2019-03-06': 404 | os.mkdir(os.path.join(root, date)) 405 | with readonly_directory(root): 406 | program = RotateBackups(force=True, rotation_scheme=dict(monthly='always')) 407 | self.assertRaises(ExternalCommandFailed, program.rotate_backups, root) 408 | 409 | def test_ensure_writable(self): 410 | """Test that ensure_writable() complains when the location isn't writable.""" 411 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 412 | for date in '2019-03-05', '2019-03-06': 413 | os.mkdir(os.path.join(root, date)) 414 | with readonly_directory(root): 415 | program = RotateBackups(rotation_scheme=dict(monthly='always')) 416 | self.assertRaises(ValueError, program.rotate_backups, root) 417 | 418 | def test_ensure_writable_optional(self): 419 | """Test that ensure_writable() isn't called when a custom removal command is used.""" 420 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 421 | for date in '2019-03-05', '2019-03-06': 422 | os.mkdir(os.path.join(root, date)) 423 | with readonly_directory(root): 424 | program = RotateBackups( 425 | removal_command=['echo', 'Deleting'], 426 | rotation_scheme=dict(monthly='always'), 427 | ) 428 | program.rotate_backups(root) 429 | 430 | def test_filename_patterns(self): 431 | """Test support for filename patterns in configuration files.""" 432 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 433 | for subdirectory in 'laptop', 'vps': 434 | os.makedirs(os.path.join(root, subdirectory)) 435 | config_file = os.path.join(root, 'rotate-backups.ini') 436 | parser = configparser.RawConfigParser() 437 | pattern = os.path.join(root, '*') 438 | parser.add_section(pattern) 439 | parser.set(pattern, 'daily', '7') 440 | parser.set(pattern, 'weekly', '4') 441 | parser.set(pattern, 'monthly', 'always') 442 | with open(config_file, 'w') as handle: 443 | parser.write(handle) 444 | # Check that the configured rotation scheme is applied. 445 | default_scheme = dict(monthly='always') 446 | program = RotateBackups(config_file=config_file, rotation_scheme=default_scheme) 447 | program.load_config_file(os.path.join(root, 'laptop')) 448 | assert program.rotation_scheme != default_scheme 449 | # Check that the available locations are matched. 450 | available_locations = [ 451 | location for location, rotation_scheme, options 452 | in load_config_file(config_file) 453 | ] 454 | assert len(available_locations) == 2 455 | assert any(location.directory == os.path.join(root, 'laptop') for location in available_locations) 456 | assert any(location.directory == os.path.join(root, 'vps') for location in available_locations) 457 | 458 | def test_custom_timestamp_pattern(self): 459 | """Test that custom timestamp patterns are properly supported.""" 460 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 461 | custom_backup_filename = os.path.join(root, 'My-File--2009-12-31--23-59-59.txt') 462 | touch(custom_backup_filename) 463 | program = RotateBackups( 464 | rotation_scheme=dict(monthly='always'), 465 | timestamp_pattern=r''' 466 | (?P\d{4}) - (?P\d{2}) - (?P\d{2}) 467 | -- 468 | (?P\d{2}) - (?P\d{2}) - (?P\d{2}) 469 | ''', 470 | ) 471 | backups = program.collect_backups(root) 472 | assert backups[0].pathname == custom_backup_filename 473 | assert backups[0].timestamp.year == 2009 474 | assert backups[0].timestamp.month == 12 475 | assert backups[0].timestamp.day == 31 476 | assert backups[0].timestamp.hour == 23 477 | assert backups[0].timestamp.minute == 59 478 | assert backups[0].timestamp.second == 59 479 | 480 | def test_invalid_timestamp_pattern(self): 481 | """Test that the capture groups in custom timestamp patterns are validated.""" 482 | self.assertRaises( 483 | ValueError, 484 | RotateBackups, 485 | rotation_scheme=dict(monthly='always'), 486 | timestamp_pattern=r'(?P\d{4})-(?P\d{2})', 487 | ) 488 | 489 | def test_optional_captures(self): 490 | """Test that the hour/minute/second captures are truly optional.""" 491 | with TemporaryDirectory(prefix='rotate-backups-', suffix='-test-suite') as root: 492 | touch(os.path.join(root, 'prometheus-grafana-production-09-04-2020.tar.gz')) 493 | program = RotateBackups( 494 | rotation_scheme=dict(monthly='always'), 495 | timestamp_pattern=r'(?P\d{2})-(?P\d{2})-(?P\d{4})', 496 | ) 497 | location = coerce_location(root) 498 | backups = program.collect_backups(location) 499 | assert len(backups) == 1 500 | assert backups[0].timestamp.day == 9 501 | assert backups[0].timestamp.month == 4 502 | assert backups[0].timestamp.year == 2020 503 | 504 | def create_sample_backup_set(self, root): 505 | """Create a sample backup set to be rotated.""" 506 | for name in SAMPLE_BACKUP_SET: 507 | os.mkdir(os.path.join(root, name)) 508 | 509 | 510 | @contextlib.contextmanager 511 | def readonly_directory(pathname): 512 | """Context manager to temporarily make something read only.""" 513 | os.chmod(pathname, 0o555) 514 | yield 515 | os.chmod(pathname, 0o775) 516 | -------------------------------------------------------------------------------- /scripts/install-on-travis.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -e 2 | 3 | main () { 4 | 5 | # On Mac OS X workers we are responsible for creating the Python virtual 6 | # environment, because we set `language: generic' in the Travis CI build 7 | # configuration file (to bypass the lack of Python runtime support). 8 | if [ "$TRAVIS_OS_NAME" = osx ]; then 9 | local environment="$HOME/virtualenv/python2.7" 10 | if [ -x "$environment/bin/python" ]; then 11 | msg "Activating virtual environment ($environment) .." 12 | source "$environment/bin/activate" 13 | else 14 | if ! which virtualenv &>/dev/null; then 15 | msg "Installing 'virtualenv' in per-user site-packages .." 16 | pip install --user virtualenv 17 | msg "Figuring out 'bin' directory of per-user site-packages .." 18 | LOCAL_BINARIES=$(python -c 'import os, site; print(os.path.join(site.USER_BASE, "bin"))') 19 | msg "Prefixing '$LOCAL_BINARIES' to PATH .." 20 | export PATH="$LOCAL_BINARIES:$PATH" 21 | fi 22 | msg "Creating virtual environment ($environment) .." 23 | virtualenv "$environment" 24 | msg "Activating virtual environment ($environment) .." 25 | source "$environment/bin/activate" 26 | msg "Checking if 'pip' executable works .." 27 | if ! pip --version; then 28 | msg "Bootstrapping working 'pip' installation using get-pip.py .." 29 | curl -s https://bootstrap.pypa.io/get-pip.py | python - 30 | fi 31 | fi 32 | fi 33 | 34 | # Install the required Python packages. 35 | pip install --requirement=requirements-travis.txt 36 | 37 | # Install the project itself, making sure that potential character encoding 38 | # and/or decoding errors in the setup script are caught as soon as possible. 39 | LC_ALL=C pip install . 40 | 41 | } 42 | 43 | msg () { 44 | echo "[install-on-travis.sh] $*" >&2 45 | } 46 | 47 | main "$@" 48 | -------------------------------------------------------------------------------- /scripts/run-on-travis.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash -e 2 | 3 | # On Mac OS X workers we are responsible for activating the Python virtual 4 | # environment, because we set `language: generic' in the Travis CI build 5 | # configuration file (to bypass the lack of Python runtime support). 6 | 7 | if [ "$TRAVIS_OS_NAME" = osx ]; then 8 | VIRTUAL_ENV="$HOME/virtualenv/python2.7" 9 | source "$VIRTUAL_ENV/bin/activate" 10 | fi 11 | 12 | eval "$@" 13 | -------------------------------------------------------------------------------- /setup.cfg: -------------------------------------------------------------------------------- 1 | # Enable universal wheels because `rotate-backups' is 2 | # pure Python and works on Python 2 and 3 alike. 3 | 4 | [wheel] 5 | universal=1 6 | -------------------------------------------------------------------------------- /setup.py: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env python 2 | 3 | # Setup script for the `rotate-backups' package. 4 | # 5 | # Author: Peter Odding 6 | # Last Change: February 18, 2020 7 | # URL: https://github.com/xolox/python-rotate-backups 8 | 9 | """ 10 | Setup script for the ``rotate-backups`` package. 11 | 12 | **python setup.py install** 13 | Install from the working directory into the current Python environment. 14 | 15 | **python setup.py sdist** 16 | Build a source distribution archive. 17 | 18 | **python setup.py bdist_wheel** 19 | Build a wheel distribution archive. 20 | """ 21 | 22 | # Standard library modules. 23 | import codecs 24 | import os 25 | import re 26 | 27 | # De-facto standard solution for Python packaging. 28 | from setuptools import find_packages, setup 29 | 30 | 31 | def get_contents(*args): 32 | """Get the contents of a file relative to the source distribution directory.""" 33 | with codecs.open(get_absolute_path(*args), 'r', 'UTF-8') as handle: 34 | return handle.read() 35 | 36 | 37 | def get_version(*args): 38 | """Extract the version number from a Python module.""" 39 | contents = get_contents(*args) 40 | metadata = dict(re.findall('__([a-z]+)__ = [\'"]([^\'"]+)', contents)) 41 | return metadata['version'] 42 | 43 | 44 | def get_requirements(*args): 45 | """Get requirements from pip requirement files.""" 46 | requirements = set() 47 | with open(get_absolute_path(*args)) as handle: 48 | for line in handle: 49 | # Strip comments. 50 | line = re.sub(r'^#.*|\s#.*', '', line) 51 | # Ignore empty lines 52 | if line and not line.isspace(): 53 | requirements.add(re.sub(r'\s+', '', line)) 54 | return sorted(requirements) 55 | 56 | 57 | def get_absolute_path(*args): 58 | """Transform relative pathnames into absolute pathnames.""" 59 | return os.path.join(os.path.dirname(os.path.abspath(__file__)), *args) 60 | 61 | 62 | setup(name="rotate-backups", 63 | version=get_version('rotate_backups', '__init__.py'), 64 | description="Simple command line interface for backup rotation", 65 | long_description=get_contents('README.rst'), 66 | url='https://github.com/xolox/python-rotate-backups', 67 | author="Peter Odding", 68 | author_email='peter@peterodding.com', 69 | license='MIT', 70 | packages=find_packages(), 71 | entry_points=dict(console_scripts=[ 72 | 'rotate-backups = rotate_backups.cli:main', 73 | ]), 74 | install_requires=get_requirements('requirements.txt'), 75 | test_suite='rotate_backups.tests', 76 | python_requires='>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*', 77 | classifiers=[ 78 | 'Development Status :: 5 - Production/Stable', 79 | 'Environment :: Console', 80 | 'Intended Audience :: Developers', 81 | 'Intended Audience :: System Administrators', 82 | 'License :: OSI Approved :: MIT License', 83 | 'Operating System :: MacOS :: MacOS X', 84 | 'Operating System :: POSIX', 85 | 'Operating System :: POSIX :: Linux', 86 | 'Operating System :: Unix', 87 | 'Programming Language :: Python', 88 | 'Programming Language :: Python :: 2', 89 | 'Programming Language :: Python :: 2.7', 90 | 'Programming Language :: Python :: 3', 91 | 'Programming Language :: Python :: 3.5', 92 | 'Programming Language :: Python :: 3.6', 93 | 'Programming Language :: Python :: 3.7', 94 | 'Programming Language :: Python :: 3.8', 95 | 'Programming Language :: Python :: Implementation :: CPython', 96 | 'Programming Language :: Python :: Implementation :: PyPy', 97 | 'Topic :: Software Development :: Libraries :: Python Modules', 98 | 'Topic :: Software Development', 99 | 'Topic :: System :: Archiving :: Backup', 100 | 'Topic :: System :: Systems Administration', 101 | ]) 102 | -------------------------------------------------------------------------------- /tox.ini: -------------------------------------------------------------------------------- 1 | # Tox (http://tox.testrun.org/) is a tool for running tests in multiple 2 | # virtualenvs. This configuration file will run the test suite on all supported 3 | # python versions. To use it, "pip install tox" and then run "tox" from this 4 | # directory. 5 | 6 | [tox] 7 | envlist = py27, py35, py36, py37, py38, pypy 8 | 9 | [testenv] 10 | deps = -rrequirements-tests.txt 11 | commands = py.test {posargs} 12 | 13 | [pytest] 14 | addopts = --verbose 15 | python_files = rotate_backups/tests.py 16 | 17 | [flake8] 18 | exclude = .tox 19 | ignore = D211,D400,D401 20 | max-line-length = 120 21 | --------------------------------------------------------------------------------