├── .gitignore ├── tests ├── data │ ├── exclude.patterns │ ├── exclude.patterns.error │ └── sudo ├── snazzer-list.bats ├── snazzer.bats ├── snazzer-prune.bats ├── fixtures.sh └── snazzer-send-wrapper.bats ├── docs ├── examples │ └── etc │ │ ├── cron.daily │ │ ├── snazzer-receive │ │ ├── snazzer │ │ └── snazzer-receive.sh │ │ ├── snazzer │ │ └── exclude.patterns │ │ ├── sudoers.d │ │ ├── snazzer-sender │ │ ├── snazzer-measurer │ │ └── snazzer-receiver │ │ └── default │ │ └── snazzer ├── snazzer-send-wrapper.md ├── snazzer-prune-candidates.md ├── snazzer-measure.md ├── snazzer.md └── snazzer-receive.md ├── AUTHORS.md ├── .travis.yml ├── LICENSE.md ├── CHANGELOG.md ├── Makefile ├── README.md ├── snazzer-send-wrapper ├── snazzer-measure ├── snazzer └── snazzer-receive /.gitignore: -------------------------------------------------------------------------------- 1 | tmp 2 | bats 3 | man 4 | -------------------------------------------------------------------------------- /tests/data/exclude.patterns: -------------------------------------------------------------------------------- 1 | /var/cache 2 | /var/lib/docker/* 3 | */.snapshots 4 | /tmp 5 | *backup* 6 | *secret* 7 | -------------------------------------------------------------------------------- /docs/examples/etc/cron.daily/snazzer-receive: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | set -e 3 | su receiveruser -c "/bin/sh snazzer-receive.sh" 4 | -------------------------------------------------------------------------------- /docs/examples/etc/snazzer/exclude.patterns: -------------------------------------------------------------------------------- 1 | /var/cache 2 | /var/lib/docker/* 3 | */.snapshots 4 | /tmp 5 | *backup* 6 | *secret* 7 | -------------------------------------------------------------------------------- /tests/data/exclude.patterns.error: -------------------------------------------------------------------------------- 1 | /var/cache 2 | /var/lib/docker/* 3 | */.snapshots 4 | /tmp 5 | foo 6 | *backup* 7 | *secret* 8 | -------------------------------------------------------------------------------- /docs/examples/etc/cron.daily/snazzer: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | set -e 3 | . /etc/default/snazzer 4 | snazzer --all 5 | snazzer --prune --force --all 6 | snazzer --measure --all 7 | -------------------------------------------------------------------------------- /AUTHORS.md: -------------------------------------------------------------------------------- 1 | SNAZZER AUTHORS 2 | =============== 3 | 4 | Compiled automatically from git history, in alphabetical order: 5 | 6 | - Florian Jacob 7 | - Paul Harvey 8 | - Tyler Langlois 9 | -------------------------------------------------------------------------------- /docs/examples/etc/sudoers.d/snazzer-sender: -------------------------------------------------------------------------------- 1 | sendinguser ALL=(root:nobody) NOPASSWD: /usr/bin/snazzer --list-snapshots * 2 | sendinguser ALL=(root:nobody) NOPASSWD:NOEXEC: \ 3 | /bin/grep -srl */.snapshotz/.measurements/, \ 4 | /sbin/btrfs send */.snapshotz/*, \ 5 | /bin/cat */.snapshotz/.measurements/* 6 | -------------------------------------------------------------------------------- /tests/data/sudo: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | case "$F" in 3 | no_args) 4 | echo $#; 5 | ;; 6 | ls_args) 7 | while [ "$#" != "0" ]; do 8 | printf '%s\n' "$1" 9 | shift 10 | done 11 | ;; 12 | *) 13 | echo "ERROR: unknown action $F" >&2 14 | exit 1 15 | ;; 16 | esac 17 | -------------------------------------------------------------------------------- /docs/examples/etc/cron.daily/snazzer-receive.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | set -e 3 | . /etc/default/snazzer 4 | . /etc/snazzer/receive-config 5 | BACKUP_ROOT=/media/backup/foo 6 | 7 | for HOST in host1 host2 host3 host4; do 8 | echo "Receiving $HOST:" 9 | if ! sudo test -e "$BACKUP_ROOT/$HOST"; then 10 | sudo btrfs subvolume create "$BACKUP_ROOT/$HOST" 11 | fi 12 | cd "$BACKUP_ROOT/$HOST" 13 | if [ "$(hostname)" = "$HOST" ]; then 14 | snazzer-receive -- --all 15 | else 16 | snazzer-receive "$HOST" --all 17 | fi 18 | done 19 | -------------------------------------------------------------------------------- /docs/examples/etc/sudoers.d/snazzer-measurer: -------------------------------------------------------------------------------- 1 | measureuser ALL=(root:nobody) NOPASSWD:NOEXEC: \ 2 | /bin/cat */.snapshotz/*/.snapshot_measurements.exclude, \ 3 | /usr/bin/du -bs --one-file-system --exclude-from * */.snapshotz/*, \ 4 | /usr/bin/find */.snapshotz/* \ 5 | -xdev -not -path /*/.snapshotz/* -printf ./%P\\\\0, \ 6 | /bin/tar --no-recursion --one-file-system --preserve-permissions \ 7 | --numeric-owner --null --create --to-stdout --directory */.snapshotz/* \ 8 | --files-from * --exclude-from */.snapshotz/*/.snapshot_measurements.exclude 9 | -------------------------------------------------------------------------------- /.travis.yml: -------------------------------------------------------------------------------- 1 | language: perl 2 | sudo: required 3 | dist: trusty 4 | perl: 5 | - "5.24" 6 | - "5.20" 7 | - "5.16" 8 | - "5.14" 9 | - "5.10" 10 | install: 11 | - sudo add-apt-repository ppa:ubuntu-lxc/buildd-backports -y 12 | - sudo apt-get update -y 13 | - sudo apt-get install btrfs-tools cabal-install -y 14 | # for some reason, this is required to get all tags 15 | # in case a tag only exists on the branch of a PR for master 16 | - git fetch --tags 17 | 18 | script: 19 | - make test 20 | 21 | after_success: 22 | - make shellcheck-tests 23 | -------------------------------------------------------------------------------- /docs/examples/etc/default/snazzer: -------------------------------------------------------------------------------- 1 | export GNUPGHOME=/etc/secrets/snazzer/.gnupg 2 | 3 | #SNAZZER_DAYLIES_TO_KEEP=31 4 | #SNAZZER_HOURLIES_TO_KEEP=24 5 | #SNAZZER_MONTHLIES_TO_KEEP=12 6 | #SNAZZER_YEARLIES_TO_KEEP=1000 7 | #MY_KEYFILES_ARE_INVINCIBLE=0 # disable snazzer-measure GPG key sanity check 8 | #SNAZZER_SIG_ENABLE=0 # disable PGP signatures in measurements 9 | #SNAZZER_MEASUREMENTS_EXCLUDE_FILE=".snapshot_measurements.exclude" # relative to snapshot root 10 | #SNAZZER_SUBVOLS_EXCLUDE_FILE=/etc/snazzer/exclude.patterns 11 | #SNAZZER_USE_UTC=1 # Use YYYY-MM-DDTHHMMSSZ rather than YYYY-MM-DDTTHHMMSS+hhmm 12 | -------------------------------------------------------------------------------- /docs/examples/etc/sudoers.d/snazzer-receiver: -------------------------------------------------------------------------------- 1 | receiveruser ALL=(root:nobody) NOPASSWD:NOEXEC: \ 2 | /usr/bin/test -e */.snapshotz*, \ 3 | /sbin/btrfs subvolume show *, \ 4 | /bin/ls */.snapshotz, \ 5 | /bin/grep -srL */.snapshotz/.measurements/, \ 6 | /bin/mkdir --mode=0755 */.snapshotz, \ 7 | /bin/mkdir --mode=0755 */.snapshotz/.measurements, \ 8 | /bin/mkdir --mode=0755 */.snapshotz/.incomplete, \ 9 | /sbin/btrfs receive */.snapshotz/.incomplete, \ 10 | /sbin/btrfs subvolume create *, \ 11 | /sbin/btrfs subvolume snapshot -r */.snapshotz/.incomplete/* */.snapshotz/,\ 12 | /sbin/btrfs subvolume delete */.snapshotz/.incomplete/*, \ 13 | /bin/rmdir */.snapshotz/.incomplete, \ 14 | /bin/mkdir -vp *, \ 15 | /bin/mkdir --mode=0755 -vp */.snapshotz, \ 16 | /usr/bin/tee -a */.snapshotz/.measurements/* 17 | -------------------------------------------------------------------------------- /LICENSE.md: -------------------------------------------------------------------------------- 1 | Copyright (C) 2015-2016, Snazzer Authors All rights reserved. 2 | 3 | Redistribution and use in source and binary forms, with or without 4 | modification, are permitted provided that the following conditions are met: 5 | 6 | 1. Redistributions of source code must retain the above copyright notice, this 7 | list of conditions and the following disclaimer. 8 | 9 | 2. Redistributions in binary form must reproduce the above copyright notice, 10 | this list of conditions and the following disclaimer in the documentation 11 | and/or other materials provided with the distribution. 12 | 13 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 14 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 15 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 16 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 17 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 18 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 19 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 20 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 21 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 22 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 23 | -------------------------------------------------------------------------------- /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # Change Log # 2 | All notable changes to this project will be documented in this file. 3 | This project adheres to [Semantic Versioning](http://semver.org/). 4 | 5 | ## [Unreleased] ## 6 | ### Added ### 7 | - For the initial creation of the `.snapshotz` folder, Snazzer now copies owner and group from the source subvolume. 8 | The default permissions stay 0700, so the owner of the source subvolume now can read and delete snapshots by default, 9 | while group and others still have no permission for anything. For example, this means that if you use home directory 10 | subvolumes, a user can now read and delete their home directory snapshots without root permissions. This only happens 11 | for new subvolumes, but you can always change `.snapshotz` ownership (and permissions) by hand as well. 12 | ### Changed ### 13 | ### Deprecated ### 14 | ### Removed ### 15 | ### Fixed ### 16 | 17 | ## [0.0.3] - 2017-06-13 ## 18 | ### Changed ### 19 | - Snazzer doesn't treat all patterns in `/` like they were implicitly pre- and suffixed with `*`. 20 | This behaviour now has to be specified explicitly. 21 | - Snazzer now explicitly makes subvolumes absolute by prefixing with `/`. 22 | - All entries in `/etc/snazzer/exclude.patterns` need to be either absolute 23 | (start with `/`) or start with a `*`. 24 | - Paths displayed by snazzer are now absolute as well. 25 | ### Fixed ### 26 | - snazzer-receive compatibility issue with btrfs-progs 4.11 onwards 27 | 28 | ## [0.0.2] - 2016-12-28 ## 29 | First official snazzer release. 30 | ### Changed ### 31 | - Renamed doc/ -> docs/ for future github-pages work 32 | 33 | ### Added ### 34 | - --version switch implemented on all scripts 35 | 36 | ## 0.0.1 - 2016-12-28 ## 37 | Unofficial pre-release of the current state of snazzer, as there will be 38 | breaking changes between the current state and official releases in the near future. 39 | 40 | 41 | 42 | This uses [Keep a CHANGELOG](http://keepachangelog.com/) as a template. 43 | 44 | [Unreleased]: https://github.com/csirac2/snazzer/compare/v0.0.3...HEAD 45 | [0.0.2]: https://github.com/csirac2/snazzer/compare/v0.0.1...v0.0.2 46 | [0.0.3]: https://github.com/csirac2/snazzer/compare/v0.0.2...v0.0.3 47 | -------------------------------------------------------------------------------- /tests/snazzer-list.bats: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bats 2 | # vi:syntax=sh 3 | 4 | load "$BATS_TEST_DIRNAME/fixtures.sh" 5 | 6 | setup() { 7 | export SNAZZER_SUBVOLS_EXCLUDE_FILE=$BATS_TEST_DIRNAME/data/exclude.patterns 8 | local TMPDIR="${BATS_TMPDIR:-/tmp}" 9 | TMPDIR=$TMPDIR/snazzer-tests 10 | export SNAP_LIST_FILE=$TMPDIR/btrfs-snapshots.list 11 | export MNT=$(SNAP_LIST_FILE=$SNAP_LIST_FILE prepare_mnt_snapshots) 12 | [ -e "$SNAZZER_SUBVOLS_EXCLUDE_FILE" ] 13 | } 14 | 15 | expected_list_subvolumes_output() { 16 | NUM_EXCL=2 17 | 18 | expected_list_subvolumes "$MNT" 19 | cat < $(expected_file) 41 | readlink -f $(which snazzer) > $(actual_file) 42 | diff -u $(expected_file) $(actual_file) 43 | } 44 | 45 | @test "snazzer --list-subvolumes --all [mountpoint]" { 46 | run snazzer --list-subvolumes --all "$MNT" 47 | expected_list_subvolumes_output > $(expected_file) 48 | echo "$output" > $(actual_file) 49 | diff -u $(expected_file) $(actual_file) 50 | [ "$status" = "0" ] 51 | } 52 | 53 | @test "snazzer --list-snapshots --all [mountpoint]" { 54 | run snazzer --list-snapshots --all "$MNT" 55 | expected_list_snapshots_output > $(expected_file) 56 | echo "$output" > $(actual_file) 57 | diff -u $(expected_file) $(actual_file) 58 | [ "$status" = "0" ] 59 | } 60 | 61 | @test "snazzer --list-snapshots --all [mountpoint/subvol]" { 62 | run snazzer --list-snapshots --all "$MNT/home" 63 | [ "$status" = "2" ] 64 | } 65 | 66 | @test "snazzer --list-snapshots [/subvol1]" { 67 | run snazzer --list-snapshots "$MNT/home" 68 | expected_list_snapshots_output | grep "^$MNT/home" > $(expected_file) 69 | echo "$output" > $(actual_file) 70 | diff -u $(expected_file) $(actual_file) 71 | [ "$status" = "0" ] 72 | } 73 | 74 | @test "snazzer --list-snapshots [/subvol1] [/subvol2] [/subvol3]" { 75 | run snazzer --list-snapshots "$MNT/home" "$MNT/srv" "$MNT/var/cache" 76 | expected_list_snapshots_output | \ 77 | grep "^$MNT/\(home\|srv\|var/cache\)/\.snapshotz" | \ 78 | sort > $(expected_file) 79 | echo "$output" > $(actual_file) 80 | diff -u $(expected_file) $(actual_file) 81 | [ "$status" = "0" ] 82 | } 83 | 84 | teardown() { 85 | teardown_mnt "$MNT" >/dev/null 2>/dev/null 86 | } 87 | -------------------------------------------------------------------------------- /tests/snazzer.bats: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bats 2 | # vi:syntax=sh 3 | 4 | load "$BATS_TEST_DIRNAME/fixtures.sh" 5 | 6 | setup() { 7 | export SNAZZER_SUBVOLS_EXCLUDE_FILE=$BATS_TEST_DIRNAME/data/exclude.patterns 8 | export SNAZZER_DATE=$(date +"%Y-%m-%dT%H%M%S%z") 9 | export MNT=$(prepare_mnt) 10 | export SNAZZER_TMP=$BATS_TMPDIR/snazzer-tests 11 | [ -e "$SNAZZER_SUBVOLS_EXCLUDE_FILE" ] 12 | } 13 | 14 | gather_snapshots() { 15 | su_do find "$MNT" | grep -v '[0-9]/' | grep '[0-9]$' 16 | } 17 | 18 | expected_snapshots() { 19 | [ -n "$MNT" -a -e "$SNAZZER_SUBVOLS_EXCLUDE_FILE" ] 20 | expected_list_subvolumes "$MNT" | sed "s|$|/.snapshotz/$SNAZZER_DATE|g" 21 | } 22 | 23 | expected_snapshots_raw() { 24 | [ -n "$SNAZZER_DATE" ] 25 | echo "$MNT/.snapshotz/$SNAZZER_DATE" 26 | gen_subvol_list | sed "s|^|$MNT/|g" | while read SUBVOL; do 27 | echo "$SUBVOL/.snapshotz/$SNAZZER_DATE" 28 | done 29 | } 30 | 31 | @test "btrfs mkfs.btrfs in PATH" { 32 | btrfs --help 33 | mkfs.btrfs --help 34 | } 35 | 36 | @test "snazzer in PATH" { 37 | readlink -f $BATS_TEST_DIRNAME/../snazzer > $(expected_file) 38 | readlink -f $(which snazzer) > $(actual_file) 39 | diff -u $(expected_file) $(actual_file) 40 | } 41 | 42 | @test "snazzer --version" { 43 | run snazzer --version 44 | git_describe_snazzer_version > $(expected_file) 45 | echo "$output" > $(actual_file) 46 | diff -u $(expected_file) $(actual_file) 47 | [ "$status" = "0" ] 48 | } 49 | 50 | @test "snazzer --all check excludefile syntax" { 51 | SNAZZER_SUBVOLS_EXCLUDE_FILE=$BATS_TEST_DIRNAME/data/exclude.patterns.error 52 | run snazzer --all --dry-run "$MNT" 53 | # 12 means that snazzer detected the errors in the file 54 | [ "$status" = "12" ] 55 | } 56 | 57 | @test "snazzer --all [mountpoint]" { 58 | run snazzer --all "$MNT" 59 | expected_snapshots | sort > $(expected_file) 60 | gather_snapshots | sort > $(actual_file) 61 | diff -u $(expected_file) $(actual_file) 62 | [ "$status" = "0" ] 63 | } 64 | 65 | @test "snazzer copies user and group of source" { 66 | run snazzer --all "$MNT" 67 | stat "$MNT" --format "%U:%G" > $(expected_file) 68 | stat "$MNT/.snapshotz" --format "%U:%G" > $(actual_file) 69 | diff -u $(expected_file) $(actual_file) 70 | [ "$status" = "0" ] 71 | } 72 | 73 | @test "snazzer --dry-run --all [mountpoint]" { 74 | run snazzer --dry-run --all "$MNT" 75 | [ "$status" = "0" ] 76 | eval "$output" 77 | expected_snapshots | sort > $(expected_file) 78 | gather_snapshots | sort > $(actual_file) 79 | diff -u $(expected_file) $(actual_file) 80 | [ "$status" = "0" ] 81 | } 82 | 83 | @test "snazzer [subvol]" { 84 | run snazzer "$MNT/home" 85 | expected_snapshots_raw | grep "^$MNT/home" > $(expected_file) 86 | gather_snapshots | sort > $(actual_file) 87 | diff -u $(expected_file) $(actual_file) 88 | [ "$status" = "0" ] 89 | } 90 | 91 | @test "snazzer [subvol1] [subvol2] [subvol3]" { 92 | run snazzer "$MNT/home" "$MNT/srv" "$MNT/var/cache" 93 | expected_snapshots_raw | grep "^$MNT/\(home\|srv\|var/cache\)/\.snapshotz" \ 94 | | sort > $(expected_file) 95 | gather_snapshots | sort > $(actual_file) 96 | diff -u $(expected_file) $(actual_file) 97 | [ "$status" = "0" ] 98 | } 99 | 100 | teardown() { 101 | teardown_mnt "$MNT" >/dev/null 2>/dev/null 102 | } 103 | -------------------------------------------------------------------------------- /tests/snazzer-prune.bats: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bats 2 | # vi:syntax=sh 3 | # SMELL: These tests assume working snazzer-prune-candidates w/default config 4 | 5 | load "$BATS_TEST_DIRNAME/fixtures.sh" 6 | 7 | setup() { 8 | export SNAZZER_SUBVOLS_EXCLUDE_FILE=$BATS_TEST_DIRNAME/data/exclude.patterns 9 | export MNT=$(prepare_mnt_snapshots) 10 | [ -e "$SNAZZER_SUBVOLS_EXCLUDE_FILE" ] 11 | } 12 | 13 | gather_snapshots() { 14 | su_do btrfs subvolume list "$MNT" | sed "s|^.*path |$MNT/|g" | \ 15 | grep '\.snapshotz' 16 | } 17 | 18 | # reduces a list of snapshot full-paths down to those that should be left after 19 | # a prune operation. snazzer-prune candidates can only handle one set of dates 20 | # (one subvol) at a time, so this takes care of input spanning multiple subvols. 21 | fake_prune() { 22 | # Dodgy EOF end-of-list hack because code rhs of pipes are in a subshell and 23 | # thus can't manipulate variables outside of their scope... 24 | echo "$1 25 | <<<>>>" | while read SNAP; do 26 | # /tmp/snazzer-tests/mnt/.snapshotz/2015-05-02T063103+1100 27 | SUBVOL=$(echo $SNAP | sed -n "s#^$MNT/\(\(.*\)/\|\)\.snapshotz.*#\2#p") 28 | if [ "$SNAP" = "<<<>>>" ]; then 29 | echo "$THIS" | snazzer-prune-candidates --invert 30 | elif [ -z "$INIT" ]; then 31 | THIS=$SNAP 32 | LAST=$SUBVOL 33 | INIT=1 34 | elif [ "$SUBVOL" = "$LAST" ]; then 35 | if [ -n "$THIS" ]; then THIS="$THIS 36 | $SNAP"; else THIS=$SNAP; fi 37 | else 38 | echo "$THIS" | snazzer-prune-candidates --invert 39 | THIS=$SNAP 40 | LAST=$SUBVOL 41 | fi 42 | done 43 | } 44 | 45 | @test "btrfs mkfs.btrfs in PATH" { 46 | btrfs --help 47 | mkfs.btrfs --help 48 | } 49 | 50 | @test "snazzer-prune-candidates in PATH" { 51 | readlink -f $BATS_TEST_DIRNAME/../snazzer-prune-candidates > $(expected_file) 52 | readlink -f $(which snazzer-prune-candidates) > $(actual_file) 53 | diff -u $(expected_file) $(actual_file) 54 | } 55 | 56 | @test "snazzer-prune-candidates --version" { 57 | run snazzer-prune-candidates --version 58 | git_describe_snazzer_version > $(expected_file) 59 | echo "$output" > $(actual_file) 60 | diff -u $(expected_file) $(actual_file) 61 | [ "$status" = "0" ] 62 | } 63 | 64 | @test "snazzer --prune --all [mountpoint]" { 65 | run snazzer --prune --all "$MNT" 66 | [ "$status" = "5" ] 67 | } 68 | 69 | @test "snazzer --prune --all --force [mountpoint]" { 70 | BEFORE=$(gather_snapshots | sort) 71 | run snazzer --prune --all --force "$MNT" 72 | fake_prune "$BEFORE" > $(expected_file) 73 | gather_snapshots | sort > $(actual_file) 74 | diff -u $(expected_file) $(actual_file) 75 | [ "$status" = "0" ] 76 | } 77 | 78 | @test "snazzer --prune --all --dry-run [mountpoint]" { 79 | BEFORE=$(gather_snapshots | sort) 80 | run snazzer --prune --all --dry-run "$MNT" 81 | [ "$status" = "0" ] 82 | eval "$output" 83 | fake_prune "$BEFORE" > $(expected_file) 84 | gather_snapshots | sort > $(actual_file) 85 | diff -u $(expected_file) $(actual_file) 86 | [ "$status" = "0" ] 87 | } 88 | 89 | @test "snazzer --prune --force [subvol]" { 90 | BEFORE=$(gather_snapshots | sort | grep "^$MNT/home/\.snapshotz") 91 | run snazzer --prune --force "$MNT/home" 92 | echo "$BEFORE" | snazzer-prune-candidates --invert > $(expected_file) 93 | gather_snapshots | sort | grep "^$MNT/home/\.snapshotz" > $(actual_file) 94 | diff -u $(expected_file) $(actual_file) 95 | [ "$status" = "0" ] 96 | } 97 | 98 | @test "snazzer --prune --force [subvol1] [subvol2] [subvol3]" { 99 | BEFORE=$(gather_snapshots | sort | grep "^$MNT/\(home\|srv\|var/cache\)/\.snapshotz") 100 | run snazzer --prune --force "$MNT/home" "$MNT/srv" "$MNT" 101 | fake_prune "$BEFORE" > $(expected_file) 102 | gather_snapshots | sort | \ 103 | grep "^$MNT/\(home\|srv\|var/cache\)/\.snapshotz" > $(actual_file) 104 | diff -u $(expected_file) $(actual_file) 105 | [ "$status" = "0" ] 106 | } 107 | 108 | teardown() { 109 | teardown_mnt "$MNT" >/dev/null 2>/dev/null 110 | } 111 | -------------------------------------------------------------------------------- /Makefile: -------------------------------------------------------------------------------- 1 | all: markdown manpages 2 | 3 | release: all AUTHORS.md CHANGELOG.md 4 | 5 | INSTALL_PREFIX:=/usr/local 6 | ls_bin :=$(shell find . -maxdepth 1 -executable -type f -printf '%P\n') 7 | ls_bin_sh :=$(shell find . -maxdepth 1 -executable -type f \ 8 | -exec sed -n '1s:^\#!.*[ /]sh$$:{}:p' {} \;) 9 | 10 | install: install-bin install-man 11 | 12 | clean: 13 | rm -f AUTHORS.md 14 | rm -f $(addprefix man/, $(addsuffix .8.gz, $(call ls_bin))) 15 | [ ! -d man ] || rmdir man 16 | for btrfs_mnt in btrfs.working.mnt btrfs-snapshots.working.mnt; do \ 17 | ! mountpoint -q "/tmp/snazzer-tests/$$btrfs_mnt" || \ 18 | umount "/tmp/snazzer-tests/$$btrfs_mnt"; \ 19 | done 20 | rm -rf /tmp/snazzer-tests 21 | 22 | distclean: clean 23 | rm -f tmp/bin/bats 24 | rm -rf tmp/bats 25 | [ ! -d tmp/bin ] || rmdir tmp/bin 26 | [ ! -d tmp ] || rmdir tmp 27 | 28 | test: bats-tests prune-tests 29 | 30 | uninstall: 31 | rm -f $(addprefix $(INSTALL_PREFIX)/bin/, $(call ls_bin)) 32 | rm -f $(addprefix $(INSTALL_PREFIX)/share/man/man8/, $(addsuffix .8.gz, \ 33 | $(call ls_bin))) 34 | 35 | install-bin: $(addprefix $(INSTALL_PREFIX)/bin/, $(call ls_bin)) 36 | 37 | $(INSTALL_PREFIX)/bin/%: % 38 | install -Dm755 $< $@ 39 | 40 | install-man: $(addprefix $(INSTALL_PREFIX)/share/man/man8/, $(addsuffix .8.gz, \ 41 | $(call ls_bin))) 42 | 43 | $(INSTALL_PREFIX)/share/man/man8/%.8.gz: man/%.8.gz 44 | install -Dm644 $< $@ 45 | 46 | bats-tests: | bats 47 | PATH=.:tmp/bin:$$PATH bats tests/ 48 | 49 | bats: 50 | @PATH=tmp/bin:$$PATH bats --help >/dev/null || (\ 51 | mkdir -p tmp/bin;\ 52 | [ -f tmp/bats/bin/bats ] ||\ 53 | git clone https://github.com/sstephenson/bats tmp/bats;\ 54 | ln -s ../bats/bin/bats tmp/bin/;\ 55 | ) 56 | 57 | prune-tests: 58 | ./snazzer-prune-candidates --tests 59 | 60 | shellcheck-tests: | shellcheck 61 | PATH=~/.cabal/bin:tmp/bin:$$PATH shellcheck $(call ls_bin_sh) 62 | 63 | shellcheck: 64 | @PATH=~/.cabal/bin/:$$PATH shellcheck --version >/dev/null || (\ 65 | mkdir -p tmp/bin;\ 66 | if ! cabal --version >/dev/null; then\ 67 | echo "ERROR: Missing cabal. Please install cabal-install" >&2;\ 68 | exit 1;\ 69 | fi;\ 70 | cabal update;\ 71 | cabal install ShellCheck;\ 72 | ) 73 | 74 | markdown: $(addprefix docs/, $(addsuffix .md, $(call ls_bin))) 75 | 76 | docs/%.md: % | docs 77 | ./$< --man-markdown >$@ 78 | 79 | docs: 80 | mkdir $@ 81 | 82 | manpages: $(addprefix man/, $(addsuffix .8.gz, $(call ls_bin))) 83 | 84 | man/%.8.gz: % | man 85 | ./$< --man-roff | gzip >$@ 86 | 87 | man: 88 | mkdir $@ 89 | 90 | AUTHORS.md: 91 | echo "SNAZZER AUTHORS" > $@ 92 | echo "===============" >>$@ 93 | echo >>$@ 94 | echo "Compiled automatically from git history, in alphabetical order:" >> $@ 95 | echo >>$@ 96 | git log --format='- %aN <%aE>' | \ 97 | sort -u |grep -v 'Paul.W Harvey ' >> $@ 98 | 99 | # Print "0.0.1" from "v0.0.1-2-g4cb93f4", see also: tests/fixtures.sh 100 | define git-describe-snazzer-version 101 | $(shell git describe --tags | sed -n 's/v\?\([0-9.]*\).*/\1/p') 102 | endef 103 | 104 | define snazzer-version-escaped 105 | $(subst .,\.,$(call git-describe-snazzer-version)) 106 | endef 107 | 108 | # Sanity-check the changelog 109 | CHANGELOG.md: 110 | @echo "Building snazzer version $(call git-describe-snazzer-version)" 111 | @printf "Checking there's a $@ entry for this version:\n " 112 | @grep '^## $(call snazzer-version-escaped) - ' $@ 113 | @printf "Checking the [Unreleased] URL:\n " 114 | @grep '^\[Unreleased\]: https://github.com/csirac2/snazzer/compare/v$(call \ 115 | snazzer-version-escaped)...HEAD' $@ 116 | 117 | .PHONY: CHANGELOG.md 118 | 119 | # assumes "#!/usr/bin/env foo", rewrites to "#!/path/to/foo" 120 | rewrite-shebangs-to-bin: 121 | for script in $(call ls_bin);\ 122 | do\ 123 | script_bin=$$(sed -n '1s:.*[ /][ /]*\([^ /]*\)$$:\1:p' "$$script");\ 124 | sed -i "1s:.*[ /][ /]*\([^ /]*\)$$:#\!$$(which $$script_bin):g"\ 125 | "$$script";\ 126 | done 127 | 128 | # assumes "#!/path/to/foo", rewrites to "#!/usr/bin/env foo" 129 | rewrite-shebangs-to-env: 130 | for script in $(call ls_bin);\ 131 | do\ 132 | script_bin=$$(sed -n '1s:.*[ /][ /]*\([^ /]*\)$$:\1:p' "$$script");\ 133 | sed -i "1s:.*[ /][ /]*\([^ /]*\)$$:#\!/usr/bin/env $$script_bin:g"\ 134 | "$$script";\ 135 | done 136 | 137 | .PHONY: install uninstall install-bin install-man markdown manpages 138 | .PHONY: all release clean distclean test bats-tests bats prune-tests 139 | .PHONY: rewrite-shebangs-to-bin rewrite-shebangs-to-env 140 | -------------------------------------------------------------------------------- /docs/snazzer-send-wrapper.md: -------------------------------------------------------------------------------- 1 | # NAME 2 | 3 | snazzer-send-wrapper - ssh forced command wrapper for snazzer-receive 4 | 5 | # SYNOPSIS 6 | 7 | SSH_ORIGINAL_COMMAND="sudo -n snazzer --list-snapshots '--all'" \ 8 | ./snazzer-send-wrapper 9 | 10 | SSH_ORIGINAL_COMMAND="sudo -n grep -srl \ 11 | 'sendinghost1' '/some/.snapshotz/.measurements/'" snazzer-send-wrapper 12 | 13 | SSH_ORIGINAL_COMMAND="sudo -n btrfs send \ 14 | '/some/.snapshotz/2015-04-01T000000Z'" snazzer-send-wrapper 15 | 16 | SSH_ORIGINAL_COMMAND="sudo -n cat \ 17 | '/some/.snapshotz/.measurements/2015-04-01T000000Z'" snazzer-send-wrapper 18 | 19 | # OPTIONS 20 | 21 | - **--help**: Brief help message 22 | - **--version**: Print version 23 | - **--man**: Full documentation 24 | - **--man-roff**: Full documentation as \*roff output, Eg: 25 | 26 | snazzer --man-roff | nroff -man 27 | 28 | - **--man-markdown**: Full documentation as markdown output, Eg: 29 | 30 | snazzer --man-markdown > snazzer-manpage.md 31 | 32 | # DESCRIPTION 33 | 34 | This is a wrapper script to be used in place of a real login shell (Eg. as an 35 | ssh(1) forced command) in order to restrict the commands available to the user 36 | account used by **snazzer-receive** to run `btrfs send`. It may be utilized by 37 | adding an entry in the `~/.ssh/authorized_keys` file on the sending host (Eg. 38 | `sendinghost1`) under the user account used by **snazzer-receive** to run 39 | `btrfs send`. `~/.ssh/authorized_keys`: 40 | 41 | command="/usr/bin/snazzer-send-wrapper",no-port-forwarding, \ 42 | no-X11-forwarding,no-pty ssh-rsa AAAA...snip...== my key 43 | 44 | And then (as an example) receive btrfs snapshots from this `sendinghost1`: 45 | 46 | snazzer-receive sendinghost1 --all 47 | 48 | # ENVIRONMENT 49 | 50 | - SSH\_ORIGINAL\_COMMAND 51 | 52 | This variable holds the original remote ssh command to be acted upon. 53 | 54 | # BUGS AND LIMITATIONS 55 | 56 | - This script tries too hard to parse normal shell commands 57 | 58 | A better design would be custom command tokens issued with more easily parsed 59 | string and argument delimeters. This would require some changes to 60 | **snazzer-receive**. 61 | 62 | A mitigating factor is that all commands are executed in the form of: 63 | 64 | foo "$@" 65 | 66 | Rather than any variant of the more exciting: 67 | 68 | foo $BAREWORD_ARGUMENTS 69 | 70 | or 71 | 72 | eval "$SSH_ORIGINAL_COMMAND" 73 | 74 | This wrapper script is also sanity-checked with bats regression tests which 75 | check that only the correct number of arguments, valid arguments, switches, 76 | path patterns and escape characters are dealt with - anything else is rejected. 77 | 78 | # EXIT STATUS 79 | 80 | **snazzer-send-wrapper** will abort with an error message printed to STDERR and 81 | non-zero exit status under the following conditions: 82 | 83 | - 2. the command string was not recognized 84 | - 98. the command string was recognized but the arguments were not safe 85 | - 99. the command string was recognized and an attempt was made to 86 | parse/re-pack the arguments however the argument string had dangling quotes or 87 | otherwise confused the parser/"$@" unpacker 88 | 89 | # SEE ALSO 90 | 91 | snazzer-receive 92 | 93 | # AUTHOR 94 | 95 | Snazzer Authors are listed in the AUTHORS.md file in the root of this 96 | distribution. See https://github.com/csirac2/snazzer for more information. 97 | NOTE: Please extend that file, not this notice. 98 | 99 | # LICENSE AND COPYRIGHT 100 | 101 | Copyright (C) 2015-2016, Snazzer Authors All rights reserved. 102 | 103 | Redistribution and use in source and binary forms, with or without 104 | modification, are permitted provided that the following conditions are met: 105 | 106 | 1\. Redistributions of source code must retain the above copyright notice, this 107 | list of conditions and the following disclaimer. 108 | 109 | 2\. Redistributions in binary form must reproduce the above copyright notice, 110 | this list of conditions and the following disclaimer in the documentation 111 | and/or other materials provided with the distribution. 112 | 113 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 114 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 115 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 116 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 117 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 118 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 119 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 120 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 121 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 122 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 123 | -------------------------------------------------------------------------------- /tests/fixtures.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | set -e 3 | 4 | su_do() { 5 | if [ "$(id -u)" = "0" ]; then 6 | "$@" 7 | else 8 | sudo "$@" 9 | fi 10 | } 11 | 12 | # Print "0.0.1" from "v0.0.1-2-g4cb93f4" 13 | git_describe_snazzer_version() { 14 | git describe --tags | sed -n 's/v\?\([0-9.]*\).*/\1/p' 15 | } 16 | 17 | gen_subvol_list() { 18 | for SUBVOL in srv 'srv/s p a c e' home var/cache var/lib/docker/btrfs \ 19 | 'echo `ls "/"; ls /;`; ~!@#$(ls)%^&*()_+-='\''[]'\''{}|:<>,./?' \ 20 | tmp_thing; 21 | do echo "$SUBVOL"; done 22 | } 23 | 24 | # As gen_subvol_list(), but filtered with exclude.patterns applied 25 | gen_subvol_list_excluded() { 26 | for SUBVOL in srv 'srv/s p a c e' home \ 27 | 'echo `ls "/"; ls /;`; ~!@#$(ls)%^&*()_+-='\''[]'\''{}|:<>,./?' \ 28 | tmp_thing; 29 | do echo "$SUBVOL"; done 30 | } 31 | 32 | populate_mnt() { 33 | local MNT="$1" 34 | [ -n "$MNT" ] 35 | shift 36 | if [ "$MNT" = "/" ]; then MNT=""; fi 37 | su_do chown "$USER" "$MNT" 38 | while read SUBVOL; do 39 | local SUBVOL_PARENT="$(dirname "$SUBVOL")" 40 | local SUBVOL_NAME="$(basename "$SUBVOL")" 41 | mkdir -p "$MNT/$SUBVOL_PARENT" 42 | su_do btrfs subvolume create "$MNT/$SUBVOL" 43 | su_do chown "$USER" "$MNT/$SUBVOL" 44 | touch "$MNT/$SUBVOL/${SUBVOL_NAME}_junk" 45 | if [ "$SUBVOL_PARENT" = "." ]; then 46 | touch "$MNT/${SUBVOL_NAME}_junk" 47 | else 48 | touch "$MNT/${SUBVOL_PARENT}_${SUBVOL_NAME}_junk" 49 | fi 50 | done 51 | } 52 | 53 | create_img() { 54 | local IMG="$1" 55 | local TMPDIR="${BATS_TMPDIR:-/tmp}" 56 | TMPDIR=$TMPDIR/snazzer-tests 57 | local MNT=$TMPDIR/btrfs.working.mnt 58 | [ -n "$MNT" -a -n "$IMG" ] 59 | mkdir -p "$MNT" 60 | if df -T "$MNT" 2>/dev/null | grep "$MNT\$" 2>/dev/null >/dev/null; then 61 | umount "$MNT" 62 | fi 63 | truncate -s 200M "$IMG" 64 | # rm .img if mkfs fails because create_img is skipped when it already exists 65 | su_do mkfs.btrfs "$IMG" || rm "$IMG" 66 | su_do mount "$IMG" "$MNT" 67 | gen_subvol_list | populate_mnt "$MNT" 68 | if [ "$DO_SNAPSHOTS" = "1" ]; then 69 | snapshot_mnt "$MNT" >/dev/null 2>/dev/null; 70 | fi 71 | su_do umount "$MNT" 72 | } 73 | 74 | _prepare_mnt() { 75 | local TMPDIR="${BATS_TMPDIR:-/tmp}" 76 | TMPDIR=$TMPDIR/snazzer-tests 77 | if [ "$DO_SNAPSHOTS" = "1" ]; then 78 | local NAME=btrfs 79 | else 80 | local NAME=btrfs-snapshots 81 | fi 82 | local MNT="$TMPDIR/${NAME}.working.mnt" 83 | local WRK="$TMPDIR/${NAME}.working.img" 84 | local IMG="$TMPDIR/${NAME}.img" 85 | teardown_mnt "$MNT" 86 | mkdir -p "$MNT" 87 | if [ ! -e "$IMG" ]; then 88 | create_img "$IMG" >/dev/null 2>/dev/null 89 | fi 90 | cp "$IMG" "$WRK" 91 | mkdir -p "$MNT" 92 | su_do mount "$WRK" "$MNT" 93 | echo "$MNT" 94 | } 95 | 96 | prepare_mnt() { 97 | DO_SNAPSHOTS=0 _prepare_mnt "$@" 98 | } 99 | 100 | prepare_mnt_snapshots() { 101 | DO_SNAPSHOTS=1 _prepare_mnt "$@" 102 | } 103 | 104 | teardown_mnt() { 105 | local MNT="$1" 106 | local IMG=$(mount |sed -n "s|^\\(.*\\) on $MNT.*|\1|p") 107 | [ -n "$MNT" ] 108 | if mountpoint -q "$MNT" 2>/dev/null; then 109 | su_do umount "$MNT" 110 | rmdir "$MNT" 111 | fi 112 | rm -f "$IMG" 113 | } 114 | 115 | gen_snapshot_dates() { 116 | cat <"$TMP_DATES" 156 | 157 | expected_list_subvolumes "$MNT" | while read SUBVOL; do 158 | mkdir -p "$SUBVOL/.snapshotz" 159 | while read DATE <&6; do 160 | su_do btrfs subvolume snapshot -r "$SUBVOL" \ 161 | "$SUBVOL/.snapshotz/$DATE" >/dev/null 2>/dev/null 162 | if [ -n "$SNAP_LIST_FILE" ]; then 163 | echo "$SUBVOL/.snapshotz/$DATE" >>"$SNAP_LIST_FILE" 164 | fi 165 | done 6<"$TMP_DATES" 166 | done 167 | 168 | rm "$TMP_DATES" 169 | } 170 | 171 | expected_file() { 172 | echo "$BATS_TMPDIR/snazzer-tests/$(basename \ 173 | $BATS_TEST_FILENAME)_${BATS_TEST_NUMBER}${1}.expected" 174 | } 175 | 176 | actual_file() { 177 | echo "$BATS_TMPDIR/snazzer-tests/$(basename \ 178 | $BATS_TEST_FILENAME)_${BATS_TEST_NUMBER}${1}.actual" 179 | } 180 | -------------------------------------------------------------------------------- /docs/snazzer-prune-candidates.md: -------------------------------------------------------------------------------- 1 | # NAME 2 | 3 | snazzer-prune-candidates - reduce a set of lines containing datetimes to only 4 | those which are no longer needed to meet retention preferences 5 | 6 | # SYNOPSIS 7 | 8 | find /some/.snapshotz -maxdepth 1 -mindepth 1 -type d | \ 9 | snazzer-prune-candidates | xargs btrfs subvolume delete 10 | 11 | echo -e "2015-02-01T000000Z\n2015-02-01T000010Z" | snazzer-prune-candidates 12 | 13 | snazzer-prune-candidates --gen-example-input | \ 14 | ./snazzer-prune-candidates --invert 15 | 16 | # OPTIONS 17 | 18 | - **--invert:** Invert output to contain only lines which should be retained 19 | - **--gen-example-input:** Generate example datetime strings for testing 20 | - **--verbose:** Verbose debugging output to STDERR 21 | - **--help:** Brief help message 22 | - **--version:** Print version number 23 | - **--man:** Full documentation 24 | - **--man-roff:** Full documentation as \*roff output, Eg: 25 | 26 | snazzer-prune-candidates --man-roff | nroff -man 27 | 28 | - **--man-markdown:** Full documentation as markdown output, Eg: 29 | 30 | snazzer-prune-candidates --man-markdown > snazzer-prune-candidates-man.md 31 | 32 | - **--tests:** Run tests (for developers/maintainers) 33 | - **--force-tty-stdin:** Skip checks for interactive tty on STDIN 34 | 35 | # ENVIRONMENT 36 | 37 | - SNAZZER\_YEARLIES\_TO\_KEEP 38 | 39 | Keep one date per year for the last N years. Default: 1000 40 | 41 | - SNAZZER\_MONTHLIES\_TO\_KEEP 42 | 43 | Keep one date per month for the last N months. Default: 12 44 | 45 | - SNAZZER\_DAYLIES\_TO\_KEEP 46 | 47 | Keep one date per day for the last N days. Default: 31 48 | 49 | - SNAZZER\_HOURLIES\_TO\_KEEP 50 | 51 | Keep one date per hour for the last N hours. Default: 24 52 | 53 | # DESCRIPTION 54 | 55 | **snazzer-prune-candidates** reads lines of input from STDIN which are expected 56 | to end in datetimes which are a subset of valid ISO 8601 strings: 57 | 58 | YYYY-MM-DD 59 | YYYY-MM-DDTHHMMSSZ 60 | YYYY-MM-DDTHHMMSS+HH 61 | YYYY-MM-DDTHHMMSS-HHMM 62 | 63 | The parsing is a dumb regex to avoid library dependencies. It is lax about what, 64 | if anything separates date or time parts - so for example, the following are 65 | also valid (and demonstrate that only the end of these lines are parsed - but 66 | note that if a line is considered unnecessary, it will be printed unchanged in 67 | full to STDOUT): 68 | 69 | /any/old/junk/YYYYMMDD 70 | /any/old/junk/YYYY_MM_DDTHH:MM:SSZ 71 | /any/old/junk/YYYY-MM-DDTHH_MM_SS+HH:MM 72 | 73 | Lines which aren't required to meet retention preferences are printed to STDOUT. 74 | 75 | **NOTE:** Command-line options override environment variables. 76 | 77 | **NOTE:** the description in [OPTIONS](https://metacpan.org/pod/OPTIONS) mentions "last N ". 78 | This refers to the period of time looking back from the most recent date seen at 79 | the input. **snazzer-prune-candidates** does not use the local system time for 80 | any decision-making part of the program. 81 | 82 | # EXIT STATUS 83 | 84 | **snazzer-prune-candidates** will abort with an error message printed to STDOUT 85 | and non-zero exit status under the following conditions: 86 | 87 | - 1. A retention preference value contains anything other than digits 88 | - 2. Line does not end in a valid datetime string pattern 89 | - 3. Datetime contains obviously non-sensical digits 90 | - 4. Detected an interactive tty and no --force-tty-stdin option was given 91 | 92 | # BUGS AND LIMITATIONS 93 | 94 | - Homebrew datetime code 95 | 96 | Due to a desire to avoid any non-core library dependencies there may be bugs 97 | with all the fun things that happen with home-brew time handling code: daylight 98 | savings, leap-years/hours/minutes/seconds and treatment of mixed timezones. 99 | 100 | A future version should try to use an appropriate datetime library to completely 101 | offload normalization, differencing and comparison of datetimes when available. 102 | 103 | - When some datetimes are close together, they mightn't be pruned 104 | 105 | **snazzer-prune-candidates** iterates over each line of the input several times: 106 | once each to mark datetimes required to be kept to meet hourly, daily, monthly 107 | and yearly retention preferences. At the end of this process, all unmarked lines 108 | may safely be dropped and those are emitted for pruning. 109 | 110 | However, rather than require exactly 60mins, 24hrs, 28/29/30/31 days etc. 111 | between snapshots - which would risk dropping some previously retained datetimes 112 | depending on how far your snapshot runs drift from their usual schedule - the 113 | algorithm instead marks whichever datetime would most closely satisfy the 114 | retention requirement relative to the previously marked item. 115 | 116 | The end result is that occasionally (for example) a snapshot which has been 117 | marked as the best choice to meet the monthly requirement isn't quite the same 118 | snapshot as the one that has already been marked to meet the daily retention 119 | requirement, although they may be very close together in time. In fact, when 120 | starting out from very few snapshots to begin with, you may find several 121 | snapshots very close together are being retained toward the end of your set of 122 | snapshots due to the coarser retention periods marking out snapshots which are 123 | only a few minutes/hours older than other marked snapshots simply because they 124 | are slightly closer to the next retention interval (even if that difference 125 | seems trivial). If this bothers you, please provide feedback or patches to the 126 | author. 127 | 128 | # SEE ALSO 129 | 130 | snazzer, snazzer-measure, snazzer-receive 131 | 132 | # AUTHOR 133 | 134 | Snazzer Authors are listed in the AUTHORS.md file in the root of this 135 | distribution. See https://github.com/csirac2/snazzer for more information. 136 | NOTE: Please extend that file, not this notice. 137 | 138 | # LICENSE AND COPYRIGHT 139 | 140 | Copyright (C) 2015-2016, Snazzer Authors All rights reserved. 141 | 142 | Redistribution and use in source and binary forms, with or without 143 | modification, are permitted provided that the following conditions are met: 144 | 145 | 1\. Redistributions of source code must retain the above copyright notice, this 146 | list of conditions and the following disclaimer. 147 | 148 | 2\. Redistributions in binary form must reproduce the above copyright notice, 149 | this list of conditions and the following disclaimer in the documentation 150 | and/or other materials provided with the distribution. 151 | 152 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 153 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 154 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 155 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 156 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 157 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 158 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 159 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 160 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 161 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 162 | -------------------------------------------------------------------------------- /docs/snazzer-measure.md: -------------------------------------------------------------------------------- 1 | # NAME 2 | 3 | snazzer-measure - report shasums & PGP signatures of content under a given path, 4 | along with commands to reproduce or verify data is unchanged 5 | 6 | # SYNOPSIS 7 | 8 | snazzer-measure /measured/path [/reported/path] >> path_measurements 9 | 10 | # DESCRIPTION 11 | 12 | Creates reproducible fingerprints of the given directory, along with commands 13 | necessary (relative to measured path, or if supplied - the optional reported 14 | path) to reproduce the measurement using only standard core GNU userland. 15 | 16 | The output includes: 17 | 18 | - hostname and datetime of **snazzer-measure** invocation 19 | - `du -bs` (bytes used) 20 | - `sha512sum` of the result of a reproducible tarball of the directory 21 | - `gpg2 --armor --sign` of the same 22 | - instructions for reproducing or verifying each of the above 23 | - `tar --version`, `tar --show-defaults` 24 | 25 | # OPTIONS 26 | 27 | - **--help**: Brief help message 28 | - **--version**: Print version 29 | - **--man**: Full documentation 30 | - **--man-roff**: Full documentation as \*roff output, Eg: 31 | 32 | snazzer-measure --man-roff | nroff -man 33 | 34 | - **--man-markdown**: Full documentation as markdown output, Eg: 35 | 36 | snazzer-measure --man-markdown > snazzer-measure-manpage.md 37 | 38 | # ENVIRONMENT 39 | 40 | - snazzer\_sig\_func 41 | 42 | Function generating PGP SIGNATURE text. Takes input from stdin, output to 43 | stdout. Signatures can be disabled with [SNAZZER\_SIG\_ENABLE](https://metacpan.org/pod/SNAZZER_SIG_ENABLE). Default: 44 | 45 | snazzer_sig_func() { 46 | gpg2 --quiet --no-greeting --batch --use-agent --armor --detach-sign - 47 | } 48 | 49 | - SNAZZER\_SIG\_ENABLE 50 | 51 | If set to 0, GPG signing is disabled and snazzer\_sig\_func() is not called. 52 | 53 | - SNAZZER\_MEASUREMENTS\_EXCLUDE\_FILE 54 | 55 | A filename within the measured directory of a newline-separated list of shell 56 | glob patterns to exclude from measurements. Default: 57 | 58 | SNAZZER_MEASUREMENTS_EXCLUDE_FILE=".snapshot_measurements.exclude" 59 | 60 | - MY\_KEYFILES\_ARE\_INVINCIBLE=1 61 | 62 | Skip sanity check/abort when gpg secret key exists on a subvolume included in 63 | default snazzer snapshots 64 | 65 | - SNAZZER\_USE\_UTC 66 | 67 | Use UTC times of the form `YYYY-MM-DDTHHMMSSZ` instead of the default local 68 | time+offset `YYYY-MM-DDTHHMMSS+hhmm` 69 | 70 | - SNAZZER\_SUBVOLS\_EXCLUDE\_FILE 71 | 72 | Filename of newline separated list of shell glob patterns of subvolume pathnames 73 | which should be excluded from `snazzer --all` invocations; compatible with 74 | `--exclude-from` for **du** and **tar**. Examples of subvolume patterns to 75 | exclude from regular snapshotting: \*secret\*, /var/cache, /var/lib/docker/\*, 76 | .snapshots. **NOTE:** `.snapshotz` is always excluded. 77 | Default: 78 | 79 | SNAZZER_SUBVOLS_EXCLUDE_FILE="/etc/snazzer/exclude.patterns" 80 | 81 | ## sudo requirements 82 | 83 | When running **snazzer-measure** as a non-root user, certain commands will be 84 | prefixed with `sudo`. The following lines in `/etc/sudoers` or 85 | `/etc/sudoers.d/snazzer` should suffice for scripted jobs such as cron (replace 86 | `measureuser` with the actual user name you are setting up for this task): 87 | 88 | measureuser ALL=(root:nobody) NOPASSWD:NOEXEC: \ 89 | /bin/cat */.snapshotz/*/.snapshot_measurements.exclude, \ 90 | /usr/bin/du -bs --one-file-system --exclude-from * */.snapshotz/*, \ 91 | /usr/bin/find */.snapshotz/* \ 92 | -xdev -not -path /*/.snapshotz/* -printf %P\\\\0, \ 93 | /bin/tar --no-recursion --one-file-system --preserve-permissions \ 94 | --numeric-owner --null --create --to-stdout \ 95 | --directory */.snapshotz/* --files-from * \ 96 | --exclude-from */.snapshotz/*/.snapshot_measurements.exclude 97 | 98 | # EXIT STATUS 99 | 100 | **snazzer-measure** will abort with an error message printed to STDERR and 101 | non-zero exit status under the following conditions: 102 | 103 | - 1. Invalid argument 104 | - 2. Path string not specified 105 | - 4. GPG signature would have been generated with a secret keyfile stored 106 | in a subvolume which has not been excluded from default snazzer snapshots, see 107 | [IMPORTANT](https://metacpan.org/pod/IMPORTANT) below 108 | - 5. Expected the .snazzer\_measurements.exclude file to contain an entry 109 | for the .snazzer\_measurements file 110 | 111 | # IMPORTANT 112 | 113 | Please note that if you are using this tool to gain some form of integrity 114 | measurement (Eg. you want to detect tampering), GPG private keys used for the 115 | signing operation mustn't be exposed among the directories being measured. 116 | 117 | Put another way: it makes no sense to GPG-sign measurements of a directory if 118 | those very same directories contain the GPG private key material required to 119 | re-sign modifications made by anyone who happens to be looking. 120 | 121 | # BUGS AND LIMITATIONS 122 | 123 | - MY\_KEYFILES\_ARE\_INVINCIBLE 124 | 125 | The sanity check for location of GPG secret keyfile may be more annoying than 126 | helpful on installations using smartcards, TPMs, or other methods of protecting 127 | keyfiles - hence the **MY\_KEYFILES\_ARE\_INVINCIBLE** work-around. 128 | 129 | - Temporary files 130 | 131 | To avoid unnecessary I/O, gpg signing and shasumming are done in parallel from 132 | the same `tar --to-stdout` pipe; this involves creating a temporary named pipe 133 | which is normally removed at the end of a successful run, but will be left 134 | behind should a failure occur. These are randomly named with `mktemp` and mode 135 | 0700, inside a `mktemp -d` directory also with 0700 permissions. 136 | 137 | # SEE ALSO 138 | 139 | snazzer, snazzer-prune-candidates, snazzer-receive 140 | 141 | # AUTHOR 142 | 143 | Snazzer Authors are listed in the AUTHORS.md file in the root of this 144 | distribution. See https://github.com/csirac2/snazzer for more information. 145 | NOTE: Please extend that file, not this notice. 146 | 147 | # LICENSE AND COPYRIGHT 148 | 149 | Copyright (C) 2015-2016, Snazzer Authors All rights reserved. 150 | 151 | Redistribution and use in source and binary forms, with or without 152 | modification, are permitted provided that the following conditions are met: 153 | 154 | 1\. Redistributions of source code must retain the above copyright notice, this 155 | list of conditions and the following disclaimer. 156 | 157 | 2\. Redistributions in binary form must reproduce the above copyright notice, 158 | this list of conditions and the following disclaimer in the documentation 159 | and/or other materials provided with the distribution. 160 | 161 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 162 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 163 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 164 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 165 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 166 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 167 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 168 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 169 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 170 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 171 | -------------------------------------------------------------------------------- /docs/snazzer.md: -------------------------------------------------------------------------------- 1 | # NAME 2 | 3 | snazzer - create read-only `/subvol/.snapshotz/[isodate]` btrfs snapshots, 4 | offers snapshot pruning and measurement 5 | 6 | # SYNOPSIS 7 | 8 | snazzer [--prune|--measure [--force ]] [--dry-run] --all 9 | 10 | snazzer [--prune|--measure [--force ]] [--dry-run] --all [mountpoint] 11 | 12 | snazzer [--prune|--measure [--force ]] [--dry-run] subvol1 [subvol2 [...]] 13 | 14 | # DESCRIPTION 15 | 16 | Examples: 17 | 18 | Snapshot all non-excluded subvols on all mounted btrfs filesystems: 19 | 20 | snazzer --all 21 | 22 | Prune all non-excluded subvols on all mounted btrfs filesystems: 23 | 24 | snazzer --prune --force --all 25 | 26 | Append output of **snazzer-measure** to 27 | `/path/to/subvol/.snapshotz/.measurements/[isodate]` for all snapshots of all 28 | subvolumes on all mounted btrfs filesytems (slow!): 29 | 30 | snazzer --measure --force --all 31 | 32 | As above, skipping snapshots already measured by this host (recommended): 33 | 34 | snazzer --measure --all 35 | 36 | Print rather than execute commands for snapshotting all non-excluded subvols for 37 | the filesystem mounted at /mnt (including /mnt itself): 38 | 39 | snazzer --dry-run --all /mnt 40 | 41 | Prune only the explicitly named subvols at /srv, /var/log and root: 42 | 43 | snazzer --prune /srv /var/log / 44 | 45 | # OPTIONS 46 | 47 | - **--all** **\[mountpoint\]**: act on all subvolumes under mountpoint. If 48 | mountpoint is omitted, **snazzer** acts on all mounted btrfs filesystems. 49 | - **--prune**: delete rather than create snapshots. Exactly which are no 50 | longer needed is **snazzer-prune-candidates**'s role, documented separately 51 | - **--measure**: append output of **snazzer-measure** to 52 | `/path/to/subvol/.snapshotz/.measurements/[isodate]` By default, only snapshots 53 | which haven't been measured by this hostname are updated - use **--force** to 54 | measure all snapshots 55 | - **--force**: required for **--prune** to carry out any pruning operation. 56 | For **--measure**, this switch overrides the default behaviour of skipping 57 | snapshots already measured by current hostname 58 | - **--list-subvolumes**: list subvolumes that would be acted on 59 | - **--list-snapshots**: list snapshots under subvolumes as above 60 | - **--dry-run**: print rather than execute commands that would be run 61 | - **--help**: Brief help message 62 | - **--version**: Print version 63 | - **--man**: Full documentation 64 | - **--man-roff**: Full documentation as \*roff output, Eg: 65 | 66 | snazzer --man-roff | nroff -man 67 | 68 | - **--man-markdown**: Full documentation as markdown output, Eg: 69 | 70 | snazzer --man-markdown > snazzer-manpage.md 71 | 72 | # ENVIRONMENT 73 | 74 | - SNAZZER\_SUBVOLS\_EXCLUDE\_FILE 75 | 76 | Filename of newline separated list of shell glob patterns of subvolume pathnames 77 | which should be excluded from `snazzer --all` invocations; compatible with 78 | `--exclude-from` for **du** and **tar**. Examples of subvolume patterns to 79 | exclude from regular snapshotting: \*secret\*, /var/cache, /var/lib/docker/\*, 80 | \*/.snapshots. Note that **NOTE:** `.snapshotz` is always excluded. 81 | Default: 82 | 83 | SNAZZER_SUBVOLS_EXCLUDE_FILE="/etc/snazzer/exclude.patterns" 84 | 85 | - SNAZZER\_USE\_UTC=1 86 | 87 | For snapshot naming and **snazzer-measure** output use UTC times of the form 88 | `YYYY-MM-DDTHHMMSSZ` instead of local time+offset `YYYY-MM-DDTHHMMSS+hhmm` 89 | 90 | # BUGS AND LIMITATIONS 91 | 92 | - Snapshot naming 93 | 94 | A choice has been made to mint a single datetime string which is used for all 95 | snapshot names in a given **snazzer** snapshot invocation, regardless of how long 96 | or at which exact time the snapshotting process takes place for each subvolume. 97 | This makes for consistency across all subvolumes and filesystems, so that 98 | identifying which snapshots were part of a given snapshotting run is possible. 99 | If the actual datetime of the snapshot event is important to you, this is 100 | available from the `btrfs subvolume show` command. 101 | 102 | - Snapshot access 103 | 104 | By default, the `.snapshotz` folder is only read- and writable by the owner 105 | of the snapshotted subvolume, group and others have no permission for 106 | anything. This is on purpose, to protect information leaks from the snapshots. 107 | If you need more open access rights, you can always change ownership and 108 | permissions by hand. 109 | 110 | - SNAZZER\_SUBVOLS\_EXCLUDE\_FILE is used with grep -f 111 | 112 | A minimal (possibly buggy/incomplete) attempt is made to convert the shell glob 113 | patterns in this file to a regex suitable for grep -f. The assumption is that 114 | the exclude patterns file should only contain "boring" paths. Obvious regex 115 | characters are escaped, however there are likely hostile path glob patterns 116 | which will break things. 117 | 118 | - .snapshot\_measurements.exclude is a work-around to the btrfs atime bug 119 | 120 | Snapshots may include empty directories under which some other subvol may have 121 | existed in the original, snapshotted subvolume. However, btrfs has a bug where 122 | these empty directories behave differently to empty directories created with 123 | `mkdir`: atimes always return with the current local time, which is obvioulsy 124 | different from one second to the next. So we have no hope of creating 125 | reproducible shasums or PGP signatures unless those directories are excluded 126 | from our measurements of the snapshot. See also: 127 | [https://bugzilla.kernel.org/show\_bug.cgi?id=95201](https://bugzilla.kernel.org/show_bug.cgi?id=95201) 128 | 129 | # EXIT STATUS 130 | 131 | **snazzer** will abort with an error message printed to STDERR and non-zero exit 132 | status under the following conditions: 133 | 134 | - 1. invalid arguments 135 | - 2. path is not a filesystem mountpoint 136 | - 3. one or more paths were not btrfs subvolumes 137 | - 4. prune expected /path/to/subvol/.snapshotz directory which was missing 138 | - 5. prune expected --dry-run or --force 139 | - 6. tried to write a .snapshot\_measurements.exclude file in the snapshot 140 | root, but it already exists in the current subvolume 141 | - 7. tried to perform snapshot measurements while existing measurements are 142 | already in progress, check lock dir at /var/run/snazzer-measure.lock 143 | - 9. tried to display man page with a formatter which is not installed 144 | - 10. missing `snazzer-measure` or `snazzer-prune-candidates` from PATH 145 | - 11. missing `btrfs` command from PATH 146 | - 12. syntax error in /etc/snazzer/exclude.patterns file. 147 | 148 | # SEE ALSO 149 | 150 | snazzer-measure, snazzer-prune-candidates, snazzer-receive 151 | 152 | # AUTHOR 153 | 154 | Snazzer Authors are listed in the AUTHORS.md file in the root of this 155 | distribution. See https://github.com/csirac2/snazzer for more information. 156 | NOTE: Please extend that file, not this notice. 157 | 158 | # LICENSE AND COPYRIGHT 159 | 160 | Copyright (C) 2015-2016, Snazzer Authors All rights reserved. 161 | 162 | Redistribution and use in source and binary forms, with or without 163 | modification, are permitted provided that the following conditions are met: 164 | 165 | 1\. Redistributions of source code must retain the above copyright notice, this 166 | list of conditions and the following disclaimer. 167 | 168 | 2\. Redistributions in binary form must reproduce the above copyright notice, 169 | this list of conditions and the following disclaimer in the documentation 170 | and/or other materials provided with the distribution. 171 | 172 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 173 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 174 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 175 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 176 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 177 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 178 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 179 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 180 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 181 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 182 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | snazzer 2 | ======= 3 | 4 | btrfs snapshotting and backup system offering snapshot measurement, transport 5 | and pruning. 6 | 7 | [![Build Status](https://travis-ci.org/csirac2/snazzer.svg?branch=master)](https://travis-ci.org/csirac2/snazzer) 8 | 9 | 10 | Features 11 | -------- 12 | 13 | * Minimal dependencies (portable-ish sh script, mostly checked with http://shellcheck.net) 14 | * Maintains snapshots for each subvol under 15 | `subvol/.snapshotz/YYYY-MM-DDTHHMMSS+hhmm` i.e. a valid isodate 16 | * Operates on specific subvols, all subvols on a filesystem, or all 17 | subvols on all mounted filesystems, Eg: `snazzer --all` 18 | * Operations include snapshotting (default), `--measure` (sha512sum & 19 | PGP signatures of snapshots), `--prune` (deleting snapshots except for 20 | those required to meet configured number of 21 | hourlies/daylies/monthlies/yearlies to keep) 22 | * `snazzer-receive` operates on remote hosts for specific subvols, all 23 | subvols on a filesystem, or all subvols on all mounted filesystems, 24 | Eg: `snazzer-receive somehost --all` (or `snazzer-receive -- --all` to 25 | receive local paths without ssh in the middle) 26 | * Automated regression testing (TODO: snazzer-receive) 27 | 28 | Getting started 29 | --------------- 30 | 31 | ### Documentation 32 | 33 | The full documentation for each part of snazzer is available as follows: 34 | 35 | snazzer --man # Create, prune and measure snapshots 36 | snazzer-receive --man # Receive remote snapshots over ssh 37 | 38 | Supporting scripts are also fully documented: 39 | 40 | snazzer-measure --man # Support script, used by snazzer 41 | snazzer-send-wrapper --man # Support script, snazzer-receive ssh wrapper 42 | snazzer-prune-candidates --man # Support script, used by snazzer[-receive] 43 | 44 | These man pages are also available at: 45 | * [docs/snazzer.md](docs/snazzer.md) 46 | * [docs/snazzer-receive.md](docs/snazzer-receive.md) 47 | * [docs/snazzer-measure.md](docs/snazzer-measure.md) 48 | * [docs/snazzer-send-wrapper.md](docs/snazzer-send-wrapper.md) 49 | * [docs/snazzer-prune-candidates.md](docs/snazzer-prune-candidates.md) 50 | 51 | ### Snapshotting and pruning 52 | 53 | Snapshots are maintained in a directory under `.snapshotz` of the root of each btrfs subvolume. Snapshots are named as valid isodates in the form of `YYYY-MM-DDTHHMMSS+hhmm` (or `YYYY-MM-DDTHHMMSSZ` if `SNAZZER_USE_UTC` is set) under this directory. Here's the output of `sudo tree -a /mnt/home` after two snapshots have been created and measured: 54 | 55 | /mnt/home 56 | ├── home_junk 57 | └── .snapshotz 58 | ├── 2015-04-16T115421+1000 59 | │   ├── home_junk 60 | │   ├── .snapshot_measurements.exclude 61 | │   └── .snapshotz 62 | ├── 2015-04-16T160810+1000 63 | │   ├── home_junk 64 | │   ├── .snapshot_measurements.exclude 65 | │   └── .snapshotz 66 | │   ├── 2015-04-16T115421+1000 67 | │   └── .measurements 68 | │   └── 2015-04-16T115421+1000 69 | └── .measurements 70 | ├── 2015-04-16T115421+1000 71 | └── 2015-04-16T160810+1000 72 | 73 | Example usage to snapshot all subvolumes in the btrfs filesystem mounted under `/mnt`: 74 | 75 | snazzer --all /mnt 76 | snazzer --all /mnt # create another snapshot 77 | snazzer --all /mnt # create another snapshot 78 | # have unneeded snapshots now, prune them: 79 | snazzer --prune --force --all /mnt 80 | 81 | ### Measuring snapshots 82 | 83 | `snazzer` offers a way to generate reproducible measurements for snapshots under its management. These measurements are reports generated by `snazzer-measure` and they include `du -bs`, `sha512sum` and `gpg2` signatures. These measurements may be performed on the original host, or any other machines receiving and handling snapshots along the way (Eg. via `snazzer-receive`). `snazzer` appends the output of `snazzer-measure` to text files in `.snapshotz/.measurements` with the same names as the snapshots they have measured under `.snapshotz`, so for example a snapshot at `/mnt/home/.snapshotz/2015-04-16T115421+1000` will have measurement results appended to `/mnt/home/.snapshotz/.measurements/2015-04-16T115421+1000`. 84 | 85 | This example will generate measurements for all snapshots of all subvolumes under the btrfs filesystem mounted at `/mnt`: 86 | 87 | snazzer --measure --all /mnt 88 | 89 | Here's an example measurement result found at `/mnt/home/.snapshotz/.measurements/2015-04-16T115421+1000` (example only). Note that the commands listed to reproduce the results (lines beginning and ending with parentheses) should work consistently regardless of whether the snapshot directory is on a btrfs filesystem or not: 90 | 91 | ################################################################################ 92 | > on host1 at 2015-04-16T155828+1000, du bytes: 93 | (du -bs --one-file-system --exclude-from '../2015-04-16T115421+1000/.snapshot_measurements.exclude' '../2015-04-16T115421+1000') 94 | 512098 /mnt/home/.snapshotz/2015-04-16T115421+1000 95 | 96 | > on host1 at 2015-04-16T155828+1000, sha512sum: 97 | (find '../2015-04-16T115421+1000' -xdev -not -path '../2015-04-16T115421+1000' -printf '%P\0' | LC_ALL=C sort -z | tar --no-recursion --one-file-system --preserve-permissions --numeric-owner --null --create --to-stdout --directory '../2015-04-16T115421+1000' --files-from - --exclude-from '../2015-04-16T115421+1000/.snapshot_measurements.exclude' | sha512sum -b) 98 | c5626e1e6036d317ac98e5ed185b9c5520e4eba67becd250fc1b6fc94574cbc483b9ca677b1f69e8691d0ad4cb17c9b07f0084271b8e11e95915fadb6ced473c *- 99 | > on host1 at 2015-04-16T155829+1000, gpg: 100 | (SIG=$(mktemp) && grep -v '/,/' '2015-04-16T115421+1000' | sed -n '/> on host1 at 2015-04-16T155829+1000, gpg:/,/-----END PGP SIGNATURE-----/ { /-----BEGIN PGP SIGNATURE-----/{x;d}; H }; ${x;p}' >"$SIG" && find '../2015-04-16T115421+1000' -xdev -not -path '../2015-04-16T115421+1000' -printf '%P\0' | LC_ALL=C sort -z | tar --no-recursion --one-file-system --preserve-permissions --numeric-owner --null --create --to-stdout --directory '../2015-04-16T115421+1000' --files-from - --exclude-from '../2015-04-16T115421+1000/.snapshot_measurements.exclude' | gpg2 --verify "$SIG" - && rm "$SIG") 101 | -----BEGIN PGP SIGNATURE----- 102 | Version: GnuPG v2 103 | 104 | -----END PGP SIGNATURE----- 105 | 106 | > on host1 at 2015-04-16T155840+1000, tar info: 107 | tar (GNU tar) 1.27.1 --format=gnu -f- -b20 --quoting-style=escape --rmt-command=/usr/lib/tar/rmt --rsh-command=/usr/bin/rsh 108 | 109 | Now observe that running the same command again, `snazzer` is smart enough to skip re-measuring snapshots which have already been measured by this host (use --force to override this behaviour): 110 | 111 | snazzer --measure --all /mnt 112 | 113 | Some observations: 114 | * Yes, the verification commands are huge and ugly, but eminently reproducible. 115 | * Each snapshot root contains a carefully maintained list of subvolumes which 116 | existed under it at the time of the snapshot in a file named 117 | `.snapshot_measurments.exclude`. This is to work around a btrfs bug which 118 | means certain empty directories within snapshots have bogus atimes, see 119 | https://bugzilla.kernel.org/show_bug.cgi?id=95201 120 | 121 | ### Receive snapshots from a remote system 122 | 123 | Receive all missing `snazzer` managed btrfs snapshots, along with any measurement files they may have, from the host `host1` via ssh to the current working directory: 124 | 125 | cd /media/backup-drive/hosts/host1 126 | snazzer-receive host1 --all 127 | 128 | The example above assumes a valid working ssh configuration and properly configured `/etc/sudoers` on `host1`. Refer to `snazzer-receive --man` for configuration hints. 129 | 130 | ### Receive snapshots from a local filesystem to local backup media 131 | 132 | Receive all missing `snazzer` managed btrfs snapshots on the local system, along with any measurement files they may have, into btrfs subvolumes maintained under the current working directory: 133 | 134 | cd /media/backup-drive/hosts/host1 135 | snazzer-receive -- --all 136 | 137 | Inspiration 138 | ----------- 139 | Most mature backup solutions do not leverage btrfs features, particularly 140 | copy-on-write snapshots or send/receive transport. This makes it too easy to end 141 | up with VMs needlessly struggling with disk I/O throughput for hours per day 142 | when a btrfs snapshot and send/receive operation would take minutes or even 143 | seconds. 144 | 145 | SuSE's `snapper` project was interesting enough to provide inspiration for the 146 | naming of `snazzer`, but seems focused on supporting recovery from sysadmin 147 | tasks and thus complements rather than provides a coherent basis for a 148 | distributed backup solution. Additionally, whilst SuSE's `snapper` has few 149 | dependencies we thought it would be possible to provide something using exactly 150 | zero dependencies beyond only very basic core utilities present on even minimal 151 | installation of any given distro. 152 | 153 | Immediate goals and assumptions 154 | ------------------------------- 155 | * Leverage btrfs (and eventually zfs?) snapshots, send/receive features as the 156 | basis for _one part_ an efficient backup system. 157 | * Provide easily reproducible sha512sum, GPG signatures etc. of snapshots to 158 | detect any btrfs shenanigans or malicious tampering. 159 | * Zero config, or at least issue helpful _easily actionable_ error messages and 160 | sanity checks along the way. 161 | * Zero dependencies, or as close as we can get. `snazzer-prune-candidates` uses 162 | perl, a core part of some distros but not others; python version coming soon. 163 | * Simple architecture without databases, XML config or daemons. 164 | 165 | Longer-term goals 166 | ----------------- 167 | * Seamlessly support ZFS On Linux instead of or in addition to btrfs 168 | * Implement `snazzer-prune-candidates` in a python version for those distros 169 | which have standardized on python rather than perl as part of base packages 170 | * Distro packaging, starting with Debian. Lots of debconf to help alleviate 171 | `snazzer-receive` config tedium. 172 | * Automated distro testing infrastructure 173 | * Remove any lingering GNU-isms and keep POSIX sh code portable to BSDs for 174 | FreeBSD and OpenIndiana compatibility (assuming `snazzer` makes sense there) 175 | 176 | License and Copyright 177 | --------------------- 178 | 179 | Copyright (C) 2015-2016, Snazzer Authors All rights reserved. Snazzer Authors 180 | are listed in the AUTHORS.md file in the root of this distribution. 181 | NOTE: Please extend that file, not this notice. 182 | 183 | This project uses the 2-clause Simplified BSD License. 184 | -------------------------------------------------------------------------------- /docs/snazzer-receive.md: -------------------------------------------------------------------------------- 1 | # NAME 2 | 3 | snazzer-receive - receive remote snazzer snapshots to current working dir 4 | 5 | # SYNOPSIS 6 | 7 | Receive snapshots from remote host via ssh: 8 | 9 | snazzer-receive [--dry-run] host --all [/path/to/btrfs/mountpoint] 10 | 11 | snazzer-receive [--dry-run] host [/remote/subvol1 [/subvol2 [..]]] 12 | 13 | Receive snapshots from local filesystem: 14 | 15 | snazzer-receive [--dry-run] -- --all [/path/to/btrfs/mountpoint] 16 | 17 | snazzer-receive [--dry-run] -- /local/subvol1 [/subvol2 [..]] 18 | 19 | # DESCRIPTION 20 | 21 | First, **snazzer-receive** obtains a list of snapshots to be received by running 22 | `snazzer --list-snapshots [args]`, where \[args\] are all **snazzer-receive** 23 | arguments after the hostname or `--` separator argument. 24 | 25 | If the first non-option positional argument is `--`, 26 | `snazzer --list-snapshots [args]` is executed locally and \[args\] will refer to 27 | local filesystem paths. Otherwise, it is taken to mean an ssh hostname which is 28 | used to run the `snazzer --list-snapshots [args]` command remotely, and \[args\] 29 | will refer to paths on that remote host. 30 | 31 | **snazzer-receive** then iterates through this list of snapshots recreating a 32 | filesystem similar to the source by creating subvolumes and `.snapshotz` 33 | directories where necessary. Missing snapshots are instantiated directly with 34 | `btrfs send` and `btrfs receive`, using `btrfs send -p [parent]` where 35 | possible to reduce transport overhead of incremental snapshots. 36 | 37 | **snazzer-receive** never deletes any snapshots from the current working dir, 38 | even if the snapshots were e.g. pruned from the source. 39 | This can be used to build a flexible, multi-tiered backup strategy with 40 | different settings of how many snapshots to keep, but means that 41 | `snazzer --prune` needs to be run on the current working directory as well for 42 | old snapshots to be deleted. 43 | 44 | Rather than offer ssh user/port/host specifications through **snazzer-receive**, 45 | it is assumed all remote hosts are properly configured through your ssh config 46 | file usually at `$HOME/.ssh/config`. 47 | 48 | # OPTIONS 49 | 50 | - **--dry-run**: print rather than execute commands that would be run 51 | - **--help**: Brief help message 52 | - **--version**: Print version 53 | - **--man**: Full documentation 54 | - **--man-roff**: Full documentation as \*roff output, Eg: 55 | 56 | snazzer-receive --man-roff | nroff -man 57 | 58 | - **--man-markdown**: Full documentation as markdown output, Eg: 59 | 60 | snazzer-receive --man-markdown > snazzer-manpage.md 61 | 62 | # ENVIRONMENT 63 | 64 | ## sudo requirements for sender/remote hosts 65 | 66 | **snazzer-receive** assumes the ssh user (or local user, if receiving a local 67 | filesystem) which will be running `btrfs send` (among other things) has 68 | passwordless sudo for the commands it needs to run. Only a few commands are 69 | necessary, the following lines in `/etc/sudoers` or `/etc/sudoers.d/snazzer` 70 | should suffice (replace "sendinguser" with the actual username you will use): 71 | 72 | sendinguser ALL=(root:nobody) NOPASSWD: /usr/bin/snazzer --list-snapshots * 73 | sendinguser ALL=(root:nobody) NOPASSWD:NOEXEC: \ 74 | /bin/grep -srl */.snapshotz/.measurements/, \ 75 | /sbin/btrfs send */.snapshotz/*, \ 76 | /bin/cat */.snapshotz/.measurements/* 77 | 78 | ## sudo and cron user requirements for receiving hosts 79 | 80 | For interactive use of **snazzer-receive**, a typical user with full sudo 81 | permissions should work out of the box. 82 | 83 | For scripted use such as a cron job, or interactive use in more restrictive 84 | environments - running ssh as the root user is generally considered a bad idea. 85 | A dedicated non-root user will require at minimum the following lines in 86 | `/etc/sudoers` or `/etc/sudoers.d/snazzer` (replace "receiveruser" with the 87 | actual username your cron job will use, and remove `NOPASSWD:` if this is for 88 | an interactive/shell user): 89 | 90 | receiveruser ALL=(root:nobody) NOPASSWD:NOEXEC: \ 91 | /usr/bin/test -e */.snapshotz*, \ 92 | /sbin/btrfs subvolume show *, \ 93 | /bin/ls */.snapshotz, \ 94 | /bin/grep -srL */.snapshotz/.measurements/, \ 95 | /bin/mkdir --mode=0755 */.snapshotz, \ 96 | /bin/mkdir --mode=0755 */.snapshotz/.measurements, \ 97 | /bin/mkdir --mode=0755 */.snapshotz/.incomplete, \ 98 | /sbin/btrfs receive */.snapshotz/.incomplete, \ 99 | /sbin/btrfs subvolume create *, \ 100 | /sbin/btrfs subvolume snapshot -r */.snapshotz/.incomplete/* */.snapshotz/,\ 101 | /sbin/btrfs subvolume delete */.snapshotz/.incomplete/*, \ 102 | /bin/rmdir */.snapshotz/.incomplete, \ 103 | /bin/mkdir -vp *, \ 104 | /bin/mkdir --mode=0755 -vp */.snapshotz, \ 105 | /usr/bin/tee -a */.snapshotz/.measurements/* 106 | 107 | # SECURITY CONSIDERATIONS 108 | 109 | ## Remote hosts 110 | 111 | **snazzer-receive** relies on running ssh remote commands. It is agnostic about 112 | the auth method used, but this documentation assumes key-based. 113 | 114 | Combined with passwordless sudo, remote hosts are vulnerable to and must have 115 | absolute trust in the ssh key-holder, user and host running **snazzer-receive**. 116 | 117 | Your deployment should include or consider the following steps, among others not 118 | listed here, to attempt to reduce the impact of or slow down an attacker which 119 | has gained control of the **snazzer-receive** user accounts or ssh keys: 120 | 121 | - Protect ssh keys 122 | 123 | The ssh key used to authenticate **snazzer-receive** typically has passwordless 124 | sudo for `btrfs send` (among other things) and you should assume that whomever 125 | wields it has access to everything: 126 | 127 | - Avoid passphraseless ssh keyfiles 128 | 129 | This should be obvious: once an attacker has copied such a keyfile they no 130 | longer need the compromised host to authenticate, and you will have a bigger, 131 | more urgent job searching for malicious use (and key removal from machines 132 | which trusted it). 133 | 134 | - Avoid ssh private keyfiles 135 | 136 | Even passphrase-protected keyfiles are vulnerable to keyloggers and 137 | memory scraping. Consider using smartcards, TPMs, Yubikeys or GoldKeys etc. to 138 | at least force an attacker to depend on whichever machine has the authentication 139 | device attached. 140 | 141 | This is especially important when passphrase-protected keyfiles are not 142 | practical (eg. scripted use of **snazzer-receive** such as cron). 143 | 144 | - Use the timeout option if using an ssh-agent 145 | 146 | - Grant minimal sudo rights 147 | 148 | Refer to "sudo requirements for remote hosts". Don't give the **snazzer-receive** 149 | user the option to run arbitrary commands remotely as root. 150 | 151 | - `~/.ssh/authorized_keys`: specify a forced-command/shell-wrapper 152 | 153 | Even if sudo is locked down, don't give the **snazzer-receive** user the option 154 | of running arbitrary commands remotely. Use a shell wrapper which permits only 155 | the required sudo commands. 156 | TODO: provide example 157 | TODO: Document shell wrapper 158 | 159 | NOTE: This does not prevent data exfiltration via `sudo btrfs send`, but 160 | may slow down an attacker who would abuse the account in other ways. 161 | 162 | - `~/.ssh/authorized_keys`: restrict originating IP address 163 | 164 | Use the `from` option to limit which machine the **snazzer-receive** host's ssh 165 | key may connect from. This might force an attacker to still depend on the 166 | **snazzer-receive** host even if they have obtained the private key somehow. 167 | TODO: provide example 168 | TODO: link to a guide on this 169 | 170 | - Disable interactive shells/logins 171 | 172 | Reduce opportunities for the **snazzer-receive** user to run arbitrary commands; 173 | remove the account password. NOTE: this doesn't stop ssh remote commands. 174 | TODO: link to a guide on this 175 | 176 | - Log remote ssh commands 177 | 178 | Most distros do zero logging of remote ssh commands. Logging such commands may 179 | be your only way to spot abuse of the **snazzer-receive** account. The 180 | `snazzer-send-wrapper` uses `logger -p user.info [cmd]` to log commands on 181 | remote hosts which are invoking `btrfs send`. 182 | TODO: link to a guide on this 183 | 184 | # BUGS AND LIMITATIONS 185 | 186 | **NOTE:** **snazzer-receive** tries to recreate a filesystem similar to that of 187 | the remote host, starting at the current working directory which represents the 188 | root filesystem. If the remote host has a root btrfs filesystem, this means that 189 | the current working directory should itself also be a btrfs subvolume in order 190 | to receive snapshots under ./.snapshotz. However, **snazzer-receive** will be 191 | unable to replace the current working directory with a btrfs subvolume if it 192 | isn't already one. 193 | 194 | Therefore, if required, ensure the current working directory is already a btrfs 195 | subvolume prior to running **snazzer-receive** if you need to receive btrfs root. 196 | 197 | # EXIT STATUS 198 | 199 | **snazzer-receive** will abort with an error message printed to STDERR and 200 | non-zero exit status under the following conditions: 201 | 202 | - 1. invalid arguments 203 | - 2. `.snapshotz/.incomplete` already exists at a given destination subvolume 204 | - 9. tried to display man page with a formatter which is not installed 205 | - 12. remote ssh sudo command failed 206 | 207 | # TODO 208 | 209 | - 1. improve fetch/append of remote host's measurements 210 | 211 | **snazzer-receive** currently does some clumsy concatenation of the remote host's 212 | measurement file onto the local measurement file for a given snapshot if the 213 | local measurement file is either missing or does not mention that remote host's 214 | hostname. Whilst this supports the simple use-case of wanting to obtain initial 215 | measurements performed on a remote host, once a remote host's measurements have 216 | been appended there is no attempt to append any further measurement results onto 217 | the local measurements file. If this bothers you, please report detailed 218 | use-cases to the author (patches welcome). 219 | 220 | - 2. include restricted wrapper script to be used as ssh forced command 221 | 222 | The snazzer project assumes that systems administrators would prefer to restrict 223 | the possible exposure of a dedicated snazzer remote user account, even if sudo 224 | is locked down. To that end, a wrapper script shall be provided which restricts 225 | possible ssh remote commands to only the few actually necessary for snazzer 226 | operation. 227 | 228 | Even so, commands which snazzer relies on such as `sudo btrfs send` are 229 | extremely dangerous no matter if it's the only command allowed by the system - 230 | securing ssh keys is of utmost importance; consider protecting ssh keys with 231 | smartcards, TPM, hardware OTP solution such as Yubi/GoldKeys etc. 232 | 233 | # SEE ALSO 234 | 235 | snazzer, snazzer-measure, snazzer-prune-candidates 236 | 237 | # AUTHOR 238 | 239 | Snazzer Authors are listed in the AUTHORS.md file in the root of this 240 | distribution. See https://github.com/csirac2/snazzer for more information. 241 | NOTE: Please extend that file, not this notice. 242 | 243 | # LICENSE AND COPYRIGHT 244 | 245 | Copyright (C) 2015-2016, Snazzer Authors All rights reserved. 246 | 247 | Redistribution and use in source and binary forms, with or without 248 | modification, are permitted provided that the following conditions are met: 249 | 250 | 1\. Redistributions of source code must retain the above copyright notice, this 251 | list of conditions and the following disclaimer. 252 | 253 | 2\. Redistributions in binary form must reproduce the above copyright notice, 254 | this list of conditions and the following disclaimer in the documentation 255 | and/or other materials provided with the distribution. 256 | 257 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 258 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 259 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 260 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 261 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 262 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 263 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 264 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 265 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 266 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 267 | -------------------------------------------------------------------------------- /tests/snazzer-send-wrapper.bats: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bats 2 | # vi:syntax=sh 3 | 4 | load "$BATS_TEST_DIRNAME/fixtures.sh" 5 | 6 | setup() { 7 | export PATH=$BATS_TMPDIR/snazzer-tests/bin:$PATH 8 | mkdir -p "$BATS_TMPDIR/snazzer-tests/bin" 9 | cp "$BATS_TEST_DIRNAME/../snazzer-send-wrapper" "$BATS_TMPDIR/snazzer-tests/bin/" 10 | chmod a+x "$BATS_TMPDIR/snazzer-tests/bin/snazzer-send-wrapper" 11 | sed -i 's/^\(export PATH=.*\)/#\1 # disabled for tests/g' \ 12 | "$BATS_TMPDIR/snazzer-tests/bin/snazzer-send-wrapper" 13 | cp "$BATS_TEST_DIRNAME/data/sudo" "$BATS_TMPDIR/snazzer-tests/bin/" 14 | } 15 | 16 | @test "snazzer-send-wrapper in PATH" { 17 | readlink -f "$BATS_TMPDIR/snazzer-tests/bin/snazzer-send-wrapper" \ 18 | > $(expected_file) 19 | readlink -f $(which snazzer-send-wrapper) > $(actual_file) 20 | diff -u $(expected_file) $(actual_file) 21 | } 22 | 23 | @test "snazzer-send-wrapper --version" { 24 | run snazzer-send-wrapper --version 25 | git_describe_snazzer_version > $(expected_file) 26 | echo "$output" > $(actual_file) 27 | diff -u $(expected_file) $(actual_file) 28 | [ "$status" = "0" ] 29 | } 30 | 31 | @test "snazzer-send-wrapper" { 32 | run snazzer-send-wrapper 33 | [ "$status" -eq "1" ] 34 | } 35 | 36 | @test "sudo -n snazzer --list-snapshots '--all'" { 37 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 38 | echo "4" > $(expected_file) 39 | echo "$output" > $(actual_file) 40 | diff -u $(expected_file) $(actual_file) 41 | [ "$status" = "0" ] 42 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=ls_args run snazzer-send-wrapper 43 | echo "-n 44 | snazzer 45 | --list-snapshots 46 | --all" > $(expected_file b) 47 | echo "$output" > $(actual_file b) 48 | diff -u $(expected_file b) $(actual_file b) 49 | [ "$status" = "0" ] 50 | } 51 | 52 | @test "sudo -n snazzer --list-snapshots '--all' '--force'" { 53 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 54 | [ "$status" = "98" ] 55 | } 56 | 57 | @test "sudo -n snazzer --list-snapshots '--force'" { 58 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 59 | [ "$status" = "98" ] 60 | } 61 | 62 | @test "sudo -n snazzer --list-snapshots '--all' 'foo=\" some stuff \"' 'hel'\\\\'' squot '\\\\''lo' 'asd \" dquot \" fgh' 'ap ple' ' bon'\\\\''squot'\\\\''jour' 'there'" { 63 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 64 | echo "10" > $(expected_file) 65 | echo "$output" > $(actual_file) 66 | diff -u $(expected_file) $(actual_file) 67 | [ "$status" = "0" ] 68 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=ls_args run snazzer-send-wrapper 69 | echo "-n 70 | snazzer 71 | --list-snapshots 72 | --all 73 | foo=\" some stuff \" 74 | hel' squot 'lo 75 | asd \" dquot \" fgh 76 | ap ple 77 | bon'squot'jour 78 | there" > $(expected_file b) 79 | echo "$output" > $(actual_file b) 80 | diff -u $(expected_file b) $(actual_file b) 81 | [ "$status" = "0" ] 82 | } 83 | 84 | # At some point we decided to error when args are switches, hence ^-prefix 85 | @test "sudo -n snazzer --list-snapshots 'bla' '^--bar' '^--cat=\" someone'\''s dog \"' '^--foo='\''a \"b\" c'\'''" { 86 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 87 | echo "7" > $(expected_file) 88 | echo "$output" > $(actual_file) 89 | diff -u $(expected_file) $(actual_file) 90 | [ "$status" = "0" ] 91 | } 92 | 93 | @test "sudo -n snazzer --list-snapshots 'unbalance'd squote'" { 94 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=ls_args run snazzer-send-wrapper 95 | [ "$status" = "99" ] 96 | } 97 | 98 | @test "sudo -n snazzer --list-snapshots 'unbalance\"d dquote'" { 99 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=ls_args run snazzer-send-wrapper 100 | [ "$status" = "0" ] 101 | } 102 | 103 | @test "sudo -n btrfs send" { 104 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 105 | [ "$status" = "2" ] 106 | } 107 | 108 | @test "sudo -n btrfs send '-/subvol/.snapshotz/FOO'" { 109 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 110 | [ "$status" = "98" ] 111 | } 112 | 113 | @test "sudo -n btrfs send '-p' '-/subvol/.snapshotz/FOO1' '/subvol/.snapshotz/FOO2'" { 114 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 115 | [ "$status" = "98" ] 116 | } 117 | 118 | @test "sudo -n btrfs send '-p' '/subvol/.snapshotz/FOO2' '/subvol/.snapshotz/FOO1' '/subvol/.snapshotz/FOO3'" { 119 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 120 | [ "$status" = "98" ] 121 | } 122 | 123 | @test "sudo -n btrfs send '/subvol/.snapshotz/FOO1' '/subvol/.snapshotz/FOO2'" { 124 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 125 | [ "$status" = "98" ] 126 | } 127 | 128 | @test "sudo -n btrfs send '/subvol/.snapshotz/FOO'" { 129 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 130 | echo "4" > $(expected_file) 131 | echo "$output" > $(actual_file) 132 | diff -u $(expected_file) $(actual_file) 133 | [ "$status" = "0" ] 134 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=ls_args run snazzer-send-wrapper 135 | echo "-n 136 | btrfs 137 | send 138 | /subvol/.snapshotz/FOO" > $(expected_file b) 139 | echo "$output" > $(actual_file b) 140 | diff -u $(expected_file b) $(actual_file b) 141 | [ "$status" = "0" ] 142 | } 143 | 144 | @test "sudo -n btrfs send '/echo \`ls \"/\"; ls /;\`; ~!@#\$(ls)%^&*()_+-='\\''[]'\\''{}|:<>,./?/.snapshotz/FOO2'" { 145 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 146 | echo "4" > $(expected_file) 147 | echo "$output" > $(actual_file) 148 | diff -u $(expected_file) $(actual_file) 149 | [ "$status" = "0" ] 150 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=ls_args run snazzer-send-wrapper 151 | echo "-n 152 | btrfs 153 | send 154 | /echo \`ls \"/\"; ls /;\`; ~!@#\$(ls)%^&*()_+-='[]'{}|:<>,./?/.snapshotz/FOO2" \ 155 | > $(expected_file b) 156 | echo "$output" > $(actual_file b) 157 | diff -u $(expected_file b) $(actual_file b) 158 | [ "$status" = "0" ] 159 | } 160 | 161 | @test "sudo -n btrfs send -p" { 162 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 163 | [ "$status" = "2" ] 164 | } 165 | 166 | @test "sudo -n btrfs send '-p' '/subvol/.snapshotz/FOO2'" { 167 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 168 | [ "$status" = "98" ] 169 | } 170 | 171 | @test "sudo -n btrfs send '-p' '/subvol/.snapshotz/FOO1' '/subvol/.snapshotz/FOO2'" { 172 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 173 | echo "6" > $(expected_file) 174 | echo "$output" > $(actual_file) 175 | diff -u $(expected_file) $(actual_file) 176 | [ "$status" = "0" ] 177 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=ls_args run snazzer-send-wrapper 178 | echo "-n 179 | btrfs 180 | send 181 | -p 182 | /subvol/.snapshotz/FOO1 183 | /subvol/.snapshotz/FOO2" > $(expected_file b) 184 | echo "$output" > $(actual_file b) 185 | diff -u $(expected_file b) $(actual_file b) 186 | [ "$status" = "0" ] 187 | } 188 | 189 | @test "sudo -n btrfs send -p '/subvol/.snapshotz/FOO1' '/subvol/.snapshotz/FOO2'" { 190 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 191 | [ "$status" = "2" ] 192 | } 193 | 194 | @test "sudo -n btrfs send '-p' '/echo \`ls \"/\"; ls /;\`; ~!@#\$(ls)%^&*()_+-='\\''[]'\\''{}|:<>,./?/.snapshotz/FOO1' '/echo \`ls \"/\"; ls /;\`; ~!@#\$(ls)%^&*()_+-='\\''[]'\\''{}|:<>,./?/.snapshotz/FOO2'" { 195 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 196 | echo "6" > $(expected_file) 197 | echo "$output" > $(actual_file) 198 | diff -u $(expected_file) $(actual_file) 199 | [ "$status" = "0" ] 200 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=ls_args run snazzer-send-wrapper 201 | echo "-n 202 | btrfs 203 | send 204 | -p 205 | /echo \`ls \"/\"; ls /;\`; ~!@#\$(ls)%^&*()_+-='[]'{}|:<>,./?/.snapshotz/FOO1 206 | /echo \`ls \"/\"; ls /;\`; ~!@#\$(ls)%^&*()_+-='[]'{}|:<>,./?/.snapshotz/FOO2" \ 207 | > $(expected_file b) 208 | echo "$output" > $(actual_file b) 209 | diff -u $(expected_file b) $(actual_file b) 210 | [ "$status" = "0" ] 211 | } 212 | 213 | @test "sudo -n btrfs send '/subvol/.snapshotz/F'\\''OO'" { 214 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 215 | echo "4" > $(expected_file) 216 | echo "$output" > $(actual_file) 217 | diff -u $(expected_file) $(actual_file) 218 | [ "$status" = "0" ] 219 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=ls_args run snazzer-send-wrapper 220 | echo "-n 221 | btrfs 222 | send 223 | /subvol/.snapshotz/F'OO" > $(expected_file b) 224 | echo "$output" > $(actual_file b) 225 | diff -u $(expected_file b) $(actual_file b) 226 | [ "$status" = "0" ] 227 | } 228 | 229 | @test "sudo -n btrfs send '/subvol/.snapshotz/F'OO'" { 230 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 231 | [ "$status" = "99" ] 232 | } 233 | 234 | # snapshots != snapshotz 235 | @test "sudo -n btrfs send '/subvol/.snapshots/FOO'" { 236 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 237 | [ "$status" = "2" ] 238 | } 239 | 240 | # snapshots != snapshotz 241 | @test "sudo -n btrfs send '-p' '/subvol/.snapshots/FOO1' '/subvol/.snapshotz/FOO2'" { 242 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 243 | [ "$status" = "98" ] 244 | } 245 | 246 | @test "sudo -n grep -srl '^> on foo1-host at ' '/subvol/.snapshotz/.measurements/'" { 247 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 248 | echo "5" > $(expected_file) 249 | echo "$output" > $(actual_file) 250 | diff -u $(expected_file) $(actual_file) 251 | [ "$status" = "0" ] 252 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=ls_args run snazzer-send-wrapper 253 | echo "-n 254 | grep 255 | -srl 256 | ^> on foo1-host at 257 | /subvol/.snapshotz/.measurements/" > $(expected_file b) 258 | echo "$output" > $(actual_file b) 259 | diff -u $(expected_file b) $(actual_file b) 260 | [ "$status" = "0" ] 261 | } 262 | 263 | @test "sudo -n grep -srl '^> on foo1-host at ' '/sub'\\''vol/.snapshotz/.measurements/'" { 264 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 265 | echo "5" > $(expected_file) 266 | echo "$output" > $(actual_file) 267 | diff -u $(expected_file) $(actual_file) 268 | [ "$status" = "0" ] 269 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=ls_args run snazzer-send-wrapper 270 | echo "-n 271 | grep 272 | -srl 273 | ^> on foo1-host at 274 | /sub'vol/.snapshotz/.measurements/" > $(expected_file b) 275 | echo "$output" > $(actual_file b) 276 | diff -u $(expected_file b) $(actual_file b) 277 | [ "$status" = "0" ] 278 | } 279 | 280 | @test "sudo -n grep -srl '^> on foo1-host at ' '/subvol/.snapshotz/.measurements/junk'" { 281 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 282 | [ "$status" = "2" ] 283 | } 284 | 285 | @test "sudo -n grep -srl '-^> on foo1-host at ' '/subvol/.snapshotz/.measurements/'" { 286 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 287 | [ "$status" = "2" ] 288 | } 289 | 290 | @test "sudo -n grep -srl '^> on foo1-host at ' '-/subvol/.snapshotz/.measurements/'" { 291 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 292 | [ "$status" = "98" ] 293 | } 294 | 295 | @test "sudo -n grep -srl '^> on foo1-host at ' '/subvol1/.snapshotz/.measurements/' '/subvol2/.snapshotz/.measurements/'" { 296 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 297 | [ "$status" = "98" ] 298 | } 299 | 300 | # snapshots!=snapshotz 301 | @test "sudo -n grep -srl '^> on foo1-host at ' '/subvol/.snapshots/.measurements/'" { 302 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 303 | [ "$status" = "2" ] 304 | } 305 | 306 | @test "sudo -n cat '/subvol/.snapshotz/.measurements/FOO'" { 307 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 308 | echo "3" > $(expected_file) 309 | echo "$output" > $(actual_file) 310 | diff -u $(expected_file) $(actual_file) 311 | [ "$status" = "0" ] 312 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=ls_args run snazzer-send-wrapper 313 | echo "-n 314 | cat 315 | /subvol/.snapshotz/.measurements/FOO" > $(expected_file b) 316 | echo "$output" > $(actual_file b) 317 | diff -u $(expected_file b) $(actual_file b) 318 | [ "$status" = "0" ] 319 | } 320 | 321 | @test "sudo -n cat '/sub'\\''vol/.snapshotz/.measurements/FOO'" { 322 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 323 | echo "3" > $(expected_file) 324 | echo "$output" > $(actual_file) 325 | diff -u $(expected_file) $(actual_file) 326 | [ "$status" = "0" ] 327 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=ls_args run snazzer-send-wrapper 328 | echo "-n 329 | cat 330 | /sub'vol/.snapshotz/.measurements/FOO" > $(expected_file b) 331 | echo "$output" > $(actual_file b) 332 | diff -u $(expected_file b) $(actual_file b) 333 | [ "$status" = "0" ] 334 | } 335 | 336 | @test "sudo -n cat" { 337 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 338 | [ "$status" = "2" ] 339 | } 340 | 341 | @test "sudo -n cat '-/subvol/.snapshotz/.measurements/FOO'" { 342 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 343 | [ "$status" = "98" ] 344 | } 345 | 346 | # .snapshots!=.snapshotz 347 | @test "sudo -n cat '/subvol/.snapshots/.measurements/FOO'" { 348 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 349 | [ "$status" = "2" ] 350 | } 351 | 352 | @test "sudo -n cat '/subvol/.snapshotz/.measurements/FOO1' '/subvol/.snapshotz/.measurements/FOO2'" { 353 | SSH_ORIGINAL_COMMAND="$BATS_TEST_DESCRIPTION" F=no_args run snazzer-send-wrapper 354 | [ "$status" = "98" ] 355 | } 356 | 357 | teardown() { 358 | rm "$BATS_TMPDIR/snazzer-tests/bin/sudo" 359 | rm "$BATS_TMPDIR/snazzer-tests/bin/snazzer-send-wrapper" 360 | rmdir "$BATS_TMPDIR/snazzer-tests/bin" 361 | } 362 | -------------------------------------------------------------------------------- /snazzer-send-wrapper: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | set -e 3 | SNAZZER_VERSION=0.0.3 4 | export PATH=/bin:/usr/bin:/sbin:/usr/local/bin 5 | 6 | log_cmd() { 7 | MSG="$0 (for $SSH_CLIENT) running: $SSH_ORIGINAL_COMMAND" 8 | logger -p user.info "$MSG" 9 | } 10 | 11 | log_cmd_and_exit() { 12 | MSG="$0 (for $SSH_CLIENT) REJECTED: $SSH_ORIGINAL_COMMAND" 13 | if [ -n "$2" ]; 14 | then MSG="$MSG ### REASON: $2" 15 | fi 16 | logger -p user.error "$MSG" 17 | echo "$MSG" >&2 18 | if [ -n "$1" ] && [ "$1" != "0" ]; then 19 | exit "$1" 20 | else 21 | exit 1 22 | fi 23 | } 24 | 25 | squote_args() { 26 | while [ -n "$1" ]; do 27 | printf "'"; printf '%s' "$1" | sed "s|'|'\\\\''|g"; printf "' " 28 | shift 29 | done 30 | } 31 | 32 | # The mission: unpack an $SSH_ORIGINAL_COMMAND-like variable with an arbitrary 33 | # number arguments to our command of interest into "$@" so that we can call our 34 | # real command safely. In other words, instead of something like: 35 | # foo $DANGEROUS_BAREWORD_STRING 36 | # we want to do instead: 37 | # foo "$@" 38 | # 39 | # ASSUMPTION: All arguments are inside single-quoted strings using '\'' to 40 | # escape internal single quotes (and we interpolate '\'' -> ' for you). So 41 | # instead of: 42 | # bla --bar --cat=" someone's dog " --foo='a "b" c' 43 | # ARGS must be of the form: 44 | # 'bla' '--bar' '--cat=" someone'\''s dog "' '--foo='\''a "b" c'\''' 45 | # 46 | # SMELL: We haven't really thought through erroneous backslashes, but that 47 | # should only cause arg mangling, shouldn't result in any shell escape vuln as 48 | # long as the real command can handle it (!) and is called using "$@" 49 | # 50 | # IMPORTANT: Refer to tests/snazzer-send-wrapper.bats test cases 51 | dispatch_cmd() { 52 | CMD=$1 53 | ARGS=$2 54 | ARGN= 55 | PHASE=right 56 | 57 | shift 58 | shift 59 | [ -n "$CMD" ] && [ -n "$ARGS" ] 60 | if ! echo "$ARGS" | grep -q "^ *'.*' *$"; then 61 | log_cmd_and_exit 99 "Args must start and end with single-quotes" 62 | fi 63 | # We parse from right-to-left, because we're (ab)using sed and it doesn't do 64 | # non-greedy regex matching. It works by chopping the end off $ARGS 65 | # progressively until there's nothing left. $ARGN accumulates the truncated 66 | # bits for the right-most argument and is reset back to the empty string 67 | # once it is done and pushed onto "$@". Am I mad? A real language'd be nice 68 | while [ -n "$ARGS" ] 69 | do 70 | case "$PHASE" in 71 | right) 72 | ARGN="$(echo "$ARGS" | sed -n "s|^.*'\([^']*\)' *$|\1|p")$ARGN" 73 | ARGS=$(echo "$ARGS" | sed "s|\([^']*\)' *$||") 74 | # If truncation up to final ' is same as truncation up to '\'', 75 | # need to split this argument so we can interpolate '\'' -> ' 76 | if [ "$(echo "$ARGS" | sed "s/^.*'//g")" = \ 77 | "$(echo "$ARGS" | sed "s/^.*'\\\\''//g")" ]; then 78 | PHASE=split 79 | # Otherwise this is a boring argument without any '\'' in it 80 | elif echo "$ARGS" | grep -q "'$"; then 81 | PHASE=left 82 | else 83 | log_cmd_and_exit 99 "This should never happen, stopped: $ARGS" 84 | fi 85 | ;; 86 | split) 87 | L=$(echo "$ARGS" | sed -n "s|^\(.*\)'\\\\''\(.*\)$|\1|p") 88 | R=$(echo "$ARGS" | sed -n "s|^\(.*\)'\\\\''\(.*\)$|\2|p") 89 | # If there are no '\'' left at all in $ARGS, stop splitting 90 | if [ -z "$L" ] && [ -z "$R" ]; then 91 | PHASE=left 92 | # If truncation up to final ' is the same as truncation up to 93 | # '\'', we're still processing current ARGN, keep splitting 94 | elif [ "$(echo "$L" | sed "s/^.*'//g")" = \ 95 | "$(echo "$L" | sed "s/^.*'\\\\''//g")" ] 96 | then 97 | ARGS="$L" 98 | ARGN="'$R${ARGN}" 99 | # Otherwise, our right-most single-quote marks that we reached 100 | # the beginning of the current ARGN, so stop splitting 101 | else 102 | PHASE=left 103 | ARGS="$L" 104 | ARGN="'$R${ARGN}" 105 | fi 106 | ;; 107 | left) 108 | ARGN="$(echo "$ARGS" | sed -n "s|^.* *'\([^']*\)$|\1|p")$ARGN" 109 | ARGS=$(echo "$ARGS" | sed "s| *'\([^']*\)$||") 110 | set -- "$ARGN" "$@" 111 | ARGN= 112 | PHASE=right 113 | ;; 114 | *) 115 | log_cmd_and_exit 99 "This should never happen, stopped: $ARGS" 116 | ;; 117 | esac 118 | done 119 | case "$CMD" in 120 | list_snapshots) 121 | log_cmd "sudo -n snazzer --list-snapshots $(squote_args "$@")" 122 | for ARG in "$@"; do 123 | if [ "$ARG" != "--all" ] && [ "$(echo "$ARG" | cut -c 1)" = "-" ]; 124 | then 125 | log_cmd_and_exit 98 "list-snapshots: non-switch args only" 126 | fi 127 | done 128 | sudo -n snazzer --list-snapshots "$@" 129 | ;; 130 | btrfs_send_p) 131 | if [ "$#" = "2" ] && \ 132 | [ "$(echo "$1" | cut -c 1)" != "-" ] && \ 133 | [ "$(echo "$2" | cut -c 1)" != "-" ]; 134 | then 135 | log_cmd "sudo -n btrfs send -p $(squote_args "$1") $(squote_args "$2")" 136 | sudo -n btrfs send -p "$1" "$2" 137 | else 138 | log_cmd_and_exit 98 "btrfs send -p X Y bad no. args or switches" 139 | fi 140 | ;; 141 | btrfs_send) 142 | if [ "$#" = "1" ] && [ "$(echo "$1" | cut -c 1)" != "-" ]; then 143 | log_cmd "sudo -n btrfs send $(squote_args "$1")" 144 | sudo -n btrfs send "$1" 145 | else 146 | log_cmd_and_exit 98 "btrfs send X bad no. args or switches" 147 | fi 148 | ;; 149 | grep_srl) 150 | if [ "$#" = "2" ] && [ "$(echo "$1" | cut -c 1)" != "-" ] && \ 151 | [ "$(echo "$2" | cut -c 1)" != "-" ]; then 152 | log_cmd "sudo -n grep -srl $(squote_args "$1") $(squote_args "$2")" 153 | sudo -n grep -srl "$1" "$2" 154 | else 155 | log_cmd_and_exit 98 "grep -srl must have two non-switch arguments" 156 | fi 157 | ;; 158 | cat_measurement) 159 | if [ "$#" = "1" ] && [ "$(echo "$1" | cut -c 1)" != "-" ]; then 160 | log_cmd "sudo -n cat $(squote_args "$1")" 161 | sudo -n cat "$1" 162 | else 163 | log_cmd_and_exit 98 "cat: single non-switch argument only" 164 | fi 165 | ;; 166 | *) 167 | log_cmd_and_exit 127 "Bad dispatch '$CMD', this should never happen" 168 | ;; 169 | esac 170 | } 171 | 172 | run_cmd() { 173 | CMD=$1 174 | case "$CMD" in 175 | "sudo -n snazzer --list-snapshots"*) 176 | ARGS="$(echo "$CMD" | sed "s|^sudo -n snazzer --list-snapshots ||g")" 177 | dispatch_cmd "list_snapshots" "$ARGS" 178 | ;; 179 | "sudo -n grep -srl '^> on "*"' '"*"/.snapshotz/.measurements/'") 180 | ARGS="$(echo "$CMD" | sed "s|^sudo -n grep -srl ||g")" 181 | dispatch_cmd "grep_srl" "$ARGS" 182 | ;; 183 | "sudo -n btrfs send '-p' '"*"/.snapshotz/"*"' '"*"/.snapshotz/"*"'") 184 | ARGS="$(echo "$CMD" | sed "s|^sudo -n btrfs send '-p' ||g")" 185 | dispatch_cmd "btrfs_send_p" "$ARGS" 186 | ;; 187 | "sudo -n btrfs send '"*"/.snapshotz/"*"'") 188 | ARGS="$(echo "$CMD" | sed "s|^sudo -n btrfs send ||g")" 189 | dispatch_cmd "btrfs_send" "$ARGS" 190 | ;; 191 | "sudo -n cat '"*"/.snapshotz/.measurements/"*"'") 192 | ARGS="$(echo "$CMD" | sed "s|^sudo -n cat ||g")" 193 | dispatch_cmd "cat_measurement" "$ARGS" 194 | ;; 195 | *) 196 | cat <&2 197 | ERROR: Unrecognized command (are paths good, arguments single-quoted?): 198 | $CMD 199 | HERE 200 | exit 2 201 | esac 202 | } 203 | 204 | case "$1" in 205 | -h | --help ) pod2usage -exit 0 "$0"; exit ;; 206 | -v | --version ) echo "$SNAZZER_VERSION"; exit; ;; 207 | --man ) pod2usage -exit 0 -verbose 3 "$0"; exit ;; 208 | --man-roff ) pod2man --release="$SNAZZER_VERSION" "$0"; exit ;; 209 | --man-markdown ) 210 | cat <new->filter('$0'); 213 | } 214 | else { 215 | print STDERR "ERROR: --man-markdown requires Pod::Markdown\n\$@\n"; 216 | exit 9; 217 | } 218 | HERE 219 | exit 220 | ;; 221 | "") 222 | if [ -n "$SSH_ORIGINAL_COMMAND" ]; then 223 | run_cmd "$SSH_ORIGINAL_COMMAND" 224 | else 225 | pod2usage -exit 1 "$0"; 226 | exit 227 | fi 228 | ;; 229 | * ) echo "ERROR: Invalid argument '$1'" >&2 ; exit ;; 230 | esac 231 | 232 | <<__DNE__ 233 | __END__ 234 | =head1 NAME 235 | 236 | snazzer-send-wrapper - ssh forced command wrapper for snazzer-receive 237 | 238 | =head1 SYNOPSIS 239 | 240 | SSH_ORIGINAL_COMMAND="sudo -n snazzer --list-snapshots '--all'" \ 241 | ./snazzer-send-wrapper 242 | 243 | SSH_ORIGINAL_COMMAND="sudo -n grep -srl \ 244 | 'sendinghost1' '/some/.snapshotz/.measurements/'" snazzer-send-wrapper 245 | 246 | SSH_ORIGINAL_COMMAND="sudo -n btrfs send \ 247 | '/some/.snapshotz/2015-04-01T000000Z'" snazzer-send-wrapper 248 | 249 | SSH_ORIGINAL_COMMAND="sudo -n cat \ 250 | '/some/.snapshotz/.measurements/2015-04-01T000000Z'" snazzer-send-wrapper 251 | 252 | =head1 OPTIONS 253 | 254 | =over 255 | 256 | =item B<--help>: Brief help message 257 | 258 | =item B<--version>: Print version 259 | 260 | =item B<--man>: Full documentation 261 | 262 | =item B<--man-roff>: Full documentation as *roff output, Eg: 263 | 264 | snazzer --man-roff | nroff -man 265 | 266 | =item B<--man-markdown>: Full documentation as markdown output, Eg: 267 | 268 | snazzer --man-markdown > snazzer-manpage.md 269 | 270 | =back 271 | 272 | =head1 DESCRIPTION 273 | 274 | This is a wrapper script to be used in place of a real login shell (Eg. as an 275 | ssh(1) forced command) in order to restrict the commands available to the user 276 | account used by B to run C. It may be utilized by 277 | adding an entry in the C<~/.ssh/authorized_keys> file on the sending host (Eg. 278 | C) under the user account used by B to run 279 | C. C<~/.ssh/authorized_keys>: 280 | 281 | command="/usr/bin/snazzer-send-wrapper",no-port-forwarding, \ 282 | no-X11-forwarding,no-pty ssh-rsa AAAA...snip...== my key 283 | 284 | And then (as an example) receive btrfs snapshots from this C: 285 | 286 | snazzer-receive sendinghost1 --all 287 | 288 | =head1 ENVIRONMENT 289 | 290 | =over 291 | 292 | =item * SSH_ORIGINAL_COMMAND 293 | 294 | This variable holds the original remote ssh command to be acted upon. 295 | 296 | =back 297 | 298 | =head1 BUGS AND LIMITATIONS 299 | 300 | =over 301 | 302 | =item * This script tries too hard to parse normal shell commands 303 | 304 | A better design would be custom command tokens issued with more easily parsed 305 | string and argument delimeters. This would require some changes to 306 | B. 307 | 308 | A mitigating factor is that all commands are executed in the form of: 309 | 310 | foo "$@" 311 | 312 | Rather than any variant of the more exciting: 313 | 314 | foo $BAREWORD_ARGUMENTS 315 | 316 | or 317 | 318 | eval "$SSH_ORIGINAL_COMMAND" 319 | 320 | This wrapper script is also sanity-checked with bats regression tests which 321 | check that only the correct number of arguments, valid arguments, switches, 322 | path patterns and escape characters are dealt with - anything else is rejected. 323 | 324 | =back 325 | 326 | =head1 EXIT STATUS 327 | 328 | B will abort with an error message printed to STDERR and 329 | non-zero exit status under the following conditions: 330 | 331 | =over 332 | 333 | =item 2. the command string was not recognized 334 | 335 | =item 98. the command string was recognized but the arguments were not safe 336 | 337 | =item 99. the command string was recognized and an attempt was made to 338 | parse/re-pack the arguments however the argument string had dangling quotes or 339 | otherwise confused the parser/"$@" unpacker 340 | 341 | =back 342 | 343 | =head1 SEE ALSO 344 | 345 | snazzer-receive 346 | 347 | =head1 AUTHOR 348 | 349 | Snazzer Authors are listed in the AUTHORS.md file in the root of this 350 | distribution. See https://github.com/csirac2/snazzer for more information. 351 | NOTE: Please extend that file, not this notice. 352 | 353 | =head1 LICENSE AND COPYRIGHT 354 | 355 | Copyright (C) 2015-2016, Snazzer Authors All rights reserved. 356 | 357 | Redistribution and use in source and binary forms, with or without 358 | modification, are permitted provided that the following conditions are met: 359 | 360 | 1. Redistributions of source code must retain the above copyright notice, this 361 | list of conditions and the following disclaimer. 362 | 363 | 2. Redistributions in binary form must reproduce the above copyright notice, 364 | this list of conditions and the following disclaimer in the documentation 365 | and/or other materials provided with the distribution. 366 | 367 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 368 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 369 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 370 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 371 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 372 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 373 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 374 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 375 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 376 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 377 | =cut 378 | __DNE__ 379 | -------------------------------------------------------------------------------- /snazzer-measure: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | set -e 4 | SNAZZER_VERSION=0.0.3 5 | 6 | # Keep in sync with the POD! 7 | default_snazzer_sig_func() { 8 | gpg2 --quiet --no-greeting --batch --use-agent --armor --detach-sign - 9 | } 10 | # Keep in sync with the POD! 11 | if [ -z "$SNAZZER_SIG_ENABLE" ]; then 12 | SNAZZER_SIG_ENABLE=1 13 | fi 14 | # Keep in sync with the POD! 15 | if [ -z "$SNAZZER_MEASUREMENTS_EXCLUDE_FILE" ]; then 16 | SNAZZER_MEASUREMENTS_EXCLUDE_FILE=".snapshot_measurements.exclude" 17 | fi 18 | 19 | # Keep in sync with the POD! 20 | if [ -z "$SNAZZER_SUBVOLS_EXCLUDE_FILE" ]; then 21 | SNAZZER_SUBVOLS_EXCLUDE_FILE="/etc/snazzer/exclude.patterns" 22 | fi 23 | 24 | if [ "$(id -u)" = "0" ]; then 25 | SUDO="" 26 | else 27 | SUDO="sudo" 28 | fi 29 | 30 | do_snazzer_sig() { 31 | if type snazzer_sig_func | grep -i function >/dev/null; then 32 | snazzer_sig_func 33 | else 34 | default_snazzer_sig_func 35 | fi 36 | } 37 | 38 | get_date() { 39 | if [ "$SNAZZER_USE_UTC" = "1" ]; then 40 | date -u +"%Y-%m-%dT%H%M%SZ" 41 | else 42 | date +"%Y-%m-%dT%H%M%S%z" 43 | fi 44 | } 45 | 46 | host_datetime() { 47 | WHAT=$1 48 | echo "> on $(hostname) at $(get_date), $WHAT:" 49 | } 50 | 51 | build_tar_cmd() { 52 | DIR_ESC=$(echo "$1" | sed "s|'|'\\\\''|g") 53 | 54 | cat <"$LIST_TMP" && \ 73 | LC_ALL=C sort -z "$LIST_TMP" > "$SORT_TMP" && \ 74 | $SUDO tar --no-recursion --one-file-system --preserve-permissions \ 75 | --numeric-owner --null --create --to-stdout \ 76 | --directory "$DIR" \ 77 | --files-from "$SORT_TMP" \ 78 | --exclude-from "$DIR/$SNAZZER_MEASUREMENTS_EXCLUDE_FILE" && \ 79 | rm "$SORT_TMP" && \ 80 | rm "$LIST_TMP" 81 | } 82 | 83 | measure_sha512sum() { 84 | DIR=$1 85 | 86 | host_datetime "sha512sum" 87 | cat <> "$EXCL_TMP" 108 | done 109 | $SUDO du -bs --one-file-system --exclude-from "$EXCL_TMP" "$DIR" 110 | rm "$EXCL_TMP" 111 | } 112 | 113 | build_find_excl_switch() { 114 | DIR=$1 115 | EXCL_FILE="$DIR/$SNAZZER_MEASUREMENTS_EXCLUDE_FILE" 116 | 117 | sed "s/'/'\\\\''/g" "$EXCL_FILE" | while read -r RULE 118 | do 119 | printf " -path '*%s' -prune -o" "$RULE" 120 | done 121 | } 122 | 123 | dry_find() { 124 | DIR_ESC=$(echo "$1" | sed "s|'|'\\\\''|g") 125 | EXCL=$(build_find_excl_switch "$1") 126 | cat <"\$SIG" && $(build_tar_cmd "$DIR") | gpg2 --verify "\$SIG" - && rm "\$SIG") 165 | CMD 166 | do_snazzer_sig 167 | echo "" 168 | } 169 | 170 | 171 | # function duplicated in snazzer 172 | glob2grep_file() { 173 | FILE=$1 174 | OUT=$(mktemp) 175 | 176 | # check if every line starts with either / or *, if not, quit with a syntax error. 177 | while read -r line; do 178 | if ! echo "$line" | grep '^[/*]' >/dev/null; then 179 | echo "SYNTAX ERROR: $1 contains line \"$line\" that starts with neither / nor *." >&2 180 | exit 12 181 | fi 182 | done < "$1" 183 | 184 | # first, escape $, . and ^ with a backslash so they're taken literally 185 | # then, extend * to .* to emulate shell globbing. 186 | # finally, add ^ and $ to the line ends so the lines are not evaluated as *line* 187 | sed 's|[$.^]|\\&|g' "$FILE" | sed 's/\*/\.*/g' | sed 's/^/\^/g' | sed 's/$/\$/g' > "$OUT" 188 | 189 | echo "$OUT" 190 | } 191 | 192 | 193 | assert_gpg_secring_excluded() { 194 | if [ "$(gpg2 --list-secret-keys | wc -l)" = "0" ]; then 195 | echo "ERROR: no gpg2 --list-secret-keys" >&2 196 | exit 10 197 | fi 198 | EXCL_FILE=$(glob2grep_file "$SNAZZER_SUBVOLS_EXCLUDE_FILE") || exit $? 199 | N=$(gpg2 --list-secret-keys | grep '^/' | xargs readlink -f | \ 200 | grep -v -f "$EXCL_FILE" | \ 201 | (xargs --no-run-if-empty df -t btrfs 2>/dev/null) | wc -l) 202 | rm "$EXCL_FILE" 203 | #N=$(gpg2 --list-secret-keys | grep '^/' | xargs readlink -f | \ 204 | # grep -v -f /etc/snazzer/exclude.patterns | wc -l) 205 | GPG_SECRING=$(gpg2 --list-secret-keys | head -n 1) 206 | 207 | if [ "$N" -ne 0 ] && ( [ -z "$MY_KEYFILES_ARE_INVINCIBLE" ] || \ 208 | [ "$MY_KEYFILES_ARE_INVINCIBLE" -ne "1" ]); then 209 | show_usage < "$TMP_RC" && exit "$?" ) | \ 250 | tee "$TMP_FIFO" | \ 251 | measure_sha512sum "$FAKE_DIR" > "$SHA512_MEASUREMENT" & 252 | sleep 1 # SMELLLLLL... avoid stupid race condition where it seems that 253 | # measure_gpg starts before tee does 254 | if [ -s "$TMP_RC" ] && [ "$(cat "$TMP_RC")" != "0" ]; then 255 | exit "$(cat "$TMP_RC")" 256 | fi 257 | measure_gpg "$FAKE_DIR" < "$TMP_FIFO" > "$GPG_MEASUREMENT" 258 | # And if we put their respective output straight to stdout, we sometimes 259 | # get their output intermixedue 260 | cat "$SHA512_MEASUREMENT" 261 | cat "$GPG_MEASUREMENT" 262 | rm "$GPG_MEASUREMENT" 263 | rm "$SHA512_MEASUREMENT" 264 | rm "$TMP_RC" 265 | rm "$TMP_FIFO" 266 | rmdir "$TMP_DIR" 267 | else 268 | do_tar "$MEAS_DIR" | measure_sha512sum "$FAKE_DIR" 269 | fi 270 | measure_tar_info 271 | } 272 | 273 | show_usage() { 274 | if [ -n "$1" ]; then 275 | pod2usage -exit 0 "$0" >&2 276 | printf "\nERROR: %s" "$1" >&2 277 | else 278 | pod2usage -exit 0 "$0" 279 | fi 280 | } 281 | 282 | case "$1" in 283 | -h | --help ) pod2usage -exit 0 "$0"; exit ;; 284 | -v | --version) echo "$SNAZZER_VERSION"; exit; ;; 285 | --man ) pod2usage -exit 0 -verbose 3 "$0"; exit ;; 286 | --man-roff ) pod2man --release=$SNAZZER_VERSION "$0"; exit ;; 287 | --man-markdown ) 288 | cat <new->filter('$0'); 291 | } 292 | else { 293 | print STDERR "ERROR: --man-markdown requires Pod::Markdown\n\$@\n"; 294 | exit 9; 295 | } 296 | HERE 297 | exit ;; 298 | -* ) echo "ERROR: Invalid argument '$1'" >&2 ; exit 1 ;; 299 | esac 300 | 301 | if [ -z "$1" ]; then 302 | show_usage "No path specified" 303 | exit 2 304 | else 305 | measure "$1" "$2" 306 | fi 307 | 308 | <<__DNE__ 309 | __END__ 310 | =head1 NAME 311 | 312 | snazzer-measure - report shasums & PGP signatures of content under a given path, 313 | along with commands to reproduce or verify data is unchanged 314 | 315 | =head1 SYNOPSIS 316 | 317 | snazzer-measure /measured/path [/reported/path] >> path_measurements 318 | 319 | =head1 DESCRIPTION 320 | 321 | Creates reproducible fingerprints of the given directory, along with commands 322 | necessary (relative to measured path, or if supplied - the optional reported 323 | path) to reproduce the measurement using only standard core GNU userland. 324 | 325 | The output includes: 326 | 327 | =over 328 | 329 | =item * hostname and datetime of B invocation 330 | 331 | =item * C (bytes used) 332 | 333 | =item * C of the result of a reproducible tarball of the directory 334 | 335 | =item * C of the same 336 | 337 | =item * instructions for reproducing or verifying each of the above 338 | 339 | =item * C, C 340 | 341 | =back 342 | 343 | =head1 OPTIONS 344 | 345 | =over 346 | 347 | =item B<--help>: Brief help message 348 | 349 | =item B<--version>: Print version 350 | 351 | =item B<--man>: Full documentation 352 | 353 | =item B<--man-roff>: Full documentation as *roff output, Eg: 354 | 355 | snazzer-measure --man-roff | nroff -man 356 | 357 | =item B<--man-markdown>: Full documentation as markdown output, Eg: 358 | 359 | snazzer-measure --man-markdown > snazzer-measure-manpage.md 360 | 361 | =back 362 | 363 | =head1 ENVIRONMENT 364 | 365 | =over 366 | 367 | =item * snazzer_sig_func 368 | 369 | Function generating PGP SIGNATURE text. Takes input from stdin, output to 370 | stdout. Signatures can be disabled with L. Default: 371 | 372 | snazzer_sig_func() { 373 | gpg2 --quiet --no-greeting --batch --use-agent --armor --detach-sign - 374 | } 375 | 376 | =item * SNAZZER_SIG_ENABLE 377 | 378 | If set to 0, GPG signing is disabled and snazzer_sig_func() is not called. 379 | 380 | =item * SNAZZER_MEASUREMENTS_EXCLUDE_FILE 381 | 382 | A filename within the measured directory of a newline-separated list of shell 383 | glob patterns to exclude from measurements. Default: 384 | 385 | SNAZZER_MEASUREMENTS_EXCLUDE_FILE=".snapshot_measurements.exclude" 386 | 387 | =item * MY_KEYFILES_ARE_INVINCIBLE=1 388 | 389 | Skip sanity check/abort when gpg secret key exists on a subvolume included in 390 | default snazzer snapshots 391 | 392 | =item * SNAZZER_USE_UTC 393 | 394 | Use UTC times of the form C instead of the default local 395 | time+offset C 396 | 397 | =item * SNAZZER_SUBVOLS_EXCLUDE_FILE 398 | 399 | Filename of newline separated list of shell glob patterns of subvolume pathnames 400 | which should be excluded from C invocations; compatible with 401 | C<--exclude-from> for B and B. Examples of subvolume patterns to 402 | exclude from regular snapshotting: *secret*, /var/cache, /var/lib/docker/*, 403 | .snapshots. B C<.snapshotz> is always excluded. 404 | Default: 405 | 406 | SNAZZER_SUBVOLS_EXCLUDE_FILE="/etc/snazzer/exclude.patterns" 407 | 408 | =back 409 | 410 | =head2 sudo requirements 411 | 412 | When running B as a non-root user, certain commands will be 413 | prefixed with C. The following lines in C or 414 | C should suffice for scripted jobs such as cron (replace 415 | C with the actual user name you are setting up for this task): 416 | 417 | measureuser ALL=(root:nobody) NOPASSWD:NOEXEC: \ 418 | /bin/cat */.snapshotz/*/.snapshot_measurements.exclude, \ 419 | /usr/bin/du -bs --one-file-system --exclude-from * */.snapshotz/*, \ 420 | /usr/bin/find */.snapshotz/* \ 421 | -xdev -not -path /*/.snapshotz/* -printf %P\\\\0, \ 422 | /bin/tar --no-recursion --one-file-system --preserve-permissions \ 423 | --numeric-owner --null --create --to-stdout \ 424 | --directory */.snapshotz/* --files-from * \ 425 | --exclude-from */.snapshotz/*/.snapshot_measurements.exclude 426 | 427 | =head1 EXIT STATUS 428 | 429 | B will abort with an error message printed to STDERR and 430 | non-zero exit status under the following conditions: 431 | 432 | =over 433 | 434 | =item 1. Invalid argument 435 | 436 | =item 2. Path string not specified 437 | 438 | =item 4. GPG signature would have been generated with a secret keyfile stored 439 | in a subvolume which has not been excluded from default snazzer snapshots, see 440 | L below 441 | 442 | =item 5. Expected the .snazzer_measurements.exclude file to contain an entry 443 | for the .snazzer_measurements file 444 | 445 | =back 446 | 447 | =head1 IMPORTANT 448 | 449 | Please note that if you are using this tool to gain some form of integrity 450 | measurement (Eg. you want to detect tampering), GPG private keys used for the 451 | signing operation mustn't be exposed among the directories being measured. 452 | 453 | Put another way: it makes no sense to GPG-sign measurements of a directory if 454 | those very same directories contain the GPG private key material required to 455 | re-sign modifications made by anyone who happens to be looking. 456 | 457 | =head1 BUGS AND LIMITATIONS 458 | 459 | =over 460 | 461 | =item * MY_KEYFILES_ARE_INVINCIBLE 462 | 463 | The sanity check for location of GPG secret keyfile may be more annoying than 464 | helpful on installations using smartcards, TPMs, or other methods of protecting 465 | keyfiles - hence the B work-around. 466 | 467 | =item * Temporary files 468 | 469 | To avoid unnecessary I/O, gpg signing and shasumming are done in parallel from 470 | the same C pipe; this involves creating a temporary named pipe 471 | which is normally removed at the end of a successful run, but will be left 472 | behind should a failure occur. These are randomly named with C and mode 473 | 0700, inside a C directory also with 0700 permissions. 474 | 475 | =back 476 | 477 | =head1 SEE ALSO 478 | 479 | snazzer, snazzer-prune-candidates, snazzer-receive 480 | 481 | =head1 AUTHOR 482 | 483 | Snazzer Authors are listed in the AUTHORS.md file in the root of this 484 | distribution. See https://github.com/csirac2/snazzer for more information. 485 | NOTE: Please extend that file, not this notice. 486 | 487 | =head1 LICENSE AND COPYRIGHT 488 | 489 | Copyright (C) 2015-2016, Snazzer Authors All rights reserved. 490 | 491 | Redistribution and use in source and binary forms, with or without 492 | modification, are permitted provided that the following conditions are met: 493 | 494 | 1. Redistributions of source code must retain the above copyright notice, this 495 | list of conditions and the following disclaimer. 496 | 497 | 2. Redistributions in binary form must reproduce the above copyright notice, 498 | this list of conditions and the following disclaimer in the documentation 499 | and/or other materials provided with the distribution. 500 | 501 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 502 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 503 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 504 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 505 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 506 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 507 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 508 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 509 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 510 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 511 | =cut 512 | __DNE__ 513 | -------------------------------------------------------------------------------- /snazzer: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | set -e 3 | SNAZZER_VERSION=0.0.3 4 | SNAZZER_MEAS_LOCK_DIR="/var/lock/snazzer-measure.lock" 5 | SNAZZER_SNAPSHOTZ_PERMS=0700 6 | 7 | # For testing 8 | if [ -n "$SNAZZER_DATE" ]; then 9 | SNAZZER_DATE=$SNAZZER_DATE 10 | elif [ "$SNAZZER_USE_UTC" = "1" ]; then 11 | SNAZZER_DATE=$(date -u +"%Y-%m-%dT%H%M%SZ") 12 | else 13 | SNAZZER_DATE=$(date +"%Y-%m-%dT%H%M%S%z") 14 | fi 15 | if [ "$(id -u)" = "0" ]; then 16 | SUDO="" 17 | else 18 | SUDO="sudo" 19 | fi 20 | if [ -z "$SNAZZER_MEASUREMENTS_EXCLUDE_FILE" ]; then 21 | SNAZZER_MEASUREMENTS_EXCLUDE_FILE=".snapshot_measurements.exclude" 22 | fi 23 | # Keep in sync with the POD! 24 | if [ -z "$SNAZZER_SUBVOLS_EXCLUDE_FILE" ]; then 25 | SNAZZER_SUBVOLS_EXCLUDE_FILE="/etc/snazzer/exclude.patterns" 26 | fi 27 | 28 | if ! $SUDO test -e "$SNAZZER_SUBVOLS_EXCLUDE_FILE"; then 29 | MISSING_SUBVOLS_EXCL_FILE="$SNAZZER_SUBVOLS_EXCLUDE_FILE" 30 | SNAZZER_SUBVOLS_EXCLUDE_FILE=$(mktemp) 31 | cat < "$SNAZZER_SUBVOLS_EXCLUDE_FILE" 32 | /var/cache 33 | /var/lib/docker/* 34 | */.snapshots 35 | /tmp 36 | *backup* 37 | *secret* 38 | HERE 39 | if [ "$DRY_RUN" = "1" ]; then printf "#"; fi 40 | cat <&2 41 | WARN: $MISSING_SUBVOLS_EXCL_FILE missing, defaulting to $SNAZZER_SUBVOLS_EXCLUDE_FILE: 42 | HERE 43 | printf " " >&2 44 | sed ':a;N;$!ba;s/\n/, /g' "$SNAZZER_SUBVOLS_EXCLUDE_FILE" >&2 45 | fi 46 | 47 | # So, mountpoint(1) returns true if the path is a btrfs subvolume root, but we 48 | # want to know if the path is a btrfs *filesystem* root. Hence the df/grep crazy 49 | assert_mountpoint() { 50 | DIR=$(readlink -f "$1") 51 | DIR_GREP=$(echo "$DIR" | sed 's|[$.^]|\\&|g') 52 | if ! mountpoint -q "$DIR" || \ 53 | ! df | grep -c "$DIR_GREP\$" >/dev/null; then 54 | echo "ERROR: $DIR is not a filesystem mountpoint" >&2 55 | exit 2 56 | fi 57 | } 58 | 59 | assert_subvolume() { 60 | DIR=$1 61 | if ! $SUDO btrfs subvolume show "$DIR" >/dev/null 2>&1; then 62 | echo "ERROR: $DIR is not a btrfs subvolume" >&2 63 | exit 3 64 | fi 65 | } 66 | 67 | assert_btrfs_tools() { 68 | if ! $SUDO btrfs subvolume snapshot --help >/dev/null; then 69 | echo 'ERROR: btrfs command not found' >&2 70 | exit 11 71 | fi 72 | } 73 | 74 | # function duplicated in snazzer-measure 75 | glob2grep_file() { 76 | FILE=$1 77 | OUT=$(mktemp) 78 | 79 | # check if every line starts with either / or *, if not, quit with a syntax error. 80 | while read -r line; do 81 | if ! echo "$line" | grep '^[/*]' >/dev/null; then 82 | echo "SYNTAX ERROR: $1 contains line \"$line\" that starts with neither / nor *." >&2 83 | exit 12 84 | fi 85 | done < "$1" 86 | 87 | # first, escape $, . and ^ with a backslash so they're taken literally 88 | # then, extend * to .* to emulate shell globbing. 89 | # finally, add ^ and $ to the line ends so the lines are not evaluated as *line* 90 | sed 's|[$.^]|\\&|g' "$FILE" | sed 's/\*/\.*/g' | sed 's/^/\^/g' | sed 's/$/\$/g' > "$OUT" 91 | 92 | echo "$OUT" 93 | } 94 | 95 | # This should only operate on a real mount(8) filesystem mountpoint, so call 96 | # assert_mountpoint before calling list_subvolumes. 97 | # the DIR argument is assumed to be without a trailing slash, i.e. "/" or "/mnt" but not "/mnt/" 98 | list_subvolumes() { 99 | DIR=$1 100 | DO_INVERT=$2 101 | if [ "$DO_INVERT" = "--excluded" ]; then 102 | GREP="grep" 103 | else 104 | GREP="grep -v" 105 | fi 106 | 107 | if [ "$DIR" != "/" ]; then 108 | PREFIX="$DIR" 109 | else 110 | PREFIX="" 111 | fi 112 | 113 | EXCL_FILE=$(glob2grep_file "$SNAZZER_SUBVOLS_EXCLUDE_FILE") || exit $? 114 | $SUDO btrfs subvolume list -t "$DIR" | tail -n+3 | \ 115 | # delete all columns except path, prefix the paths with a / 116 | sed 's|^[0-9]*[ \t]*[0-9]*[ \t]*[0-9]*[ \t]*|/|g' | \ 117 | # add / line, as the root subvolume is not included in btrfs subvolume list 118 | (echo "/"; cat) | \ 119 | $GREP -f "$EXCL_FILE" | grep -v '\.snapshotz' | \ 120 | while read -r SUBVOL; do echo "${PREFIX}$SUBVOL"; done 121 | rm "$EXCL_FILE" 122 | } 123 | 124 | report_subvols_excluded() { 125 | DIR=$1 126 | assert_mountpoint "$DIR" 127 | SUBVOLS=$(list_subvolumes "$DIR" --excluded) || exit $? 128 | # command substitution removes trailing newlines, we have to add it again for wc -l to give correct results. 129 | NUM=$(printf "%s\n" "$SUBVOLS" | wc -l) 130 | 131 | if [ "$NUM" != "0" ]; then 132 | if [ "$DRY_RUN" = "1" ]; then printf "#"; fi 133 | echo "$NUM subvolumes excluded in $DIR by $SNAZZER_SUBVOLS_EXCLUDE_FILE." 134 | fi 135 | } 136 | 137 | # Get a list of mountpoints from the unique list of devices showing up as 138 | # mounted btrfs filesystems. We don't want to list bind mounts or manually 139 | # mounted subvols (who already have their parent/container filesystems mounted) 140 | # as separate filesystems, that would result in multiple snapshotting of those 141 | # subvols later on. 142 | # SMELL: what if a subvol is mounted some place other than its path name? 143 | list_btrfs_mountpoints() { 144 | EXCL_FILE=$(glob2grep_file "$SNAZZER_SUBVOLS_EXCLUDE_FILE") || exit $? 145 | df -t btrfs 2>/dev/null | tail -n+2 | awk '{ print $1 }' | sort | uniq | \ 146 | while read -r DEV; do df --output=target "$DEV" | tail -n+2 ; done | \ 147 | grep -v '\.snapshotz' | grep -v -f "$EXCL_FILE" || true 148 | rm "$EXCL_FILE" 149 | } 150 | 151 | get_subvol_path() { 152 | DIR=$1 153 | UUID=$($SUDO btrfs subvolume show "$DIR" | \ 154 | sed -n 's/^[ \t]*uuid:[ \t]*\(.*\)/\1/p') 155 | if [ -n "$UUID" ]; then 156 | $SUDO btrfs subvolume list -u "$DIR" | \ 157 | sed -n "s/^.*$UUID[ \t]*path[ \t]*\\(.*\\)/\\1/p" 158 | fi 159 | } 160 | 161 | # List the excluded subvols relative to DIR 162 | list_subvols() { 163 | DIR=$1 164 | SUBVOL_PATH_ESC=$(get_subvol_path "$DIR" | sed -e 's/[]$*.^|[]/\\&/g') 165 | $SUDO btrfs subvolume list -o "$DIR" | \ 166 | sed -n "s|.*path $SUBVOL_PATH_ESC[/]*\\(.*\\)|\\1|p" 167 | } 168 | 169 | assert_missing() { 170 | THING=$1 171 | if $SUDO test -e "$THING"; then 172 | cat <&2 173 | ERROR: $THING exists, aborting 174 | HERE 175 | exit 6 176 | fi 177 | } 178 | 179 | assert_exists() { 180 | THING=$1 181 | if ! $SUDO test -e "$THING"; then 182 | cat <&2 183 | ERROR: $THING is missing, aborting 184 | HERE 185 | exit 4 186 | fi 187 | } 188 | 189 | report_snapshot_dirs() { 190 | SUBVOL=$1 191 | 192 | if [ "$DO_ACTION" = "snapshot" ] && [ "$SUBVOL" = "/" ]; then 193 | SUBVOL_MNT=$(df --output=target "$SUBVOL" | tail -n+2) 194 | for DIR in "/tmp" "/var/tmp" "/var/cache" "/var/log"; do 195 | DIR_MNT=$(df --output=target "$DIR" | tail -n+2) 196 | if [ "$DIR_MNT" = "$SUBVOL_MNT" ]; then 197 | if [ "$DRY_RUN" = "1" ]; then printf "#"; fi 198 | cat </dev/null 219 | $( list_subvols "$SUBVOL" | sed -e 's|[`$]|\\&|g') 220 | EXCL 221 | $SUDO btrfs subvolume snapshot -r '$SUBVOL_ESC' '$SNAP_DIR_ESC' 222 | $SUDO rm '$EXCL_FILE_ESC' 223 | CMD 224 | } 225 | 226 | do_snapshot() { 227 | SUBVOL=$1 228 | SNAP_DIR=$2 229 | if [ "$SUBVOL" = "/" ]; then 230 | EXCL_FILE="/$SNAZZER_MEASUREMENTS_EXCLUDE_FILE" 231 | else 232 | EXCL_FILE="$SUBVOL/$SNAZZER_MEASUREMENTS_EXCLUDE_FILE" 233 | fi 234 | 235 | # The cat | tee below, is in case we're leaning on sudo... 236 | cat </dev/null 237 | $( list_subvols "$SUBVOL" | sed -e 's|[`$]|\\&|g') 238 | EXCL 239 | $SUDO btrfs subvolume snapshot -r "$SUBVOL" "$SNAP_DIR" 240 | $SUDO rm "$EXCL_FILE" 241 | } 242 | 243 | snapshot() { 244 | SUBVOL=$1 245 | assert_subvolume "$SUBVOL" 246 | assert_missing "$SUBVOL/$SNAZZER_MEASUREMENTS_EXCLUDE_FILE" 247 | if [ "$SUBVOL" = "/" ]; then 248 | DEST="/.snapshotz" 249 | else 250 | DEST="$SUBVOL/.snapshotz" 251 | fi 252 | $SUDO mkdir -p "$DEST" --mode="$SNAZZER_SNAPSHOTZ_PERMS" 253 | $SUDO chown --reference "$SUBVOL" "$DEST" 254 | $SUDO chgrp --reference "$SUBVOL" "$DEST" 255 | if [ "$DRY_RUN" = "1" ]; then 256 | dry_snapshot "$SUBVOL" "$DEST/$SNAZZER_DATE" 257 | else 258 | do_snapshot "$SUBVOL" "$DEST/$SNAZZER_DATE" 259 | fi 260 | assert_missing "$SUBVOL/$SNAZZER_MEASUREMENTS_EXCLUDE_FILE" 261 | } 262 | 263 | list_dirs() { 264 | DIR=$1 265 | $SUDO ls -F "$DIR" | sed -n 's|^\(.*\)/$|\1|p' 266 | } 267 | 268 | # List directories to be pruned under a foo/.snapshotz dir, newline separated. 269 | prune_shotz_dirs() { 270 | SUBVOL_SHOTZ=$1 271 | assert_missing "$SUBVOL_SHOTZ/.incomplete" 272 | $SUDO ls "$SUBVOL_SHOTZ" | snazzer-prune-candidates | \ 273 | while read -r SNAP_NAME; do echo "$SUBVOL_SHOTZ/$SNAP_NAME"; done 274 | } 275 | 276 | dry_prune() { 277 | PRUNE_LIST=$1 278 | shift 279 | cat <>>>" | while read -r ITEM; do 284 | if [ "$ITEM" = "<<<>>>" ]; then 285 | echo $SUDO btrfs subvolume delete "$@" 286 | else 287 | set -- "$@" "'$(echo "$ITEM" | sed "s|'|'\\\\''|g")'" 288 | fi 289 | done 290 | ) 291 | CMD 292 | } 293 | 294 | # SMELL: Ugly "while read -r" variable scope limitation work-around. Build up "$@" 295 | # to be used safely w/btrfs. <<>> is an unlikely string marking end of list 296 | # at which point we make use of "$@" while it's still in scope. 297 | do_prune() { 298 | PRUNE_LIST=$1 299 | shift 300 | ( 301 | IFS= 302 | echo "$PRUNE_LIST 303 | <<<>>>" | while read -r ITEM; do 304 | if [ "$ITEM" = "<<<>>>" ]; then 305 | $SUDO btrfs subvolume delete "$@" 306 | else 307 | set -- "$@" "$ITEM" 308 | fi 309 | done 310 | ) 311 | } 312 | 313 | prune() { 314 | SUBVOL=$1 315 | if [ "$SUBVOL" = "/" ]; then 316 | SUBVOL_SHOTZ="/.snapshotz" 317 | else 318 | SUBVOL_SHOTZ="$SUBVOL/.snapshotz" 319 | fi 320 | assert_subvolume "$SUBVOL" 321 | assert_exists "$SUBVOL_SHOTZ" 322 | PRUNE_LIST=$(prune_shotz_dirs "$SUBVOL_SHOTZ") 323 | NUM_PRUNE=$(echo "$PRUNE_LIST" | grep -c . || true) 324 | NUM_AVAIL=$(list_dirs "$SUBVOL_SHOTZ" | grep -c . || true ) 325 | if [ "$DRY_RUN" = "1" ]; then printf "#"; fi 326 | echo "$SUBVOL_SHOTZ: pruning $NUM_PRUNE of $NUM_AVAIL" 327 | if [ "$DRY_RUN" != "1" ] && [ "$DO_FORCE" != "1" ]; then 328 | echo "ERROR: --prune expected --force or --dry-run" >&2 329 | exit 5 330 | fi 331 | if [ "$NUM_PRUNE" != "0" ]; then 332 | if [ -n "$PRUNE_LIST" ]; then 333 | if [ "$DRY_RUN" = "1" ]; then 334 | dry_prune "$PRUNE_LIST" 335 | else 336 | do_prune "$PRUNE_LIST" | \ 337 | grep -v 'Transaction commit: none (default)' || true 338 | fi 339 | fi 340 | fi 341 | } 342 | 343 | measure_lock() { 344 | if ! $SUDO test -e "$SNAZZER_MEAS_LOCK_DIR"; then 345 | true 346 | elif ! $SUDO test -e "$SNAZZER_MEAS_LOCK_DIR/pid"; then 347 | echo "ERROR: $SNAZZER_MEAS_LOCK_DIR exists without pidfile 'pid'" >&2 348 | echo " Please report this to the author and/or remove the dir" >&2 349 | exit 7 350 | else 351 | OLD_PID=$(cat "$SNAZZER_MEAS_LOCK_DIR/pid") 352 | if ps --pid="$OLD_PID" >/dev/null; then 353 | echo "ERROR: A snazzer --measure invocation is already running" >&2 354 | echo " $SNAZZER_MEAS_LOCK_DIR/pid is $OLD_PID" >&2 355 | exit 7 356 | else 357 | if [ "$DRY_RUN" = "1" ]; then printf "#"; fi 358 | cat <&2 359 | WARN: Previous snazzer --measure invocation PID $OLD_PID exited ungracefully; 360 | $SNAZZER_MEAS_LOCK_DIR/pid stale 361 | HERE 362 | $SUDO rm -v "$SNAZZER_MEAS_LOCK_DIR/pid" 363 | $SUDO rmdir -v "$SNAZZER_MEAS_LOCK_DIR" 364 | fi 365 | fi 366 | $SUDO mkdir "$SNAZZER_MEAS_LOCK_DIR" 367 | echo "$$" | $SUDO tee "$SNAZZER_MEAS_LOCK_DIR/pid" >/dev/null 368 | } 369 | 370 | measure_unlock() { 371 | $SUDO rm "$SNAZZER_MEAS_LOCK_DIR/pid" 372 | $SUDO rmdir "$SNAZZER_MEAS_LOCK_DIR" 373 | } 374 | 375 | dry_measure() { 376 | SUBVOL_ESC=$(echo "$1" | sed "s/'/'\\\\''/g") 377 | REPORT_ESC=$(echo "$2" | sed "s/'/'\\\\''/g") 378 | FAKEPATH_ESC=$(echo "$3" | sed "s/'/'\\\\''/g") 379 | 380 | cat <> '$REPORT_ESC'; 383 | HERE 384 | } 385 | 386 | do_measure() { 387 | SUBVOL=$1 388 | REPORT=$2 389 | FAKEPATH=$3 390 | REPORT_TMP=$(mktemp) 391 | # Avoid putting this in a pipe in case snazzer-measure aborts with error. 392 | # Also, using a tempfile avoids writing partial measurements to the report. 393 | SNAZZER_SUBVOLS_EXCLUDE_FILE="$SNAZZER_SUBVOLS_EXCLUDE_FILE" \ 394 | snazzer-measure "$SUBVOL" "$FAKEPATH" > "$REPORT_TMP" 395 | $SUDO tee -a "$REPORT" <"$REPORT_TMP" >/dev/null 396 | rm "$REPORT_TMP" 397 | } 398 | 399 | # If DO_FORCE=1, measure all snapshots regardless of whether the hostname seems 400 | # to already have made a measurement in the past 401 | measure() { 402 | SUBVOL=$1 403 | if [ "$SUBVOL" = "/" ]; then 404 | SUBVOL_SHOTZ="/.snapshotz" 405 | else 406 | SUBVOL_SHOTZ="$SUBVOL/.snapshotz" 407 | fi 408 | assert_subvolume "$SUBVOL" 409 | assert_exists "$SUBVOL_SHOTZ" 410 | measure_lock 411 | if ! $SUDO test -e "$SUBVOL_SHOTZ/.measurements"; then 412 | $SUDO mkdir "$SUBVOL_SHOTZ/.measurements" 413 | fi 414 | SNAP_LIST_AVAIL=$($SUDO ls "$SUBVOL_SHOTZ") 415 | if [ "$DO_FORCE" = "1" ]; then 416 | SNAP_LIST="$SNAP_LIST_AVAIL" 417 | else 418 | # Collect measurements that don't mention our hostname 419 | SNAP_LIST=$($SUDO grep -rsL "^> on $(hostname) at" \ 420 | "$SUBVOL_SHOTZ/.measurements" | \ 421 | while read -r SNAP_FULLPATH; do basename "$SNAP_FULLPATH"; done) 422 | # Collect snapshot names without any measurements at all 423 | # SMELL: What a nasty way to do this. We're trying to limit the number 424 | # of different things a $SUDO user would need to open up in sudoers, but 425 | # that's a pretty terribly tedious use-case. 426 | SNAP_MISSING=$(echo "$SNAP_LIST_AVAIL" | while read -r SNAP_NAME 427 | do $SUDO test -e \ 428 | "$SUBVOL_SHOTZ/.measurements/$SNAP_NAME" || \ 429 | echo "$SNAP_NAME" 430 | done) 431 | SNAP_LIST=$(echo "$SNAP_LIST"; echo "$SNAP_MISSING") 432 | fi 433 | # grep . so we don't count final newline 434 | NUM_MEAS=$( echo "$SNAP_LIST" | grep -c . || true ) 435 | NUM_AVAIL=$( echo "$SNAP_LIST_AVAIL" | grep -c . || true ) 436 | echo "$SUBVOL: measuring $NUM_MEAS of $NUM_AVAIL" 437 | if [ -n "$SNAP_LIST" ]; then 438 | if [ "$DRY_RUN" = "1" ]; then 439 | echo "$SNAP_LIST" | grep . | while read -r SNAP_NAME; do 440 | echo " $SNAP_NAME" 441 | SNAP_PATH_ESC=$(echo "$SUBVOL_SHOTZ/$SNAP_NAME" | sed "s|'|'\\\\''|g") 442 | REPORT_PATH_ESC=$(echo "$SUBVOL_SHOTZ/.measurements/$SNAP_NAME" | sed "s|'|'\\\\''|g") 443 | dry_measure "$SNAP_PATH_ESC" "$REPORT_PATH_ESC" "../$SNAP_NAME" 444 | done 445 | else 446 | echo "$SNAP_LIST" | grep . | while read -r SNAP_NAME; do 447 | echo " $SNAP_NAME" 448 | SNAP_PATH="$SUBVOL_SHOTZ/$SNAP_NAME" 449 | REPORT_PATH="$SUBVOL_SHOTZ/.measurements/$SNAP_NAME" 450 | do_measure "$SNAP_PATH" "$REPORT_PATH" "../$SNAP_NAME" 451 | done 452 | fi 453 | fi 454 | measure_unlock 455 | } 456 | 457 | do_multiple() { 458 | DO_ACTION=$1 459 | shift 460 | for SUBVOL in "$@"; do 461 | # Remove trailing slash 462 | SUBVOL=$(echo "$SUBVOL" | sed 's|\(.\)/$|\1|g') 463 | case "$DO_ACTION" in 464 | 'snapshot') snapshot "$SUBVOL" 465 | ;; 466 | 'prune') if ! which snazzer-prune-candidates >/dev/null; then 467 | echo 'ERROR: snazzer-prune-candidates not found' >&2 468 | exit 10 469 | fi 470 | prune "$SUBVOL" 471 | ;; 472 | 'measure') if ! which snazzer-measure >/dev/null; then 473 | echo 'ERROR: snazzer-measure not found' >&2 474 | exit 10 475 | fi 476 | measure "$SUBVOL" 477 | ;; 478 | 'list-subvolumes') echo "$SUBVOL" 479 | ;; 480 | 'list-snapshots') 481 | if [ "$SUBVOL" = "/" ]; then 482 | $SUDO btrfs subvolume list -o "$SUBVOL" | \ 483 | grep '\.snapshotz' | \ 484 | sed -n "s|.*path .*\.snapshotz/\(.*\)|/.snapshotz/\1|p" 485 | else 486 | MOUNTPOINT_ESC=$(echo "$SUBVOL" | \ 487 | sed 's|[&*$.^\|]|\\&|g') 488 | $SUDO btrfs subvolume list -o "$SUBVOL" | \ 489 | grep '\.snapshotz' | \ 490 | sed -n "s|^.*path .*\.snapshotz/\(.*\)|$MOUNTPOINT_ESC/\.snapshotz/\1|p" 491 | fi 492 | ;; 493 | *) echo "ERROR: invalid cmd '$DO_ACTION'" 494 | exit 1 495 | ;; 496 | esac 497 | done 498 | sync 499 | } 500 | 501 | DO_ACTION="snapshot" 502 | DO_FORCE=0 503 | DRY_RUN=0 504 | DO_ALL=0 505 | 506 | while [ "$(echo "$1" | grep -c "^-" || true)" != 0 ] 507 | do 508 | case "$1" in 509 | -v | --version ) echo "$SNAZZER_VERSION"; exit; ;; 510 | -h | --help ) pod2usage -exit 0 "$0"; exit ;; 511 | --man ) pod2usage -exit 0 -verbose 3 "$0"; exit ;; 512 | --man-roff ) pod2man --release=$SNAZZER_VERSION "$0"; exit ;; 513 | --man-markdown ) 514 | cat <new->filter('$0'); 517 | } 518 | else { 519 | print STDERR "ERROR: --man-markdown requires Pod::Markdown\n\$@\n"; 520 | exit 9; 521 | } 522 | HERE 523 | exit ;; 524 | -p | --prune ) DO_ACTION="prune"; ;; 525 | -m | --measure ) DO_ACTION="measure"; ;; 526 | --list-subvolumes ) DO_ACTION="list-subvolumes"; ;; 527 | --list-snapshots ) DO_ACTION="list-snapshots"; ;; 528 | -f | --force ) DO_FORCE=1; ;; 529 | -d | --dry-run ) DRY_RUN=1; ;; 530 | -a | --all ) DO_ALL=1; ;; 531 | * ) echo "ERROR: Invalid argument '$1'" >&2 ; exit ;; 532 | esac 533 | shift 534 | done 535 | 536 | do_mountpoint() { 537 | MOUNTPOINT=$1 538 | TMP_EXCL=$2 539 | TMP_DIRS=$3 540 | # Remove trailing slash 541 | if [ "$MOUNTPOINT" != "/" ]; then 542 | MOUNTPOINT=$(echo "$MOUNTPOINT" | sed 's|/$||g') 543 | fi 544 | report_subvols_excluded "$MOUNTPOINT" >> "$TMP_EXCL" 545 | assert_mountpoint "$MOUNTPOINT" 546 | SUBVOLS=$(list_subvolumes "$MOUNTPOINT") || exit $? 547 | printf "%s\n" "$SUBVOLS" | while read -r SUBVOL; do 548 | do_multiple "$DO_ACTION" "$SUBVOL" 549 | report_snapshot_dirs "$SUBVOL" >> "$TMP_DIRS" 550 | done 551 | } 552 | 553 | do_mountpoints() { 554 | TMP_EXCL=$(mktemp) 555 | TMP_DIRS=$(mktemp) 556 | if [ "$#" = 0 ]; then 557 | MOUNTPOINTS=$(list_btrfs_mountpoints) || exit $? 558 | printf "%s\n" "$MOUNTPOINTS" | while read -r MOUNTPOINT; do 559 | do_mountpoint "$MOUNTPOINT" "$TMP_EXCL" "$TMP_DIRS" 560 | done 561 | else 562 | for MOUNTPOINT in "$@"; do 563 | do_mountpoint "$MOUNTPOINT" "$TMP_EXCL" "$TMP_DIRS" 564 | done 565 | fi 566 | if [ -s "$TMP_EXCL" ]; then 567 | echo "" >&2 568 | cat "$TMP_EXCL" >&2 569 | fi 570 | if [ -s "$TMP_DIRS" ]; then 571 | echo "" 572 | cat "$TMP_DIRS" 573 | fi 574 | rm "$TMP_EXCL" 575 | rm "$TMP_DIRS" 576 | } 577 | 578 | if [ -z "$1" ] && [ "$DO_ALL" != "1" ]; then 579 | pod2usage -exit 0 "$0" 580 | echo "ERROR: Missing argument" >&2 581 | exit 1 582 | elif [ "$DO_FORCE" = "1" ] && [ "$DRY_RUN" = "1" ]; then 583 | pod2usage -exit 0 "$0" 584 | echo "ERROR: --force and --dry-run are incompatible" >&2 585 | exit 1 586 | elif [ "$DO_ALL" = "1" ]; then 587 | assert_btrfs_tools 588 | if [ -z "$1" ]; then 589 | do_mountpoints 590 | elif [ -n "$1" ] && [ -z "$2" ]; then 591 | do_mountpoints "$1" 592 | else 593 | pod2usage -exit 0 "$0" 594 | echo "ERROR: Extraneous argument '$2'" >&2 595 | exit 1 596 | fi 597 | else 598 | assert_btrfs_tools 599 | do_multiple "$DO_ACTION" "$@" 600 | fi 601 | if [ -n "$MISSING_SUBVOLS_EXCL_FILE" ]; then 602 | rm "$SNAZZER_SUBVOLS_EXCLUDE_FILE"; 603 | fi 604 | 605 | <<__DNE__ 606 | __END__ 607 | =head1 NAME 608 | 609 | snazzer - create read-only C btrfs snapshots, 610 | offers snapshot pruning and measurement 611 | 612 | =head1 SYNOPSIS 613 | 614 | snazzer [--prune|--measure [--force ]] [--dry-run] --all 615 | 616 | snazzer [--prune|--measure [--force ]] [--dry-run] --all [mountpoint] 617 | 618 | snazzer [--prune|--measure [--force ]] [--dry-run] subvol1 [subvol2 [...]] 619 | 620 | =head1 DESCRIPTION 621 | 622 | Examples: 623 | 624 | Snapshot all non-excluded subvols on all mounted btrfs filesystems: 625 | 626 | snazzer --all 627 | 628 | Prune all non-excluded subvols on all mounted btrfs filesystems: 629 | 630 | snazzer --prune --force --all 631 | 632 | Append output of B to 633 | C for all snapshots of all 634 | subvolumes on all mounted btrfs filesytems (slow!): 635 | 636 | snazzer --measure --force --all 637 | 638 | As above, skipping snapshots already measured by this host (recommended): 639 | 640 | snazzer --measure --all 641 | 642 | Print rather than execute commands for snapshotting all non-excluded subvols for 643 | the filesystem mounted at /mnt (including /mnt itself): 644 | 645 | snazzer --dry-run --all /mnt 646 | 647 | Prune only the explicitly named subvols at /srv, /var/log and root: 648 | 649 | snazzer --prune /srv /var/log / 650 | 651 | =head1 OPTIONS 652 | 653 | =over 654 | 655 | =item B<--all> B<[mountpoint]>: act on all subvolumes under mountpoint. If 656 | mountpoint is omitted, B acts on all mounted btrfs filesystems. 657 | 658 | =item B<--prune>: delete rather than create snapshots. Exactly which are no 659 | longer needed is B's role, documented separately 660 | 661 | =item B<--measure>: append output of B to 662 | C By default, only snapshots 663 | which haven't been measured by this hostname are updated - use B<--force> to 664 | measure all snapshots 665 | 666 | =item B<--force>: required for B<--prune> to carry out any pruning operation. 667 | For B<--measure>, this switch overrides the default behaviour of skipping 668 | snapshots already measured by current hostname 669 | 670 | =item B<--list-subvolumes>: list subvolumes that would be acted on 671 | 672 | =item B<--list-snapshots>: list snapshots under subvolumes as above 673 | 674 | =item B<--dry-run>: print rather than execute commands that would be run 675 | 676 | =item B<--help>: Brief help message 677 | 678 | =item B<--version>: Print version 679 | 680 | =item B<--man>: Full documentation 681 | 682 | =item B<--man-roff>: Full documentation as *roff output, Eg: 683 | 684 | snazzer --man-roff | nroff -man 685 | 686 | =item B<--man-markdown>: Full documentation as markdown output, Eg: 687 | 688 | snazzer --man-markdown > snazzer-manpage.md 689 | 690 | =back 691 | 692 | =head1 ENVIRONMENT 693 | 694 | =over 695 | 696 | =item * SNAZZER_SUBVOLS_EXCLUDE_FILE 697 | 698 | Filename of newline separated list of shell glob patterns of subvolume pathnames 699 | which should be excluded from C invocations; compatible with 700 | C<--exclude-from> for B and B. Examples of subvolume patterns to 701 | exclude from regular snapshotting: *secret*, /var/cache, /var/lib/docker/*, 702 | */.snapshots. Note that B C<.snapshotz> is always excluded. 703 | Default: 704 | 705 | SNAZZER_SUBVOLS_EXCLUDE_FILE="/etc/snazzer/exclude.patterns" 706 | 707 | =item * SNAZZER_USE_UTC=1 708 | 709 | For snapshot naming and B output use UTC times of the form 710 | C instead of local time+offset C 711 | 712 | =back 713 | 714 | =head1 BUGS AND LIMITATIONS 715 | 716 | =over 717 | 718 | =item * Snapshot naming 719 | 720 | A choice has been made to mint a single datetime string which is used for all 721 | snapshot names in a given B snapshot invocation, regardless of how long 722 | or at which exact time the snapshotting process takes place for each subvolume. 723 | This makes for consistency across all subvolumes and filesystems, so that 724 | identifying which snapshots were part of a given snapshotting run is possible. 725 | If the actual datetime of the snapshot event is important to you, this is 726 | available from the C command. 727 | 728 | =item * Snapshot access 729 | 730 | By default, the C<.snapshotz> folder is only read- and writable by the owner 731 | of the snapshotted subvolume, group and others have no permission for 732 | anything. This is on purpose, to protect information leaks from the snapshots. 733 | If you need more open access rights, you can always change ownership and 734 | permissions by hand. 735 | 736 | =item * SNAZZER_SUBVOLS_EXCLUDE_FILE is used with grep -f 737 | 738 | A minimal (possibly buggy/incomplete) attempt is made to convert the shell glob 739 | patterns in this file to a regex suitable for grep -f. The assumption is that 740 | the exclude patterns file should only contain "boring" paths. Obvious regex 741 | characters are escaped, however there are likely hostile path glob patterns 742 | which will break things. 743 | 744 | =item * .snapshot_measurements.exclude is a work-around to the btrfs atime bug 745 | 746 | Snapshots may include empty directories under which some other subvol may have 747 | existed in the original, snapshotted subvolume. However, btrfs has a bug where 748 | these empty directories behave differently to empty directories created with 749 | C: atimes always return with the current local time, which is obvioulsy 750 | different from one second to the next. So we have no hope of creating 751 | reproducible shasums or PGP signatures unless those directories are excluded 752 | from our measurements of the snapshot. See also: 753 | L 754 | 755 | =back 756 | 757 | =head1 EXIT STATUS 758 | 759 | B will abort with an error message printed to STDERR and non-zero exit 760 | status under the following conditions: 761 | 762 | =over 763 | 764 | =item 1. invalid arguments 765 | 766 | =item 2. path is not a filesystem mountpoint 767 | 768 | =item 3. one or more paths were not btrfs subvolumes 769 | 770 | =item 4. prune expected /path/to/subvol/.snapshotz directory which was missing 771 | 772 | =item 5. prune expected --dry-run or --force 773 | 774 | =item 6. tried to write a .snapshot_measurements.exclude file in the snapshot 775 | root, but it already exists in the current subvolume 776 | 777 | =item 7. tried to perform snapshot measurements while existing measurements are 778 | already in progress, check lock dir at /var/run/snazzer-measure.lock 779 | 780 | =item 9. tried to display man page with a formatter which is not installed 781 | 782 | =item 10. missing C or C from PATH 783 | 784 | =item 11. missing C command from PATH 785 | 786 | =item 12. syntax error in /etc/snazzer/exclude.patterns file. 787 | 788 | =back 789 | 790 | =head1 SEE ALSO 791 | 792 | snazzer-measure, snazzer-prune-candidates, snazzer-receive 793 | 794 | =head1 AUTHOR 795 | 796 | Snazzer Authors are listed in the AUTHORS.md file in the root of this 797 | distribution. See https://github.com/csirac2/snazzer for more information. 798 | NOTE: Please extend that file, not this notice. 799 | 800 | =head1 LICENSE AND COPYRIGHT 801 | 802 | Copyright (C) 2015-2016, Snazzer Authors All rights reserved. 803 | 804 | Redistribution and use in source and binary forms, with or without 805 | modification, are permitted provided that the following conditions are met: 806 | 807 | 1. Redistributions of source code must retain the above copyright notice, this 808 | list of conditions and the following disclaimer. 809 | 810 | 2. Redistributions in binary form must reproduce the above copyright notice, 811 | this list of conditions and the following disclaimer in the documentation 812 | and/or other materials provided with the distribution. 813 | 814 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 815 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 816 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 817 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 818 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 819 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 820 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 821 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 822 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 823 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 824 | =cut 825 | __DNE__ 826 | -------------------------------------------------------------------------------- /snazzer-receive: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | set -e 3 | SNAZZER_VERSION=0.0.3 4 | SNAZZER_SNAPSHOTZ_PERMS=0755 5 | 6 | if [ "$(id -u)" = "0" ]; then 7 | SUDO="" 8 | else 9 | SUDO="sudo" 10 | fi 11 | 12 | squote_args() { 13 | while [ -n "$1" ]; do 14 | printf "'"; printf '%s' "$1" | sed "s|'|'\\\\''|g"; printf "' " 15 | shift 16 | done 17 | } 18 | 19 | check_ssh_err() { 20 | HOST=$1 21 | CMD=$2 22 | STDERR=$3 23 | RC=$4 24 | 25 | if [ -s "$STDERR" ] && grep '^sudo:' "$STDERR"; then 26 | cat <&2 27 | 28 | ERROR running: 29 | ssh "$HOST" "$CMD" 30 | $(cat "$STDERR") 31 | 32 | Is the sudo invocation above permitted to run passwordless on this host? Check 33 | sudoers config against the snazzer documentation (hint: snazzer-receive --man) 34 | HERE 35 | exit 12 36 | elif [ "$RC" = "0" ]; then 37 | cat "$STDERR" >&2 38 | else 39 | cat <&2 40 | 41 | ERROR running: 42 | ssh "$HOST" "$CMD" 43 | $(cat "$STDERR") 44 | HERE 45 | exit "$RC" 46 | fi 47 | } 48 | 49 | do_ssh() { 50 | HOST=$1 51 | CMD=$2 52 | STDERR=$(mktemp) 53 | RC=0 54 | 55 | ssh "$HOST" "$CMD" 2>"$STDERR" || RC=$? 56 | check_ssh_err "$HOST" "$CMD" "$STDERR" "$RC" 57 | 58 | rm "$STDERR" 59 | } 60 | 61 | dry_list_snapshots() { 62 | HOST=$1 63 | shift 64 | if [ "$LOCAL_RUN" = "1" ]; then 65 | cat < on $HOST at " \ 94 | "$SUBVOL/.snapshotz/.measurements/" || true) 95 | else 96 | # Beware whitespace in do_ssh arg affects send-wrapper sanity checks 97 | MEAS_LIST=$(do_ssh "$HOST" "sudo -n grep -srl '^> on $HOST_ESC at ' \ 98 | '$SUBVOL_ESC/.snapshotz/.measurements/'" || ( \ 99 | RC=$? 100 | if [ "$RC" != "1" ]; then 101 | exit "$RC" 102 | fi)) 103 | fi 104 | echo "$MEAS_LIST" | sed "s|^.*/||g" 105 | } 106 | 107 | dry_send() { 108 | HOST=$1 109 | SNAP_ABSPATH=$2 110 | SNAP_ABSPATH_ESC=$(echo "$SNAP_ABSPATH" | sed "s|'|'\\\\''|g") 111 | SNAP_PARENT_PATH=$3 112 | SNAP_PARENT_PATH_ESC=$(echo "$SNAP_PARENT_PATH" | sed "s|'|'\\\\''|g") 113 | DEST_SUBVOL=$(echo "$SNAP_ABSPATH" | sed 's|^/*\(.*\)/\.snapshotz/.*|\1|g') 114 | DEST_SUBVOL_ESC=$(echo "$SNAP_ABSPATH_ESC" | sed 's|^/*\(.*\)/\.snapshotz/.*|\1|g') 115 | if [ -z "$DEST_SUBVOL" ]; then 116 | DEST_SUBVOL="." 117 | DEST_SUBVOL_ESC="." 118 | fi 119 | SNAP_NAME=$(basename "$SNAP_ABSPATH") 120 | # Don't forget to escape \$(mktemp) and delete usage of \$SEND_ERR! 121 | # Don't forget to use _ESC paths! 122 | cat </dev/null 151 | $SUDO btrfs subvolume delete \ 152 | '$DEST_SUBVOL_ESC/.snapshotz/.incomplete/$SNAP_NAME' >/dev/null 153 | $SUDO rmdir '$DEST_SUBVOL_ESC/.snapshotz/.incomplete' 154 | CMD 155 | } 156 | 157 | do_send() { 158 | # Don't forget to copy the lines below verbatim into dry_send() 159 | HOST=$1 160 | SNAP_ABSPATH=$2 161 | SNAP_ABSPATH_ESC=$(echo "$SNAP_ABSPATH" | sed "s|'|'\\\\''|g") 162 | SNAP_PARENT_PATH=$3 163 | SNAP_PARENT_PATH_ESC=$(echo "$SNAP_PARENT_PATH" | sed "s|'|'\\\\''|g") 164 | DEST_SUBVOL=$(echo "$SNAP_ABSPATH" | sed 's|^/*\(.*\)/\.snapshotz/.*|\1|g') 165 | DEST_SUBVOL_ESC=$(echo "$SNAP_ABSPATH_ESC" | sed 's|^/*\(.*\)/\.snapshotz/.*|\1|g') 166 | if [ -z "$DEST_SUBVOL" ]; then 167 | DEST_SUBVOL="." 168 | DEST_SUBVOL_ESC="." 169 | fi 170 | SNAP_NAME=$(basename "$SNAP_ABSPATH") 171 | # Can't use do_ssh below, we must stream ssh output to btrfs receive 172 | # Don't forget to copy the lines below into dry_send() between CMD heredoc 173 | $SUDO mkdir --mode="$SNAZZER_SNAPSHOTZ_PERMS" \ 174 | "$DEST_SUBVOL/.snapshotz/.incomplete" 175 | if [ "$LOCAL_RUN" = "1" ]; then 176 | if [ -n "$SNAP_PARENT_PATH" ]; then 177 | # Don't forget to copy the lines below into dry_send() between CMD2 heredoc 178 | $SUDO btrfs send -p "$SNAP_PARENT_PATH" "$SNAP_ABSPATH" | \ 179 | $SUDO btrfs receive "$DEST_SUBVOL/.snapshotz/.incomplete" 180 | else 181 | # Don't forget to copy the lines below into dry_send() between CMD3 heredoc 182 | $SUDO btrfs send "$SNAP_ABSPATH" | \ 183 | $SUDO btrfs receive "$DEST_SUBVOL/.snapshotz/.incomplete" 184 | fi 185 | else 186 | # Don't forget to copy the lines below into dry_send() between CMD4 heredoc 187 | if [ -n "$SNAP_PARENT_PATH" ]; then 188 | CMD="sudo -n btrfs send '-p' '$SNAP_PARENT_PATH_ESC' '$SNAP_ABSPATH_ESC'" 189 | else 190 | CMD="sudo -n btrfs send '$SNAP_ABSPATH_ESC'" 191 | fi 192 | SEND_ERR=$(mktemp) 193 | (ssh "$HOST" "$CMD" 2>"$SEND_ERR" || \ 194 | check_ssh_err "$HOST" "$CMD" "$SEND_ERR" "$?") | \ 195 | $SUDO btrfs receive "$DEST_SUBVOL/.snapshotz/.incomplete" 196 | rm "$SEND_ERR" 197 | fi 198 | # Don't forget to copy the lines below into dry_send() between CMD heredoc 199 | $SUDO btrfs subvolume snapshot -r \ 200 | "$DEST_SUBVOL/.snapshotz/.incomplete/$SNAP_NAME" \ 201 | "$DEST_SUBVOL/.snapshotz/" >/dev/null 202 | $SUDO btrfs subvolume delete \ 203 | "$DEST_SUBVOL/.snapshotz/.incomplete/$SNAP_NAME" >/dev/null 204 | $SUDO rmdir "$DEST_SUBVOL/.snapshotz/.incomplete" 205 | } 206 | 207 | dry_copy() { 208 | HOST=$1 209 | SUBVOL_REMOTEPATH=$2 210 | SUBVOL_REMOTEPATH_ESC=$(echo "$SUBVOL_REMOTEPATH" | sed "s|'|'\\\\''|g") 211 | SUBVOL_LOCALPATH=$3 212 | SNAP_NAME=$4 213 | # Don't forget to escape \$(mktemp), \$MEAS_TMP, \$CMD! 214 | cat <"\$MEAS_TMP" 222 | CMD2 223 | else 224 | cat <"\$MEAS_TMP" 227 | CMD3 228 | fi) 229 | $SUDO tee -a "$SUBVOL_LOCALPATH/.snapshotz/.measurements/$SNAP_NAME" \ 230 | <"\$MEAS_TMP" >/dev/null 231 | rm "\$MEAS_TMP" 232 | CMD 233 | } 234 | 235 | do_copy() { 236 | # Don't forget to copy the lines below verbatim into dry_copy() 237 | HOST=$1 238 | SUBVOL_REMOTEPATH=$2 239 | SUBVOL_REMOTEPATH_ESC=$(echo "$SUBVOL_REMOTEPATH" | sed "s|'|'\\\\''|g") 240 | SUBVOL_LOCALPATH=$3 241 | SNAP_NAME=$4 242 | # We're using tee below to avoid giving sudo for cp 243 | # Don't forget to copy the lines below verbatim into dry_copy() between CMD 244 | $SUDO mkdir -vp --mode="$SNAZZER_SNAPSHOTZ_PERMS" \ 245 | "$SUBVOL_REMOTEPATH/.snapshotz/.measurements" 246 | MEAS_TMP=$(mktemp) 247 | if [ "$LOCAL_RUN" = "1" ]; then 248 | # Don't forget to copy the lines below verbatim into dry_copy() between CMD2 249 | $SUDO cat "$SUBVOL_REMOTEPATH/.snapshotz/.measurements/$SNAP_NAME" \ 250 | >"$MEAS_TMP" 251 | else 252 | # Don't forget to copy the lines below verbatim into dry_copy() between CMD3 253 | CMD="cat '$SUBVOL_REMOTEPATH_ESC/.snapshotz/.measurements/$SNAP_NAME'" 254 | do_ssh "$HOST" "sudo -n $CMD" >"$MEAS_TMP" 255 | fi 256 | # Don't forget to copy the lines below verbatim into dry_copy() between CMD 257 | $SUDO tee -a "$SUBVOL_LOCALPATH/.snapshotz/.measurements/$SNAP_NAME" \ 258 | <"$MEAS_TMP" >/dev/null 259 | rm "$MEAS_TMP" 260 | } 261 | 262 | create_subvol() { 263 | DIR="$1" 264 | PARENT=$(dirname "$DIR") 265 | if [ "$DIR" != "$PARENT" ] && [ "$PARENT" != "." ]; then 266 | $SUDO mkdir -vp "$PARENT" 267 | fi 268 | $SUDO btrfs subvolume create "$DIR" 269 | $SUDO mkdir --mode="$SNAZZER_SNAPSHOTZ_PERMS" -vp "$DIR/.snapshotz" 270 | } 271 | 272 | is_subvol() { 273 | DIR=$1 274 | 275 | if $SUDO btrfs subvolume show "$DIR" >/dev/null 2>&1; then 276 | echo 1 277 | else 278 | echo 0 279 | fi 280 | } 281 | 282 | dispatch_copy() { 283 | HOST=$1 284 | SUBVOL_REMOTEPATH=$2 285 | SUBVOL_LOCALPATH=$3 286 | SNAP_NAME=$4 287 | if [ "$DRY_RUN" != "0" ]; then 288 | dry_copy "$HOST" "$SUBVOL_REMOTEPATH" "$SUBVOL_LOCALPATH" "$SNAP_NAME" 289 | else 290 | do_copy "$HOST" "$SUBVOL_REMOTEPATH" "$SUBVOL_LOCALPATH" "$SNAP_NAME" 291 | fi 292 | } 293 | 294 | do_subvolume_measurements() { 295 | HOST="$1" 296 | SUBVOL="$2" 297 | if [ -z "$SUBVOL" ]; then 298 | SUBVOL_RELPATH="." 299 | else 300 | SUBVOL_RELPATH="$SUBVOL" 301 | fi 302 | SUBVOL_RELPATH_ESC=$(echo "$SUBVOL_RELPATH" | sed "s|'|'\\\\''|g") 303 | printf " appending measurements..." 304 | if $SUDO test -e "$SUBVOL_RELPATH/.snapshotz/.incomplete"; then 305 | cat <&2 306 | 307 | ERROR: $SUBVOL_RELPATH/.snapshotz/.incomplete exists. Another instance is already 308 | running, or a previous invocation was interrupted. If you are sure no other 309 | instances are already running, remove this directory and snapshots under it: 310 | 311 | btrfs subvolume delete '$SUBVOL_RELPATH_ESC/.snapshotz/.incomplete'/* 312 | rmdir '$SUBVOL_RELPATH_ESC/.snapshotz/.incomplete' 313 | HERE 314 | exit 2 315 | fi 316 | if [ "$(is_subvol "$SUBVOL_RELPATH")" != "1" ]; then 317 | echo "ERROR: $SUBVOL_RELPATH not a subvolume, this shouldn't happen." 318 | exit 319 | fi 320 | if ! $SUDO test -e "$SUBVOL_RELPATH/.snapshotz/.measurements"; then 321 | $SUDO mkdir --mode="$SNAZZER_SNAPSHOTZ_PERMS" \ 322 | "$SUBVOL_RELPATH/.snapshotz/.measurements" 323 | fi 324 | SNAPSHOTS=$($SUDO ls "$SUBVOL_RELPATH/.snapshotz") 325 | MEAS_WANT=$(mktemp) 326 | MEAS_ABSENT=$($SUDO grep -srL "^> on $HOST at " \ 327 | "$SUBVOL_RELPATH/.snapshotz/.measurements/" | \ 328 | sed -n 's|^.*/\([^/]*\)|\1|p') 329 | MEAS_REMOTE=$(list_remote_snapshot_measurements "$HOST" "/$SUBVOL_RELPATH") 330 | # 1. Print snapshot measurements that we have locally but are missing 331 | # mentions of remote $HOST; 332 | # 2. Print snapshot measurements available on the remote $HOST 333 | # 3. Write the union of these two lists (duplicates only) as a list of the 334 | # snapshot names which we should grab remote $HOST's measurements of 335 | if [ -n "$MEAS_ABSENT" ] && [ -n "$MEAS_REMOTE" ]; then 336 | printf "%s\n%s" "$MEAS_ABSENT" "$MEAS_REMOTE" | sort | uniq -d \ 337 | > "$MEAS_WANT" 338 | else 339 | printf "%s%s" "$MEAS_ABSENT" "$MEAS_REMOTE" | sort | uniq -d \ 340 | > "$MEAS_WANT" 341 | fi 342 | NUM_WANT=$(wc -l "$MEAS_WANT" | cut -d ' ' -f 1) 343 | NUM_SNAP=$(echo "$SNAPSHOTS" | wc -l | cut -d ' ' -f 1) 344 | NUM_COPY=0 345 | NUM_REMOTE=$(echo "$MEAS_REMOTE" | wc -l | cut -d ' ' -f 1) 346 | while read -r SNAP_NAME <&4 347 | do 348 | dispatch_copy "$HOST" "/$SUBVOL_RELPATH" "$SUBVOL_RELPATH" "$SNAP_NAME" 349 | NUM_COPY=$((NUM_COPY + 1)) 350 | done 4<"$MEAS_WANT" 351 | echo "$MEAS_REMOTE" > "$MEAS_WANT" 352 | while read -r SNAP_NAME <&4 353 | do 354 | if $SUDO test -e "$SUBVOL_RELPATH/.snapshotz/$SNAP_NAME" && \ 355 | ! $SUDO test -e \ 356 | "$SUBVOL_RELPATH/.snapshotz/.measurements/$SNAP_NAME"; then 357 | dispatch_copy "$HOST" "/$SUBVOL_RELPATH" "$SUBVOL_RELPATH" "$SNAP_NAME" 358 | NUM_COPY=$((NUM_COPY + 1)) 359 | fi 360 | done 4<"$MEAS_WANT" 361 | rm "$MEAS_WANT" 362 | echo " $NUM_COPY of $NUM_REMOTE." 363 | } 364 | 365 | do_subvolume() { 366 | HOST=$1 367 | SUBVOL=$2 368 | if [ -z "$SUBVOL" ]; then 369 | SUBVOL="." 370 | fi 371 | SUBVOL_ESC=$(echo "$SUBVOL" | sed "s|'|'\\\\''|g") 372 | echo "subvolume $SUBVOL:" 373 | if $SUDO test -e "$SUBVOL/.snapshotz/.incomplete"; then 374 | cat <&2 375 | 376 | ERROR: $SUBVOL/.snapshotz/.incomplete exists. Another instance is already 377 | running, or a previous invocation was interrupted. If you are sure no other 378 | instances are already running, remove this directory and snapshots under it: 379 | 380 | btrfs subvolume delete '$SUBVOL_ESC/.snapshotz/.incomplete'/* 381 | rmdir '$SUBVOL_ESC/.snapshotz/.incomplete' 382 | HERE 383 | exit 2 384 | fi 385 | if [ "$(is_subvol "$SUBVOL")" != "1" ]; then 386 | create_subvol "$SUBVOL" 387 | fi 388 | if ! $SUDO test -e "$SUBVOL/.snapshotz"; then 389 | $SUDO mkdir --mode="$SNAZZER_SNAPSHOTZ_PERMS" \ 390 | "$SUBVOL/.snapshotz" 391 | fi 392 | # Last snapshot seen in the local target fs 393 | PREV_SNAP= 394 | if [ "$SUBVOL" = "." ]; then 395 | SNAPSHOTS=$(do_list_snapshots "$HOST" "/") 396 | else 397 | SNAPSHOTS=$(do_list_snapshots "$HOST" "/$SUBVOL") 398 | fi 399 | SNAP_WANT=$(mktemp) 400 | echo "$SNAPSHOTS" | snazzer-prune-candidates --invert > "$SNAP_WANT" 401 | NUM_WANT=$(wc -l "$SNAP_WANT" | cut -d ' ' -f 1) 402 | NUM_SNAP=$(echo "$SNAPSHOTS" | wc -l | cut -d ' ' -f 1) 403 | NUM_PRUN=$((NUM_SNAP - NUM_WANT)) 404 | NUM_RECV=0 405 | NUM_SKIP=0 406 | while read -r SNAPSHOT <&4 407 | do 408 | SNAP_RELPATH=$(echo "$SNAPSHOT" | sed 's|^/*||g') 409 | if $SUDO test -e "$SNAP_RELPATH"; then 410 | NUM_SKIP=$((NUM_SKIP + 1)) 411 | else 412 | NUM_RECV=$((NUM_RECV + 1)) 413 | if [ "$DRY_RUN" != "0" ]; then 414 | dry_send \ 415 | "$HOST" "$SNAPSHOT" "$PREV_SNAP" 416 | else 417 | do_send \ 418 | "$HOST" "$SNAPSHOT" "$PREV_SNAP" 419 | fi 420 | fi 421 | PREV_SNAP="$SNAPSHOT" 422 | done 4<"$SNAP_WANT" 423 | rm "$SNAP_WANT" 424 | printf " %s of %s snapshots received " "$NUM_RECV" "$NUM_SNAP" 425 | echo "($NUM_PRUN pruned, $NUM_WANT considered, $NUM_SKIP skipped)" 426 | } 427 | 428 | #SMELL: Assumes --list-snapshots lines are grouped by subvol, ordered by date 429 | #FIXME: Subvols containing mixed timezone snapshots will use suboptimal parents 430 | do_host() { 431 | HOST=$1 432 | if [ "$LOCAL_RUN" = "1" ]; then 433 | HOST=$(hostname) 434 | elif [ "$HOST" = "--" ]; then 435 | echo "ERROR: This should never happen" >&2 436 | exit 127 437 | fi 438 | NUM_SUBVOL=0 439 | LIST_ERR=$(mktemp) 440 | LIST_OUT=$(mktemp) 441 | shift 442 | # This is messy because we want to capture stderr but at the same time still 443 | # emit it should do_list_snapshots actually exit with a non-zero return code 444 | LIST=$( (do_list_snapshots "$HOST" "$@" 2>"$LIST_ERR") || \ 445 | (RC=$$; cat "$LIST_ERR" >&2; exit "$RC" ) ) 446 | echo "$LIST" | \ 447 | sed 's|^/*\(.*\)/\.snapshotz/.*|\1|g' | sort | uniq > "$LIST_OUT" 448 | NUM_SUBVOL=$(wc -l "$LIST_OUT" | cut -d ' ' -f 1) 449 | while read -r SUBVOL <&5 450 | do 451 | do_subvolume "$HOST" "$SUBVOL" 452 | do_subvolume_measurements "$HOST" "$SUBVOL" 453 | done 5<"$LIST_OUT" 454 | printf "Processed %s subvolume(s)." "$NUM_SUBVOL" 455 | if [ -s "$LIST_ERR" ]; then 456 | echo " Additional output from:" 457 | printf " " 458 | dry_list_snapshots "$HOST" "$@" 459 | grep . "$LIST_ERR" | sed 's/^/ /g' 460 | else 461 | echo "" 462 | fi 463 | rm "$LIST_OUT" 464 | rm "$LIST_ERR" 465 | } 466 | 467 | DRY_RUN=0 468 | LOCAL_RUN=0 469 | 470 | while [ "$(echo "$1" | grep -c "^-")" != "0" ] && [ "$LOCAL_RUN" != "1" ] 471 | do 472 | case "$1" in 473 | -v | --version ) 474 | echo "$SNAZZER_VERSION" 475 | exit 476 | ;; 477 | -h | --help ) 478 | pod2usage -exit 0 "$0" 479 | exit 480 | ;; 481 | --man ) 482 | pod2usage -exit 0 -verbose 3 "$0" 483 | exit 484 | ;; 485 | --man-roff ) 486 | pod2man --release=$SNAZZER_VERSION "$0" 487 | exit 488 | ;; 489 | --man-markdown ) 490 | cat <new->filter('$0'); 493 | } 494 | else { 495 | print STDERR "ERROR: --man-markdown requires Pod::Markdown\n\$@\n"; 496 | exit 9; 497 | } 498 | HERE 499 | exit 500 | ;; 501 | -d | --dry-run ) 502 | DRY_RUN=1; 503 | shift 504 | ;; 505 | -- ) 506 | LOCAL_RUN=1 507 | ;; 508 | * ) 509 | echo "ERROR: Invalid argument '$1'" >&2 510 | exit 511 | ;; 512 | esac 513 | done 514 | 515 | if [ -z "$1" ]; then 516 | pod2usage -exit 0 "$0" 517 | echo "ERROR: Missing argument" >&2 518 | exit 1 519 | fi 520 | 521 | do_host "$@" 522 | 523 | <<__DNE__ 524 | __END__ 525 | =head1 NAME 526 | 527 | snazzer-receive - receive remote snazzer snapshots to current working dir 528 | 529 | =head1 SYNOPSIS 530 | 531 | Receive snapshots from remote host via ssh: 532 | 533 | snazzer-receive [--dry-run] host --all [/path/to/btrfs/mountpoint] 534 | 535 | snazzer-receive [--dry-run] host [/remote/subvol1 [/subvol2 [..]]] 536 | 537 | Receive snapshots from local filesystem: 538 | 539 | snazzer-receive [--dry-run] -- --all [/path/to/btrfs/mountpoint] 540 | 541 | snazzer-receive [--dry-run] -- /local/subvol1 [/subvol2 [..]] 542 | 543 | =head1 DESCRIPTION 544 | 545 | First, B obtains a list of snapshots to be received by running 546 | C, where [args] are all B 547 | arguments after the hostname or C<--> separator argument. 548 | 549 | If the first non-option positional argument is C<-->, 550 | C is executed locally and [args] will refer to 551 | local filesystem paths. Otherwise, it is taken to mean an ssh hostname which is 552 | used to run the C command remotely, and [args] 553 | will refer to paths on that remote host. 554 | 555 | B then iterates through this list of snapshots recreating a 556 | filesystem similar to the source by creating subvolumes and C<.snapshotz> 557 | directories where necessary. Missing snapshots are instantiated directly with 558 | C and C, using C where 559 | possible to reduce transport overhead of incremental snapshots. 560 | 561 | B never deletes any snapshots from the current working dir, 562 | even if the snapshots were e.g. pruned from the source. 563 | This can be used to build a flexible, multi-tiered backup strategy with 564 | different settings of how many snapshots to keep, but means that 565 | C needs to be run on the current working directory as well for 566 | old snapshots to be deleted. 567 | 568 | 569 | Rather than offer ssh user/port/host specifications through B, 570 | it is assumed all remote hosts are properly configured through your ssh config 571 | file usually at C<$HOME/.ssh/config>. 572 | 573 | =head1 OPTIONS 574 | 575 | =over 576 | 577 | =item B<--dry-run>: print rather than execute commands that would be run 578 | 579 | =item B<--help>: Brief help message 580 | 581 | =item B<--version>: Print version 582 | 583 | =item B<--man>: Full documentation 584 | 585 | =item B<--man-roff>: Full documentation as *roff output, Eg: 586 | 587 | snazzer-receive --man-roff | nroff -man 588 | 589 | =item B<--man-markdown>: Full documentation as markdown output, Eg: 590 | 591 | snazzer-receive --man-markdown > snazzer-manpage.md 592 | 593 | =back 594 | 595 | =head1 ENVIRONMENT 596 | 597 | =head2 sudo requirements for sender/remote hosts 598 | 599 | B assumes the ssh user (or local user, if receiving a local 600 | filesystem) which will be running C (among other things) has 601 | passwordless sudo for the commands it needs to run. Only a few commands are 602 | necessary, the following lines in C or C 603 | should suffice (replace "sendinguser" with the actual username you will use): 604 | 605 | sendinguser ALL=(root:nobody) NOPASSWD: /usr/bin/snazzer --list-snapshots * 606 | sendinguser ALL=(root:nobody) NOPASSWD:NOEXEC: \ 607 | /bin/grep -srl */.snapshotz/.measurements/, \ 608 | /sbin/btrfs send */.snapshotz/*, \ 609 | /bin/cat */.snapshotz/.measurements/* 610 | 611 | =head2 sudo and cron user requirements for receiving hosts 612 | 613 | For interactive use of B, a typical user with full sudo 614 | permissions should work out of the box. 615 | 616 | For scripted use such as a cron job, or interactive use in more restrictive 617 | environments - running ssh as the root user is generally considered a bad idea. 618 | A dedicated non-root user will require at minimum the following lines in 619 | C or C (replace "receiveruser" with the 620 | actual username your cron job will use, and remove C if this is for 621 | an interactive/shell user): 622 | 623 | receiveruser ALL=(root:nobody) NOPASSWD:NOEXEC: \ 624 | /usr/bin/test -e */.snapshotz*, \ 625 | /sbin/btrfs subvolume show *, \ 626 | /bin/ls */.snapshotz, \ 627 | /bin/grep -srL */.snapshotz/.measurements/, \ 628 | /bin/mkdir --mode=0755 */.snapshotz, \ 629 | /bin/mkdir --mode=0755 */.snapshotz/.measurements, \ 630 | /bin/mkdir --mode=0755 */.snapshotz/.incomplete, \ 631 | /sbin/btrfs receive */.snapshotz/.incomplete, \ 632 | /sbin/btrfs subvolume create *, \ 633 | /sbin/btrfs subvolume snapshot -r */.snapshotz/.incomplete/* */.snapshotz/,\ 634 | /sbin/btrfs subvolume delete */.snapshotz/.incomplete/*, \ 635 | /bin/rmdir */.snapshotz/.incomplete, \ 636 | /bin/mkdir -vp *, \ 637 | /bin/mkdir --mode=0755 -vp */.snapshotz, \ 638 | /usr/bin/tee -a */.snapshotz/.measurements/* 639 | 640 | 641 | =head1 SECURITY CONSIDERATIONS 642 | 643 | =head2 Remote hosts 644 | 645 | B relies on running ssh remote commands. It is agnostic about 646 | the auth method used, but this documentation assumes key-based. 647 | 648 | Combined with passwordless sudo, remote hosts are vulnerable to and must have 649 | absolute trust in the ssh key-holder, user and host running B. 650 | 651 | Your deployment should include or consider the following steps, among others not 652 | listed here, to attempt to reduce the impact of or slow down an attacker which 653 | has gained control of the B user accounts or ssh keys: 654 | 655 | =over 656 | 657 | =item * Protect ssh keys 658 | 659 | The ssh key used to authenticate B typically has passwordless 660 | sudo for C (among other things) and you should assume that whomever 661 | wields it has access to everything: 662 | 663 | =over 664 | 665 | =item * Avoid passphraseless ssh keyfiles 666 | 667 | This should be obvious: once an attacker has copied such a keyfile they no 668 | longer need the compromised host to authenticate, and you will have a bigger, 669 | more urgent job searching for malicious use (and key removal from machines 670 | which trusted it). 671 | 672 | =item * Avoid ssh private keyfiles 673 | 674 | Even passphrase-protected keyfiles are vulnerable to keyloggers and 675 | memory scraping. Consider using smartcards, TPMs, Yubikeys or GoldKeys etc. to 676 | at least force an attacker to depend on whichever machine has the authentication 677 | device attached. 678 | 679 | This is especially important when passphrase-protected keyfiles are not 680 | practical (eg. scripted use of B such as cron). 681 | 682 | =item * Use the timeout option if using an ssh-agent 683 | 684 | =back 685 | 686 | =item * Grant minimal sudo rights 687 | 688 | Refer to "sudo requirements for remote hosts". Don't give the B 689 | user the option to run arbitrary commands remotely as root. 690 | 691 | =item * C<~/.ssh/authorized_keys>: specify a forced-command/shell-wrapper 692 | 693 | Even if sudo is locked down, don't give the B user the option 694 | of running arbitrary commands remotely. Use a shell wrapper which permits only 695 | the required sudo commands. 696 | TODO: provide example 697 | TODO: Document shell wrapper 698 | 699 | NOTE: This does not prevent data exfiltration via C, but 700 | may slow down an attacker who would abuse the account in other ways. 701 | 702 | =item * C<~/.ssh/authorized_keys>: restrict originating IP address 703 | 704 | Use the C option to limit which machine the B host's ssh 705 | key may connect from. This might force an attacker to still depend on the 706 | B host even if they have obtained the private key somehow. 707 | TODO: provide example 708 | TODO: link to a guide on this 709 | 710 | =item * Disable interactive shells/logins 711 | 712 | Reduce opportunities for the B user to run arbitrary commands; 713 | remove the account password. NOTE: this doesn't stop ssh remote commands. 714 | TODO: link to a guide on this 715 | 716 | =item * Log remote ssh commands 717 | 718 | Most distros do zero logging of remote ssh commands. Logging such commands may 719 | be your only way to spot abuse of the B account. The 720 | C uses C to log commands on 721 | remote hosts which are invoking C. 722 | TODO: link to a guide on this 723 | 724 | =back 725 | 726 | =head1 BUGS AND LIMITATIONS 727 | 728 | B B tries to recreate a filesystem similar to that of 729 | the remote host, starting at the current working directory which represents the 730 | root filesystem. If the remote host has a root btrfs filesystem, this means that 731 | the current working directory should itself also be a btrfs subvolume in order 732 | to receive snapshots under ./.snapshotz. However, B will be 733 | unable to replace the current working directory with a btrfs subvolume if it 734 | isn't already one. 735 | 736 | Therefore, if required, ensure the current working directory is already a btrfs 737 | subvolume prior to running B if you need to receive btrfs root. 738 | 739 | =head1 EXIT STATUS 740 | 741 | B will abort with an error message printed to STDERR and 742 | non-zero exit status under the following conditions: 743 | 744 | =over 745 | 746 | =item 1. invalid arguments 747 | 748 | =item 2. C<.snapshotz/.incomplete> already exists at a given destination subvolume 749 | 750 | =item 9. tried to display man page with a formatter which is not installed 751 | 752 | =item 12. remote ssh sudo command failed 753 | 754 | =back 755 | 756 | =head1 TODO 757 | 758 | =over 759 | 760 | =item 1. improve fetch/append of remote host's measurements 761 | 762 | B currently does some clumsy concatenation of the remote host's 763 | measurement file onto the local measurement file for a given snapshot if the 764 | local measurement file is either missing or does not mention that remote host's 765 | hostname. Whilst this supports the simple use-case of wanting to obtain initial 766 | measurements performed on a remote host, once a remote host's measurements have 767 | been appended there is no attempt to append any further measurement results onto 768 | the local measurements file. If this bothers you, please report detailed 769 | use-cases to the author (patches welcome). 770 | 771 | =item 2. include restricted wrapper script to be used as ssh forced command 772 | 773 | The snazzer project assumes that systems administrators would prefer to restrict 774 | the possible exposure of a dedicated snazzer remote user account, even if sudo 775 | is locked down. To that end, a wrapper script shall be provided which restricts 776 | possible ssh remote commands to only the few actually necessary for snazzer 777 | operation. 778 | 779 | Even so, commands which snazzer relies on such as C are 780 | extremely dangerous no matter if it's the only command allowed by the system - 781 | securing ssh keys is of utmost importance; consider protecting ssh keys with 782 | smartcards, TPM, hardware OTP solution such as Yubi/GoldKeys etc. 783 | 784 | =back 785 | 786 | =head1 SEE ALSO 787 | 788 | snazzer, snazzer-measure, snazzer-prune-candidates 789 | 790 | =head1 AUTHOR 791 | 792 | Snazzer Authors are listed in the AUTHORS.md file in the root of this 793 | distribution. See https://github.com/csirac2/snazzer for more information. 794 | NOTE: Please extend that file, not this notice. 795 | 796 | =head1 LICENSE AND COPYRIGHT 797 | 798 | Copyright (C) 2015-2016, Snazzer Authors All rights reserved. 799 | 800 | Redistribution and use in source and binary forms, with or without 801 | modification, are permitted provided that the following conditions are met: 802 | 803 | 1. Redistributions of source code must retain the above copyright notice, this 804 | list of conditions and the following disclaimer. 805 | 806 | 2. Redistributions in binary form must reproduce the above copyright notice, 807 | this list of conditions and the following disclaimer in the documentation 808 | and/or other materials provided with the distribution. 809 | 810 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND 811 | ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED 812 | WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 813 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 814 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 815 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 816 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 817 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 818 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 819 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 820 | =cut 821 | __DNE__ 822 | --------------------------------------------------------------------------------