├── README.md ├── bkp-env.sh ├── bkp-mount.sh ├── bkp-openwrt.sh ├── bkp-prune.sh ├── bkp-pull.sh ├── bkp-status.sh ├── bkp-stream.sh └── bkp.sh /README.md: -------------------------------------------------------------------------------- 1 | # Restic backup 2 | 3 | ### Usage 4 | 5 | `bkp.sh` - execute restic backup according to configuration 6 | 7 | This script executes `sudo restic`, so it is a good idea to add NOPASSWD rule to sudoers configuration: 8 | 9 | Cmnd_Alias BACKUP = /usr/bin/restic, /snap/bin/restic, /usr/local/bin/restic 10 | Defaults!BACKUP env_keep += "RESTIC_PASSWORD RESTIC_REPOSITORY" 11 | %wheel ALL=(ALL) NOPASSWD: BACKUP 12 | 13 | To install `restic` on remote linux amd64 machine, run: 14 | 15 | curl -s https://api.github.com/repos/restic/restic/releases/latest | grep 'browser_download_url.*\.bz2"' | grep amd64 | grep linux | cut -d : -f 2,3 | tr -d \" | xargs -n 1 curl -sSL > /tmp/restic.bz2 && sudo sh -c 'bzip2 -dc /tmp/restic.bz2 > /usr/local/bin/restic && chmod +x /usr/local/bin/restic' 16 | 17 | # Restic pull backup 18 | 19 | ### Usage: 20 | 21 | `bkp-pull.sh --help` - help and usage 22 | 23 | `bkp-pull.sh host` - open ssh connection to a host and execute restic remotely 24 | 25 | This script executes `sudo restic` remotely, so it is a good idea to add NOPASSWD rule to sudoers configuration. 26 | 27 | ### How does it work? 28 | 29 | It opens SSH connection to the host and redirects a port (`60008` in the example) 30 | to the restic rest server (`backuphost:8000` in the example). 31 | Thanks to that, restic rest server can be in the local network without 32 | any port exposed publicly. 33 | Next, it creates temporary directory on the host with configuration files. 34 | Include and exclude lists are normal files. 35 | However, repository location, password and main script are FIFO sockets. 36 | It means, that they can be read only once. 37 | The last step is execution of the main script on the remote host, 38 | which performs backup. 39 | 40 | If the host has different location of restic main executable, 41 | not available in the `PATH`, it can be 42 | overriden (e.g. in the host specific configuration file): 43 | 44 | export RESTIC_EXE="/opt/bin/restic" 45 | 46 | If the host does not have or need to use `sudo`, 47 | it also can be overriden: 48 | 49 | export SUDO_RESTIC_EXE="/opt/bin/restic" 50 | 51 | # Configuration 52 | 53 | `$CONFIGURATION_DIR` is `$HOME/.config/bkp-restic`. 54 | 55 | Main configuration file: `$CONFIGURATION_DIR/main.conf` 56 | 57 | Host specific configuration file: `$CONFIGURATION_DIR/host.conf` 58 | 59 | Host specific configuration overrides main configuration. 60 | 61 | In case of `bkp-pull.sh`, `host` is the same value as the first argument to the script. 62 | 63 | In case of `bkp.sh`, `host` is a local hostname (evaluated `$(hostnamectl status --transient)`). 64 | 65 | ##### Restic backup - example configuration 66 | 67 | ``` 68 | export BKP_RESTIC_PASSWORD='abc' 69 | 70 | export BKP_REST_RESTIC_REPOSITORY="rest:http://admin:password@backuphost:8000/$REMOTE_HOST" 71 | 72 | export BKP_RESTIC_INCLUDE_FILES="$CONFIGURATION_DIR/bkp-include.txt" 73 | export BKP_RESTIC_EXCLUDE_FILES="$CONFIGURATION_DIR/bkp-exclude.txt" 74 | ``` 75 | 76 | `backuphost:8000` is the actual address of restic-rest server instance. 77 | `BKP_RESTIC_INCLUDE_FILES` and `BKP_RESTIC_EXCLUDE_FILES` contain include and exclude patterns, 78 | exclude file patterns are case insensitive. 79 | 80 | ##### Restic pull backup - example configuration 81 | 82 | ``` 83 | export BKP_RESTIC_PASSWORD='abc' 84 | 85 | export BKP_FORWARDED_RESTIC_REPOSITORY="rest:http://admin:password@localhost:60008/$REMOTE_HOST" 86 | export BKP_SSH_FORWARD_RULE='60008:backuphost:8000' 87 | 88 | export BKP_RESTIC_INCLUDE_FILES="$CONFIGURATION_DIR/bkp-include.txt" 89 | export BKP_RESTIC_EXCLUDE_FILES="$CONFIGURATION_DIR/bkp-exclude.txt" 90 | ``` 91 | 92 | `BKP_RESTIC_PASSWORD` and `BKP_FORWARDED_RESTIC_REPOSITORY` are stored as FIFO files in the remote temporary directory. 93 | `BKP_SSH_FORWARD_RULE` is the SSH remote forward rule. 94 | `backuphost:8000` is the actual address of restic-rest server instance. 95 | 96 | ### Example bkp-include.txt 97 | 98 | ``` 99 | /etc 100 | /home 101 | /root 102 | ``` 103 | 104 | ### Example bkp-exclude.txt 105 | 106 | ``` 107 | cache 108 | *cache 109 | caches 110 | Cache 111 | *Cache 112 | CacheStorage 113 | .ds_store 114 | backup 115 | backups 116 | bkp-*.log 117 | bkp-*.log.* 118 | *crash*.log 119 | log 120 | logs 121 | .m2 122 | .gradle 123 | .cargo 124 | go 125 | target 126 | *.class 127 | *.pyc 128 | *.o 129 | *~ 130 | *.bak 131 | *.old 132 | *.tmp 133 | *.temp 134 | *.lock 135 | *.part 136 | .ptmp* 137 | tmp 138 | temp 139 | pkg 140 | .trash* 141 | .local 142 | lost+found 143 | random* 144 | generated 145 | thumbs 146 | thumbs.db 147 | thumb 148 | thumbnails 149 | .thumbnails 150 | ``` 151 | 152 | # bkp-env.sh 153 | 154 | The script imports the backup environment for a given host. When called without arguments, 155 | it imports the environment for the current host (`hostnamectl status --transient`). 156 | 157 | After running the script (`. bkp-env.sh`), current environment will contain 158 | `RESTIC_REPOSITORY` and `RESTIC_PASSWORD` variables, 159 | so it will be possible to run all `restic` commands without specyfing repository or password. 160 | 161 | # bkp-prune.sh 162 | 163 | It keeps repositories nice and tidy. Intended to be run periodically. 164 | The script requires configuration variable `BKP_REAL_PATH_RESTIC_REPOSITORY` to be set 165 | to the filesystem path with repositories. 166 | 167 | # bkp-status.sh 168 | 169 | It prints all snapshots for all repositories found 170 | in the filesystem path `BKP_REAL_PATH_RESTIC_REPOSITORY`. 171 | 172 | # bkp-stream.sh 173 | 174 | Backup input stream. E.g. `tar -c /home | $(basename $0) home.tar` will create compressed `home.tar.zst` in the repository `hostname-streams` (repository can be configured by `BKP_REST_RESTIC_REPOSITORY` property). 175 | 176 | # bkp-openwrt.sh 177 | 178 | Backup openwrt router using lede `/cgi-bin/cgi-backup` endpoint. Default repository is `openwrt_IP`. It logs in using `root` user name and property `LUCI_PASSWORD`. 179 | 180 | # bkp-mount.sh 181 | 182 | Mount specific host backup. 183 | -------------------------------------------------------------------------------- /bkp-env.sh: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | 3 | CONFIGURATION_DIR="$HOME/.config/bkp-restic" 4 | 5 | show_help() { 6 | cat << EOF 7 | Restic backup environment. 8 | 9 | Usage: 10 | $(basename $0) --help - help and usage 11 | . $(basename $0) --no-auto - import \`bkp_setup_env\` function for later use 12 | . $(basename $0) host - import backup environment for a given host 13 | EOF 14 | } 15 | 16 | bkp_load_env() { 17 | export REMOTE_HOST="$1" 18 | echo "Loading configuration for $REMOTE_HOST" 19 | if [ ! -d $CONFIGURATION_DIR ] ; then 20 | echo "Configuration folder $CONFIGURATION_DIR is not present" 21 | exit 2 22 | fi 23 | if [ -f $CONFIGURATION_DIR/main.conf ] ; then 24 | echo "Loading $CONFIGURATION_DIR/main.conf" 25 | pushd "$CONFIGURATION_DIR" >/dev/null 2>&1 26 | . "main.conf" 27 | popd >/dev/null 2>&1 28 | fi 29 | if [ -f $CONFIGURATION_DIR/$REMOTE_HOST.conf ] ; then 30 | echo "Loading $CONFIGURATION_DIR/$REMOTE_HOST.conf" 31 | pushd "$CONFIGURATION_DIR" >/dev/null 2>&1 32 | . "$REMOTE_HOST.conf" 33 | popd >/dev/null 2>&1 34 | fi 35 | } 36 | 37 | bkp_verify_env() { 38 | if [ -z ${BKP_RESTIC_PASSWORD+x} ]; then 39 | echo "BKP_RESTIC_PASSWORD is unset" 40 | exit 3 41 | fi 42 | if [ -z ${BKP_REST_RESTIC_REPOSITORY+x} ]; then 43 | echo "BKP_REST_RESTIC_REPOSITORY is unset" 44 | exit 3 45 | fi 46 | 47 | export RESTIC_REPOSITORY="${BKP_REST_RESTIC_REPOSITORY}" 48 | export RESTIC_PASSWORD="${BKP_RESTIC_PASSWORD}" 49 | } 50 | 51 | bkp_setup_env() { 52 | bkp_load_env "$1" 53 | bkp_verify_env 54 | 55 | if [ ! -z ${BKP_REAL_PATH_RESTIC_REPOSITORY+x} ]; then 56 | if [ -d "${BKP_REAL_PATH_RESTIC_REPOSITORY}${REMOTE_HOST}" ]; then 57 | echo "Using existing directory ${BKP_REAL_PATH_RESTIC_REPOSITORY}${REMOTE_HOST}" 58 | export RESTIC_REPOSITORY="${BKP_REAL_PATH_RESTIC_REPOSITORY}${REMOTE_HOST}" 59 | fi 60 | fi 61 | } 62 | 63 | if [[ "$1" == "--help" ]]; then 64 | show_help 65 | elif [[ "$1" == "--no-auto" ]]; then 66 | true 67 | elif [ $# -eq 1 ]; then 68 | bkp_setup_env $1 69 | else 70 | REMOTE_HOST="$(hostnamectl status --transient 2>/dev/null || hostname)" 71 | [ -n "${REMOTE_HOST}" ] && \ 72 | bkp_setup_env "$REMOTE_HOST" || \ 73 | echo >&2 "Cannot get hostname." 74 | fi 75 | -------------------------------------------------------------------------------- /bkp-mount.sh: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | 3 | set -e 4 | 5 | command -v restic >/dev/null 2>&1 || { echo >&2 "Required command restic is not installed."; exit 1; } 6 | 7 | CONFIGURATION_DIR="$HOME/.config/bkp-restic" 8 | 9 | show_help() { 10 | cat << EOF 11 | Mount restic backup 12 | 13 | Usage: 14 | $(basename $0) --help - help and usage 15 | $(basename $0) my-pc - mount 'my-pc' backup at temporary directory 16 | $(basename $0) my-pc ./my-pc-backup - mount 'my-pc' backup at './my-pc-backup' directory 17 | 18 | For more information, see https://github.com/gdmn/bkp-restic/blob/main/README.md 19 | EOF 20 | } 21 | 22 | if [ $# -lt 1 ]; then 23 | show_help 24 | exit 1 25 | fi 26 | if [[ "$1" == "--help" ]]; then 27 | show_help 28 | exit 0 29 | fi 30 | 31 | REMOTE_HOST="$1" 32 | MOUNT_DIR="${2:-$( mktemp -d )}" 33 | RESTIC_EXE="restic" 34 | SUDO_RESTIC_EXE="sudo restic" 35 | 36 | if [ ! -d "$MOUNT_DIR" ] ; then 37 | echo "Directory does not exist: $MOUNT_DIR" 38 | fi 39 | 40 | SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" 41 | if command -v "bkp-env.sh" >/dev/null 2>&1 ; then 42 | CMD_BKP_ENV="bkp-env.sh" 43 | elif command -v "${SCRIPT_DIR}/bkp-env.sh" >/dev/null 2>&1 ; then 44 | CMD_BKP_ENV="${SCRIPT_DIR}/bkp-env.sh" 45 | fi 46 | if [ -z ${CMD_BKP_ENV+x} ]; then 47 | echo "Can not find bkp-env.sh" 48 | exit 3 49 | fi 50 | . $CMD_BKP_ENV --no-auto 51 | bkp_load_env "$REMOTE_HOST" 52 | bkp_verify_env 53 | 54 | $RESTIC_EXE version 55 | 56 | $SUDO_RESTIC_EXE mount --allow-other=true "$MOUNT_DIR" 57 | 58 | -------------------------------------------------------------------------------- /bkp-openwrt.sh: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | 3 | set -e 4 | 5 | command -v restic >/dev/null 2>&1 || { echo >&2 "Required command restic is not installed."; exit 1; } 6 | command -v zstd >/dev/null 2>&1 || { echo >&2 "Required command zstd is not installed."; exit 1; } 7 | 8 | show_usage() { 9 | cat << EOF 10 | Restic backup openwrt router using /cgi-bin/cgi-backup endpoint. 11 | 12 | Usage: 13 | $(basename $0) host - backup given host 14 | EOF 15 | } 16 | 17 | if [ $# -lt 1 ]; then 18 | show_usage 19 | exit 1 20 | fi 21 | 22 | IP="$1" 23 | if ! ping -c 1 -n -w 1 $IP >/dev/null 2>&1 ; then 24 | echo "Could not ping $IP" 25 | exit 1 26 | fi 27 | 28 | SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" 29 | if command -v "bkp-env.sh" >/dev/null 2>&1 ; then 30 | CMD_BKP_ENV="bkp-env.sh" 31 | elif command -v "${SCRIPT_DIR}/bkp-env.sh" >/dev/null 2>&1 ; then 32 | CMD_BKP_ENV="${SCRIPT_DIR}/bkp-env.sh" 33 | fi 34 | if [ -z ${CMD_BKP_ENV+x} ]; then 35 | echo "Can not find bkp-env.sh" 36 | exit 3 37 | fi 38 | . $CMD_BKP_ENV --no-auto 39 | bkp_load_env "openwrt_$IP" 40 | bkp_verify_env 41 | 42 | if [ -z ${LUCI_PASSWORD+x} ]; then 43 | echo "LUCI_PASSWORD is unset" 44 | exit 3 45 | fi 46 | 47 | NAME="openwrt_${IP}-$(date +%Y%m%d_%H%M%S)" 48 | log="${HOME}/bkp-${NAME}.log.zst" 49 | echo "LOG: $log" 50 | echo "NAME: $NAME" 51 | 52 | restic version \ 53 | 2>&1 | tee >(zstd -T0 --long >> "$log") 54 | restic init \ 55 | 2>&1 | tee >(zstd -T0 --long >> "$log") \ 56 | || true 57 | 58 | URL="http://${IP}/cgi-bin" 59 | f=$(mktemp) 60 | routerpswd="luci_username=root&luci_password=$LUCI_PASSWORD" 61 | sessionid=$(curl --silent --data-raw "$routerpswd" -c - "${URL}/luci" | grep sysauth | sed -r 's/.*sysauth.*\s+//') 62 | curl "${URL}/cgi-backup" --data-raw "sessionid=${sessionid}" \ 63 | | restic \ 64 | --verbose \ 65 | --no-cache \ 66 | --tag="openwrt" \ 67 | --tag="stream" \ 68 | --tag="${IP}" \ 69 | --stdin \ 70 | --stdin-filename="${NAME}.tgz" \ 71 | backup \ 72 | 2>&1 | tee >(zstd -T0 --long >> "$log") 73 | 74 | exit 0 75 | -------------------------------------------------------------------------------- /bkp-prune.sh: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | 3 | set -e 4 | 5 | command -v restic >/dev/null 2>&1 || { echo >&2 "Required command restic is not installed."; exit 1; } 6 | command -v zstd >/dev/null 2>&1 || { echo >&2 "Required command zstd is not installed."; exit 1; } 7 | 8 | SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" 9 | if command -v "bkp-env.sh" >/dev/null 2>&1 ; then 10 | CMD_BKP_ENV="bkp-env.sh" 11 | elif command -v "${SCRIPT_DIR}/bkp-env.sh" >/dev/null 2>&1 ; then 12 | CMD_BKP_ENV="${SCRIPT_DIR}/bkp-env.sh" 13 | fi 14 | if [ -z ${CMD_BKP_ENV+x} ]; then 15 | echo "Can not find bkp-env.sh" 16 | exit 3 17 | fi 18 | . $CMD_BKP_ENV --no-auto 19 | bkp_load_env bkp-prune 20 | 21 | if [ -z ${BKP_RESTIC_PASSWORD+x} ]; then 22 | echo "BKP_RESTIC_PASSWORD is unset" 23 | exit 3 24 | fi 25 | if [ -z ${BKP_REAL_PATH_RESTIC_REPOSITORY+x} ]; then 26 | echo "Repository folder $BKP_REAL_PATH_RESTIC_REPOSITORY is unset" 27 | exit 3 28 | fi 29 | if [ ! -d "${BKP_REAL_PATH_RESTIC_REPOSITORY}" ]; then 30 | echo "Repository folder $BKP_REAL_PATH_RESTIC_REPOSITORY is not present" 31 | exit 2 32 | fi 33 | export RESTIC_PASSWORD="${BKP_RESTIC_PASSWORD}" 34 | log="$HOME/bkp-prune-$(date +%Y%m%d_%H%M%S).log.zst" 35 | echo "LOG: $log" 36 | 37 | processRepoClean() { 38 | restic unlock \ 39 | -r "$1" \ 40 | 2>&1 | tee >(zstd -T0 --long >> "$log") 41 | 42 | # keep daily snapshots for a week, weekly for a month, monthly for a year and yearly for 2 years: 43 | ionice -c 2 -n 7 nice -n 19 \ 44 | restic forget --verbose \ 45 | --cleanup-cache \ 46 | -r "$1" \ 47 | --group-by host,tags \ 48 | --keep-within-daily 7d --keep-within-weekly 1m --keep-within-monthly 1y --keep-within-yearly 2y \ 49 | --keep-last 7 \ 50 | 2>&1 | tee >(zstd -T0 --long >> "$log") 51 | 52 | ionice -c 2 -n 7 nice -n 19 \ 53 | restic prune --max-unused 0 \ 54 | -r "$1" \ 55 | 2>&1 | tee >(zstd -T0 --long >> "$log") 56 | 57 | ionice -c 2 -n 7 nice -n 19 \ 58 | restic check --read-data-subset=9.9% \ 59 | -r "$1" \ 60 | 2>&1 | tee >(zstd -T0 --long >> "$log") 61 | } 62 | 63 | processRepoKeepOneClean() { 64 | restic unlock \ 65 | -r "$1" \ 66 | 2>&1 | tee >(zstd -T0 --long >> "$log") 67 | 68 | ionice -c 2 -n 7 nice -n 19 \ 69 | restic forget --keep-last 1 \ 70 | -r "$1" \ 71 | 2>&1 | tee >(zstd -T0 --long >> "$log") 72 | 73 | ionice -c 2 -n 7 nice -n 19 \ 74 | restic prune --max-unused 0 \ 75 | -r "$1" \ 76 | 2>&1 | tee >(zstd -T0 --long >> "$log") 77 | } 78 | 79 | find "$BKP_REAL_PATH_RESTIC_REPOSITORY" -mindepth 0 -maxdepth 3 -name 'snapshots' -type d | \ 80 | while read k ; do 81 | dir="$(dirname $k)" 82 | if [ -r "${dir}/config" ] ; then 83 | echo "${dir}" 84 | fi 85 | done | \ 86 | while read repo ; do 87 | processRepoClean "$repo" 88 | #processRepoKeepOneClean "$repo" 89 | done 90 | 91 | -------------------------------------------------------------------------------- /bkp-pull.sh: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | 3 | CONFIGURATION_DIR="$HOME/.config/bkp-restic" 4 | 5 | show_usage() { 6 | cat << EOF 7 | Restic pull backup 8 | 9 | Usage: 10 | $(basename $0) --help - help and usage 11 | $(basename $0) host - open ssh connection to host and execute restic remotely 12 | 13 | Main configuration file: $CONFIGURATION_DIR/main.conf 14 | Host specific configuration file: $CONFIGURATION_DIR/host.conf 15 | ("host" is the same value as the first argument to this script) 16 | EOF 17 | } 18 | 19 | show_help() { 20 | cat << EOF 21 | 22 | Example configuration: 23 | export BKP_RESTIC_PASSWORD='abc' 24 | export BKP_FORWARDED_RESTIC_REPOSITORY="rest:http://admin:password@localhost:60008/\$REMOTE_HOST" 25 | export BKP_SSH_FORWARD_RULE='60008:backuphost:8000' 26 | export BKP_RESTIC_INCLUDE_FILES="\$CONFIGURATION_DIR/bkp-include.txt" 27 | export BKP_RESTIC_EXCLUDE_FILES="\$CONFIGURATION_DIR/bkp-exclude.txt" 28 | 29 | Example configuration explenation: 30 | BKP_SSH_FORWARD_RULE is the SSH remote forward rule. 31 | "backuphost:8000" is the actual address of restic-rest server instance. 32 | BKP_RESTIC_INCLUDE_FILES and BKP_RESTIC_EXCLUDE_FILES contain include 33 | and exclude patterns, exclude file patterns are case insensitive. 34 | 35 | For more information, see https://github.com/gdmn/bkp-restic/blob/main/README.md 36 | EOF 37 | } 38 | 39 | set -e 40 | 41 | if [ $# -lt 1 ]; then 42 | show_usage 43 | exit 1 44 | fi 45 | 46 | if [[ "$1" == "--help" ]]; then 47 | show_usage 48 | show_help 49 | exit 0 50 | fi 51 | 52 | REMOTE_HOST="$1" 53 | RESTIC_EXE="restic" 54 | SUDO_RESTIC_EXE="sudo restic" 55 | SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" 56 | 57 | if command -v "bkp-env.sh" >/dev/null 2>&1 ; then 58 | CMD_BKP_ENV="bkp-env.sh" 59 | elif command -v "${SCRIPT_DIR}/bkp-env.sh" >/dev/null 2>&1 ; then 60 | CMD_BKP_ENV="${SCRIPT_DIR}/bkp-env.sh" 61 | fi 62 | if [ -z ${CMD_BKP_ENV+x} ]; then 63 | echo "Can not find bkp-env.sh" 64 | exit 3 65 | fi 66 | . $CMD_BKP_ENV --no-auto 67 | bkp_load_env "$REMOTE_HOST" 68 | 69 | if [ -z ${BKP_RESTIC_PASSWORD+x} ]; then 70 | echo "BKP_RESTIC_PASSWORD is unset" 71 | exit 3 72 | fi 73 | if [ -z ${BKP_FORWARDED_RESTIC_REPOSITORY+x} ]; then 74 | echo "BKP_FORWARDED_RESTIC_REPOSITORY is unset" 75 | exit 3 76 | fi 77 | if [ -z ${BKP_SSH_FORWARD_RULE+x} ]; then 78 | echo "BKP_SSH_FORWARD_RULE is unset" 79 | exit 3 80 | fi 81 | if [ -z ${BKP_RESTIC_INCLUDE_FILES+x} ]; then 82 | echo "BKP_RESTIC_INCLUDE_FILES is unset" 83 | exit 3 84 | fi 85 | if [ -z ${BKP_RESTIC_EXCLUDE_FILES+x} ]; then 86 | echo "BKP_RESTIC_EXCLUDE_FILES is unset" 87 | exit 3 88 | fi 89 | if [ ! -r ${BKP_RESTIC_INCLUDE_FILES} ]; then 90 | echo "$BKP_RESTIC_INCLUDE_FILES is not readable" 91 | exit 3 92 | fi 93 | if [ ! -r ${BKP_RESTIC_EXCLUDE_FILES} ]; then 94 | echo "$BKP_RESTIC_EXCLUDE_FILES is not readable" 95 | exit 3 96 | fi 97 | 98 | SSH_CONTROL_SOCKET="$(mktemp)" 99 | SSH="ssh -R $BKP_SSH_FORWARD_RULE -S $SSH_CONTROL_SOCKET $REMOTE_HOST" 100 | 101 | echo -n 'waiting for ssh...' 102 | until $SSH -o ConnectTimeout=1 -o ConnectionAttempts=1 -t true >/dev/null 2>&1; do 103 | echo -n '.' 104 | sleep 3 105 | done 106 | echo ' ok' 107 | 108 | REMOTE_TEMP_DIR=$($SSH "mktemp -d") 109 | cat "$BKP_RESTIC_INCLUDE_FILES" | $SSH "cat > ${REMOTE_TEMP_DIR}/include.txt" 110 | cat "$BKP_RESTIC_EXCLUDE_FILES" | $SSH "cat > ${REMOTE_TEMP_DIR}/exclude.txt" 111 | $SSH "mkfifo $REMOTE_TEMP_DIR/repository1" 112 | echo "$BKP_FORWARDED_RESTIC_REPOSITORY" | $SSH "cat > ${REMOTE_TEMP_DIR}/repository1" & 113 | $SSH "mkfifo $REMOTE_TEMP_DIR/repository2" 114 | echo "$BKP_FORWARDED_RESTIC_REPOSITORY" | $SSH "cat > ${REMOTE_TEMP_DIR}/repository2" & 115 | $SSH "mkfifo $REMOTE_TEMP_DIR/password1" 116 | echo "$BKP_RESTIC_PASSWORD" | $SSH "cat > ${REMOTE_TEMP_DIR}/password1" & 117 | $SSH "mkfifo $REMOTE_TEMP_DIR/password2" 118 | echo "$BKP_RESTIC_PASSWORD" | $SSH "cat > ${REMOTE_TEMP_DIR}/password2" & 119 | 120 | $SSH "mkfifo $REMOTE_TEMP_DIR/fifo" 121 | cat << EOF | $SSH "cat > ${REMOTE_TEMP_DIR}/fifo" & 122 | set -e 123 | command -v $RESTIC_EXE >/dev/null 2>&1 || { echo >&2 "Required command restic is not installed."; exit 1; } 124 | 125 | A='script' 126 | echo \$A loaded 127 | 128 | $RESTIC_EXE version 129 | $RESTIC_EXE init \ 130 | --repository-file "${REMOTE_TEMP_DIR}/repository1" \ 131 | --password-file "${REMOTE_TEMP_DIR}/password1" \ 132 | || true 133 | $SUDO_RESTIC_EXE -vvv backup \ 134 | --no-scan --read-concurrency 10 \ 135 | --repository-file "${REMOTE_TEMP_DIR}/repository2" \ 136 | --password-file "${REMOTE_TEMP_DIR}/password2" \ 137 | --files-from "${REMOTE_TEMP_DIR}/include.txt" \ 138 | --iexclude-file "${REMOTE_TEMP_DIR}/exclude.txt" \ 139 | --exclude-if-present .exclude_from_restic_bkp \ 140 | --one-file-system \ 141 | 2>&1 \ 142 | | grep -v '^unchanged ' | grep -v '0 B added' 143 | echo 'Restic finished' 144 | EOF 145 | 146 | $SSH -t "sh ${REMOTE_TEMP_DIR}/fifo" \ 147 | || true 148 | $SSH "rm -rf $REMOTE_TEMP_DIR" 149 | $SSH -O exit 150 | rm -f $SSH_CONTROL_SOCKET 151 | -------------------------------------------------------------------------------- /bkp-status.sh: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | 3 | set -e 4 | 5 | command -v restic >/dev/null 2>&1 || { echo >&2 "Required command restic is not installed."; exit 1; } 6 | 7 | SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" 8 | if command -v "bkp-env.sh" >/dev/null 2>&1 ; then 9 | CMD_BKP_ENV="bkp-env.sh" 10 | elif command -v "${SCRIPT_DIR}/bkp-env.sh" >/dev/null 2>&1 ; then 11 | CMD_BKP_ENV="${SCRIPT_DIR}/bkp-env.sh" 12 | fi 13 | if [ -z ${CMD_BKP_ENV+x} ]; then 14 | echo "Can not find bkp-env.sh" 15 | exit 3 16 | fi 17 | . $CMD_BKP_ENV --no-auto 18 | bkp_load_env bkp-prune 19 | 20 | if [ -z ${BKP_RESTIC_PASSWORD+x} ]; then 21 | echo "BKP_RESTIC_PASSWORD is unset" 22 | exit 3 23 | fi 24 | if [ -z ${BKP_REAL_PATH_RESTIC_REPOSITORY+x} ]; then 25 | echo "Repository folder $BKP_REAL_PATH_RESTIC_REPOSITORY is unset" 26 | exit 3 27 | fi 28 | if [ ! -d "${BKP_REAL_PATH_RESTIC_REPOSITORY}" ]; then 29 | echo "Repository folder $BKP_REAL_PATH_RESTIC_REPOSITORY is not present" 30 | exit 2 31 | fi 32 | export RESTIC_PASSWORD="${BKP_RESTIC_PASSWORD}" 33 | 34 | processRepoSnapshots() { 35 | du -xsh "$1" || \ 36 | echo "$1:" 37 | if command -v jq >/dev/null 2>&1 ; then 38 | b=$(basename "$1") 39 | restic snapshots --no-lock --quiet --json -r "$1" | jq '.[] | .time' | sed 's/\..*//' | sed 's/\"//' | sed 's/T/ /' | sed "s/^/$b\t/" 40 | else 41 | restic snapshots \ 42 | --no-lock --compact --quiet \ 43 | -r "$1" || true 44 | fi 45 | echo '' 46 | } 47 | 48 | find "$BKP_REAL_PATH_RESTIC_REPOSITORY" -mindepth 0 -maxdepth 3 -name 'snapshots' -type d | \ 49 | while read k ; do 50 | dir="$(dirname $k)" 51 | if [ -r "${dir}/config" ] ; then 52 | echo "${dir}" 53 | fi 54 | done | \ 55 | while read repo ; do 56 | processRepoSnapshots "$repo" 57 | done 58 | -------------------------------------------------------------------------------- /bkp-stream.sh: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | 3 | set -e 4 | 5 | command -v restic >/dev/null 2>&1 || { echo >&2 "Required command restic is not installed."; exit 1; } 6 | command -v zstd >/dev/null 2>&1 || { echo >&2 "Required command zstd is not installed."; exit 1; } 7 | 8 | show_help() { 9 | cat << EOF 10 | Restic backup stdin 11 | 12 | Usage: 13 | $(basename $0) name - backup stdin as "name" 14 | 15 | Example: 16 | tar -c /home | $(basename $0) home.tar 17 | date | $(basename $0) date.txt 18 | 19 | To clean: 20 | filter="--tag stream"; restic forget --keep-last 1 \$filter; restic snapshots \$filter | grep -E '^[0-9a-f]{8}.*stream.*' | sed -E 's/(^[0-9a-f]{8}).*/\\1/' | while read id; do restic forget \$id; done; restic prune --max-unused 0 21 | EOF 22 | } 23 | 24 | if [ $# -lt 1 ]; then 25 | show_help 26 | exit 1 27 | fi 28 | 29 | if [[ "$1" == "--help" ]]; then 30 | show_help 31 | exit 0 32 | fi 33 | 34 | export REMOTE_HOST="$(hostnamectl status --transient 2>/dev/null || hostname)-streams" 35 | SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" 36 | if command -v "bkp-env.sh" >/dev/null 2>&1 ; then 37 | CMD_BKP_ENV="bkp-env.sh" 38 | elif command -v "${SCRIPT_DIR}/bkp-env.sh" >/dev/null 2>&1 ; then 39 | CMD_BKP_ENV="${SCRIPT_DIR}/bkp-env.sh" 40 | fi 41 | if [ -z ${CMD_BKP_ENV+x} ]; then 42 | echo "Can not find bkp-env.sh" 43 | exit 3 44 | fi 45 | . $CMD_BKP_ENV --no-auto 46 | bkp_load_env "$REMOTE_HOST" 47 | bkp_verify_env 48 | NAME="$1" 49 | 50 | log="${HOME}/bkp-${REMOTE_HOST}-$(date +%Y%m%d_%H%M%S).log.zst" 51 | echo "LOG: $log" 52 | echo "NAME: $NAME" 53 | 54 | restic version \ 55 | 2>&1 | tee >(zstd -T0 --long >> "$log") 56 | restic init \ 57 | 2>&1 | tee >(zstd -T0 --long >> "$log") \ 58 | || true 59 | 60 | if command -v ionice >/dev/null 2>&1; then 61 | lowprio="ionice -c 2 -n 7 nice -n 19" 62 | else 63 | lowprio="nice -n 19" 64 | fi 65 | 66 | cat | \ 67 | $lowprio \ 68 | restic \ 69 | --verbose \ 70 | --no-cache \ 71 | --no-scan \ 72 | --tag="${NAME}" \ 73 | --tag="stream" \ 74 | --stdin \ 75 | --stdin-filename="${NAME}" \ 76 | backup \ 77 | 2>&1 | tee >(zstd -T0 --long >> "$log") 78 | 79 | exit 0 80 | 81 | -------------------------------------------------------------------------------- /bkp.sh: -------------------------------------------------------------------------------- 1 | #! /usr/bin/env bash 2 | 3 | set -e 4 | 5 | command -v restic >/dev/null 2>&1 || { echo >&2 "Required command restic is not installed."; exit 1; } 6 | command -v zstd >/dev/null 2>&1 || { echo >&2 "Required command zstd is not installed."; exit 1; } 7 | 8 | CONFIGURATION_DIR="$HOME/.config/bkp-restic" 9 | 10 | show_help() { 11 | cat << EOF 12 | Restic backup 13 | 14 | Usage: 15 | $(basename $0) --help - help and usage 16 | $(basename $0) - execute restic backup 17 | $(basename $0) alternative-conf - execute restic backup for alternative configuration 18 | instead of current host specific configuration, i.e. "REMOTE_HOST=\$1" in that case 19 | 20 | Main configuration file: $CONFIGURATION_DIR/main.conf 21 | Host specific configuration file: $CONFIGURATION_DIR/$(hostnamectl status --transient 2>/dev/null || hostname).conf 22 | 23 | Example configuration: 24 | export BKP_RESTIC_PASSWORD='abc' 25 | export BKP_REST_RESTIC_REPOSITORY="rest:http://admin:password@backuphost:8000/\$REMOTE_HOST" 26 | export BKP_RESTIC_INCLUDE_FILES="\$CONFIGURATION_DIR/bkp-include.txt" 27 | export BKP_RESTIC_EXCLUDE_FILES="\$CONFIGURATION_DIR/bkp-exclude.txt" 28 | 29 | For more information, see https://github.com/gdmn/bkp-restic/blob/main/README.md 30 | EOF 31 | } 32 | 33 | if [[ "$1" == "--help" ]]; then 34 | show_help 35 | exit 0 36 | fi 37 | 38 | export REMOTE_HOST="${1:-$(hostnamectl status --transient 2>/dev/null || hostname)}" 39 | RESTIC_EXE="restic" 40 | SUDO_RESTIC_EXE="sudo restic" 41 | 42 | SCRIPT_DIR="$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )" 43 | if command -v "bkp-env.sh" >/dev/null 2>&1 ; then 44 | CMD_BKP_ENV="bkp-env.sh" 45 | elif command -v "${SCRIPT_DIR}/bkp-env.sh" >/dev/null 2>&1 ; then 46 | CMD_BKP_ENV="${SCRIPT_DIR}/bkp-env.sh" 47 | fi 48 | if [ -z ${CMD_BKP_ENV+x} ]; then 49 | echo "Can not find bkp-env.sh" 50 | exit 3 51 | fi 52 | . $CMD_BKP_ENV --no-auto 53 | bkp_load_env "$REMOTE_HOST" 54 | bkp_verify_env 55 | if [ -z ${BKP_RESTIC_INCLUDE_FILES+x} ]; then 56 | echo "BKP_RESTIC_INCLUDE_FILES is unset" 57 | exit 3 58 | fi 59 | if [ -z ${BKP_RESTIC_EXCLUDE_FILES+x} ]; then 60 | echo "BKP_RESTIC_EXCLUDE_FILES is unset" 61 | exit 3 62 | fi 63 | if [ ! -r ${BKP_RESTIC_INCLUDE_FILES} ]; then 64 | echo "$BKP_RESTIC_INCLUDE_FILES is not readable" 65 | exit 3 66 | fi 67 | if [ ! -r ${BKP_RESTIC_EXCLUDE_FILES} ]; then 68 | echo "$BKP_RESTIC_EXCLUDE_FILES is not readable" 69 | exit 3 70 | fi 71 | 72 | log="${HOME}/bkp-${REMOTE_HOST}-$(date +%Y%m%d_%H%M%S).log.zst" 73 | echo "LOG: $log" 74 | 75 | if [[ "$RESTIC_REPOSITORY" == "sftp:"* ]] ; then 76 | dir=${RESTIC_REPOSITORY//*:/} 77 | srv=${RESTIC_REPOSITORY//:\/*/} 78 | srv=${srv//*:/} 79 | echo "dir: $dir" 80 | echo "srv: $srv" 81 | ssh -t $srv "mkdir -p $dir" 82 | fi 83 | 84 | $RESTIC_EXE version \ 85 | 2>&1 | tee >(zstd -T0 --long >> "$log") 86 | $RESTIC_EXE init \ 87 | 2>&1 | tee >(zstd -T0 --long >> "$log") \ 88 | || true 89 | 90 | if command -v ionice >/dev/null 2>&1; then 91 | lowprio="ionice -c 2 -n 7 nice -n 19" 92 | else 93 | lowprio="nice -n 19" 94 | fi 95 | 96 | $lowprio \ 97 | $SUDO_RESTIC_EXE -vvv backup \ 98 | --no-scan --read-concurrency 10 \ 99 | --files-from "$BKP_RESTIC_INCLUDE_FILES" \ 100 | --iexclude-file "$BKP_RESTIC_EXCLUDE_FILES" \ 101 | --exclude-if-present .exclude_from_restic_bkp \ 102 | --one-file-system \ 103 | 2>&1 | tee >(zstd -T0 --long >> "$log") \ 104 | | grep -v '^unchanged ' | grep -v '0 B added' 105 | --------------------------------------------------------------------------------