├── .circleci └── config.yml ├── .env.example ├── .gitignore ├── LICENSE ├── README.md ├── cleanup-old-snapshots ├── dev-scripts └── check-bash ├── flake.lock ├── flake.nix ├── replicate-full-snapshots ├── replicate-incremental-snapshots └── snapshot-to-dataset /.circleci/config.yml: -------------------------------------------------------------------------------- 1 | version: 2.1 2 | jobs: 3 | check_bash: 4 | docker: 5 | - image: koalaman/shellcheck-alpine:v0.9.0 6 | steps: 7 | - run: 8 | name: Install dependencies 9 | command: apk add bash git openssh-client grep 10 | - checkout 11 | - run: 12 | name: Create .env file from example 13 | command: mv .env.example .env 14 | - run: 15 | name: Run static analysis on bash scripts 16 | command: ./dev-scripts/check-bash 17 | workflows: 18 | test: 19 | jobs: 20 | - check_bash 21 | -------------------------------------------------------------------------------- /.env.example: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | readonly POOL="mypool" 4 | readonly BASE_DIR="/mnt/${POOL}/encrypted-backups" 5 | # shellcheck disable=SC2034 # Not unused 6 | readonly FULL_SNAPSHOTS_DIR="${BASE_DIR}/full-snapshots" 7 | # shellcheck disable=SC2034 # Not unused 8 | readonly INCREMENTAL_SNAPSHOTS_DIR="${BASE_DIR}/incremental-snapshots" 9 | 10 | DATASETS=() 11 | DATASETS+=("documents") 12 | DATASETS+=("music") 13 | DATASETS+=("emails") 14 | readonly DATASETS 15 | 16 | # Optional: Specify a Cronitor base URLs for job monitoring. 17 | readonly CRONITOR_FULL_JOB_URL='' 18 | readonly CRONITOR_INCREMENTAL_JOB_URL='' 19 | -------------------------------------------------------------------------------- /.gitignore: -------------------------------------------------------------------------------- 1 | .env* 2 | !.env.example 3 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | This is free and unencumbered software released into the public domain. 2 | 3 | Anyone is free to copy, modify, publish, use, compile, sell, or 4 | distribute this software, either in source code form or as a compiled 5 | binary, for any purpose, commercial or non-commercial, and by any 6 | means. 7 | 8 | In jurisdictions that recognize copyright laws, the author or authors 9 | of this software dedicate any and all copyright interest in the 10 | software to the public domain. We make this dedication for the benefit 11 | of the public at large and to the detriment of our heirs and 12 | successors. We intend this dedication to be an overt act of 13 | relinquishment in perpetuity of all present and future rights to this 14 | software under copyright law. 15 | 16 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 17 | EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF 18 | MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 19 | IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR 20 | OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, 21 | ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR 22 | OTHER DEALINGS IN THE SOFTWARE. 23 | 24 | For more information, please refer to 25 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # zfs-encrypted-backup 2 | 3 | [![CircleCI](https://circleci.com/gh/mtlynch/zfs-encrypted-backup.svg?style=svg)](https://circleci.com/gh/mtlynch/zfs-encrypted-backup) 4 | [![License](https://img.shields.io/badge/license-Unlicense-blue)](LICENSE) 5 | 6 | ## Overview 7 | 8 | Example scripts to demonstrate how to back up encrypted ZFS backups to an unencrypted dataset. 9 | 10 | ## Setup 11 | 12 | 1. Copy `.env.example` to `.env` 13 | 1. Customize the values in `.env` according to your system. 14 | 15 | ## Perform a full backup 16 | 17 | To perform a full backup of your dataset snapshots, run the following script: 18 | 19 | ```bash 20 | ./replicate-full-snapshots 21 | ``` 22 | 23 | You'll likely want to run this as a cron job weekly or monthly. 24 | 25 | ## Perform an incremental backup 26 | 27 | To perform incremental backups relative to your latest full backups (above), run the following command: 28 | 29 | ```bash 30 | ./replicate-incremental-snapshots 31 | ``` 32 | 33 | You'll likely want to run this as a cron job daily or weekly. 34 | 35 | ## Restore from backup 36 | 37 | To recover from an encrypted dataset backup, run the following script: 38 | 39 | ```bash 40 | ./snapshot-to-dataset \ 41 | new-dataset-name \ 42 | full-snapshot-path \ 43 | [incremental-snapshot-path] 44 | ``` 45 | 46 | For example, if you were recovering a dataset from encrypted and incremental backup files, you'd run the following: 47 | 48 | ```bash 49 | ./snapshot-to-dataset \ 50 | documents-recovered \ 51 | /mnt/pool1/secure-backups/full-snapshots/documents@manual-2022-07-02_22-18 \ 52 | /mnt/pool1/secure-backups/incremental-snapshots/documents@auto-2022-07-05_00-00 53 | ``` 54 | -------------------------------------------------------------------------------- /cleanup-old-snapshots: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # Delete snapshots and files older than three months for datasets in DATASETS 4 | # array. Cleans up both full and incremental snapshots and their corresponding 5 | # files. 6 | 7 | set -eux 8 | 9 | # Parse command line arguments 10 | DRY_RUN=false 11 | if [[ "${1:-}" == "--dry-run" ]]; then 12 | DRY_RUN=true 13 | echo "DRY RUN MODE: No actual deletions will be performed" 14 | fi 15 | 16 | # Change working directory to script directory. 17 | cd "$(dirname "${BASH_SOURCE[0]}")" 18 | 19 | # shellcheck disable=SC1091 20 | . .env 21 | 22 | # Calculate cutoff date (3 months ago) 23 | CUTOFF_DATE=$(date -d "3 months ago" +%s) 24 | readonly CUTOFF_DATE 25 | 26 | echo "Cleaning up snapshots and files older than $(date -d "@${CUTOFF_DATE}" \ 27 | -Iseconds)" 28 | if [[ "${DRY_RUN}" == "true" ]]; then 29 | echo "*** DRY RUN MODE - No actual deletions will be performed ***" 30 | fi 31 | 32 | # Clean up ZFS snapshots 33 | echo "Cleaning up ZFS snapshots..." 34 | # shellcheck disable=SC2153 # Not a misspelling 35 | for DATASET in "${DATASETS[@]}"; do 36 | echo "Processing dataset: ${DATASET}" 37 | 38 | # Get all snapshots for this dataset 39 | zfs list -H -t snapshot -o name,creation -s creation \ 40 | "${POOL}/${DATASET}" 2>/dev/null | \ 41 | while IFS=$'\t' read -r snapshot_name creation_time; do 42 | # Convert ZFS creation time to epoch 43 | snapshot_epoch=$(date -d "${creation_time}" +%s 2>/dev/null || echo "0") 44 | 45 | if [[ ${snapshot_epoch} -lt ${CUTOFF_DATE} && ${snapshot_epoch} -gt 0 ]]; then 46 | if [[ "${DRY_RUN}" == "true" ]]; then 47 | echo "[DRY RUN] Would destroy old snapshot: ${snapshot_name} " \ 48 | "(created: ${creation_time})" 49 | else 50 | echo "Destroying old snapshot: ${snapshot_name} " \ 51 | "(created: ${creation_time})" 52 | zfs destroy "${snapshot_name}" 53 | fi 54 | fi 55 | done 56 | done 57 | 58 | # Function to clean up snapshot files in a directory 59 | cleanup_snapshot_files() { 60 | local dir="$1" 61 | local dir_type="$2" 62 | 63 | if [[ -d "${dir}" ]]; then 64 | echo "Cleaning up ${dir_type} snapshot files in ${dir}..." 65 | 66 | find "${dir}" -type f -name "*@*" | while read -r file; do 67 | # Extract timestamp from filename 68 | filename=$(basename "${file}") 69 | timestamp_part="${filename##*@}" 70 | 71 | # Handle different timestamp formats: 72 | # Format 1: YYYY-MMDDTHHMMSSZ (from original script) 73 | # Format 2: YYYY-MM-DDTHHMMSS-HHMM (with timezone offset) 74 | file_epoch=0 75 | 76 | if [[ ${timestamp_part} =~ ^[0-9]{4}-[0-9]{2}[0-9]{2}T[0-9]{6}Z$ ]]; then 77 | # Format: YYYY-MMDDTHHMMSSZ 78 | formatted_timestamp="${timestamp_part:0:4}-${timestamp_part:4:2}-" 79 | formatted_timestamp+="${timestamp_part:6:2}T${timestamp_part:9:2}:" 80 | formatted_timestamp+="${timestamp_part:11:2}:${timestamp_part:13:2}Z" 81 | file_epoch=$(date -d "${formatted_timestamp}" +%s 2>/dev/null || \ 82 | echo "0") 83 | elif [[ ${timestamp_part} =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{6}[-+][0-9]{4}$ ]]; then 84 | # Format: YYYY-MM-DDTHHMMSS-HHMM or YYYY-MM-DDTHHMMSS+HHMM 85 | # Insert colons into the time part 86 | date_part="${timestamp_part:0:11}" # YYYY-MM-DDT 87 | time_part="${timestamp_part:11:6}" # HHMMSS 88 | tz_part="${timestamp_part:17}" # -HHMM or +HHMM 89 | 90 | formatted_timestamp="${date_part}${time_part:0:2}:${time_part:2:2}:" 91 | formatted_timestamp+="${time_part:4:2}${tz_part}" 92 | file_epoch=$(date -d "${formatted_timestamp}" +%s 2>/dev/null || \ 93 | echo "0") 94 | fi 95 | 96 | if [[ ${file_epoch} -gt 0 ]]; then 97 | if [[ ${file_epoch} -lt ${CUTOFF_DATE} ]]; then 98 | if [[ "${DRY_RUN}" == "true" ]]; then 99 | echo "[DRY RUN] Would remove old ${dir_type} snapshot file: " \ 100 | "${file} (timestamp: ${timestamp_part})" 101 | else 102 | echo "Removing old ${dir_type} snapshot file: ${file} " \ 103 | "(timestamp: ${timestamp_part})" 104 | rm -f "${file}" 105 | fi 106 | fi 107 | else 108 | echo "Warning: Could not parse timestamp from filename: ${filename}" 109 | fi 110 | done 111 | else 112 | echo "Directory ${dir} does not exist, skipping ${dir_type} file cleanup" 113 | fi 114 | } 115 | 116 | # Clean up full snapshot files 117 | cleanup_snapshot_files "${FULL_SNAPSHOTS_DIR}" "full" 118 | 119 | # Clean up incremental snapshot files 120 | cleanup_snapshot_files "${INCREMENTAL_SNAPSHOTS_DIR}" "incremental" 121 | 122 | if [[ "${DRY_RUN}" == "true" ]]; then 123 | echo "*** DRY RUN COMPLETED - No actual deletions were performed ***" 124 | else 125 | echo "Finished cleaning up old snapshots and files" 126 | fi 127 | -------------------------------------------------------------------------------- /dev-scripts/check-bash: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # Run static analysis on bash scripts. 4 | 5 | # Exit on first failing command. 6 | set -e 7 | 8 | # Exit on unset variable. 9 | set -u 10 | 11 | BASH_SCRIPTS=() 12 | 13 | while read -r filepath; do 14 | # Check shebang for bash. 15 | if head -n 1 "${filepath}" | grep --quiet --regexp 'bash'; then 16 | BASH_SCRIPTS+=("${filepath}") 17 | fi 18 | done < <(git ls-files) 19 | 20 | readonly BASH_SCRIPTS 21 | 22 | # Echo commands before executing them, by default to stderr. 23 | set -x 24 | 25 | shellcheck "${BASH_SCRIPTS[@]}" 26 | -------------------------------------------------------------------------------- /flake.lock: -------------------------------------------------------------------------------- 1 | { 2 | "nodes": { 3 | "flake-utils": { 4 | "inputs": { 5 | "systems": "systems" 6 | }, 7 | "locked": { 8 | "lastModified": 1705309234, 9 | "narHash": "sha256-uNRRNRKmJyCRC/8y1RqBkqWBLM034y4qN7EprSdmgyA=", 10 | "owner": "numtide", 11 | "repo": "flake-utils", 12 | "rev": "1ef2e671c3b0c19053962c07dbda38332dcebf26", 13 | "type": "github" 14 | }, 15 | "original": { 16 | "owner": "numtide", 17 | "repo": "flake-utils", 18 | "type": "github" 19 | } 20 | }, 21 | "root": { 22 | "inputs": { 23 | "flake-utils": "flake-utils", 24 | "shellcheck_dep": "shellcheck_dep" 25 | } 26 | }, 27 | "shellcheck_dep": { 28 | "locked": { 29 | "lastModified": 1695132891, 30 | "narHash": "sha256-cJR9AFHmt816cW/C9necLJyOg/gsnkvEeFAfxgeM1hc=", 31 | "owner": "NixOS", 32 | "repo": "nixpkgs", 33 | "rev": "8b5ab8341e33322e5b66fb46ce23d724050f6606", 34 | "type": "github" 35 | }, 36 | "original": { 37 | "owner": "NixOS", 38 | "repo": "nixpkgs", 39 | "rev": "8b5ab8341e33322e5b66fb46ce23d724050f6606", 40 | "type": "github" 41 | } 42 | }, 43 | "systems": { 44 | "locked": { 45 | "lastModified": 1681028828, 46 | "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=", 47 | "owner": "nix-systems", 48 | "repo": "default", 49 | "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e", 50 | "type": "github" 51 | }, 52 | "original": { 53 | "owner": "nix-systems", 54 | "repo": "default", 55 | "type": "github" 56 | } 57 | } 58 | }, 59 | "root": "root", 60 | "version": 7 61 | } 62 | -------------------------------------------------------------------------------- /flake.nix: -------------------------------------------------------------------------------- 1 | { 2 | description = "Dev environment for zfs-encrypted-backup"; 3 | 4 | inputs = { 5 | flake-utils.url = "github:numtide/flake-utils"; 6 | 7 | # 0.9.0 release 8 | shellcheck_dep.url = "github:NixOS/nixpkgs/8b5ab8341e33322e5b66fb46ce23d724050f6606"; 9 | }; 10 | 11 | outputs = { self, flake-utils, shellcheck_dep }@inputs : 12 | flake-utils.lib.eachDefaultSystem (system: 13 | let 14 | shellcheck_dep = inputs.shellcheck_dep.legacyPackages.${system}; 15 | in 16 | { 17 | devShells.default = shellcheck_dep.mkShell { 18 | packages = [ 19 | shellcheck_dep.shellcheck 20 | ]; 21 | 22 | shellHook = '' 23 | echo "shellcheck" "$(shellcheck --version | grep '^version:')" 24 | ''; 25 | }; 26 | }); 27 | } 28 | -------------------------------------------------------------------------------- /replicate-full-snapshots: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # Create full snapshots of datasets in DATASETS array. 4 | 5 | set -eux 6 | 7 | # Change working directory to script directory. 8 | cd "$(dirname "${BASH_SOURCE[0]}")" 9 | 10 | # shellcheck disable=SC1091 11 | . .env 12 | 13 | if [[ -n "${CRONITOR_FULL_JOB_URL}" ]]; then 14 | curl "${CRONITOR_FULL_JOB_URL}?state=run" 15 | fi 16 | 17 | mkdir -p "${FULL_SNAPSHOTS_DIR}" 18 | 19 | TIMESTAMP="$(date -Iseconds | sed 's/://g' | sed 's/+0000/Z/g')" 20 | readonly TIMESTAMP 21 | 22 | # shellcheck disable=SC2153 # Not a misspelling 23 | for DATASET in "${DATASETS[@]}"; do 24 | # Take a snapshot. 25 | SNAPSHOT_NAME="${POOL}/${DATASET}@${TIMESTAMP}" 26 | zfs snapshot "${SNAPSHOT_NAME}" 27 | 28 | # Write the snapshot to a file. 29 | OUTPUT_FILENAME="${SNAPSHOT_NAME//${POOL}\//}" 30 | zfs send --raw --verbose "${SNAPSHOT_NAME}" > "${FULL_SNAPSHOTS_DIR}/${OUTPUT_FILENAME}" 31 | done 32 | 33 | echo "Finished replicating full snapshots" 34 | 35 | if [[ -n "${CRONITOR_FULL_JOB_URL}" ]]; then 36 | curl "${CRONITOR_FULL_JOB_URL}?state=complete" 37 | fi 38 | -------------------------------------------------------------------------------- /replicate-incremental-snapshots: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # Create incremental snapshots of datasets in DATASETS array relative to their 4 | # last full snapshot. 5 | 6 | set -eux 7 | 8 | # Change working directory to script directory. 9 | cd "$(dirname "${BASH_SOURCE[0]}")" 10 | 11 | # shellcheck disable=SC1091 12 | . .env 13 | 14 | if [[ -n "${CRONITOR_INCREMENTAL_JOB_URL}" ]]; then 15 | curl "${CRONITOR_INCREMENTAL_JOB_URL}?state=run" 16 | fi 17 | 18 | mkdir -p "${INCREMENTAL_SNAPSHOTS_DIR}" 19 | 20 | TIMESTAMP="$(date -Iseconds | sed 's/://g' | sed 's/+0000/Z/g')" 21 | readonly TIMESTAMP 22 | 23 | # shellcheck disable=SC2153 # Not a misspelling 24 | for DATASET in "${DATASETS[@]}"; do 25 | # Take a snapshot. 26 | INCREMENTAL_SNAPSHOT="${POOL}/${DATASET}@${TIMESTAMP}" 27 | zfs snapshot "${INCREMENTAL_SNAPSHOT}" 28 | 29 | # Find the most recent full snapshot. 30 | # shellcheck disable=SC2012 # ls is better than find in this context 31 | BASE_SNAPSHOT_FILENAME="$(basename "$(ls -tr "${FULL_SNAPSHOTS_DIR}/${DATASET}"* | tail -1)")" 32 | if [[ -z "${BASE_SNAPSHOT_FILENAME}" ]]; then 33 | >&2 echo "Couldn't find full snapshot for dataset: ${DATASET}" 34 | exit 1 35 | fi 36 | BASE_SNAPSHOT="${POOL}/${BASE_SNAPSHOT_FILENAME}" 37 | 38 | # Write the incremental snapshot to a file. 39 | OUTPUT_FILENAME="${INCREMENTAL_SNAPSHOT//${POOL}\//}" 40 | OUTPUT_PATH="${INCREMENTAL_SNAPSHOTS_DIR}/${OUTPUT_FILENAME}" 41 | if [[ -f "${OUTPUT_PATH}" ]]; then 42 | echo "${OUTPUT_PATH} already exists, skipping..." 43 | continue 44 | fi 45 | zfs send --raw --verbose -i "${BASE_SNAPSHOT}" "${INCREMENTAL_SNAPSHOT}" \ 46 | > "${OUTPUT_PATH}" 47 | done 48 | 49 | echo "Finished replicating incremental snapshots" 50 | 51 | if [[ -n "${CRONITOR_INCREMENTAL_JOB_URL}" ]]; then 52 | curl "${CRONITOR_INCREMENTAL_JOB_URL}?state=complete" 53 | fi 54 | -------------------------------------------------------------------------------- /snapshot-to-dataset: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | # 3 | # Recover a dataset from an encrypted snapshot. 4 | # 5 | # Usage: 6 | # ./snapshot-to-dataset.sh new-dataset-name full-snapshot-path [incremental-snapshot-path] 7 | # 8 | # Example: 9 | # ./snapshot-to-dataset.sh \ 10 | # documents-recovered \ 11 | # /mnt/pool1/secure-backups/full-snapshots/documents@manual-2022-07-02_22-18 \ 12 | # /mnt/pool1/secure-backups/incremental-snapshots/documents@auto-2022-07-05_00-00 13 | 14 | 15 | set -ex 16 | 17 | # Change working directory to script directory. 18 | cd "$(dirname "${BASH_SOURCE[0]}")" 19 | 20 | # shellcheck disable=SC1091 21 | . .env 22 | 23 | NEW_DATASET_NAME="$1" 24 | readonly NEW_DATASET_NAME 25 | 26 | FULL_SNAPSHOT_PATH="$2" 27 | readonly FULL_SNAPSHOT_PATH 28 | 29 | INCREMENTAL_SNAPSHOT_PATH="$3" 30 | readonly INCREMENTAL_SNAPSHOT_PATH 31 | 32 | set -u 33 | 34 | # Restore from base snapshot 35 | zfs receive "${POOL}/${NEW_DATASET_NAME}" < "${FULL_SNAPSHOT_PATH}" 36 | 37 | if [[ -n "${INCREMENTAL_SNAPSHOT_PATH}" ]]; then 38 | # Update dataset to latest incremental snapshot 39 | zfs receive "${POOL}/${NEW_DATASET_NAME}" < "${INCREMENTAL_SNAPSHOT_PATH}" 40 | fi 41 | --------------------------------------------------------------------------------