├── README.org ├── examples ├── README.org ├── clear_USB_test_backup.sh ├── clear_remote_test_backup.sh ├── filter_rules ├── filter_rules_excld ├── filter_rules_recovery_plan ├── job_backup_recovery_plan_to_remote.sh ├── job_backup_to_USB.sh ├── job_backup_to_USB_excld.sh ├── job_backup_to_USB_minimal.sh ├── job_backup_to_remote.sh ├── job_restore_directory_from_remote.sh ├── job_restore_last_snapshot.sh └── setup_test_data_directory.sh ├── rclone_jobber.sh ├── rclone_jobber_tutorial.org └── test_suite ├── README.org ├── filter_rules ├── job_1d_backup_to_dated_directory.sh ├── job_1d_restore_from_dated_directory.sh ├── job_USB_backup_to_USB.sh ├── job_USB_restore_from_USB.sh ├── job_check_empty_source.sh ├── job_check_null_dest.sh ├── job_check_null_source.sh ├── job_dir_backup_to_dated_directory.sh ├── job_dir_restore_from_dated_directory.sh ├── job_snapshot_backup_old_data_to_delete.sh ├── job_snapshot_restore_from_last_snapshot.sh ├── test_1d.sh ├── test_USB.sh ├── test_check_range.sh ├── test_last_snapshot.sh └── test_restore_deleted_directory.sh /README.org: -------------------------------------------------------------------------------- 1 | * About rclone_jobber backup script 2 | [[./rclone_jobber.sh][rclone_jobber.sh]] is a shell script to make backups. 3 | It performs backup jobs by calling rclone. 4 | [[https://rclone.org/][Rclone]] is a mature tool and many [[https://github.com/ncw/rclone/wiki/Third-Party-Integrations-with-rclone][third-party tools use rclone]]. 5 | 6 | The [[file:rclone_jobber_tutorial.org][rclone jobber tutorial]] includes backup-job and restore-job examples for a home computer. 7 | 8 | *rclone_jobber.sh features:* 9 | - Tested on Linux, and should also run on macOS and [[https://docs.microsoft.com/en-us/windows/wsl/about][Windows 10 wsl]] 10 | - Options to archive old backup files in their original hierarchy 11 | - Can backup to multiple destinations for redundant backups (one backup job per destination) 12 | - Aborts if the backup job is already running (maybe the previous run didn't finish) 13 | - Pop-up for error conditions 14 | - Option for a cron-monitoring service 15 | - Logging 16 | - POSIX compliant shell script so it is easy to customize 17 | - Free (open source Creative Commons Zero license) 18 | 19 | *Rclone features:* 20 | - Back up data from local machine 21 | - Back up data from a cloud provider (Google Drive for example) 22 | - Back up to local storage 23 | - Back up to remote cloud storage (safe from local disaster) 24 | - Over 30 cloud-storage providers to choose from (so you're never locked into a provider) 25 | - Optional encryption (Crypt) 26 | - MD5/SHA1 hashes checked at all times for file integrity 27 | - Timestamps preserved on files 28 | - Filter rules (to exclude or include files) 29 | - rsync-like algorithm and interface 30 | - Sync (one way) mode to make a directory identical 31 | - Partial syncs supported on a whole file basis 32 | - Free (open source MIT license) 33 | 34 | Both rclone and rclone_jobber.sh are command line tools. 35 | If you prefer a GUI, checkout [[https://mmozeiko.github.io/RcloneBrowser/][RcloneBrowser]]. 36 | 37 | * Example job scripts 38 | rclone_jobber.sh is already developed, documented, and tested. 39 | Only the backup-job files need to be written. 40 | 41 | Example minimal backup-job script for rclone_jobber: 42 | : #!/usr/bin/env sh 43 | : 44 | : source="$HOME" 45 | : dest="$usb/rclone_jobber_backup" 46 | : 47 | : $rclone_jobber/rclone_jobber.sh "$source" "$dest" 48 | 49 | Example backup job with all the rclone_jobber.sh arguments defined: 50 | : #!/usr/bin/env sh 51 | : 52 | : source="$HOME" 53 | : dest="$usb/rclone_jobber_backup" 54 | : move_old_files_to="dated_files" 55 | : options="--filter-from=$rclone_jobber/filter_rules --checksum --dry-run" 56 | : monitoring_URL="https://monitor.io/12345678-1234-1234-1234-1234567890ab" 57 | : 58 | : $rclone_jobber/rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" "$monitoring_URL" 59 | 60 | * Contact 61 | Please tell me if something is wrong, you’re helping me improve this project. 62 | 63 | Submit issues to https://github.com/wolfv6/rclone_jobber/issues. 64 | Pull requests are welcome. 65 | 66 | * Licenses 67 | [[http://creativecommons.org/licenses/by-nc-sa/4.0/][https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png]]\\ 68 | The content of the tutorial, including associated setup_test_data_directory.sh, are licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 license. 69 | 70 | [[http://creativecommons.org/publicdomain/zero/1.0/][http://i.creativecommons.org/p/zero/1.0/88x31.png]]\\ 71 | rclone_jobber.sh source code, including associated job files, filter_rules file, and test_suite, are licensed under the Creative Commons Zero 1.0 license. 72 | 73 | Rclone_jobber is not affiliated with rclone. 74 | -------------------------------------------------------------------------------- /examples/README.org: -------------------------------------------------------------------------------- 1 | The files in this directory are examples used by rclone_jobber_tutorial.org. 2 | 3 | The example scripts use the following environment path variables: 4 | - HOME 5 | - rclone_jobber 6 | - usb 7 | - remote 8 | 9 | Detailed instructions are in the tutorial's "Setup test directories and path variables" section. 10 | -------------------------------------------------------------------------------- /examples/clear_USB_test_backup.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | # deletes USB test backup directory; rclone_jobber.sh can create a new one as needed 4 | # this script uses these user-defined environment variables: usb 5 | ############################################ 6 | 7 | #delete test backup 8 | rm -r $usb/test_rclone_backup 9 | -------------------------------------------------------------------------------- /examples/clear_remote_test_backup.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | # clears remote test backup for rclone_jobber.sh 4 | # this script uses these user-defined environment variables: remote 5 | ############################################### 6 | 7 | #delete test backup 8 | rclone purge ${remote}: 9 | 10 | #verify empty, should say "Failed to ls: directory not found" 11 | rclone ls ${remote}: 12 | -------------------------------------------------------------------------------- /examples/filter_rules: -------------------------------------------------------------------------------- 1 | ################################### license ################################## 2 | # filter_rules tell rclone which files to include or exclude. 3 | # Written in 2018 by Wolfram Volpi, contact at https://github.com/wolfv6/rclone_jobber/issues. 4 | # To the extent possible under law, the author(s) have dedicated all copyright and related and neighboring rights to this software to the public domain worldwide. 5 | # This software is distributed without any warranty. 6 | # You should have received a copy of the CC0 Public Domain Dedication along with this software. If not, see http://creativecommons.org/publicdomain/zero/1.0/. 7 | # rclone_jobber is not affiliated with rclone. 8 | ############################################################################## 9 | 10 | # paths in this file are from source="~/test_rclone_data/" 11 | 12 | ########################## 1) includes ####################################### 13 | + direc0/** 14 | 15 | ########################## 2) exclude everything else ######################## 16 | - * 17 | -------------------------------------------------------------------------------- /examples/filter_rules_excld: -------------------------------------------------------------------------------- 1 | ################################### license ################################## 2 | # filter_rules tell rclone which files to include or exclude. 3 | # Written in 2018 by Wolfram Volpi, contact at https://github.com/wolfv6/rclone_jobber/issues. 4 | # To the extent possible under law, the author(s) have dedicated all copyright and related and neighboring rights to this software to the public domain worldwide. 5 | # This software is distributed without any warranty. 6 | # You should have received a copy of the CC0 Public Domain Dedication along with this software. If not, see http://creativecommons.org/publicdomain/zero/1.0/. 7 | # rclone_jobber is not affiliated with rclone. 8 | ############################################################################## 9 | 10 | # Paths in this file are from source="~/test_rclone_data/". 11 | # The sections alternate between include and exclude, progressing from fine to coarse grained. 12 | 13 | ########################## 1) include specific files ######################### 14 | # Include specific files before they are excluded by section 2. 15 | 16 | # config files 17 | + .bashrc 18 | + direc1/.thingrc 19 | 20 | # hidden directories to backup 21 | + .git/** 22 | 23 | ########################## 2) exclude patterns ############################### 24 | # Exclude these patterns before they are included by section 3. 25 | # Exclude directories first because filter is faster when directories are not entered. 26 | # Then exclude file types. 27 | 28 | # directory names ending in "_exc" 29 | - *_exc/** 30 | 31 | # hidden directories 32 | - .*/** 33 | 34 | # file names with extension 35 | - *.bak 36 | - *.log 37 | - *.tmp 38 | 39 | # file names ending in "_exc." and any extension 40 | - *_exc.* 41 | 42 | # hidden files 43 | - .* 44 | 45 | ########################## 3) include directories ############################ 46 | # Include these directories before they are excluded by section 4. 47 | + direc_empty/** 48 | + direc0/** 49 | + direc1/** 50 | 51 | ########################## 4) exclude everything else ######################## 52 | # Exclude all top directories and files. 53 | - * 54 | -------------------------------------------------------------------------------- /examples/filter_rules_recovery_plan: -------------------------------------------------------------------------------- 1 | #this file is for rclone --filter-from 2 | #used by job_backup_recovery_plan_to_remote.sh 3 | #paths in this file are from source="/home/wolfv/Documents/pc_maintenance" 4 | 5 | ########################## include specific files ###################### 6 | + /backup_systems/rclone_jobber/rclone_jobber_tutorial.org 7 | + /backup_systems/rclone_wolf/job_restore_last_snapshot.sh 8 | + /linux_notes/GPT_fdisk.txt 9 | + /linux_notes/GPT_gdisk.txt 10 | + /linux_notes/GPT_sgdisk.bin 11 | + /linux_notes/debian_9_install.org 12 | + /linux_notes/linux_install_notes.org 13 | 14 | ########################## exclude everything else ##################### 15 | - * -------------------------------------------------------------------------------- /examples/job_backup_recovery_plan_to_remote.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | rclone_wolf="/home/wolfv/Documents/pc_maintenance/backup_systems/rclone_wolf" 4 | 5 | source="/home/wolfv/Documents/pc_maintenance" 6 | dest="onedrive:recovery_plan" 7 | move_old_files_to="dated_directory" 8 | options="--filter-from=$rclone_wolf/filter_rules_recovery_plan --checksum" 9 | monitoring_URL="https://monitor.io/12345678-1234-1234-1234-1234567890ab" 10 | 11 | $rclone_wolf/rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" "$monitoring_URL" 12 | -------------------------------------------------------------------------------- /examples/job_backup_to_USB.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | ################################### license ################################## 4 | # job_backup_to_USB.sh calls rclone_jobber to perform a backup to USB. 5 | # Written in 2018 by Wolfram Volpi, contact at https://github.com/wolfv6/rclone_jobber/issues. 6 | # To the extent possible under law, the author(s) have dedicated all copyright and related and neighboring rights to this software to the public domain worldwide. 7 | # This software is distributed without any warranty. 8 | # You should have received a copy of the CC0 Public Domain Dedication along with this software. If not, see http://creativecommons.org/publicdomain/zero/1.0/. 9 | # rclone_jobber is not affiliated with rclone. 10 | ############################################################################## 11 | 12 | #this script uses these user-defined environment variables: rclone_jobber, usb 13 | rclone_jobber=$rclone_jobber #path to rclone_jobber directory 14 | 15 | source="$HOME/test_rclone_data" 16 | dest="$usb/test_rclone_backup" 17 | move_old_files_to="dated_files" 18 | options="--filter-from=$rclone_jobber/examples/filter_rules --checksum" 19 | monitoring_URL="" 20 | 21 | $rclone_jobber/rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" "$monitoring_URL" 22 | 23 | #display test directories (display for tutorial only, don't do this in production) 24 | tree -a $source 25 | tree -a $dest 26 | -------------------------------------------------------------------------------- /examples/job_backup_to_USB_excld.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | ################################### license ################################## 4 | # job_backup_to_USB.sh calls rclone_jobber to perform a backup to USB. 5 | # Written in 2018 by Wolfram Volpi, contact at https://github.com/wolfv6/rclone_jobber/issues. 6 | # To the extent possible under law, the author(s) have dedicated all copyright and related and neighboring rights to this software to the public domain worldwide. 7 | # This software is distributed without any warranty. 8 | # You should have received a copy of the CC0 Public Domain Dedication along with this software. If not, see http://creativecommons.org/publicdomain/zero/1.0/. 9 | # rclone_jobber is not affiliated with rclone. 10 | ############################################################################## 11 | 12 | #this script uses these user-defined environment variables: rclone_jobber, usb 13 | rclone_jobber=$rclone_jobber #path to rclone_jobber directory 14 | 15 | source="$HOME/test_rclone_data" 16 | dest="$usb/test_rclone_backup" 17 | move_old_files_to="dated_files" 18 | options="--filter-from=$rclone_jobber/examples/filter_rules_excld --checksum" 19 | monitoring_URL="" 20 | 21 | $rclone_jobber/rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" "$monitoring_URL" 22 | 23 | #display test directories (display for tutorial only, don't do this in production) 24 | tree -a $source 25 | tree -a $dest 26 | -------------------------------------------------------------------------------- /examples/job_backup_to_USB_minimal.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | ################################### license ################################## 4 | # job_backup_to_USB_minimal.sh calls rclone_jobber to perform a backup to USB. 5 | # Written in 2018 by Wolfram Volpi, contact at https://github.com/wolfv6/rclone_jobber/issues. 6 | # To the extent possible under law, the author(s) have dedicated all copyright and related and neighboring rights to this software to the public domain worldwide. 7 | # This software is distributed without any warranty. 8 | # You should have received a copy of the CC0 Public Domain Dedication along with this software. If not, see http://creativecommons.org/publicdomain/zero/1.0/. 9 | # rclone_jobber is not affiliated with rclone. 10 | ############################################################################## 11 | 12 | #this script uses these user-defined environment variables: rclone_jobber, usb 13 | source="$HOME/test_rclone_data" 14 | dest="$usb/test_rclone_backup" 15 | options="--dry-run" 16 | 17 | $rclone_jobber/rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" 18 | 19 | #display test directories (display for tutorial only, don't do this in production) 20 | tree -a $source 21 | tree -a $dest 22 | -------------------------------------------------------------------------------- /examples/job_backup_to_remote.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | ################################### license ################################## 4 | # job_backup_to_remote.sh calls rclone_jobber to perform a backup to remote. 5 | # Written in 2018 by Wolfram Volpi, contact at https://github.com/wolfv6/rclone_jobber/issues. 6 | # To the extent possible under law, the author(s) have dedicated all copyright and related and neighboring rights to this software to the public domain worldwide. 7 | # This software is distributed without any warranty. 8 | # You should have received a copy of the CC0 Public Domain Dedication along with this software. If not, see http://creativecommons.org/publicdomain/zero/1.0/. 9 | # rclone_jobber is not affiliated with rclone. 10 | ############################################################################## 11 | 12 | #this script uses these user-defined environment variables: rclone_jobber, remote 13 | rclone_jobber=$rclone_jobber #path to rclone_jobber directory 14 | 15 | source="$HOME/test_rclone_data" 16 | dest="${remote}:" 17 | move_old_files_to="dated_directory" 18 | options="--filter-from=$rclone_jobber/examples/filter_rules" 19 | monitoring_URL="" 20 | 21 | $rclone_jobber/rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" "$monitoring_URL" 22 | 23 | #display test directories (display for tutorial only, don't do this in production) 24 | tree -a $source 25 | rclone ls $dest 26 | -------------------------------------------------------------------------------- /examples/job_restore_directory_from_remote.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | ################################### license ################################## 4 | # job_restore_directory_from_remote.sh restores directory in path from onedrive to it's original location, with "_last_snapshot" appended to directory name 5 | # Written in 2018 by Wolfram Volpi, contact at https://github.com/wolfv6/rclone_jobber/issues. 6 | # To the extent possible under law, the author(s) have dedicated all copyright and related and neighboring rights to this software to the public domain worldwide. 7 | # This software is distributed without any warranty. 8 | # You should have received a copy of the CC0 Public Domain Dedication along with this software. If not, see http://creativecommons.org/publicdomain/zero/1.0/. 9 | # rclone_jobber is not affiliated with rclone. 10 | ############################################################################## 11 | 12 | #assign relative path of directory to restore 13 | path="relative_path_of_directory_to_restore" 14 | 15 | #this script uses these user-defined environment variables: remote 16 | source="${remote}:last_snapshot/$path" 17 | destination="$HOME/${path}_last_snapshot" 18 | 19 | cmd="rclone copy $source $destination --dry-run" 20 | 21 | echo "$cmd" 22 | echo ">>>>>>>>>>>>>>> Run the above rclone command? (y) <<<<<<<<<<<<<<<<< " 23 | read -n 1 -r 24 | echo 25 | if [[ $REPLY =~ ^[Yy]$ ]] 26 | then 27 | eval $cmd #restore last_snapshot 28 | fi 29 | -------------------------------------------------------------------------------- /examples/job_restore_last_snapshot.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | ################################### license ################################## 4 | # job_restore_last_snapshot.sh restores last_snapshot to home, with "_last_snapshot" appended to user name 5 | # Written in 2018 by Wolfram Volpi, contact at https://github.com/wolfv6/rclone_jobber/issues. 6 | # To the extent possible under law, the author(s) have dedicated all copyright and related and neighboring rights to this software to the public domain worldwide. 7 | # This software is distributed without any warranty. 8 | # You should have received a copy of the CC0 Public Domain Dedication along with this software. If not, see http://creativecommons.org/publicdomain/zero/1.0/. 9 | # rclone_jobber is not affiliated with rclone. 10 | ############################################################################## 11 | 12 | #this script uses these user-defined environment variables: remote 13 | 14 | #uncomment the source variable to restore data from: 15 | source="$usb/test_rclone_backup/last_snapshot" 16 | #source="${remote}:last_snapshot" 17 | 18 | destination="$HOME/last_snapshot" 19 | 20 | cmd="rclone copy $source $destination" 21 | 22 | echo "$cmd" 23 | echo ">>>>>>>>>>>>>>> Run the above rclone command? (y) <<<<<<<<<<<<<<<<< " 24 | read -n 1 -r 25 | echo 26 | if [[ $REPLY =~ ^[Yy]$ ]] 27 | then 28 | eval $cmd #restore last_snapshot 29 | fi 30 | -------------------------------------------------------------------------------- /examples/setup_test_data_directory.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | ########################## License ######################################## 4 | # setup_test_data_directory.sh by Wolfram Volpi is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. 5 | # Based on a work at https://github.com/wolfv6/rclone_jobber. 6 | # Permissions beyond the scope of this license may be available at https://github.com/wolfv6/rclone_jobber/issues. 7 | ########################################################################### 8 | 9 | # setup ~/test_rclone_data directory for rclone_jobber.sh 10 | ######################################################### 11 | 12 | #delete and recreate test_rclone_data 13 | rm -r ~/test_rclone_data 14 | mkdir ~/test_rclone_data 15 | 16 | mkdir ~/test_rclone_data/direc_empty 17 | 18 | mkdir ~/test_rclone_data/direc0 19 | touch ~/test_rclone_data/direc0/f0 20 | 21 | mkdir ~/test_rclone_data/direc1 22 | touch ~/test_rclone_data/direc1/f1 23 | 24 | mkdir ~/test_rclone_data/direc1/direc1a 25 | touch ~/test_rclone_data/direc1/direc1a/f1a 26 | 27 | mkdir ~/test_rclone_data/direc1/direc1b 28 | touch ~/test_rclone_data/direc1/direc1b/f1b 29 | 30 | #files and directory names ending in "_exc" 31 | mkdir ~/test_rclone_data/direc1/direc1c_exc 32 | touch ~/test_rclone_data/direc1/direc1c_exc/f1c 33 | 34 | touch ~/test_rclone_data/direc1/f2_exc 35 | touch ~/test_rclone_data/direc1/f3_exc.txt 36 | touch ~/test_rclone_data/direc1/f4_excy 37 | touch ~/test_rclone_data/direc1/f5_exc.not.txt 38 | touch ~/test_rclone_data/direc1/f6.bak 39 | 40 | #hidden directories and files 41 | touch ~/test_rclone_data/.bashrc 42 | touch ~/test_rclone_data/direc1/.hidden 43 | touch ~/test_rclone_data/direc1/.thingrc 44 | mkdir ~/test_rclone_data/direc1/.hide 45 | touch ~/test_rclone_data/direc1/.hide/f8 46 | mkdir ~/test_rclone_data/direc1/.git 47 | touch ~/test_rclone_data/direc1/.git/f7 48 | 49 | #display test_data directory 50 | tree -a ~/test_rclone_data 51 | -------------------------------------------------------------------------------- /rclone_jobber.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | # rclone_jobber.sh version 1.5.6 3 | # Tutorial, backup-job examples, and source code at https://github.com/wolfv6/rclone_jobber 4 | # Logging options are headed by "# set log". Details are in the tutorial's "Logging options" section. 5 | 6 | ################################### license ################################## 7 | # rclone_jobber.sh is a script that calls rclone sync to perform a backup. 8 | # Written in 2018 by Wolfram Volpi, contact at https://github.com/wolfv6/rclone_jobber/issues 9 | # To the extent possible under law, the author(s) have dedicated all copyright and related and 10 | # neighboring rights to this software to the public domain worldwide. 11 | # This software is distributed without any warranty. 12 | # You should have received a copy of the CC0 Public Domain Dedication along with this software. 13 | # If not, see http://creativecommons.org/publicdomain/zero/1.0/. 14 | # rclone_jobber is not affiliated with rclone. 15 | 16 | ################################# parameters ################################# 17 | source="$1" #the directory to back up (without a trailing slash) 18 | dest="$2" #the directory to back up to (without a trailing slash or "last_snapshot") destination=$dest/last_snapshot 19 | move_old_files_to="$3" #move_old_files_to is one of: 20 | # "dated_directory" - move old files to a dated directory (an incremental backup) 21 | # "dated_files" - move old files to old_files directory, and append move date to file names (an incremental backup) 22 | # "" - old files are overwritten or deleted (a plain one-way sync backup) 23 | options="$4" #rclone options like "--filter-from=filter_patterns --checksum --log-level="INFO" --dry-run" 24 | #do not put these in options: --backup-dir, --suffix, --log-file 25 | job_name="$5" #job_name="$(basename $0)" 26 | monitoring_URL="$6" #cron monitoring service URL to send email if cron failure or other error prevented back up 27 | 28 | ################################ set variables ############################### 29 | # $new is the directory name of the current snapshot 30 | # $timestamp is time that old file was moved out of new (not time that file was copied from source) 31 | new="last_snapshot" 32 | timestamp="$(date +%F_%T)" 33 | #timestamp="$(date +%F_%H%M%S)" #time w/o colons if thumb drive is FAT format, which does not allow colons in file name 34 | 35 | # set log_file path 36 | path="$(realpath "$0")" #this will place log in the same directory as this script 37 | log_file="${path%.*}.log" #replace path extension with "log" 38 | #log_file="/var/log/rclone_jobber.log" #for Logrotate 39 | 40 | # set log_option for rclone 41 | log_option="--log-file=$log_file" #log to log_file 42 | #log_option="--syslog" #log to systemd journal 43 | 44 | ################################## functions ################################# 45 | send_to_log() 46 | { 47 | msg="$1" 48 | 49 | # set log - send msg to log 50 | echo "$msg" >> "$log_file" #log msg to log_file 51 | #printf "$msg" | systemd-cat -t RCLONE_JOBBER -p info #log msg to systemd journal 52 | } 53 | 54 | # print message to echo, log, and popup 55 | print_message() 56 | { 57 | urgency="$1" 58 | msg="$2" 59 | message="${urgency}: $job_name $msg" 60 | 61 | echo "$message" 62 | send_to_log "$(date +%F_%T) $message" 63 | warning_icon="/usr/share/icons/Adwaita/32x32/emblems/emblem-synchronizing.png" #path in Fedora 28 64 | # notify-send is a popup notification on most Linux desktops, install libnotify-bin 65 | command -v notify-send && notify-send --urgency critical --icon "$warning_icon" "$message" 66 | } 67 | 68 | ################################# range checks ################################ 69 | # if source is empty string 70 | if [ -z "$source" ]; then 71 | print_message "ERROR" "aborted because source is empty string." 72 | exit 1 73 | fi 74 | 75 | # if dest is empty string 76 | if [ -z "$dest" ]; then 77 | print_message "ERROR" "aborted because dest is empty string." 78 | exit 1 79 | fi 80 | 81 | # if source is empty 82 | if ! test "rclone lsf --max-depth 1 $source"; then # rclone lsf requires rclone 1.40 or later 83 | print_message "ERROR" "aborted because source is empty." 84 | exit 1 85 | fi 86 | 87 | # if job is already running (maybe previous run didn't finish) 88 | # https://github.com/wolfv6/rclone_jobber/pull/9 said this is not working in macOS 89 | if pidof -o $PPID -x "$job_name"; then 90 | print_message "WARNING" "aborted because it is already running." 91 | exit 1 92 | fi 93 | 94 | ############################### move_old_files_to ############################# 95 | # deleted or changed files are removed or moved, depending on value of move_old_files_to variable 96 | # default move_old_files_to="" will remove deleted or changed files from backup 97 | if [ "$move_old_files_to" = "dated_directory" ]; then 98 | # move deleted or changed files to archive/$(date +%Y)/$timestamp directory 99 | backup_dir="--backup-dir=$dest/archive/$(date +%Y)/$timestamp" 100 | elif [ "$move_old_files_to" = "dated_files" ]; then 101 | # move deleted or changed files to old directory, and append _$timestamp to file name 102 | backup_dir="--backup-dir=$dest/old_files --suffix=_$timestamp" 103 | elif [ "$move_old_files_to" != "" ]; then 104 | print_message "WARNING" "Parameter move_old_files_to=$move_old_files_to, but should be dated_directory or dated_files.\ 105 | Moving old data to dated_directory." 106 | backup_dir="--backup-dir=$dest/$timestamp" 107 | fi 108 | 109 | ################################### back up ################################## 110 | cmd="rclone sync $source $dest/$new $backup_dir $log_option $options" 111 | 112 | # progress message 113 | echo "Back up in progress $timestamp $job_name" 114 | echo "$cmd" 115 | 116 | # set logging to verbose 117 | #send_to_log "$timestamp $job_name" 118 | #send_to_log "$cmd" 119 | 120 | eval $cmd 121 | exit_code=$? 122 | 123 | ############################ confirmation and logging ######################## 124 | if [ "$exit_code" -eq 0 ]; then #if no errors 125 | confirmation="$(date +%F_%T) completed $job_name" 126 | echo "$confirmation" 127 | send_to_log "$confirmation" 128 | send_to_log "" 129 | if [ ! -z "$monitoring_URL" ]; then 130 | wget --quiet $monitoring_URL -O /dev/null 131 | fi 132 | exit 0 133 | else 134 | print_message "ERROR" "failed. rclone exit_code=$exit_code" 135 | send_to_log "" 136 | exit 1 137 | fi 138 | -------------------------------------------------------------------------------- /rclone_jobber_tutorial.org: -------------------------------------------------------------------------------- 1 | * About this rclone_jobber backup script tutorial 2 | This tutorial shows you how to setup and test the rclone_jobber backup script. 3 | The final product will perform local and remote backups automatically. 4 | 5 | The tutorial is written for Linux and most of the content is also useful for macOS and [[https://docs.microsoft.com/en-us/windows/wsl/about][Windows 10 wsl]]. 6 | 7 | Both rclone and rclone_jobber.sh are command line tools. 8 | This tutorial assumes that the reader has basic command-line skills. 9 | 10 | The example jobs scripts make backups for a home PC. 11 | You can adapt rclone_jobber.sh and the example job scripts to suit your own backup needs. 12 | 13 | Backup trivia: the verb form to "back up" is two words, whereas the noun is "backup". 14 | 15 | *Disclaimer:* 16 | Some of the scripts used in this tutorial delete or overwrite data. 17 | Unless you know exactly what you are doing, please work this tutorial on a spare computer that doesn't contain data you don't want to lose. 18 | This tutorial and associated scripts are distributed without any warranty. 19 | I am not responsible for any lost data. 20 | 21 | * Table of Contents :TOC_2: 22 | - [[#about-this-rclone_jobber-backup-script-tutorial][About this rclone_jobber backup script tutorial]] 23 | - [[#install-rclone][Install rclone]] 24 | - [[#install-rclone_jobber][Install rclone_jobber]] 25 | - [[#setup-environment-path-variables][Setup environment path variables]] 26 | - [[#setup-a-test_data-directory][Setup a test_data directory]] 27 | - [[#take-rclone_jobbersh-for-a-test-drive][Take rclone_jobber.sh for a test drive]] 28 | - [[#backup-job-and-rclone_jobbersh-parameters][Backup job and rclone_jobber.sh parameters]] 29 | - [[#backup-job-script][Backup job script]] 30 | - [[#1-source][1) source]] 31 | - [[#2-dest][2) dest]] 32 | - [[#if-source-or-dest-paths-contain-a-space][If source or dest paths contain a space]] 33 | - [[#3-move_old_files_to][3) move_old_files_to]] 34 | - [[#4-options][4) options]] 35 | - [[#5-job_name][5) job_name]] 36 | - [[#6-monitoring_url][6) monitoring_URL]] 37 | - [[#filter-rules][Filter rules]] 38 | - [[#select-a-cloud-storage-provider][Select a cloud storage provider]] 39 | - [[#configure-a-remote][Configure a remote]] 40 | - [[#configure-a-crypt][Configure a crypt]] 41 | - [[#schedule-backup-jobs-to-run-automatically][Schedule backup jobs to run automatically]] 42 | - [[#logging-options][Logging options]] 43 | - [[#-set-logging-to-verbose][# set logging to verbose]] 44 | - [[#-set-log_file-path][# set log_file path]] 45 | - [[#-set-log_file-path-to-varlogrclone_jobberlog-linux][# set log_file path to /var/log/rclone_jobber.log (Linux)]] 46 | - [[#logrotate-varlogrclone_jobberlog-linux][Logrotate /var/log/rclone_jobber.log (Linux)]] 47 | - [[#send-varlogrclone_jobberlog-to-systemd-journal-linux][Send /var/log/rclone_jobber.log to systemd journal (Linux)]] 48 | - [[#back-up-and-restore][Back up and restore]] 49 | - [[#example-backup-jobs][Example backup jobs]] 50 | - [[#example-restore-data][Example restore data]] 51 | - [[#test-backup-jobs-and-test-restore-data-jobs][Test backup jobs and test restore-data jobs]] 52 | - [[#monitor-your-backups][Monitor your backups]] 53 | - [[#data-recovery-plan][Data recovery plan]] 54 | - [[#data-recovery-plan-documents][Data recovery plan documents]] 55 | - [[#data-recovery-plan-monitoring-checklist][Data-recovery-plan monitoring checklist]] 56 | - [[#license][License]] 57 | 58 | * Install rclone 59 | Install the latest rclone from https://rclone.org/downloads/ 60 | 61 | * Install rclone_jobber 62 | Download or clone the [[https://github.com/wolfv6/rclone_jobber][rclone_jobber repository]]. 63 | 64 | The "examples" directory contains all the scripts used by this tutorial. 65 | 66 | * Setup environment path variables 67 | Most tutorials use notation to indicate substitution. 68 | This tutorial uses environment path variables to automate the substitutions. 69 | Thus, all the examples in this tutorial can run on your system without you having to edit the scripts. 70 | 71 | The example scripts use the following environment path variables: 72 | - HOME 73 | - rclone_jobber 74 | - usb 75 | - remote (explained in "Configure a remote" section) 76 | 77 | The above "rclone_jobber" path is the location of the rclone_jobber repository you downloaded. 78 | 79 | Tutorial examples use the "usb" path variable to demonstrate backup to local mass storage. 80 | The "usb" path doesn't actually have to be a USB drive. 81 | 82 | If you're on Linux, add these lines to your ~/.profile (or ~/.bash_profile) file, but with your system paths: 83 | : export rclone_jobber="/home/wolfv/rclone_jobber" 84 | : export usb="/media/wolfv/USB_device_name" 85 | 86 | Reload your .profile: 87 | : $ source ~/.profile 88 | 89 | Now Linux shell should substitute $rclone_jobber and $usb. 90 | 91 | * Setup a test_data directory 92 | This tutorial's example scripts backup a small test directory. 93 | 94 | The [[./examples/setup_test_data_directory.sh][setup_test_data_directory.sh]] script will setup the small source directory. 95 | It recursively deletes the ~/test_rclone_data directory and rebuilds a fresh copy. 96 | To setup a test directory from the command line: 97 | : $ $rclone_jobber/examples/setup_test_data_directory.sh 98 | 99 | * Take rclone_jobber.sh for a test drive 100 | Once you have the path variables and test_data directories setup, you can take rclone_jobber for a test drive. 101 | 102 | Here is a minimal backup-job script for rclone_jobber: 103 | : #!/usr/bin/env sh 104 | : 105 | : source="$HOME/test_rclone_data" 106 | : dest="$usb/test_rclone_backup" 107 | : 108 | : $rclone_jobber/rclone_jobber.sh "$source" "$dest" 109 | 110 | That last line calls rclone_jobber.sh with arguments =source= and =dest=. 111 | 112 | Open the [[./examples/job_backup_to_USB_minimal.sh][examples/job_backup_to_USB_minimal.sh]] in your favorite text editor. 113 | Set options to =--dry-run=: 114 | : options="--dry-run" 115 | 116 | Run the backup job: 117 | : $ $rclone_jobber/examples/job_backup_to_USB_minimal.sh 118 | 119 | The backup, $dest, was not created because --dry-run. 120 | 121 | *Important:* A bad backup job can cause data loss. 122 | First test with the =--dry-run= flag to see exactly what would be copied and deleted. 123 | 124 | Here are some more things you can try with rclone_jobber: 125 | 1. Open rclone_jobber.log (rclone_jobber.log is in the same directory as rclone_jobber.sh). 126 | 2. Run the backup job again, this time without =--dry-run=. 127 | 3. Inspect changes in the destination files. 128 | 4. Change some files in source: 129 | - delete a file 130 | - edit a file 131 | - add a file 132 | - move a file 133 | And run the backup job again. 134 | 135 | * Backup job and rclone_jobber.sh parameters 136 | ** Backup job script 137 | Each backup job contains arguments and a call to rclone_jobber.sh. 138 | Here is an example backup job with all the rclone_jobber.sh arguments defined: 139 | : #!/usr/bin/env sh 140 | : 141 | : source="$HOME/test_rclone_data" 142 | : dest="$usb/test_rclone_backup" 143 | : move_old_files_to="dated_files" 144 | : options="--filter-from=$rclone_jobber/examples/filter_rules --checksum --dry-run" 145 | : monitoring_URL="https://monitor.io/12345678-1234-1234-1234-1234567890ab" 146 | : 147 | : $rclone_jobber/rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" "$monitoring_URL" 148 | 149 | The last line calls rclone_jobber.sh. 150 | =source= and =dest= are required, the remaining arguments can be empty string \"\" or undefined. 151 | 152 | The next sections describe rclone_jobber.sh parameters: 153 | 1) source 154 | 2) dest 155 | 3) move_old_files_to 156 | 4) options 157 | 5) job_name 158 | 6) monitoring_URL 159 | 160 | ** 1) source 161 | =source= is the directory to back up. 162 | 163 | Example =source= argument: 164 | : source="/home/wolfv" 165 | 166 | ** 2) dest 167 | =dest= is the directory to back up to. 168 | Data is backed up to =destination=$dest/last_snapshot=. 169 | 170 | Example =dest= argument for [[https://rclone.org/local/][local file system]] data storage: 171 | : dest="/media/wolfv/USB/wolfv_backup" 172 | 173 | Example =dest= for remote data storage: 174 | : dest="onedrive_wolfv_backup_crypt:" 175 | 176 | ** If source or dest paths contain a space 177 | If your path contains a space, then you must use extra quotes. 178 | 179 | For Linux / OSX =source= argument: 180 | : source="'/home/wolf v'" 181 | 182 | For Windows =source= argument: 183 | : source='"/home/wolf v"' 184 | 185 | Details at https://rclone.org/docs/#quoting-and-the-shell. 186 | 187 | ** 3) move_old_files_to 188 | When a file is changed or deleted, the old version already in backup is either moved or removed. 189 | The =move_old_files_to= parameter specifies what happens to the old files. 190 | 191 | *** move_old_files_to=\"dated_directory\" 192 | Argument to move deleted or changed files to a dated directory: 193 | : move_old_files_to="dated_directory" 194 | 195 | Old files are moved to the dated directory in their original hierarchy. 196 | This makes it easy to restore a deleted sub-directory. 197 | Also convenient to manually delete a directory from a previous year. 198 | : backup 199 | : ├── archive <<<<<<<< archive contains dated directories 200 | : │ └── 2018 201 | : │ ├── 2018-02-22_14:00:14 202 | : │ │ └── direc1 203 | : │ │ └── f1 204 | : │ └── 2018-02-22_15:00:14 <<<<<<<< old files were moved here on dated_directory's date 205 | : │ └── direc1 206 | : │ └── f1 <<<<<<<< old version of file f1 207 | : └── last_snapshot <<<<<<<< last_snapshot directory contains the most recent backup 208 | : └── direc1 209 | : └── f1 210 | 211 | *** move_old_files_to=\"dated_files\" 212 | Argument to move old files to old_files directory, and append move date to file names: 213 | : move_old_files_to="dated_files" 214 | 215 | Old files are moved to the old_files directory in their original hierarchy. 216 | This makes it easy to browse a file's history, and restore a particular version of a file. 217 | : backup 218 | : ├── last_snapshot <<<<<<<< last_snapshot directory contains the most recent backup 219 | : │ └── direc1 220 | : │ └── f1 221 | : └── old_files <<<<<<<< old_files directory contains old dated_files 222 | : └── direc1 223 | : ├── f1_2018-02-22_14:00:14 224 | : └── f1_2018-02-22_15:00:14 <<<<<<<<< old version of file f1 moved here on appended date 225 | 226 | *** move_old_files_to=\"\" 227 | Argument to remove old files from backup: 228 | : move_old_files_to="" 229 | 230 | Only the most recent version of each file remains in the backup. 231 | This can save a little storage space. 232 | Useful for making an extra backup before OS upgrade or OS clean install. 233 | : backup 234 | : └── last_snapshot <<<<<<<< last_snapshot directory contains the most recent backup 235 | : └── direc1 236 | : └── f1 <<<<<<<< old versions of file f1 were overwritten or removed 237 | 238 | ** 4) options 239 | The =options= argument can contain any number of rclone options. 240 | You can put any [[https://rclone.org/docs/#options][rclone options]] in the options argument, except for these three: 241 | : --backup-dir 242 | : --suffix 243 | : --log-file 244 | 245 | Those options are set in rclone_jobber.sh. 246 | 247 | Example options argument containing four rclone options: 248 | : options="--filter-from=filter_rules --checksum --log-level=INFO --dry-run" 249 | 250 | Rclone options used in this tutorial are: 251 | - [[https://rclone.org/filtering/#filter-from-read-filtering-patterns-from-a-file][--filter-from]] (discussed in the "filter rules" section) 252 | - [[https://rclone.org/docs/#c-checksum][--checksum]] 253 | - [[https://rclone.org/docs/#log-level-level][--log-level]] 254 | - [[https://rclone.org/docs/#n-dry-run][--dry-run]] 255 | 256 | ** 5) job_name 257 | The =job_name= argument specifies the job's file name: 258 | : job_name="$(basename $0)" 259 | 260 | The shell command "$(basename $0)" will fill in the job's file name for you. 261 | 262 | rclone_jobber.sh guards against =job_name= running again before the previous run is finished. 263 | 264 | rclone_jobber.sh prints =job_name= in warnings and log entries. 265 | 266 | ** 6) monitoring_URL 267 | The =monitoring_URL= argument specifies a ping URL for a cron-monitoring service. 268 | 269 | =monitoring_URL= is optional. 270 | This is redundant if the remote data-storage provider offers an integrated monitoring service. 271 | 272 | Example =monitoring_URL=: 273 | : monitoring_URL="https://monitor.io/12345678-1234-1234-1234-1234567890ab" 274 | 275 | Every time rclone_jobber.sh completes a job without error, it pings the monitoring_URL. 276 | If the cron monitoring service hasn't been pinged within a set amount of time, then it sends you an email alert. 277 | Many cron monitoring services offer free plans. 278 | 279 | No two jobs should share the same =monitoring_URL=. 280 | 281 | * Filter rules 282 | Rclone has a sophisticated set of [[https://rclone.org/filtering/][filter rules]]. 283 | Filter rules tell rclone which files to include or exclude. 284 | 285 | Open the [[./examples/filter_rules_excld][examples/filter_rules_excld]] file. 286 | Each rule starts with a "+ " or "- ", followed by a pattern. 287 | - a leading "+" means include if the pattern matches 288 | - a leading "-" means exclude if the pattern matches 289 | 290 | For each file in source, filter rules are processed in the order that they are defined. 291 | If the matcher fails to find a match after testing all the filter rules, then the path is included. 292 | Read the [[./examples/filter_rules_excld][examples/filter_rules_excld]] file to see how this works. 293 | 294 | Lines starting with '#' are comments. 295 | Comment at the end of a rule is not supported because file names can contain a '#'. 296 | 297 | The rclone_jobber =options= argument specifies the filter_rules_excld file like this: 298 | : options="--filter-from filter_rules_excld" 299 | 300 | To see the example filter_rules_excld file in action, run: 301 | : $ $rclone_jobber/examples/clear_USB_test_backup.sh 302 | : $ $rclone_jobber/examples/job_backup_to_USB_excld.sh 303 | 304 | * Select a cloud storage provider 305 | Rclone uses cloud storage providers to backup data to an off-site storage system. 306 | Off-site storage systems are safe from local disaster. 307 | 308 | All rclone cloud-storage providers are listed on https://rclone.org/. 309 | Some of the cloud-storage-providers' features are listed in two tables on https://rclone.org/overview/. 310 | Most cloud-storage providers offer small storage capacities for free. 311 | Pick one. 312 | You can always try other cloud-storage providers after you finish this tutorial. 313 | 314 | * Configure a remote 315 | Once you have an account with your chosen cloud-storage provider, the next step is to configure its remote. 316 | 317 | There is one page of configuration instructions for each cloud-storage provider. 318 | Links to the configuration instructions are at https://rclone.org/docs/#configure and https://rclone.org/. 319 | Follow the instructions to configure your remote now. 320 | : $ rclone config 321 | 322 | Rclone stores all the configuration information you entered. 323 | The default location is ~/.config/rclone/rclone.conf. 324 | The remote's password is stored in the rclone.conf file, so be careful about giving people access to it. 325 | 326 | To list all your rclone remotes: 327 | : $ rclone listremotes 328 | 329 | Here is how to run the tutorial's example remote backup job on Linux 330 | (for tutorial scripts only, don't do this for production). 331 | Add this line to your ~/.profile file, but with your remote path: 332 | : export remote="onedrive_test_rclone_backup" 333 | and reload .profile: 334 | : $ source ~/.profile 335 | 336 | To use a tutorial example script as a template for production backups, edit the tutorial scripts: 337 | replace occurrences of "${remote}" with your remote path. 338 | 339 | To test your remote, run: 340 | : $ $rclone_jobber/examples/job_backup_to_remote.sh 341 | 342 | * Configure a crypt 343 | "crypt" is a kind of remote that: 344 | - encrypts and decrypts the data stream for an underlying remote 345 | - performs encryption and decryption on the client side 346 | - uses the same command interface as other kinds of remotes 347 | 348 | Instructions for configuring a crypt remote are at https://rclone.org/crypt/ and https://rclone.org/docs/#configuration-encryption. 349 | 350 | When configuring a crypt remote, rclone will ask you to give it a name. 351 | Put some thought into naming your remotes. 352 | : name> myremote_myfolder_crypt 353 | 354 | And then rclone will ask for the underlying remote. 355 | This example will encrypt myfolder in myremote: 356 | : remote> myremote:myfolder 357 | You can always rename a remote later via rclone config. 358 | 359 | To list all your rclone remotes: 360 | : $ rclone listremotes 361 | 362 | Most remote cloud-storage providers allow you to view your directory names and file names in a web browser. 363 | But that's not very useful if rclone encrypted the directory and file names. 364 | Use rclone to browse encrypted directory and file names. 365 | 366 | To list directories in remote: 367 | : $ rclone lsd remote: 368 | : $ rclone lsd remote:path 369 | 370 | To list objects and directories of path (requires rclone-v1.40 or later): 371 | : $ rclone lsf remote:path 372 | 373 | To list top-level files in path: 374 | : $ rclone ls remote:path --max-depth 1 375 | 376 | To list all files in path recursively: 377 | : $ rclone ls remote:path 378 | 379 | [[./examples/job_backup_to_remote.sh][/examples/job_backup_to_remote.sh]] uses a remote, which could be of type crypt. 380 | 381 | To test your crypt remote, set the path variable as described in the "Configure a remote" section, and then run: 382 | : $ $rclone_jobber/examples/job_backup_to_remote.sh 383 | 384 | *** pathIsTooLong error 385 | Most cloud storage providers have a 254 character-path-length limit. 386 | Crypt limits encrypted paths to 151 characters with some cloud storage providers (this is a [[https://github.com/ncw/rclone/issues/637][known crypt issue]]). 387 | If the path is too long, rclone returns this ERROR: 388 | : Failed to copy: invalidRequest: pathIsTooLong: Path exceeds maximum length 389 | 390 | There are 3 work-a-rounds: 391 | - turn off "encrypt directory names" in rclone config (file content can still be encrypted) 392 | - shorten your paths 393 | - Long Path Tool (I have not tried this) 394 | 395 | *** Backblaze b2 lifecycle 396 | rclone crypt file-name and directory-name encryption don’t work with Backblaze b2 lifecycle. 397 | This is because: 398 | - b2 lifecycle appends date to end of file names 399 | - b2 doesn’t strip off the appended date before passing the file name back to rclone 400 | 401 | So then rclone can’t decrypt the file names. 402 | 403 | There are 3 work-a-rounds: 404 | - turn off "encrypt file names" and "encrypt directory names" in rclone config (file content can still be encrypted) 405 | - turn off b2 lifecycle and 406 | - set move_old_files_to=\"dated_directory\" in the backup job 407 | - manually delete old files at end of life 408 | - use a different remote data-storage provider 409 | 410 | * Schedule backup jobs to run automatically 411 | After you schedule backup jobs, you will have an automated backup system with this workflow: 412 | 1. a job scheduler calls a backup job script 413 | 2. the job script calls rclone_jobber.sh 414 | 3. rclone_jobber.sh calls rclone 415 | 4. rclone consults your filter rules, connects to a backup storage, and uploads modified files 416 | 417 | Schedule your backup jobs in your favorite job scheduler. 418 | 419 | The following example schedules jobs on cron (cron is a popular job scheduler installed on Linux). 420 | The first line runs a local job every hour on the hour. 421 | The second line runs a remote job every hour, 30 minutes past the hour. 422 | The third line runs at 3:18 and 15:18 every day 423 | : $ crontab -e 424 | : 00 * * * * /home/wolfv/rclone_jobber/job_backup_to_USB.sh 425 | : 30 * * * * /home/wolfv/rclone_jobber/job_backup_to_remote.sh 426 | : 18 3,15 * * * /home/wolfv/rclone_jobber/job_backup_recovery_plan_to_remote.sh 427 | 428 | The initial backup will take a long time (subsequent backups are much shorter). 429 | If your computer goes to sleep while a backup is in progress, the backup will not finish. 430 | Consider disabling sleep on your computer for the initial backup. 431 | On Linux Gnome desktop: 432 | : right click > Settings > Power > Automatic suspend: Off 433 | 434 | * Logging options 435 | rclone_jobber.sh default behavior places rclone_jobber.log in the same directory as rclone_jobber.sh. 436 | Read this section if you want the log in a different location. 437 | 438 | Logging options are set in rclone_jobber.sh, headed by "# set log" comments. 439 | To change logging behavior, search for "# set log" and change the default values. 440 | 441 | Logging options are described in the next 5 sections. 442 | 443 | ** # set logging to verbose 444 | To send more information to the log, use the send_to_log function in rclone_jobber.sh: 445 | : # set logging to verbose 446 | : send_to_log "$timestamp $job_name" 447 | : send_to_log "$cmd" 448 | 449 | Additionally, you can set [[https://rclone.org/docs/#log-level-level][--log-level]] in the job's "options" parameter. 450 | 451 | ** # set log_file path 452 | In rclone_jobber.sh, variable log_file contains the log file's path. 453 | The default behavior places rclone_jobber.log in the same directory as rclone_jobber.sh: 454 | : # set log_file path 455 | : path="$(realpath "$0")" #path of this script 456 | : log_file="${path%.*}.log" #replace path extension with "log" 457 | You can change log_file to any path you like. 458 | 459 | ** # set log_file path to /var/log/rclone_jobber.log (Linux) 460 | To set the rclone_jobber log location to /var/log/, create the log file and give it the user's ownership and read-write permission. 461 | In this example, rclone_jobber.log ownership is given to wolfv: 462 | : $ sudo touch /var/log/rclone_jobber.log 463 | : $ sudo chown wolfv /var/log/rclone_jobber.log 464 | : $ sudo chmod 0666 /var/log/rclone_jobber.log 465 | : $ sudo ls -l /var/log/rclone_jobber.log 466 | : -rw-rw-rw-. 1 wolfv root 19 Mar 21 13:58 /var/log/rclone_jobber.log 467 | 468 | In rclone_jobber.sh, set the new log_file path: 469 | : # set log_file path 470 | : log_file="/var/log/rclone_jobber.log" 471 | 472 | ** Logrotate /var/log/rclone_jobber.log (Linux) 473 | Over time a log file can grow to unwieldy size. 474 | The logrotate utility can automatically archive the current log, start a fresh log, and delete older logs. 475 | 476 | To setup logrotate, set log_file path to /var/log/rclone_jobber.log (described in previous section). 477 | Then create a logrotate configuration file in /etc/logrotate.d: 478 | : $ sudo vi /etc/logrotate.d/rclone_jobber 479 | 480 | And paste this text into the logrotate configuration file: 481 | : /var/log/rclone_jobber.log { 482 | : monthly 483 | : rotate 2 484 | : size 1M 485 | : compress 486 | : delaycompress 487 | : } 488 | 489 | More options are listed in man: 490 | : $ man logrotate 491 | 492 | Execute a dry-run to see what logrotate would do: 493 | : $ logrotate -d /etc/logrotate.d/rclone_jobber 494 | 495 | ** Send /var/log/rclone_jobber.log to systemd journal (Linux) 496 | Linux and macOS can send all log output to systemd journal. 497 | To do so, make these two changes to rclone_jobber.sh script: 498 | 1. change log_option to --syslog 499 | : # set log_option for rclone 500 | : log_option="--syslog" 501 | 2. send msg to systemd journal (sending msg to log_file is optional, and is commented in this example) 502 | : # set log - send msg to log 503 | : #echo "$msg" >> "$log_file" #send msg to log_file 504 | : printf "$msg" | systemd-cat -t RCLONE_JOBBER -p info #send msg to systemd journal 505 | 506 | * Back up and restore 507 | ** Example backup jobs 508 | The following system uses two backup jobs with complementary attributes (this is how I backup my home PC). 509 | The latest snapshot can be easily restored from either backup. 510 | 511 | [[./examples/job_backup_to_USB.sh][examples/job_backup_to_USB.sh]] has attributes that make it convenient to browse file history: 512 | - local storage (for fast navigation) 513 | - move_old_files_to=\"dated_files\" (will group old versions of a file together) 514 | - not encrypted (easy to browse files in a file manager) 515 | (unencrypted local storage is OK if storage is safe from theft, and useful if the remote storage password is lost) 516 | - schedule hourly, on the hour (this assumes the USB drive is always plugged in and mounted) 517 | 518 | [[./examples/job_backup_to_remote.sh][/examples/job_backup_to_remote.sh]] has attributes that make it secure, and easy to restore a deleted sub-directory: 519 | - remote storage (off-site is safe from on-site disaster) 520 | - move_old_files_to=\"dated_directory\" (easy to restore a deleted sub-directory e.g. Documents) 521 | - encrypted (please keep your password in a safe place) 522 | - schedule hourly, 30 min past the hour (for a back up every 30 minutes when combined with job_backup_to_USB.sh) 523 | 524 | In addition, job_backup_recovery_plan_to_remote.sh stores recovery-plan files off-site unencrypted. 525 | Recovery-plan files are listed in the "Data recovery plan" section. 526 | 527 | Backup to both local and remote locations in case disaster destroys one. 528 | If the Internet connection fails, local backup is still made. 529 | 530 | ** Example restore data 531 | To restore data, copy files from backup to destination. 532 | 533 | You can use cp (shell command) to restore data from local unencrypted backup.\\ 534 | To copy a single file from local backup: 535 | : $ cp -p local_backup_path dest_path 536 | To copy a last_snapshot directory from local backup: 537 | : $ cp -a local_backup/last_snapshot dest_path 538 | 539 | Use rclone to restore data from remote or encrypted backup.\\ 540 | To copy a single file from remote backup: 541 | : $ rclone copy remote:source_path dest:dest_path 542 | To copy a single file from remote backup, use one of these scripts: 543 | - [[./examples/job_restore_last_snapshot.sh][examples/job_restore_last_snapshot.sh]] 544 | - [[./examples/job_restore_directory_from_remote.sh][examples/job_restore_directory_from_remote.sh]] 545 | 546 | ** Test backup jobs and test restore-data jobs 547 | The following commands test the example backup and restore jobs. 548 | They test your entire data recovery system end to end, testing both the data backup and data recovery together. 549 | Don't worry, the tutorial's environment is setup to make testing painless. 550 | 551 | Clear and setup test directories in preparation for a new test run: 552 | : $ $rclone_jobber/examples/clear_USB_test_backup.sh 553 | : $ $rclone_jobber/examples/clear_remote_test_backup.sh 554 | : $ $rclone_jobber/examples/setup_test_data_directory.sh 555 | 556 | Back up data: 557 | : $ $rclone_jobber/examples/job_backup_to_USB.sh 558 | : $ $rclone_jobber/examples/job_backup_to_remote.sh 559 | 560 | In job_restore_last_snapshot.sh, uncomment source variable to restore data from. 561 | Then restore data: 562 | : $ $rclone_jobber/examples/job_restore_last_snapshot.sh 563 | 564 | Verify that the files were faithfully restored: 565 | : $ diff -r $HOME/test_rclone_data/direc0 $HOME/last_snapshot/direc0 566 | 567 | Notice that rclone does not back up empty directories. 568 | 569 | Follow a similar test procedure when practicing your recovery plan, but with real data. 570 | 571 | ** Monitor your backups 572 | Monitor your backups to make that data is actually being backed up. 573 | Do not rely solely on warning messages or rclone_jobber.log. 574 | They do not prove that data was saved to destination. 575 | 576 | Manually run a checklist script once per month, similar to this monitor_backups.sh: 577 | 578 | #+BEGIN_EXAMPLE 579 | #!/bin/bash 580 | 581 | echo "" 582 | echo ">>>> Check recently changed file time in local backup:" 583 | ls -l /run/media/wolfv/big_stick/wolfv_backup/last_snapshot/DATA/Documents/tasks/tasks.org 584 | 585 | echo "" 586 | echo ">>>> Check recently changed file time in remote backup:" 587 | rclone lsl onedrive_wolfv_backup_crypt:last_snapshot/DATA/Documents/tasks --max-depth 1 588 | 589 | echo "" 590 | echo ">>>> Check last log time:" 591 | tail -5 /var/log/rclone_jobber.log 592 | 593 | echo "" 594 | echo ">>>> Check my Monthly Report emailed from my monitoring service." 595 | 596 | echo "" 597 | echo ">>>> Check space usage and available space." 598 | #+END_EXAMPLE 599 | 600 | * Data recovery plan 601 | ** Data recovery plan documents 602 | A data recovery plan is a documented process to recover and protect data in the event of a disaster. 603 | The data recovery plan presented here also includes re-installing the operating system. 604 | 605 | Example data recovery plan: 606 | 1. Retrieve recovery-plan files from an on-site or off-site location 607 | - notes for installing OS 608 | - recovery plan (this file) 609 | - job_restore_last_snapshot.sh 610 | - ~/.config/rclone/rclone.conf 611 | 2. Install operating system 612 | 3. Install rclone 613 | 4. Restore ~/.config/rclone/rclone.conf 614 | 5. Edit source variable in job_restore_last_snapshot.sh, and then run job_restore_last_snapshot.sh 615 | 616 | The rclone.conf configuration file contains the encryption key for backup. 617 | Keep it in a secure location. 618 | I keep my backup rclone.conf in a password manager (LastPass). 619 | The other recovery-plan files (listed in item 1.) are not encrypted so that they can be accessed before rclone is installed. 620 | With this setup, all I need to bootstrap the recovery process is a web browser and my LastPass master password. 621 | 622 | Schedule the backup of your backup recovery plan. 623 | This ensures that your backup recovery-plan files are always up-to-date. 624 | Do not encrypt the recovery-plan files so that they can be accessed before installing rclone. 625 | For each backup location, place the recovery-plan files in a directory to be backed up. 626 | - If a backup is not encrypted, then the recovery-plan files will be accessible in the backup. 627 | - If a backup is encrypted, create an unencrypted backup job to the same underlying remote. 628 | Like this example: 629 | - [[./examples/job_backup_recovery_plan_to_remote.sh][job_backup_recovery_plan_to_remote.sh]] 630 | - [[./examples/filter_rules_recovery_plan][filter_rules_recovery_plan]] 631 | - and schedule job_backup_recovery_plan_to_remote.sh to run automatically 632 | 633 | Practice the recovery plan. 634 | Start from scratch with a blank environment (or use a different location on the current machine). 635 | You’ll run into snags, and that is the point. Workout the snags BEFORE data is lost. 636 | If you have enough disk space, restore all your data to a different directory. 637 | And then use diff to verify the accuracy of the restored data. 638 | 639 | ** Data-recovery-plan monitoring checklist 640 | Example annual recovery-plan checklist: 641 | 1. review your recovery plan 642 | 2. make sure the recovery-plan files are still accessible and up-to-date (listed in previous section) 643 | - on-site copy 644 | - off-site copy 645 | 3. practice restore-data on small test directory, from $rclone_jobber/examples: 646 | 1. setup_test_data_directory.sh 647 | 2. job_backup_to_USB.sh 648 | 3. job_backup_to_remote.sh 649 | 4. delete the ~/test_data_directory 650 | 5. job_restore_last_snapshot.sh 651 | 652 | * License 653 | [[http://creativecommons.org/licenses/by-nc-sa/4.0/][https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png]]\\ 654 | rclone_jobber_tutorial.org by Wolfram Volpi is licensed under a [[http://creativecommons.org/licenses/by-nc-sa/4.0/][Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License]]. 655 | Based on a work at https://github.com/wolfv6/rclone_jobber. 656 | Permissions beyond the scope of this license may be available at https://github.com/wolfv6/rclone_jobber/issues. 657 | 658 | rclone_jobber is not affiliated with rclone. 659 | -------------------------------------------------------------------------------- /test_suite/README.org: -------------------------------------------------------------------------------- 1 | * How the rclone_jobber test suite works 2 | Each of the following sections describes one test case. 3 | 4 | Test-script names start with "test_". 5 | Each test_ script creates two directories: 6 | - test_rclone_data 7 | - test_rclone_backup 8 | 9 | Each test_ script tests two or three job_ scripts. 10 | Job-script names start with "job_". 11 | 12 | Test procedure for each test script: 13 | 1. run the test script from the terminal 14 | 2. manually inspect output for expected results (expected results are listed in test output and enclosed in parenthesis) 15 | 16 | rlcone_jobber.sh was tested on Linux Fedora 27, Gnome 3.26.2, rclone v1.39 17 | rlcone_jobber.sh was tested on Linux Debian 9, Gnome 3.22.2, rclone v1.41 18 | 19 | * test_last_snapshot.sh 20 | Testing: 21 | - filter exclude files ending with listed suffix 22 | - attempt to restore deleted sub-directory from last snapshot 23 | 24 | Passed 5/27/18 25 | 26 | Expected result: 27 | - file names ending in "_exe" are excluded from backup 28 | - deleted directory direc1/direc1b/ can not be restored because move_old_files_to=\"\" 29 | 30 | Dependencies: 31 | - job_snapshot_backup_old_data_to_delete.sh 32 | - job_snapshot_restore_from_last_snapshot.sh 33 | - rclone_jobber.sh 34 | - filter_rules 35 | 36 | * test_restore_deleted_directory.sh 37 | This test is similar to test_onedrive.sh but w/o encryption. 38 | 39 | Testing: 40 | - move_old_files_to="typo" should give pop-up warning and default to "dated_directory" 41 | - attempt to back up a deleted top directory gets rclone message, and does not delete files 42 | - restore deleted sub-directory 43 | 44 | Passed 5/27/18 45 | 46 | Expected result: 47 | - 2 pop-up warnings: 48 | : WARNING: Parameter move_old_files_to=typo, but should be... 49 | : WARNING: Parameter move_old_files_to=typo, but should be... 50 | - rclone_jobber.log (located in same directory as rclone_jobber.sh) contains a WARNINGs and a INFOs: 51 | : WARNING: Parameter move_old_files_to=typo, but should be... 52 | : ... 53 | : 2018/02/19 16:17:51 INFO : Local file system at /home/wolfv/test_rclone_backup/last_snapshot: ... 54 | : ... 55 | - test will prompt for date_time of direc1b to restore 56 | - data contains restored direc1/direc1b_$timestamp/f1b 57 | 58 | Dependencies: 59 | - job_dir_backup_to_dated_directory.sh 60 | - job_dir_restore_from_dated_directory.sh 61 | - rclone_jobber.sh 62 | 63 | * test_USB.sh 64 | This test script uses environment variable USB, set it to your USB's path. 65 | For example, in Linux add this line to ~./profile file: 66 | : export USB="/run/media/user/myUSB" 67 | 68 | and then source from command line: 69 | : $ source ~/.profile 70 | : $ echo $USB 71 | : /run/media/user/myUSB 72 | 73 | Testing: 74 | - backup to USB 75 | - move_old_data="dated_files" 76 | - restore dated file 77 | - --checksum option 78 | 79 | Passed 5/27/18 80 | 81 | Expected result: 82 | - old f1 file is restored in f1_ 83 | 84 | Dependencies: 85 | - job_USB_backup_to_USB.sh 86 | - job_USB_restore_from_USB.sh 87 | - rclone_jobber.sh 88 | 89 | * test_1d.sh 90 | "1d" means OneDrive remote. 91 | This test is similar to test_restore_deleted_directory.sh but on encrypted OneDrive. 92 | 93 | To set up a remote, follow instructions on https://rclone.org/onedrive/ substituting this value: 94 | - name> onedrive 95 | 96 | To set up encryption, follow instructions on https://rclone.org/crypt/ substituting these values: 97 | - name> onedrive_crypt 98 | - remote> onedrive:test_rclone_backup 99 | 100 | Testing: 101 | - backup to encrypted OneDrive 102 | - same as [[*test_restore_deleted_directory.sh][test_restore_deleted_directory.sh]] 103 | - abort if job is already running (maybe previous run didn't finish) 104 | - --checksum option 105 | 106 | Open two terminals 107 | Manually test "job aborted because it is already running." 108 | : $ ./test_1d.sh 109 | 110 | It takes 1 minute for a remote job to complete. 111 | While that's running, call the backup job again from the other terminal: 112 | : $ ./job_1d_backup_to_dated_directory.sh 113 | : job_1d_backup_to_dated_directory.sh aborted because it is already running. 114 | 115 | Passed 5/27/18 116 | 117 | Expected result: 118 | - see expected result listing in [[*test_restore_deleted_directory.sh][test_restore_deleted_directory.sh]] 119 | - the OneDrive backup has two copies of deleted files: dated directory and OneDrive Recycle bin 120 | - deleted files are moved to a dated directory, which is use for restoring a deleted directory 121 | - deleted files are moved to the OneDrive Recycle bin (most cloud storage providers don't or provide a way to turn it off) 122 | - you can manually empty the OneDrive Recycle bin from the OneDrive website 123 | 124 | Dependencies: 125 | - job_1d_backup_to_dated_directory.sh 126 | - job_1d_restore_from_dated_directory.sh 127 | - rclone_jobber.sh 128 | 129 | * test_check_range.sh 130 | Testing: 131 | - empty source 132 | - null source string 133 | - null dest string 134 | - monitoring_URL 135 | 136 | Passed 5/27/18 137 | 138 | Expected result: 139 | - all 3 backup jobs generate an error message, no backups are made 140 | - monitoring_URL sends an email after 24 hrs grace period 141 | 142 | Dependencies: 143 | - job_check_empty_source.sh 144 | - job_check_null_source.sh 145 | - job_check_null_dest.sh 146 | - rclone_jobber.sh 147 | 148 | * License 149 | [[http://creativecommons.org/publicdomain/zero/1.0/][http://i.creativecommons.org/p/zero/1.0/88x31.png]]\\ 150 | This README file and other files contained in this test_suite directory are for testing rclone_jobber.sh. 151 | Written in 2018 by Wolfram Volpi, contact at https://github.com/wolfv6/rclone_jobber/issues. 152 | To the extent possible under law, the author(s) have dedicated all copyright and related and neighboring rights to this software to the public domain worldwide. 153 | This software is distributed without any warranty. 154 | You should have received a copy of the CC0 Public Domain Dedication along with this software. If not, see http://creativecommons.org/publicdomain/zero/1.0/. 155 | 156 | rclone_jobber is not affiliated with rclone. 157 | -------------------------------------------------------------------------------- /test_suite/filter_rules: -------------------------------------------------------------------------------- 1 | #The rules are processed in the order that they are defined. 2 | 3 | ########## include files not in top directories ########### 4 | #e.g. config files 5 | + direc1/.bashrc 6 | 7 | ##################### exclude patterns #################### 8 | 9 | #file names with extension 10 | - *.bak 11 | - *.log 12 | - *.tmp 13 | 14 | #file names ending in "_exc" w/o extension 15 | - *_exc 16 | 17 | #file names ending in "_exc." and any extension 18 | - *_exc.* 19 | 20 | #directory names ending in "_exc" 21 | - *_exc/** 22 | 23 | #EMACS LOCKS exclude made redundant by HIDDEN FILES 24 | #- .#* 25 | 26 | #HIDDEN FILES 27 | - .* 28 | 29 | ################ include top directories ############### 30 | + direc1/** 31 | 32 | #exclude all other top directories and files 33 | - * -------------------------------------------------------------------------------- /test_suite/job_1d_backup_to_dated_directory.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | source="$HOME/test_rclone_data" 4 | dest="onedrive_test_rclone_backup_crypt:" 5 | move_old_files_to="typo" #should give pop-up warning and default to "dated_directory" 6 | options="--checksum" 7 | 8 | ../rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" 9 | -------------------------------------------------------------------------------- /test_suite/job_1d_restore_from_dated_directory.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | #restore direc1 directory to it's original location 3 | 4 | #source and destination paths 5 | echo ">>>>>>>>>>>>>>>>>>> Enter time stamp of direc1b to restore <<<<<<<<<<<<<<<<<<<<" 6 | read old 7 | source="onedrive_test_rclone_backup_crypt:$old/direc1/direc1b" 8 | dest="$HOME/test_rclone_data/direc1/direc1b_$old" 9 | 10 | printf "\n*** restoring directory direc1b ***\n" 11 | rclone copy $source $dest 12 | 13 | #################### output for testing ################ 14 | printf "*** data directory ***\n" 15 | tree ~/test_rclone_data 16 | 17 | f1b="$HOME/test_rclone_data/direc1/direc1b_$old/f1b" 18 | echo "contents of $f1b: $(cat $f1b)" 19 | -------------------------------------------------------------------------------- /test_suite/job_USB_backup_to_USB.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | source="$HOME/test_rclone_data" 4 | dest="$USB/test_rclone_backup" 5 | move_old_files_to="dated_files" 6 | options="--checksum" 7 | 8 | ../rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" 9 | -------------------------------------------------------------------------------- /test_suite/job_USB_restore_from_USB.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | #restore old version of f1 to it's original location 3 | 4 | #source and destination paths 5 | source="$USB/test_rclone_backup/old_files/direc1/f1_*" 6 | dest="$HOME/test_rclone_data/direc1" 7 | 8 | #restore file 9 | rclone copy $source $dest 10 | -------------------------------------------------------------------------------- /test_suite/job_check_empty_source.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | source="$HOME/test_rclone_data" 4 | dest="$USB/test_rclone_backup" 5 | move_old_files_to="dated_files" 6 | monitoring_URL="https://hchk.io/c74b2d3c-dfb9-4cdb-b81a-872618dfd0b8" 7 | 8 | ../rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" "$monitoring_URL" 9 | -------------------------------------------------------------------------------- /test_suite/job_check_null_dest.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | source="$HOME/test_rclone_data" 4 | dest="" 5 | move_old_files_to="dated_files" 6 | monitoring_URL="https://hchk.io/c74b2d3c-dfb9-4cdb-b81a-872618dfd0b8" 7 | 8 | ../rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" "$monitoring_URL" 9 | -------------------------------------------------------------------------------- /test_suite/job_check_null_source.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | source="" 4 | dest="$USB/test_rclone_backup" 5 | move_old_files_to="dated_files" 6 | monitoring_URL="https://hchk.io/c74b2d3c-dfb9-4cdb-b81a-872618dfd0b8" 7 | 8 | ../rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" "$monitoring_URL" 9 | -------------------------------------------------------------------------------- /test_suite/job_dir_backup_to_dated_directory.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | source="$HOME/test_rclone_data" 4 | dest="$HOME/test_rclone_backup" 5 | move_old_files_to="typo" #should give pop-up warning and move old data to dated_directory 6 | 7 | ../rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" 8 | -------------------------------------------------------------------------------- /test_suite/job_dir_restore_from_dated_directory.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | #restore direc1 directory to it's original location 3 | 4 | #source and destination paths 5 | echo "\n>>>>>>>>>>>>>> Enter time stamp of direc1b to restore <<<<<<<<<<<<<<<<<" 6 | read old 7 | source="$HOME/test_rclone_backup/$old/direc1/direc1b" 8 | dest="$HOME/test_rclone_data/direc1/direc1b_$old" 9 | 10 | printf "\n*** restoring deleted directory direc1b ***\n" 11 | rclone copy $source $dest 12 | -------------------------------------------------------------------------------- /test_suite/job_snapshot_backup_old_data_to_delete.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | source="$HOME/test_rclone_data" 4 | dest="$HOME/test_rclone_backup" 5 | options="--filter-from=filter_rules" 6 | 7 | ../rclone_jobber.sh "$source" "$dest" "$move_old_files_to" "$options" "$(basename $0)" 8 | -------------------------------------------------------------------------------- /test_suite/job_snapshot_restore_from_last_snapshot.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | #restore direc1 directory to it's original location 3 | 4 | #source and destination paths 5 | source="$HOME/test_rclone_backup/last_snapshot/direc1/direc1b" 6 | dest="$HOME/test_rclone_data/direc1/direc1b_last_snapshot" 7 | 8 | #restore directory 9 | rclone copy $source $dest 10 | -------------------------------------------------------------------------------- /test_suite/test_1d.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | ############## setup directories ############# 4 | 5 | #delete and recreate test_rclone_data 6 | rm -r ~/test_rclone_data 7 | mkdir ~/test_rclone_data 8 | 9 | mkdir ~/test_rclone_data/direc0 10 | touch ~/test_rclone_data/direc0/f0 11 | 12 | mkdir ~/test_rclone_data/direc1 13 | touch ~/test_rclone_data/direc1/f1 14 | 15 | mkdir ~/test_rclone_data/direc1/direc1a 16 | touch ~/test_rclone_data/direc1/direc1a/f1a 17 | 18 | mkdir ~/test_rclone_data/direc1/direc1b 19 | printf "text" > ~/test_rclone_data/direc1/direc1b/f1b 20 | 21 | #delete test_rclone_backup 22 | rclone purge onedrive_test_rclone_backup_crypt: 23 | 24 | ################## call jobs ##################### 25 | 26 | printf "*** backup (should be: directory not found) ***\n" 27 | rclone ls onedrive_test_rclone_backup_crypt: 28 | 29 | printf "\n*** performing first backup (should have pop-up WARNING: job_1d_backup_to_dated_directory.sh) ***\n" 30 | ./job_1d_backup_to_dated_directory.sh 31 | 32 | printf "\n*** backup ***\n" 33 | rclone ls onedrive_test_rclone_backup_crypt: 34 | 35 | printf "\n*** data directory ***\n" 36 | tree ~/test_rclone_data 37 | 38 | printf "\n*** deleting top directory direc0 ***\n" 39 | rm -r ~/test_rclone_data/direc0 40 | 41 | printf "*** deleting sub-directory direc1b ***\n" 42 | rm -r ~/test_rclone_data/direc1/direc1b 43 | 44 | printf "*** data directory (should be missing direc0 and direc1b) ***\n" 45 | tree ~/test_rclone_data 46 | 47 | printf "\n*** performing second backup (should have pop-up WARNING: job_1d_backup_to_dated_directory.sh) ***\n" 48 | ./job_1d_backup_to_dated_directory.sh 49 | 50 | printf "\n*** backup (f0 and f1b should be moved to dated directory) ***\n" 51 | rclone ls onedrive_test_rclone_backup_crypt: 52 | 53 | #prompt for deleted direc1b timestamp, restore, and output data directory 54 | ./job_1d_restore_from_dated_directory.sh 55 | -------------------------------------------------------------------------------- /test_suite/test_USB.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | ############## setup directories ############# 4 | 5 | #delete and recreate test_rclone_data 6 | rm -r ~/test_rclone_data 7 | mkdir ~/test_rclone_data 8 | 9 | mkdir ~/test_rclone_data/direc0 10 | touch ~/test_rclone_data/direc0/f0 11 | 12 | mkdir ~/test_rclone_data/direc1 13 | printf "original" > ~/test_rclone_data/direc1/f1 14 | 15 | mkdir ~/test_rclone_data/direc1/direc1a 16 | touch ~/test_rclone_data/direc1/direc1a/f1a 17 | 18 | mkdir ~/test_rclone_data/direc1/direc1b 19 | touch ~/test_rclone_data/direc1/direc1b/f1b 20 | 21 | #delete and recreate test_rclone_backup 22 | rm -r $usb/test_rclone_backup 23 | mkdir $usb/test_rclone_backup 24 | 25 | ################## call jobs ##################### 26 | 27 | printf "*** backup (should be empty) ***\n" 28 | tree $usb/test_rclone_backup 29 | 30 | printf "\n*** performing first backup ***\n" 31 | ./job_USB_backup_to_USB.sh 32 | 33 | printf "\n*** editing f1 file ***\n" 34 | printf "edited" > ~/test_rclone_data/direc1/f1 35 | 36 | printf "*** data directory ***\n" 37 | tree $HOME/test_rclone_data 38 | 39 | printf "\n*** performing second backup ***\n" 40 | ./job_USB_backup_to_USB.sh 41 | 42 | printf "\n*** backup (f1_ should be in old_files) ***\n" 43 | tree $usb/test_rclone_backup 44 | 45 | printf "\n*** restoring old f1 ***\n" 46 | ./job_USB_restore_from_USB.sh 47 | 48 | printf "*** data directory (f1_ and f1 should be in direc1) ***\n" 49 | tree $HOME/test_rclone_data 50 | -------------------------------------------------------------------------------- /test_suite/test_check_range.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | ############## setup directories ############# 4 | 5 | #delete and recreate test_rclone_data 6 | rm -r ~/test_rclone_data 7 | mkdir ~/test_rclone_data 8 | 9 | touch ~/test_rclone_data/f 10 | 11 | mkdir ~/test_rclone_data/direc0 12 | touch ~/test_rclone_data/direc0/f0 13 | 14 | #delete and recreate test_rclone_backup 15 | rm -r $usb/test_rclone_backup 16 | mkdir $usb/test_rclone_backup 17 | 18 | ################## call jobs ##################### 19 | 20 | printf "*** backup (should be empty) ***\n" 21 | tree $usb/test_rclone_backup 22 | 23 | printf "\n*** performing backup from null string (should have pop-up ERROR: job_check_null_source.sh aborted) ***\n" 24 | ./job_check_null_source.sh 25 | 26 | printf "\n*** backup (should be empty) ***\n" 27 | tree $usb/test_rclone_backup 28 | 29 | printf "\n*** performing backup to null string (should have pop-up ERROR: job_check_null_source.sh aborted) ***\n" 30 | ./job_check_null_dest.sh 31 | 32 | printf "\n*** backup (should be empty) ***\n" 33 | tree $usb/test_rclone_backup 34 | 35 | printf "\n*** deleting data directory ***\n" 36 | rm -r ~/test_rclone_data 37 | mkdir ~/test_rclone_data 38 | 39 | printf "*** performing backup from empty source (should have pop-up ERROR: job_check_null_source.sh aborted) ***\n" 40 | ./job_check_empty_source.sh 41 | 42 | printf "\n*** backup (should be empty) ***\n" 43 | tree $usb/test_rclone_backup 44 | -------------------------------------------------------------------------------- /test_suite/test_last_snapshot.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | ############## setup test directories ############# 4 | 5 | #delete and recreate test_rclone_data 6 | rm -r ~/test_rclone_data 7 | mkdir ~/test_rclone_data 8 | 9 | mkdir ~/test_rclone_data/direc0 10 | touch ~/test_rclone_data/direc0/f0 11 | 12 | mkdir ~/test_rclone_data/direc1 13 | touch ~/test_rclone_data/direc1/f1 14 | 15 | mkdir ~/test_rclone_data/direc1/direc1a 16 | touch ~/test_rclone_data/direc1/direc1a/f1a 17 | 18 | mkdir ~/test_rclone_data/direc1/direc1b 19 | touch ~/test_rclone_data/direc1/direc1b/f1b 20 | 21 | mkdir ~/test_rclone_data/direc1/direc1c 22 | touch ~/test_rclone_data/direc1/direc1c/f1c 23 | 24 | #files and directory names ending in "_exc" 25 | touch ~/test_rclone_data/direc1/f1_exc 26 | touch ~/test_rclone_data/direc1/f2_exc.txt 27 | touch ~/test_rclone_data/direc1/f3_excy 28 | touch ~/test_rclone_data/direc1/f4_exc.not.txt 29 | mkdir ~/test_rclone_data/direc1/direc5_exc 30 | touch ~/test_rclone_data/direc1/direc5_exc/f5 31 | touch ~/test_rclone_data/direc1/f6.bak 32 | 33 | touch ~/test_rclone_data/direc1/.hidden 34 | touch ~/test_rclone_data/direc1/.bashrc 35 | touch ~/test_rclone_data/direc1/.#lock.sh 36 | 37 | #delete and recreate test_rclone_backup 38 | rm -r ~/test_rclone_backup 39 | mkdir ~/test_rclone_backup 40 | 41 | ################## call jobs ##################### 42 | 43 | printf "\n*** data directory, with _exc files to be excluded from backup ***\n" 44 | tree -a ~/test_rclone_data 45 | 46 | printf "\n*** backup (should be empty) ***\n" 47 | tree -a ~/test_rclone_backup 48 | 49 | printf "\n*** performing first backup ***\n" 50 | ./job_snapshot_backup_old_data_to_delete.sh 51 | 52 | printf "\n*** backup (file names ending in _exc should be excluded) ***\n" 53 | tree -a ~/test_rclone_backup 54 | 55 | printf "\n*** removing direc1b ***\n" 56 | rm -r ~/test_rclone_data/direc1/direc1b 57 | 58 | printf "*** performing second backup ***\n" 59 | ./job_snapshot_backup_old_data_to_delete.sh 60 | 61 | printf "\n*** backup (direc1b should be missing) ***\n" 62 | tree -a ~/test_rclone_backup 63 | 64 | printf "\n*** attempting to restore direc1 from last snapshot ***\n" 65 | ./job_snapshot_restore_from_last_snapshot.sh 66 | 67 | printf "\n*** data directory (direc1b should be missing) ***\n" 68 | tree -a ~/test_rclone_data 69 | 70 | -------------------------------------------------------------------------------- /test_suite/test_restore_deleted_directory.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env sh 2 | 3 | ############## setup directories ############# 4 | 5 | #delete and recreate test_rclone_data 6 | rm -r ~/test_rclone_data 7 | mkdir ~/test_rclone_data 8 | 9 | mkdir ~/test_rclone_data/direc0 10 | touch ~/test_rclone_data/direc0/f0 11 | 12 | mkdir ~/test_rclone_data/direc1 13 | touch ~/test_rclone_data/direc1/f1 14 | 15 | mkdir ~/test_rclone_data/direc1/direc1a 16 | touch ~/test_rclone_data/direc1/direc1a/f1a 17 | 18 | mkdir ~/test_rclone_data/direc1/direc1b 19 | touch ~/test_rclone_data/direc1/direc1b/f1b 20 | 21 | #delete and recreate test_rclone_backup 22 | rm -r ~/test_rclone_backup 23 | mkdir ~/test_rclone_backup 24 | 25 | ################## call jobs ##################### 26 | 27 | printf "\n*** backup (should be empty) ***\n" 28 | tree ~/test_rclone_backup 29 | 30 | printf "\n*** performing first backup (should have pop-up WARNING: Parameter move_old_files_to=typo) ***\n" 31 | ./job_dir_backup_to_dated_directory.sh 32 | 33 | printf "\n*** backup ***\n" 34 | tree ~/test_rclone_backup 35 | 36 | printf "\n*** deleting top directory direc0 ***\n" 37 | rm -r ~/test_rclone_data/direc0 38 | 39 | printf "*** deleting sub-directory direc1b ***\n" 40 | rm -r ~/test_rclone_data/direc1/direc1b 41 | 42 | printf "*** data directory (should be missing direc0 and direc1b) ***\n" 43 | tree ~/test_rclone_data 44 | 45 | printf "\n*** performing second backup (should have pop-up WARNING: Parameter move_old_files_to=typo) ***\n" 46 | ./job_dir_backup_to_dated_directory.sh 47 | 48 | printf "\n*** backup (direc0 and direc1b should be moved to dated directory) ***\n" 49 | tree ~/test_rclone_backup 50 | 51 | #prompts for date_time of direc1b 52 | ./job_dir_restore_from_dated_directory.sh 53 | 54 | printf "*** data directory (should have restored directory direc1b_) ***\n" 55 | tree ~/test_rclone_data 56 | --------------------------------------------------------------------------------