├── LICENSE
├── README.md
├── rsync-invoke.sh
└── rsync-module.sh


/LICENSE:
--------------------------------------------------------------------------------
 1 | MIT License
 2 | 
 3 | Copyright (c) 2020 Keith Nash
 4 | 
 5 | Permission is hereby granted, free of charge, to any person obtaining a copy
 6 | of this software and associated documentation files (the "Software"), to deal
 7 | in the Software without restriction, including without limitation the rights
 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 9 | copies of the Software, and to permit persons to whom the Software is
10 | furnished to do so, subject to the following conditions:
11 | 
12 | The above copyright notice and this permission notice shall be included in all
13 | copies or substantial portions of the Software.
14 | 
15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21 | SOFTWARE.
22 | 


--------------------------------------------------------------------------------
/README.md:
--------------------------------------------------------------------------------
  1 | # Scripts for using rsync
  2 | 
  3 | The __rsync__ command -- whether used standalone or installed as a module -- is a powerful tool for copying filesets between UNIX-style systems. It is very efficient in that it only copies new or changed data to the target system.
  4 | 
  5 | ### Purpose
  6 | 
  7 | I wrote these wrapper scripts for the purpose of duplicating datasets from my primary to my secondary FreeNAS server. Because FreeNAS is based on FreeBSD, these scripts _might_ be somewhat FreeBSD-centric. But __rsync__ is a common tool on all UNIX-style systems, so it shouldn't take much effort to port the scripts to run successfully on Linux distributions, and in fact, I have used earlier versions of these scripts to transfer data to a Linux-based Synology Diskstation NAS system.
  8 | 
  9 | My goal was to copy new or changed files and also to delete files on the target that don't exist on the source. If your needs are different, particularly if you want to keep files on the target that have been deleted on the source, then you will need to remove the `--inplace` and  `--delete-during` options used in the scripts.
 10 | 
 11 | ### The Scripts
 12 | There are two scripts in this repository: one for use with modules (__rsync-module.sh__) and one to run __rsync__ directly (__rsync-invoke.sh__). Both scripts require 3 command-line arguments:
 13 | - The source specification, including username and hostname for remote systems - example: `/mnt/tank/foo/`
 14 | - The target specification, including username and hostname for remote systems - example: `root@boomer:/mnt/tank/foo`
 15 | - A log filename
 16 | 
 17 | Telling __rsync__ what to copy is a bit arcane: you have to be careful about placing the '/' character correctly. Basically, to copy a dataset from the source to the target you add a trailing '/' to the source specification and leave it off the target.
 18 | 
 19 | This is easier to explain with an example: to use __rsync-invoke.sh__ to copy local dataset __foo__ to remote server 'BOOMER', use this command line:
 20 | 
 21 | `./rsync-invoke.sh /mnt/tank/foo/ root@boomer:/mnt/tank/foo /mnt/tank/bandit/log/rsync.log`
 22 | 
 23 | Modules work a little differently when specifying the path. You don't separate the user and server names from the target path with a colon, and instead of providing the full path of the target, you specify the module name. Again, this is easier to demonstrate with an example:
 24 | 
 25 | `./rsync-module.sh /mnt/tank/foo/ root@boomer/tank/foo /mnt/tank/bandit/log/rsync.log`
 26 | 
 27 | After either of the examples above complete, you can examine log file `/mnt/tank/bandit/log/rsync.log` for results, which will look something like this:
 28 | ```
 29 | +---------------------------------------------------------------------------------
 30 | + Mon Aug 10 04:00:05 CDT 2020: Copy /mnt/tank/foo/ to root@boomer/tank/foo
 31 | +---------------------------------------------------------------------------------
 32 | 2020/08/10 04:00:06 [10732] building file list
 33 | 2020/08/10 04:00:37 [10732] *deleting   ONYX/Macrium/onyx-system3-00-00.mrimg
 34 | 2020/08/10 04:00:37 [10732] *deleting   ONYX/Macrium/onyx-data3-00-00.mrimg
 35 | 2020/08/10 04:00:37 [10732] .d..t...... ONYX/Macrium/
 36 | 2020/08/10 04:00:37 [10732] <f.st...... ONYX/Macrium/file-groom.log
 37 | 2020/08/10 04:08:02 [10732] <f+++++++++ ONYX/Macrium/onyx-data1-00-00.mrimg
 38 | 2020/08/10 04:10:35 [10732] <f+++++++++ ONYX/Macrium/onyx-system2-00-00.mrimg
 39 | 2020/08/10 04:10:36 [10732] sent 129.62G bytes  received 5.02K bytes  205.26M bytes/sec
 40 | 2020/08/10 04:10:36 [10732] total size is 2.93T  speedup is 22.58
 41 | + Mon Aug 10 04:10:36 CDT 2020 Transfer completed
 42 | ```
 43 | 
 44 | ### Push or Pull?
 45 | My use of these scripts is strictly _push_ oriented: the 'source' I specify is always a local dataset; the 'target' is always on a remote system. But note that you can use __rsync-invoke.sh__ to _pull_ data from a remote system as well.
 46 | 
 47 | Example: this command will copy dataset 'foo' from remote server 'BOOMER' to the local system:
 48 | 
 49 | `./rsync-invoke.sh root@boomer:/mnt/tank/foo/ /mnt/tank/foo /mnt/tank/bandit/log/rsync.log`
 50 | 
 51 | The module script (__rsync-module.sh__) is strictly _push_ oriented: it can only be used to copy data from the local system to a remote rsync module because the target specifier has the `rsync://` prefix hard-coded. But you could easily modify a copy of this script and put the `rsync://` prefix on the source specifier if you need _pull_ capability.
 52 | 
 53 | ### Prerequisites: SSH
 54 | Since __rsync__ typically uses __ssh__, you will need to configure __ssh__ key-based authentication to allow logging on to your target servers without having to enter a password.
 55 | 
 56 | ### Prerequisites: rsync modules
 57 | You will need to configure __rsync__ modules if you plan to use them as targets. 
 58 | 
 59 | Example: on my FreeNAS server 'BOOMER' I have configured a single __rsync__ module named 'tank', with a path of '/mnt/tank', access mode of 'Read and Write', user 'root', and group 'wheel'.
 60 | 
 61 | ### Slow Network Performance
 62 | 
 63 | Just about every __rsync__ user notices how slow it is at transferring data. This is usually due to using __ssh__ as the transport protocol, with its attendant encryption. A common approach to overcoming slow transfers is to use less CPU-intensive encryption algorithms or to do away with encryption altogether. I have found that using __rsync__ modules is much faster than standalone mode, and that disabling encryption speeds up standalone transfers.
 64 | 
 65 | On my 10Gb network, I get transfer rates of up to 2Gb/s using __rsync-module.sh__, quite a bit faster than the typical __rsync-invoke.sh__ rate of roughly 800Mb/s.
 66 | 
 67 | ### Windows ACL Data
 68 | 
 69 | Copying Windows ACLs can be a problem on some systems, particularly FreeNAS/FreeBSD, and I have selected options to avoid problems with this issue. On FreeNAS this means avoiding these options:
 70 | 
 71 | ```
 72 | -a, --archive Equals -rlptgoD (no -H, -A, -X)
 73 | -p, --perms   Preserve permissions
 74 | -A, --acls    Preserve ACLs (implies -p)
 75 | ```
 76 | See my post ["Impaired rsync permissions support for Windows datasets"](https://www.ixsystems.com/community/threads/impaired-rsync-permissions-support-for-windows-datasets.43973) on the iXsystem FreeNAS forum for further discussion on this issue.
 77 | 
 78 | On some Linux distributions, __rsync__ may support copying Windows ACLs directly, while on others it will not. In the latter case, users have reported success using __robocopy__ in conjunction with __rsync__ to transfer Windows ACL data.
 79 | 
 80 | To determine whether your environment supports copying Windows ACLs, explore the options above along with:
 81 | - `-X --xattrs preserve extended attributes`
 82 | 
 83 | ### Options
 84 | 
 85 | These are the __rsync__ options used in both scripts:
 86 | ```
 87 | -r  recurse into directories
 88 | -l  copy symlinks as symlinks
 89 | -t  preserve modification times
 90 | -g  preserve group
 91 | -o  preserve owner
 92 | -D  preserve device and special files
 93 | -h  human readable progress
 94 | -v  increase verbosity
 95 | --delete-during   receiver deletes during the transfer
 96 | --inplace         write updated data directly to destination file
 97 | --progress        show progress during transfer  
 98 | --log-file        specify log file
 99 | --exclude         exclude files
100 | ```
101 | Both scripts exclude the following files; modify or remove to suit your needs:
102 | ```
103 | vmware.log      VMware virtual machine log files
104 | vmware-*.log
105 | @eaDir/         Synology extended attributes
106 | @eaDir
107 | Thumbs.db       Windows folder-related system file
108 | ```
109 | The __rsync-invoke.sh__ script disables __SSH__ encryption and compression with these settings:
110 | ```
111 | -e "ssh -T -c none -o Compression=no -x"
112 | ```
113 | If your system does not support `none` as an encryption scheme, try `arcfour` or another low-cost encryption algorithm.
114 | 
115 | ## Example Usage
116 | 
117 | I run this `cron` script early every morning to synchronize datasets from my primary FreeNAS server 'BANDIT' to my secondary server 'BOOMER', using the __rsync__ module installed on 'BOOMER'. On both servers the datasets are stored on a pool named 'tank':
118 | 
119 | ```
120 | #!/bin/sh
121 | 
122 | # Synchronize all tank datasets from BANDIT to BOOMER
123 | 
124 | logfile=/mnt/tank/bandit/log/bandit-to-boomer.log
125 | 
126 | datasets="archives backups devtools domains hardware media music ncs opsys photo systools web"
127 | 
128 | rm ${logfile}
129 | for dataset in $datasets; do
130 |     # Use rsync-module.sh to target the rsync module on the remote server:
131 |     /mnt/tank/systems/scripts/rsync-module.sh /mnt/tank/$dataset/ root@boomer/tank/$dataset ${logfile}
132 | done
133 | ```
134 | 
135 | This script does exactly the same thing, only using __rsync-invoke.sh__ to call __rsync__ directly:
136 | ```
137 | #!/bin/sh
138 | 
139 | # Synchronize all tank datasets from BANDIT to BOOMER
140 | 
141 | logfile=/mnt/tank/bandit/log/bandit-to-boomer.log
142 | 
143 | datasets="archives backups devtools domains hardware media music ncs opsys photo systools web"
144 | 
145 | rm ${logfile}
146 | for dataset in $datasets; do
147 |     # Use rsync-invoke.sh to run rsync directly:
148 |     /mnt/tank/systems/scripts/rsync-invoke.sh /mnt/tank/$dataset/ root@boomer:/mnt/tank/$dataset ${logfile}
149 | done
150 | ```
151 | 
152 | 


--------------------------------------------------------------------------------
/rsync-invoke.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/sh
 2 | 
 3 | # Invoke rsync directly, via SSH
 4 | 
 5 | if [ $# -ne 3 ]; then
 6 |   echo "Error: not enough arguments!"
 7 |   echo "Usage is: $0 r_src r_dest r_logfile"
 8 |   exit 2
 9 | fi
10 | 
11 | R_SRC=$1
12 | R_DEST=$2
13 | R_LOGFILE=$3
14 | 
15 | # Options:
16 | 
17 | R_OPTIONS="-rltgoDhv --delete-during --inplace --progress --log-file="${R_LOGFILE}
18 | 
19 | # Files to exclude:
20 | 
21 | R_EXCLUDE="--exclude vmware.log --exclude vmware-*.log --exclude @eaDir/ --exclude @eaDir --exclude Thumbs.db"
22 | 
23 | echo "+---------------------------------------------------------------------------------" | tee -a "${R_LOGFILE}"
24 | echo "+ $(date): Copy $R_SRC to $R_DEST" | tee -a "${R_LOGFILE}"
25 | echo "+---------------------------------------------------------------------------------" | tee -a "${R_LOGFILE}"
26 | rsync ${R_OPTIONS} ${R_EXCLUDE} -e "ssh -T -c none -o Compression=no -x" ${R_SRC} ${R_DEST}
27 | echo "+ $(date) Transfer completed" | tee -a "${R_LOGFILE}"
28 | exit
29 | 


--------------------------------------------------------------------------------
/rsync-module.sh:
--------------------------------------------------------------------------------
 1 | #!/bin/sh
 2 | 
 3 | # Push to rsync module on target
 4 | 
 5 | if [ $# -ne 3 ]; then
 6 |   echo "Error: not enough arguments!"
 7 |   echo "Usage is: $0 r_src r_dest r_logfile"
 8 |   exit 2
 9 | fi
10 | 
11 | R_SRC=$1
12 | R_DEST=$2
13 | R_LOGFILE=$3
14 | 
15 | # Options:
16 | 
17 | R_OPTIONS="-rltgoDhv --delete-during --inplace --progress --log-file="${R_LOGFILE}
18 | 
19 | # Files to exclude:
20 | 
21 | R_EXCLUDE="--exclude vmware.log --exclude vmware-*.log --exclude @eaDir/ --exclude @eaDir --exclude Thumbs.db"
22 | 
23 | echo "+---------------------------------------------------------------------------------" | tee -a "${R_LOGFILE}"
24 | echo "+ $(date): Copy $R_SRC to $R_DEST" | tee -a "${R_LOGFILE}"
25 | echo "+---------------------------------------------------------------------------------" | tee -a "${R_LOGFILE}"
26 | rsync ${R_OPTIONS} ${R_EXCLUDE} ${R_SRC} rsync://${R_DEST}
27 | echo "+ $(date) Transfer completed" | tee -a "${R_LOGFILE}"
28 | exit
29 | 


--------------------------------------------------------------------------------