├── CHANGELOG.md ├── HOWTO.md ├── LICENSE ├── README.md ├── RESULTS-20220819.md ├── RESULTS-20220906.md └── script ├── backup-bench.conf └── backup-bench.sh /CHANGELOG.md: -------------------------------------------------------------------------------- 1 | # current master 2 | 3 | - Added new parameter --backup-id-timestamp to deal with multiple non git backups 4 | - Tuned SSH a bit more (mostly done in sshd anyway) 5 | 6 | # version 2022100201 7 | 8 | - Tune ZFS filesystem settings 9 | - Updated borg beta to version 2.0.0b2 10 | - Added debian compatibility (not tested) 11 | - Added duplicacy --threads 8 flag 12 | - Updated kopia to version 0.12.0 13 | - Try kopia s2-default compression algorith 14 | - Revert back to zstd since file size was 60% bigger, meaning we'd get biased comparaison 15 | - Added kopia --parallel 8 flag 16 | 17 | # version 2022090601 18 | 19 | - Converted restic HTTP backend to HTTPS 20 | 21 | # version 2022090501 22 | 23 | - Improved bupstash, borg, kopia, restic and duplicacy install process by downloading github releases instead of various installation scenarios 24 | - Removed restic_beta since new restic release 0.14.0 now has compression support 25 | - Added kopia HTTPS backend 26 | - Added RSA certificate generator 27 | - Added restic HTTP backend 28 | - Added new --config parameter to load different config files 29 | - Increased backup benchmark timeout from 5 to 10 hours 30 | - Added new --stop-http-serve parameter to kill kopia and restic servers 31 | - Lots of small fixes 32 | 33 | What isn't tested: 34 | - Failed commands that should stop execution (failed SSH copies, failed repo inits) 35 | 36 | What could be improved: 37 | - Check for SELinux labels before relabeling so we don't get errors (cosmetic only) 38 | - Check for existing user directories before creating them so we don't get errors (cosmetic only) 39 | 40 | # version 2022081901 41 | 42 | - Initial version 43 | -------------------------------------------------------------------------------- /HOWTO.md: -------------------------------------------------------------------------------- 1 | # How to setup backup-bench.sh yourself 2 | 3 | The backup-bench script supposes you have a source system with a RHEL 8/9 clone installed (PRs for other systems are welcome). 4 | The default configuration will delete and create the following folders: 5 | 6 | - /opt/backup_test as the git dataset download folder 7 | - /backup-bench-repos as the folder which will contain backup repositories (local or on remote target) 8 | - /tmp/backup-bench-restore as the folder where backup restoration tests happen 9 | 10 | You can customize those settings in `backup-bench.conf` file. 11 | 12 | ## Local benchmarks 13 | 14 | The script must prepare your machine by installing the requested backup software. You can do so with: 15 | 16 | ``` 17 | ./backup-bench.sh --setup-source 18 | ``` 19 | 20 | It can run as local backup benchmark solution only, in that case you should run the following commands: 21 | ``` 22 | ./backup-bench.sh --clear-repos 23 | ./backup-bench.sh --init-repos --git 24 | ./backup-bench.sh --benchmarks 25 | ``` 26 | 27 | The `--git` parameter for `--init-repos` command instructs the script to fetch the linux kernel source as backup source. 28 | This allows to have the same datasets for different workloads. 29 | You may also configure `BACKUP_ROOT` variable in `backup-bench.conf` to point to specific dataset and avoid using `--git` parameter. 30 | 31 | 32 | You might want to run multiple iterations of backups. 33 | In that case, you can run the following 34 | 35 | ``` 36 | ./backup-bench.sh --clear-repos 37 | ./backup-bench.sh --init-repos --git 38 | ./backup-bench.sh --benchmarks --git 39 | ``` 40 | 41 | Results can be found in `/var/log/backup-bench.log` and `/var/log/backup-bench.results.csv`. 42 | 43 | Moreever, backup solution log files can be found in `/var/log/backup-bench.[BACKUP SOLUTION].log` 44 | 45 | ## Remote benchmarks using SSH / SFTP backends 46 | 47 | Remote benchmarks assume you have a second (target) machine. 48 | Both source and target machines must be reachable via SSH. 49 | 50 | After having setup the necessary FQDN and ports in `backup-bench.conf`, you can initialize the target with: 51 | 52 | ``` 53 | ./backup-bench.sh --setup-target 54 | ``` 55 | 56 | The target machine will connect to your source server to upload the ssh keys necessary for the source machine to connect to your target. This requires you to enter the password once. 57 | Once this is setup, the target cannot connect to source anymore. 58 | 59 | Once this is done, the source machine can use the uploaded ssh keys to connect to the remote repositories on the target system. 60 | You can then prepare the benchmarks with 61 | 62 | ``` 63 | ./backup-bench.sh --clear-repos --remote 64 | ``` 65 | 66 | Optional step if using kopia / restic HTTP servers, on target: 67 | ``` 68 | ./backup-bench.sh --serve-http-targets 69 | ``` 70 | 71 | On source 72 | ``` 73 | ./backup-bench.sh --init-repos --remote 74 | ./backup-bench.sh --benchmarks --remote 75 | ``` 76 | 77 | Again, you can run multiple backup iterations with: 78 | ``` 79 | ./backup-bench.sh --clear-repos --remote 80 | ``` 81 | 82 | Optional step if using kopia / restic HTTP servers, on target: 83 | ``` 84 | ./backup-bench.sh --serve-http-targets 85 | ``` 86 | 87 | ``` 88 | ./backup-bench.sh --init-repos --remote --git 89 | ./backup-bench.sh --benchmarks --remote --git 90 | ``` 91 | 92 | > :warning: 93 | > duplicity benchmarks will fail if you do initialize repositories locally and try remote backup benchmarks or vice verca. 94 | 95 | ## Remote benchmarks using alternative backends 96 | 97 | There is a work in progress to support restic and kopia http servers, which have not been tested yet. 98 | You're welcome to help to automate those. 99 | Script will assume restic http, restic_beta http and kopia http ports are reachable from source to target. As of today, no auth mechanism is used in script, so please make sure you know what you're doing when using http backends. 100 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2022, Orsiris de Jong 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | 1. Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | 2. Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | 3. Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # backup-bench 2 | Quick and dirty backup tool benchmark with reproducible results 3 | 4 | ** This is a one page entry with benchmarks (see below), previous versions are available via git versioning.** 5 | 6 | ## What 7 | 8 | This repo aims to compare different backup solutions among: 9 | 10 | - [borg backup](https://www.borgbackup.org) 11 | - [bupstash](https://bupstash.io) 12 | - [restic](https://restic.net) 13 | - [kopia](https://www.kopia.io) 14 | - [duplicacy](https://duplicacy.com) 15 | - your tool (PRs to support new backup tools are welcome) 16 | 17 | The idea is to have a script that executes all backup programs on the same datasets. 18 | 19 | We'll use a quite big (and popular) git repo as first dataset so results can be reproduced by checking out branches (and ignoring .git directory). 20 | I'll also use another (not public) dataset which will be some qcow2 files which are in use. 21 | 22 | Time spent by the backup program is measured by the script so we get as accurate as possible results (time is measured from process beginning until process ends, with a 1 second granularity). 23 | 24 | While backups are done, cpu/memory/disk metrics are saved so we know how "resource hungry" a backup program can be. 25 | 26 | All backup programs are setup to use SSH in order to compare their performance regardless of the storage backend. 27 | 28 | When available, we'll tune the encryption algorithm depending on the results of a benchmark. For instance, kopia has a `kopia benchmark compression --data-file=/some/big/data/file` option to find out which compression / crypto works best on the current architecture. 29 | This is *REALLY NICE TO HAVE* when choices need to be made, aware of current architecture. 30 | As of the current tests, Borg v2.0.0-b1 also has a `borg benchmark cpu` option. 31 | 32 | ## Why 33 | 34 | I am currently using multiple backup programs to achieve my needs. As of today, I use Graham Keeling's burp https://github.com/grke/burp to backup windows machines, and borg backup to backup QEMU VM images. Graham decided to remove its deduplication (protocol 2) and stick with rsync based backups (protocol 1), which isn't compatible with my backup strategies. 35 | I've also tried out bupstash which I found to be quite quick, but which produces bigger backups remotely when dealing with small files (probably because of the chunk size?). 36 | 37 | Anyway, I am searching for a good allrounder, so I decided to give all the deduplicating backup solutions a try, and since I am configuring them all, I thought why not make my results available to anyone, with a script so everything can be reproduced easily. 38 | 39 | As of today I use the script on my lab hypervisor, which runs AlmaLinux 8.6. The script should run on other distros, although I didn't test it. 40 | 41 | I'll try to be as little biased as possible when doing my backup tests. 42 | If you feel that I didn't give a specific program enough attention, feel free to open an issue. 43 | 44 | # In depth comparison of backup solutions 45 | 46 | Last update: 03 October 2022 47 | 48 | |Backup software|Version| 49 | |------------------|--------| 50 | |borg|1.2.2| 51 | |borg beta|2.0.0b2| 52 | |restic|0.14.0| 53 | |kopia|0.12.0| 54 | |bupstash|0.11.1| 55 | |duplicacy|2.7.2| 56 | 57 | The following list is my personal shopping list when it comes to backup solutions, and might not be complete, you're welcome to provide PRs to update it. ;) 58 | 59 | | **Goal** | **Functionality** | **borg** | **restic** | **kopia** | **bupstash** | **duplicacy** | 60 | |------------------------------------|--------------------------------------------------------------------------|-----------------------|----------------|--------------------------------------------|-----------------------|---------------| 61 | | **Reliability** | Redundant index copies | ? | ? | Yes | yes, redundant + sync | No indexes used| 62 | | **Reliability** | Continue restore on bad blocks in repository | ? | ? | Yes (can ignore errors when restoring) | No | Yes, [erasure coding](https://forum.duplicacy.com/t/new-feature-erasure-coding/4168)| 63 | | **Reliability** | Data checksumming | Yes (CRC & HMAC) | ? | No (Reed–Solomon in the works) | HMAC | Yes | 64 | | **Reliability** | Backup coherency (detecting in flight file changes while backing up) | [Yes](https://github.com/deajan/backup-bench/issues/5#issue-1363881841) | [Yes](https://forum.restic.net/t/what-happens-if-file-changes-during-backup/264/2) | ? | [No](https://bupstash.io/doc/guides/Filesystem%20Backups.html) | ? | 65 | | **Restoring Data** | Backup mounting as filesystem | Yes | Yes | Yes | No | No | 66 | | **File management** | File includes / excludes bases on regexes | Yes | ? | ? | ? | Yes | 67 | | **File management** | Supports backup XATTRs | Yes | ? | No | Yes | ? | 68 | | **File management** | Supports backup ACLs | Yes | ? | No | Yes | ? | 69 | | **File management** | Supports hardlink identification (no multiple stored hardlinked files | No ([borg2 will](https://github.com/borgbackup/borg/issues/2379) | [Yes](https://forum.restic.net/t/trying-to-understand-how-hard-links-are-handled-by-restic/3785) | [No](https://github.com/kopia/kopia/issues/544#issuecomment-988329366) | [Yes](https://github.com/deajan/backup-bench/issues/13#issue-1363979532) | [No](https://forum.duplicacy.com/t/hard-links-not-properly-restored/962/3) | 70 | | **File management** | Supports sparse files (thin provisionned files on disk) | [Yes](https://github.com/borgbackup/borg/pull/5561) | [Yes](https://github.com/restic/restic/pull/3854) | [Yes](https://github.com/kopia/kopia/pull/1823) | [Yes](https://bupstash.io/doc/man/bupstash-restore.html) | ? | 71 | | **File management** | Can exclude CACHEDIR.TAG(3) directories | Yes | Yes | Yes | [Yes](https://github.com/andrewchambers/bupstash/commit/2ecaab63d178bc26198855a8313ab6288544ecd4/) | No | 72 | | **Dedup & compression efficiency** | Is data compressed | Yes | Yes | Yes | Yes | Yes | 73 | | **Dedup & compression efficiency** | Uses newer compression algorithms (ie zstd) | Yes | Yes | Yes | Yes | Yes | 74 | | **Dedup & compression efficiency** | Can files be excluded from compression by extension | ? | No | Yes | No | No | 75 | | **Dedup & compression efficiency** | Is data deduplicated | Yes | Yes | Yes | Yes | Yes | 76 | | **Platform support** | Programming lang | Python | Go | Go | Rust | Go | 77 | | **Platform support** | Unix Prebuilt binaries | Yes | Yes | Yes | No | Yes | 78 | | **Platform support** | Windows support | Yes (WSL) | Yes | Yes | No | Yes | 79 | | **Platform support** | Windows first class support (PE32 binary) | No | Yes | Yes | No | Yes | 80 | | **Platform support** | Unix snapshot support where snapshot path prefix is removed | ? | ? | ? | ? | ? | 81 | | **Platform support** | Windows VSS snapshot support where snapshot path prefix is removed | No | Yes | No, but pre-/post hook VSS script provided | No | Yes | 82 | | **WAN Support** | Can backups be sent to a remote destination without keeping a local copy | Yes | Yes | Yes | Yes | Yes | 83 | | **WAN Support** | What other remote backends are supported ? | rclone | (1) | (2) | None | (1) | 84 | | **Security** | Are encryption protocols secure (AES-256-GCM / PolyChaCha / etc ) ? | Yes, AES-256-GCM | Yes, AES-256 | Yes, AES-256-GCM or Chacha20Poly1305 | Yes, Chacha20Poly1305 | Yes, AES-256-GCM| 85 | | **Security** | Are metadatas encrypted too ? | ? | [Yes](https://restic.readthedocs.io/en/latest/100_references.html#threat-model) | ? | Yes | Yes | 86 | | **Security** | Can encrypted / compressed data be guessed (CRIME/BREACH style attacks)? | [No](https://github.com/borgbackup/borg/issues/3687) | [No](https://restic.readthedocs.io/en/latest/100_references.html#threat-model) | ? | No (4) | ? | 87 | | **Security** | Can a compromised client delete backups? | No (append mode) | [No](https://github.com/restic/restic/issues/3917#issuecomment-1242772365) (append mode)| Supports optional object locking | No (ssh restriction ) | No [pubkey](https://forum.duplicacy.com/t/new-feature-rsa-encryption/2662) + immutable targets| 88 | | **Security** | Can a compromised client restore encrypted data? | Yes | ? | ? | No | No [pubkey](https://forum.duplicacy.com/t/new-feature-rsa-encryption/2662) | 89 | | **Security** | Are pull backup scenarios possible? | Yes | No | No | No, planned | ? | 90 | | **Misc** | Does the backup software support pre/post execution hooks? | ? | ? | Yes | No | [Yes](https://forum.duplicacy.com/t/pre-command-and-post-command-scripts/1100) | 91 | | **Misc** | Does the backup software provide an API for their client ? | Yes (JSON cmd) | No, but REST API on server | No, but REST API on server | No | No | 92 | | **Misc** | Does the backup sofware provide an automatic GFS system ? | Yes | [Yes](https://restic.readthedocs.io/en/stable/060_forget.html#removing-snapshots-according-to-a-policy) | Yes | No | ? | 93 | | **Misc** | Does the backup sofware provide a crypto benchmark ? | No, available in beta | No | Yes | Undocumented | No, [generic benchmark](https://forum.duplicacy.com/t/benchmark-command-details/1078)| 94 | | **Misc** | Can a repo be synchronized to another repo ? | ? | ? | Yes | Yes | Yes | 95 | 96 | - (1) SFTP/S3/Wasabi/B2/Aliyun/Swift/Azure/Google Cloud 97 | - (2) SFTP/Google Cloud/S3 and S3-compatible storage like Wasabi/B2/Azure/WebDav/rclone* 98 | - (3) see https://bford.info/cachedir/ 99 | - (4) For bupstash, CRIME/BREACH style attacks are mitigated if you disable read access for backup clients, and keep decryption keys off server. 100 | 101 | A quick word about backup coherence: 102 | 103 | While some backup tools might detect filesysetm changes inflight, it's usually the burden of a snapshot system (zfs, bcachefs, lvm, btrfs, vss...) to provide the backup program a reliable static version of the filesystem. 104 | Still it's a really nice to have in order to detect problems on backups without those snapshot aware tools, like plain XFS/EXT4 partitions. 105 | 106 | # Results 107 | 108 | ## 2022-10-02 109 | 110 | ### Used system specs 111 | 112 | - Source system: Xeon E3-1275, 64GB RAM, 2x SSD 480GB (for git dataset and local target), 2x4TB disks 7.2krpm (for bigger dataset), using XFS, running AlmaLinux 8.6 113 | - Remote target system: AMD Turion(tm) II Neo N54L Dual-Core Processor (yes, this is old), 6GB RAM, 2x4TB WD RE disks 7.2krpm using ZFS 2.1.5, 1x 1TB WD Blue using XFS, running AlmaLinux 8.6 114 | 115 | - Target system has a XFS filesystem as target for the linux kernel backup tests 116 | - Target system has a ZFS filesystem as target for the qemu backup tests. ZFS has been configured as follows: 117 | - `zfs set xattr=off backup` 118 | - `zfs set compression=off backup` # Since we already compress, we don't want to add another layer here 119 | - `zfs set atime=off backup` 120 | - `zfs set recordsize=1M backup` # This could be tuned as per backup program... 121 | 122 | 123 | ### source data for local and remote multiple git repo versions backup benchmarks 124 | 125 | Linux kernel sources, initial git checkout v5.19, then changed to v5.18, 4.18 and finally v3.10 for the last run. 126 | Initial git directory totals 4.1GB, for 5039 directories and 76951 files. Using `env GZIP=-9 tar cvzf kernel.tar.gz /opt/backup_test/linux` produced a 2.8GB file. Again, using "best" compression with `tar cf - /opt/backup_test/linux | xz -9e -T4 -c - > kernel.tar.bz` produces a 2.6GB file, so there's probably big room for deduplication in the source files, even without running multiple consecutive backups on different points in time of the git repo. 127 | 128 | ### backup multiple git repo versions to local repositories 129 | 130 | ![image](https://user-images.githubusercontent.com/4681318/193457878-3f9816d0-9853-42bf-a9f7-59c0560b9fe4.png) 131 | 132 | Numbers: 133 | | Operation | bupstash 0.11.1 | borg 1.2.2 | borg\_beta 2.0.0b2 | kopia 0.12.0 | restic 0.14.0 | duplicacy 2.7.2 | 134 | | -------------- | --------------- | ---------- | ------------------ | ------------ | ------------- | --------------- | 135 | | backup 1st run | 9 | 41 | 55 | 10 | 23 | 32 | 136 | | backup 2nd run | 11 | 22 | 25 | 4 | 8 | 13 | 137 | | backup 3rd run | 7 | 28 | 39 | 7 | 17 | 23 | 138 | | backup 4th run | 5 | 20 | 29 | 6 | 13 | 16 | 139 | | restore | 4 | 16 | 17 | 5 | 9 | 11 | 140 | | size 1st run | 213268 | 257300 | 265748 | 259780 | 260520 | 360200 | 141 | | size 2nd run | 375776 | 338760 | 348248 | 341088 | 343060 | 480600 | 142 | | size 3rd run | 538836 | 527732 | 543432 | 529812 | 531892 | 722176 | 143 | | size 4th run | 655836 | 660812 | 680092 | 666408 | 668404 | 894984 | 144 | 145 | Remarks: 146 | - kopia was the best allround performer on local backups when it comes to speed, but is quite CPU intensive. 147 | - bupstash was the most space efficient tool and is not CPU hungry. 148 | - For the next instance, I'll need to post CPU / Memory / Disk IO usage graphs from my Prometheus instance. 149 | 150 | ### backup multiple git repo versions to remote repositories 151 | 152 | - Remote repositories are SSH (+ binary) for bupstash and burp. 153 | - Remote repository is SFTP for duplicacy. 154 | - Remote repository is HTTPS for kopia (kopia server with 2048 bit RSA certificate) 155 | - Remote repository is HTTPS for restic (rest-server 0.11.0 with 2048 bit RSA certificate) 156 | 157 | ![image](https://user-images.githubusercontent.com/4681318/193457882-7228cba5-5ed3-4ffa-b2e0-4b863ef78df0.png) 158 | 159 | Numbers: 160 | 161 | | Operation | bupstash 0.11.1 | borg 1.2.2 | borg\_beta 2.0.0b2 | kopia 0.12.0 | restic 0.14.0 | duplicacy 2.7.2 | 162 | | -------------- | --------------- | ---------- | ------------------ | ------------ | ------------- | --------------- | 163 | | backup 1st run | 10 | 47 | 67 | 72 | 24 | 32 | 164 | | backup 2nd run | 12 | 25 | 30 | 32 | 10 | 15 | 165 | | backup 3rd run | 9 | 36 | 47 | 54 | 19 | 23 | 166 | | backup 4th run | 7 | 31 | 50 | 46 | 21 | 23 | 167 | | restore | 170 | 244 | 243 | 258 | 28 | 940 | 168 | | size 1st run | 213240 | 257288 | 265716 | 255852 | 260608 | 360224 | 169 | | size 2nd run | 375720 | 338720 | 348260 | 336440 | 342848 | 480856 | 170 | | size 3rd run | 538780 | 527620 | 543204 | 522512 | 531820 | 722448 | 171 | | size 4th run | 655780 | 660708 | 679868 | 657196 | 668436 | 895248 | 172 | 173 | Remarks: 174 | - With restic's recent release 0.14.0, the remote speeds using rest-server increased dramatically and are onpar with local backup results. 175 | - All other programs take about 5-10x more time to restore than the initial backup, except for duplicacy, which has a 30x factor which is really bad 176 | - Since [last benchmark series](RESULTS-20220906.md), kopia 0.2.0 was released which resolves the [remote bottleneck](https://github.com/kopia/kopia/issues/2372) 177 | - I finally switchted from ZFS to XFS remote filesystem so we have comparable file sizes between local and remote backups 178 | - Noticing bad restore results, I've tried to tweak the SSH server: 179 | - The best cipher algorithm on my repository server was chacha-poly1305 (found with https://gist.github.com/joeharr4/c7599c52f9fad9e53f62e9c8ae690e6b) 180 | - Compression disabled 181 | - X11 Forwarding disabled (was already disabled) 182 | - The above settings were applied to sshd, so even duplicacy gets to use them, since I didn't find a way to configure those settings for duplicacy 183 | 184 | ### backup private qemu disk images to remote repositories 185 | 186 | Source data are 8 qemu qcow2 files, and 7 virtual machines description JSON files for a total of 366GB. 187 | Remote repositories are configured as above, except that I used ZFS as a backing filesystem. 188 | 189 | ![image](https://user-images.githubusercontent.com/4681318/193459995-b07cdc75-f98d-4334-9fe3-26b4d9d0ba1e.png) 190 | 191 | Numbers: 192 | 193 | | Operation | bupstash 0.11.1 | borg 1.2.2 | borg\_beta 2.0.0b2 | kopia 0.12.0 | restic 0.14.0 | duplicacy 2.7.2 | 194 | | -------------- | --------------- | ---------- | ------------------ | ------------ | ------------- | --------------- | 195 | | initial backup | 4699 | 7044 | 6692 | 12125 | 8848 | 5889 | 196 | | initial size | 121167779 | 123111836 | 122673523 | 139953808 | 116151424 | 173809600 | 197 | 198 | Remarks: 199 | 200 | As I did the backup benchmarks, I computed the average size of the files in each repository using 201 | ``` 202 | find /path/to/repository -type f -printf '%s\n' | awk '{s+=$0} 203 | END {printf "Count: %u\nAverage size: %.2f\n", NR, s/NR}' 204 | ``` 205 | 206 | Results for the linux kernel sources backups: 207 | 208 | | Software | Source | bupstash 0.11.1 | borg 1.2.2 | borg\_beta 2.0.0b2 | kopia 0.12 | restic 0.14.0 | duplicacy 2.7.2 | 209 | |----------|----------------|-----------------|------------|--------------------|------------|---------------|-----------------| 210 | | File count | 61417 | 2727 | 12 | 11 | 23 | 14 | 89 | 211 | | Avg file size (kb) | 62 | 42 | 12292 | 13839 | 6477 | 10629 | 2079 | 212 | 213 | I also computed the average file sizes in each repository for my private qemu images which I backup with all the tools using backup-bench. 214 | 215 | Results for the qemu images backups: 216 | 217 | | Software | Source | bupstash 0.11.1 | borg 1.2.2 | borg\_beta 2.0.0b2 | kopia 0.12 | restic 0.14.0 | duplicacy 2.7.2 | 218 | |----------|----------------|-----------------|------------|--------------------|------------|---------------|-----------------| 219 | | File count | 15 | 136654 | 239 | 267 | 6337 | 66000 | 41322 | 220 | | Avg file size (kb) | 26177031 | 850 | 468088 | 469933 | 22030 | 17344875 | 3838 | 221 | 222 | Interesting enough, bupstash is the only software that produces sub megabyte chunks. Of the above 136654 files, only 39443 files weight more than 1MB. 223 | The qemu disk images are backed up to a ZFS filesystem with recordsize=1M. 224 | In order to measure the size difference, I created a ZFS filesystem with a 128k recordsize, and copied the bupstash repo to that filesystem. 225 | This resulted in bupstash repo size being roughly 13% smaller (137364728kb to 121167779kb). 226 | Since bupstash uses smaller chunk file sizes, I will continue using the 128k recordsize for the ZFS bupstash repository. 227 | 228 | ## Footnotes 229 | 230 | - Getting restic SFTP to work with a different SSH port made me roam restic forums and try various setups. Didn't succeed in getting RESTIC_REPOSITORY variable to work with that configuration. 231 | - duplicacy wasn't as easy to script as the other tools, since it modifies the source directory (by adding .duplicacy folder) so I had to exclude that one from all the other backup tools. 232 | - The necessity for duplicacy to cd into the directory to backup/restore doesn't feel natural to me. 233 | 234 | ## EARLIER RESULTS 235 | 236 | - [2022-09-06](RESULTS-20220906.md) 237 | - [2022-08-19](RESULTS-20220819.md) 238 | 239 | ## Links 240 | 241 | As of 6 September 2022, I've posted an issue to every backup program's git asking if they could review this benchmark repo: 242 | 243 | - bupstash: https://github.com/andrewchambers/bupstash/issues/335 244 | - restic: https://github.com/restic/restic/issues/3917 245 | - borg: https://github.com/borgbackup/borg/issues/7007 246 | - duplicacy: https://github.com/gilbertchen/duplicacy/issues/635 247 | - kopia: https://github.com/kopia/kopia/issues/2375 248 | -------------------------------------------------------------------------------- /RESULTS-20220819.md: -------------------------------------------------------------------------------- 1 | ## 2022-08-19 2 | 3 | ### Source system: Xeon E3-1275, 64GB RAM, 2x SSD 480GB (for git dataset and local target), 2x4TB disks 7.2krpm (for bigger dataset), using XFS, running AlmaLinux 8.6 4 | ### Target system: AMD Turion(tm) II Neo N54L Dual-Core Processor (yes, this is old), 6GB RAM, 2x4TB WD RE disks 7.2krpm, using ZFS 2.1.5, running AlmaLinux 8.6 5 | 6 | #### source data 7 | 8 | Linux kernel sources, initial git checkout v5.19, then changed to v5.18, 4.18 and finally v3.10 for the last run. 9 | Initial git directory totals 4.1GB, for 5039 directories and 76951 files. Using `env GZIP=-9 tar cvzf kernel.tar.gz /opt/backup_test/linux` produced a 2.8GB file. Again, using "best" compression with `tar cf - /opt/backup_test/linux | xz -9e -T4 -c - > kernel.tar.bz` produces a 2.6GB file, so there's probably big room for deduplication in the source files, even without running multiple consecutive backups on different points in time of the git repo. 10 | 11 | #### backup multiple git repo versions to local repositories 12 | ![image](https://user-images.githubusercontent.com/4681318/185691430-d597ecd1-880e-474b-b015-27ed6a02c7ea.png) 13 | 14 | Numbers: 15 | | Operation | bupstash 0.11.0 | borg 1.2.1 | borg\_beta 2.0.0b1 | kopia 0.11.3 | restic 0.13.1 | restic\_beta 0.13.1-dev | duplicacy 2.7.2 | 16 | | -------------- | --------------- | ---------- | ------------------ | ------------ | ------------- | ----------------------- | --------------- | 17 | | backup 1st run | 9 | 38 | 54 | 9 | 22 | 24 | 30 | 18 | | backup 2nd run | 13 | 19 | 23 | 4 | 8 | 9 | 12 | 19 | | backup 3rd run | 8 | 25 | 37 | 7 | 16 | 17 | 21 | 20 | | backup 4th run | 6 | 18 | 26 | 6 | 13 | 13 | 16 | 21 | | restore | 3 | 15 | 17 | 6 | 7 | 9 | 17 | 22 | | size 1st run | 213208 | 257256 | 259600 | 259788 | 1229540 | 260488 | 360244 | 23 | | size 2nd run | 375680 | 338796 | 341488 | 341088 | 1563592 | 343036 | 480888 | 24 | | size 3rd run | 538724 | 527808 | 532256 | 529804 | 2256348 | 532512 | 723556 | 25 | | size 4th run | 655712 | 660896 | 665864 | 666408 | 2732740 | 669124 | 896312 | 26 | 27 | Remarks: 28 | - It seems that current stable restic version (without compression) uses huge amounts of disk space, hence the test with current restic beta that supports compression. 29 | - kopia was the best allround performer on local backups when it comes to speed 30 | - bupstash was the most space efficient tool (beats borg beta by about 1MB) 31 | 32 | #### backup multiple git repo versions to remote repositories 33 | ![image](https://user-images.githubusercontent.com/4681318/185691444-b57ec8dc-9221-46d4-bbb6-94e1f6471d9e.png) 34 | 35 | Remote repositories are SSH (+ binary) for bupstash and burp. 36 | Remote repositories are SFTP for kopia, restic and duplicacy. 37 | 38 | Numbers: 39 | | Operation | bupstash 0.11.0 | borg 1.2.1 | borg\_beta 2.0.0b1 | kopia 0.11.3 | restic 0.13.1 | restic\_beta 0.13.1-dev | duplicacy 2.7.2 | 40 | | -------------- | --------------- | ---------- | ------------------ | ------------ | ------------- | ----------------------- | --------------- | 41 | | backup 1st run | 22 | 44 | 63 | 101 | 764 | 86 | 116 | 42 | | backup 2nd run | 15 | 22 | 30 | 59 | 229 | 33 | 42 | 43 | | backup 3rd run | 16 | 32 | 47 | 61 | 473 | 76 | 76 | 44 | | backup 4th run | 13 | 25 | 35 | 68 | 332 | 55 | 53 | 45 | | restore | 172 | 251 | 256 | 749 | 1451 | 722 | 1238 | 46 | | size 1st run | 250098 | 257662 | 259710 | 268300 | 1256836 | 262960 | 378792 | 47 | | size 2nd run | 443119 | 339276 | 341836 | 352507 | 1607666 | 346000 | 505072 | 48 | | size 3rd run | 633315 | 528738 | 532970 | 547279 | 2312586 | 536675 | 756943 | 49 | | size 4th run | 770074 | 661848 | 666848 | 688184 | 2801249 | 674189 | 936291 | 50 | 51 | Remarks: 52 | - Very bad restore results can be observed across all backup solutions, we'll need to investigate this: 53 | - Both links are monitored by dpinger, which shows no loss 54 | - Target server, although being (really) old, has no observed bottlenecks (monitored, no iowait, disk usage nor cpu is skyrocketing) 55 | - kopia, restic and duplicacy seem to not cope well SFTP, whereas borg and bupstash are advantaged since they run a ssh deamon on the target 56 | - I have chosen to use SFTP to make sure ssh overhead is similar between all solutions 57 | - It would be a good idea to setup kopia and restic HTTP servers and redo the remote repository tests 58 | - Strangely, the repo sizes of bupstash and duplicacy are quite larger than local repos for the same data, probably because of some chunking algorithm that changes chuck sizes depending on transfer rate or so ? That could be discussed by the solution's developers. 59 | 60 | #### Notes 61 | Disclaimers: 62 | - The script has run on a lab server that hold about 10VMs. I've made sure that CPU/MEM/DISK WAIT stayed the same between all backup tests, nevertheless, some deviances may have occured while measuring. 63 | - Bandwidth between source and target is 1Gbit/s theoretically. Nevertheless, I've made a quick iperf3 test to make sure that bandwidth is available between both servers. 64 | 65 | `iperf3 -c targetfqdn` results 66 | ``` 67 | [ ID] Interval Transfer Bitrate Retr 68 | [ 5] 0.00-10.00 sec 545 MBytes 457 Mbits/sec 23 sender 69 | [ 5] 0.00-10.04 sec 544 MBytes 455 Mbits/sec receiver 70 | ``` 71 | `iperf3 -c targetfqdn -R` results 72 | ``` 73 | [ ID] Interval Transfer Bitrate Retr 74 | [ 5] 0.00-10.04 sec 530 MBytes 443 Mbits/sec 446 sender 75 | [ 5] 0.00-10.00 sec 526 MBytes 442 Mbits/sec receiver 76 | ``` 77 | - Deterministic results cannot be achieved since too much external parameters come in when running the benchmarks. Nevertheless, the systems are monitored, and the tests were done when no cpu/ram/io spikes where present, and no bandwidth problem was detected. -------------------------------------------------------------------------------- /RESULTS-20220906.md: -------------------------------------------------------------------------------- 1 | ## 2022-09-06 2 | 3 | ### Source system: Xeon E3-1275, 64GB RAM, 2x SSD 480GB (for git dataset and local target), 2x4TB disks 7.2krpm (for bigger dataset), using XFS, running AlmaLinux 8.6 4 | ### Target system: AMD Turion(tm) II Neo N54L Dual-Core Processor (yes, this is old), 6GB RAM, 2x4TB WD RE disks 7.2krpm, using ZFS 2.1.5, running AlmaLinux 8.6 5 | 6 | 7 | #### source data 8 | 9 | Linux kernel sources, initial git checkout v5.19, then changed to v5.18, 4.18 and finally v3.10 for the last run. 10 | Initial git directory totals 4.1GB, for 5039 directories and 76951 files. Using `env GZIP=-9 tar cvzf kernel.tar.gz /opt/backup_test/linux` produced a 2.8GB file. Again, using "best" compression with `tar cf - /opt/backup_test/linux | xz -9e -T4 -c - > kernel.tar.bz` produces a 2.6GB file, so there's probably big room for deduplication in the source files, even without running multiple consecutive backups on different points in time of the git repo. 11 | 12 | Note: I removed restic_beta benchmark since restic 0.14.0 with compression support is officially released. 13 | 14 | #### backup multiple git repo versions to local repositories 15 | 16 | ![image](https://user-images.githubusercontent.com/4681318/188726855-2813d297-3349-4849-9ac7-c58caa58a72d.png) 17 | 18 | Numbers: 19 | | Operation | bupstash 0.11.0 | borg 1.2.2 | borg\_beta 2.0.0b1 | kopia 0.11.3 | restic 0.14.0 | duplicacy 2.7.2 | 20 | | -------------- | --------------- | ---------- | ------------------ | ------------ | ------------- | --------------- | 21 | | backup 1st run | 9 | 38 | 54 | 9 | 23 | 30 | 22 | | backup 2nd run | 13 | 21 | 25 | 4 | 9 | 14 | 23 | | backup 3rd run | 9 | 29 | 39 | 7 | 18 | 24 | 24 | | backup 4th run | 6 | 21 | 29 | 5 | 13 | 16 | 25 | | restore | 3 | 17 | 16 | 6 | 10 | 16 | 26 | | size 1st run | 213220 | 257224 | 259512 | 259768 | 260588 | 360160 | 27 | | size 2nd run | 375712 | 338792 | 341096 | 341060 | 343356 | 480392 | 28 | | size 3rd run | 538768 | 527716 | 531980 | 529788 | 532216 | 722420 | 29 | | size 4th run | 655764 | 660808 | 665692 | 666396 | 668840 | 895192 | 30 | 31 | Remarks: 32 | - kopia was the best allround performer on local backups when it comes to speed, but is quite CPU intensive. 33 | - bupstash was the most space efficient tool (beats borg beta by about 1MB), and is not CPU hungry. 34 | - For the next instance, I'll need to post CPU / Memory / Disk IO usage graphs. 35 | 36 | #### backup multiple git repo versions to remote repositories 37 | 38 | - Remote repositories are SSH (+ binary) for bupstash and burp. 39 | - Remote repositories is SFTP for duplicacy. 40 | - Remote repository is HTTPS for kopia (kopia server with 2048 bit RSA certificate) 41 | - Remote repository is HTTP for restic (rest-server 0.11.0) 42 | - [Update] I've also redone the same tests in HTTPS with `--insecure-tls` which is documented on restic docs but not visible when using `restic --help`. 43 | 44 | ![image](https://user-images.githubusercontent.com/4681318/188742959-cb114ccd-0f03-47df-a07c-1d31ae8853a7.png) 45 | 46 | Numbers: 47 | 48 | | Operation | bupstash 0.11.0 | borg 1.2.2 | borg\_beta 2.0.0b1 | kopia 0.11.3 | restic 0.14.0 | duplicacy 2.7.2 | 49 | | -------------- | --------------- | ---------- | ------------------ | ------------ | ------------- | --------------- | 50 | | backup 1st run | 23 | 62 | 64 | 1186 | 17 | 107 | 51 | | backup 2nd run | 16 | 25 | 29 | 292 | 9 | 44 | 52 | | backup 3rd run | 19 | 37 | 48 | 904 | 14 | 60 | 53 | | backup 4th run | 15 | 29 | 36 | 800 | 11 | 47 | 54 | | restore | 161 | 255 | 269 | 279 | 20 | 1217 | 55 | | size 1st run | 250012 | 257534 | 260090 | 257927 | 262816 | 382572 | 56 | | size 2nd run | 443008 | 339276 | 341704 | 339181 | 346264 | 508655 | 57 | | size 3rd run | 633083 | 528482 | 532710 | 526723 | 536403 | 761362 | 58 | | size 4th run | 769681 | 661720 | 666588 | 662558 | 673989 | 941247 | 59 | 60 | Remarks: 61 | - Very bad restore results can be observed across all backup solutions (except restic), we'll need to investigate this: 62 | - Both links are monitored by dpinger, which shows no loss. 63 | - Target server, although being (really) old, has no observed bottlenecks (monitored, no iowait, disk usage nor cpu is skyrocketing) 64 | - Since [last benchmark series](RESULTS-20220819.md), I changed Kopia's backend from SFTP to HTTPS. There must be a bottlebeck since backup times are really bad, but restore times improved. 65 | - I opened an issue at https://github.com/kopia/kopia/issues/2372 to see whether I configured kopia poorly. 66 | - CPU usage on target is quite intensive when backing up via HTTPS contrary to SFTP backend. I need to investigate. 67 | - Since last benchmark series, I changed restic's backend from SFTP to HTTP. There's a *REALLY* big speed improvement, and numbers are comparable to local repositories. 68 | - I must add HTTPS encryption so we can compare what's comparable. [UPDATE]: Done, same results + or - a couple of seconds, table and image is updated 69 | - Indeed I checked that those numbers are really bound to remote repository, I can confirm, restic with rest-server is an all over winner when dealing with remote repositories. 70 | - Strangely, the repo sizes of bupstash and duplicacy are quite larger than local repos for the same data, I discussed the subject at https://github.com/andrewchambers/bupstash/issues/26 . 71 | - I think this might be ZFS related. The remote target has a default recordsize of 128KB. I think I need to redo a next series of benchmarks with XFS as remote filesystem for repositories. 72 | -------------------------------------------------------------------------------- /script/backup-bench.conf: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | #### backup-bench.sh configuration file 4 | 5 | BACKUP_SOFTWARES=(bupstash borg borg_beta kopia restic rustic duplicacy) 6 | 7 | CSV_RESULT_FILE="/var/log/${PROGRAM}.results.csv" 8 | 9 | #### SOURCE SETTINGS 10 | 11 | # User on source system that will execute backup tools 12 | SOURCE_USER=root 13 | SOURCE_USER_HOMEDIR=$(eval echo ~${SOURCE_USER}) 14 | 15 | SOURCE_FQDN="source.example.tld" # FQDN of source server so target can upload ssh keys, password will be asked when --setup-target is executed 16 | SOURCE_SSH_PORT="22" 17 | 18 | # git dataset (this dataset will be downloaded and used as primary test dataset) 19 | GIT_DATASET_REPOSITORY="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git" 20 | GIT_TAGS=(v5.19 v5.18 v4.18 v3.10) # List of tags to backup 21 | GIT_ROOT_DIRECTORY="linux" # Name of the git repo directory 22 | 23 | # Where source dataset is stored (when using git, includes git root directory) 24 | BACKUP_ROOT="/opt/backup_test/linux" 25 | 26 | # alternative (non-git) dataset, can be pointed to big files 27 | #BACKUP_ROOT="/data/backups" 28 | 29 | # Directory where to restore data on source system, make sure you have enough disk space 30 | RESTORE_DIR=/tmp/backup-bench-restore 31 | 32 | 33 | #### TARGET SETTINGS 34 | 35 | TARGET_ROOT=/opt/backup-bench-repos # Root of target repositories, make sure you have enough disk space 36 | HAVE_ZFS=false # Disable to ignore zfs setup 37 | 38 | #### REMOTE TARGET SETTINGS 39 | 40 | # Additional SSH tuning 41 | SSH_OPTS="-o Compression=no -c chacha20-poly1305@openssh.com -x -T" 42 | 43 | REMOTE_TARGET_FQDN="target.example.tld" # FQDN of target server so source can upload data to 44 | REMOTE_TARGET_SSH_PORT="22" 45 | 46 | # This runner allows to connect from source to target as root in order to clear / repos 47 | REMOTE_SSH_RUNNER="ssh -i ${SOURCE_USER_HOMEDIR}/.ssh/backup-bench.key -p ${REMOTE_TARGET_SSH_PORT} $SSH_OPTS -o StrictHostKeyChecking=accept-new root@${REMOTE_TARGET_FQDN}" 48 | 49 | # BORG SPECIFIC SETTINGS 50 | BORG_STABLE_REPO_LOCAL="${TARGET_ROOT}/borg/data" 51 | BORG_BETA_REPO_LOCAL="${TARGET_ROOT}/borg_beta/data" 52 | BORG_STABLE_REPO_REMOTE="borg_user@${REMOTE_TARGET_FQDN}:${TARGET_ROOT}/borg/data" 53 | BORG_BETA_REPO_REMOTE="ssh://borg_beta_user@${REMOTE_TARGET_FQDN}${TARGET_ROOT}/borg_beta/data" 54 | export BORG_PASSPHRASE=SOMEPASSWORD 55 | 56 | # BUPSTASH SPECIFIC SETTINGS 57 | BUPSTASH_REPOSITORY_LOCAL="${TARGET_ROOT}/bupstash/data" 58 | BUPSTASH_REPOSITORY_COMMAND_REMOTE="ssh -i ${SOURCE_USER_HOMEDIR}/.ssh/bupstash.key -p ${REMOTE_TARGET_SSH_PORT} $SSH_OPTS -o StrictHostKeyChecking=accept-new -T bupstash_user@${REMOTE_TARGET_FQDN}" 59 | export BUPSTASH_KEY="${SOURCE_USER_HOMEDIR}/bupstash.store.key" 60 | 61 | # KOPIA SPECIFIC SETTINGS 62 | KOPIA_USE_HTTP=false 63 | KOPIA_HTTP_PORT=37890 64 | export KOPIA_PASSWORD=SOMEPASSWORD 65 | KOPIA_HTTP_USERNAME=backup-bench 66 | KOPIA_HTTP_PASSWORD=SOMEHTTPPASSWORD 67 | export KOPIA_SERVER_CONTROL_USER=masteruser # This allows to interact with kopia server (for server refresh) 68 | export KOPIA_SERVER_CONTROL_PASSWORD=SOMEMASTERPASSWORD 69 | 70 | # RESTIC SPECIFC SETTINGS 71 | export RESTIC_PASSWORD=SOMEPASSWORD 72 | RESTIC_USE_HTTP=false 73 | RESTIC_HTTP_PORT=37891 74 | 75 | # RUSTIC SPECIFC SETTINGS 76 | export RUSTIC_PASSWORD=SOMEPASSWORD 77 | RUSTIC_USE_HTTP=false 78 | RUSTIC_HTTP_PORT=37892 79 | 80 | # DUPLICACY_SPECIFIC_SETTINGS 81 | export DUPLICACY_PASSWORD=SOMEPASSWORD 82 | export DUPLICACY_SSH_KEY_FILE=${SOURCE_USER_HOMEDIR}/.ssh/duplicacy.key 83 | -------------------------------------------------------------------------------- /script/backup-bench.sh: -------------------------------------------------------------------------------- 1 | #!/usr/bin/env bash 2 | 3 | # This script is a (very) simple backup benchmark for the following backup programs: 4 | # bupstash 5 | # borg 6 | # kopia 7 | # restic 8 | # rustic 9 | # duplicacy 10 | 11 | # It (should) allow to produce reproductible results, and give an idea of what program is the fastest and creates the smallest backups 12 | # Results can be found in /var/log as pseudo-CSV file 13 | # Should be executed together with a monitoring system that matches cpu/ram/io usage against the running backup solution (disclaimer: I use netdata) 14 | 15 | # Script tested on RHEL 8 & RHEL 9 (would probably work on DEB based distros too) 16 | # requires bash >=4.2, and yum or apt 17 | 18 | # So why do we have multiple functions that could be factored into one ? Because each backup program might get different settings at some time, so it's easier to have one function per program 19 | 20 | PROGRAM="backup-bench" 21 | AUTHOR="(C) 2022-2024 by Orsiris de Jong" 22 | PROGRAM_BUILD=2024112301 23 | 24 | function self_setup { 25 | echo "Setting up ofunctions" 26 | ofunctions_path="/tmp/ofunctions.sh" 27 | 28 | # Download copy of ofunctions.sh so we get Logger and ExecTasks functions 29 | [ ! -f "${ofunctions_path}" ] && curl -L https://raw.githubusercontent.com/deajan/ofunctions/main/ofunctions.sh -o "${ofunctions_path}" 30 | source "${ofunctions_path}" || exit 99 31 | # Don't polluate RUN_DIR since we won't need alerts 32 | _LOGGER_WRITE_PARTIAL_LOGS=false 33 | } 34 | 35 | function download_prerequisites { 36 | local nodeps="${1:-false}" 37 | 38 | local result=true # did we succeed in installing our stuff 39 | 40 | if command dnf > /dev/null 2>&1; then 41 | Logger "Installing packages tar, bzip2, git using dnf" "NOTICE" 42 | dnf install -y tar bzip2 git || result=false 43 | 44 | # bupstash specific since we need to build it from source 45 | dnf install -y rust cargo pkgconfig libsodium-devel || result=false 46 | 47 | elif command apt > /dev/null 2>&1; then 48 | Logger "Installing packages tar, bzip2, git using apt" "NOTICE" 49 | apt install -y tar bzip2 git || result=false 50 | 51 | # bupstash specific since we need to build it from source 52 | apt install -y rustc cargo pkgconf libsodium-dev || result=false 53 | else 54 | result=false 55 | fi 56 | 57 | # Detect selinux and install semanage 58 | if type -p getenforce > /dev/null 2>&1; then 59 | # Is Selinux enabled 60 | if [ "$(getenforce)" == "Enforcing" ]; then 61 | Logger "Installing SELinux package policycoreutils-python-utils using dnf" "NOTICE" 62 | dnf install -y policycoreutils-python-utils || result=false 63 | else 64 | Logger "Skipping SELinux setup since it's disabled or permissive" "NOTICE" 65 | fi 66 | fi 67 | 68 | if [ "${result}" == false ]; then 69 | if [ "${nodeps}" == false ]; then 70 | Logger "Could not install required packages. We need the following: tar, bzip2, . You can bypass required packages install by specifying --no-deps" "NOTICE" 71 | else 72 | Logger "Required packages install bypassed" "NOTICE" 73 | fi 74 | else 75 | Logger "Successfully installed required packages." "NOTICE" 76 | fi 77 | } 78 | 79 | function get_lastest_git_release { 80 | local org="${1}" 81 | local repo="${2}" 82 | 83 | LASTEST_VERSION=$(curl -s https://api.github.com/repos/${org}/${repo}/releases/latest | grep "tag_name" | cut -d'"' -f4) 84 | echo ${LASTEST_VERSION} 85 | } 86 | 87 | function get_remote_certificate_fingerprint { 88 | # Used for kopia server certificate authentication 89 | local fqdn="${1}" 90 | local port="${2}" 91 | 92 | echo $(openssl s_client -connect ${fqdn}:${port} < /dev/null 2>/dev/null | openssl x509 -fingerprint -sha256 -noout -in /dev/stdin | cut -d'=' -f2 | tr -d ':') 93 | } 94 | 95 | function get_certificate_fingerprint { 96 | local file="${1}" 97 | 98 | echo $(openssl x509 -fingerprint -sha256 -noout -in "${file}" | cut -d'=' -f2 | tr -d ':') 99 | } 100 | 101 | function create_certificate { 102 | # Create a RSA certificate for kopia 103 | local name="${1}" 104 | 105 | openssl req -nodes -new -x509 -days 7300 -newkey rsa:2048 -keyout "${HOME}/${name}.key" -subj "/C=FR/O=SOMEORG/CN=FQDN/OU=RD/L=City/ST=State/emailAddress=contact@example.tld" -out "${HOME}/${name}.crt" 106 | } 107 | 108 | function clear_users { 109 | # clean users on target system when using remote repositories 110 | for backup_software in "${BACKUP_SOFTWARES[@]}"; do 111 | userdel -r "${backup_software}"_user 112 | done 113 | } 114 | 115 | function setup_root_access { 116 | # Quick and dirty ssh root setup on target sysetem when using remote repositories 117 | # This allows the source machine to access target 118 | ssh-keygen -b 2048 -t rsa -f /root/.ssh/backup-bench.rsa -q -N "" 119 | cat /root/.ssh/backup-bench.rsa.pub > /root/.ssh/authorized_keys && chmod 600 /root/.ssh/authorized_keys 120 | type -p semanage > /dev/null 2>&1 && semanage fcontext -a -t ssh_home_t /root/.ssh/authorized_keys 121 | type -p restorecon > /dev/null 2>&1 && restorecon -v /root/.ssh/authorized_keys 122 | 123 | cat /root/.ssh/backup-bench.rsa | ssh ${SOURCE_USER}@${SOURCE_FQDN} -p ${SOURCE_SSH_PORT} -o ControlMaster=auto -o ControlPersist=yes -o ControlPath=/tmp/$PROGRAM.ctrlm.%r@%h.$$ "cat > ${SOURCE_USER_HOMEDIR}/.ssh/backup-bench.key; chmod 600 ${SOURCE_USER_HOMEDIR}/.ssh/backup-bench.key" 124 | if [ "$?" != 0 ]; then 125 | echo "Failed to setup root access to target" 126 | echo "Please copy file \"/root/.ssh/backup-bench.rsa\" to source system in \"${SOURCE_USER_HOMEDIR}/.ssh/backup-bench.key\" and execute \"chmod 600 ${SOURCE_USER_HOMEDIR}/.ssh/backup-bench.key\"" 127 | fi 128 | } 129 | 130 | function setup_target_local_repos { 131 | for backup_software in "${BACKUP_SOFTWARES[@]}"; do 132 | [ -d ${TARGET_ROOT}/"${backup_software}" ] && rm -rf ${TARGET_ROOT:?}/"${backup_software}" 133 | mkdir -p ${TARGET_ROOT}/"${backup_software}" 134 | done 135 | } 136 | 137 | function setup_target_remote_repos { 138 | # Quick and dirty ssh repo setup on target system 139 | for backup_software in "${BACKUP_SOFTWARES[@]}"; do 140 | if [ "${HAVE_ZFS}" == true ]; then 141 | zfs create backup/"${backup_software}" 142 | zfs set compression=off backup/"${backup_software}" 143 | zfs set xattr=off backup/"${backup_software}" 144 | zfs set atime=off backup/"${backup_software}" 145 | # The following setting is targeted at qcow file backups 146 | # bupstash tends to create smaller files than others 147 | # We might want to set recordsize=128k for linux kernel benchmarks 148 | zfs set recordsize=1M backup/"${backup_software}" 149 | else 150 | mkdir -p ${TARGET_ROOT} || exit 127 151 | fi 152 | useradd -d ${TARGET_ROOT}/"${backup_software}" -m -r -U "${backup_software}"_user 153 | mkdir -p ${TARGET_ROOT}/"${backup_software}"/data 154 | mkdir -p ${TARGET_ROOT}/"${backup_software}"/.ssh && chmod 700 ${TARGET_ROOT}/"${backup_software}"/.ssh 155 | ssh-keygen -b 2048 -t rsa -f ${TARGET_ROOT}/"${backup_software}"/.ssh/"${backup_software}".rsa -q -N "" 156 | cat ${TARGET_ROOT}/"${backup_software}"/.ssh/"${backup_software}".rsa.pub > ${TARGET_ROOT}/"${backup_software}"/.ssh/authorized_keys && chmod 600 ${TARGET_ROOT}/"${backup_software}"/.ssh/authorized_keys 157 | chown "${backup_software}"_user -R ${TARGET_ROOT}/"${backup_software}" 158 | type -p semanage > /dev/null 2>&1 && semanage fcontext -a -t ssh_home_t ${TARGET_ROOT}/"${backup_software}"/.ssh/authorized_keys 159 | type -p restorecon > /dev/null 2>&1 && restorecon -v ${TARGET_ROOT}/"${backup_software}"/.ssh/authorized_keys 160 | done 161 | for backup_software in "${BACKUP_SOFTWARES[@]}"; do 162 | Logger "Copying RSA key for ${backup_software} to source into [${SOURCE_USER_HOMEDIR}/.ssh/${backup_software}.key]" "NOTICE" 163 | cat "${TARGET_ROOT}/${backup_software}/.ssh/${backup_software}.rsa" | ssh ${SOURCE_USER}@${SOURCE_FQDN} -p ${SOURCE_SSH_PORT} -o ControlMaster=auto -o ControlPersist=yes -o ControlPath=/tmp/$PROGRAM.ctrlm.%r@%h.$$ "cat > ${SOURCE_USER_HOMEDIR}/.ssh/${backup_software}.key; chmod 600 ${SOURCE_USER_HOMEDIR}/.ssh/${backup_software}.key" 164 | if [ "$?" != 0 ]; then 165 | echo "Failed to copy ssh key to source system" 166 | echo "Please copy file \"${TARGET_ROOT}/${backup_software}/.ssh/${backup_software}.rsa\" to source system in \"${SOURCE_USER_HOMEDIR}/.ssh/${backup_software}.key\" and execute chmod 600 \"${SOURCE_USER_HOMEDIR}/.ssh/${backup_software}.key\"" 167 | else 168 | rm -f /tmp/$PROGRAM.ctrlm.%r@%h.$$ 169 | fi 170 | done 171 | } 172 | 173 | function install_bupstash { 174 | ORG=andrewchambers 175 | REPO=bupstash 176 | lastest_version=$(get_lastest_git_release $ORG $REPO) 177 | 178 | Logger "Installing bupstash ${lastest_version}" "NOTICE" 179 | #dnf install -y rust cargo pkgconfig libsodium-devel tar # now installed in specific function 180 | mkdir -p /opt/bupstash/bupstash-"${lastest_version}" && cd /opt/bupstash/bupstash-"${lastest_version}" || exit 127 181 | curl -OL https://github.com/$ORG/$REPO/releases/download/"${lastest_version}"/bupstash-"${lastest_version}"-src+deps.tar.gz 182 | tar xvf bupstash-"${lastest_version}"-src+deps.tar.gz 183 | cargo build --release 184 | cp target/release/bupstash /usr/local/bin/ 185 | 186 | Logger "Installed bupstash $(get_version_bupstash)" "NOTICE" 187 | } 188 | 189 | function get_version_bupstash { 190 | echo "$(bupstash --version | awk -F'-' '{print $2}')" 191 | } 192 | 193 | function setup_ssh_bupstash_server { 194 | echo "$(echo -n "command=\"cd ${TARGET_ROOT}/bupstash; bupstash serve ${TARGET_ROOT}/bupstash/data\",no-port-forwarding,no-x11-forwarding,no-agent-forwarding,no-pty,no-user-rc "; cat ${TARGET_ROOT}/bupstash/.ssh/authorized_keys)" > ${TARGET_ROOT}/bupstash/.ssh/authorized_keys 195 | } 196 | 197 | function init_bupstash_repository { 198 | local remotely="${1:-false}" 199 | 200 | Logger "Initializing bupstash repository. Remote: ${remotely}." "NOTICE" 201 | if [ "${remotely}" == true ]; then 202 | export BUPSTASH_REPOSITORY_COMMAND=${BUPSTASH_REPOSITORY_COMMAND_REMOTE} 203 | unset BUPSTASH_REPOSITORY 204 | else 205 | export BUPSTASH_REPOSITORY="${BUPSTASH_REPOSITORY_LOCAL}" 206 | unset BUPSTASH_REPOSITORY_COMMAND 207 | fi 208 | 209 | bupstash init 210 | result=$? 211 | if [ "${result}" -ne 0 ]; then 212 | Logger "Failure with exit code $result" "CRITICAL" 213 | exit 125 214 | fi 215 | } 216 | 217 | function clear_bupstash_repository { 218 | local remotely="${1:-false}" 219 | 220 | # bupstash expects the directory to not already exist in order to server it bia bupstash / or even just to make an init since v0.12 221 | Logger "Clearing bupstash repository. Remote: ${remotely}." "NOTICE" 222 | cmd="rm -rf \"${TARGET_ROOT:?}/bupstash\"; mkdir ${TARGET_ROOT:?}/bupstash; if getent passwd | grep bupstash_user > /dev/null; then chown bupstash_user \"${TARGET_ROOT}/bupstash\"; fi" 223 | if [ "${remotely}" == true ]; then 224 | $REMOTE_SSH_RUNNER $cmd 225 | else 226 | eval "${cmd}" 227 | fi 228 | } 229 | 230 | function install_borg { 231 | ORG=borgbackup 232 | REPO=borg 233 | lastest_version=$(get_lastest_git_release $ORG $REPO) 234 | 235 | Logger "Installing borg ${lastest_version}" "NOTICE" 236 | 237 | # Earlier borg backup install commands 238 | #dnf install -y libacl-devel openssl-devel gcc-c++ 239 | #dnf -y install python39 python39-devel 240 | #python3.9 -m pip install --upgrade pip setuptools wheel 241 | #python3.9 -m pip install borgbackup 242 | 243 | # borg-linuxnew64 uses GLIBC 2.39 as of 20220905 whereas RHEL9 uses GLIBC 2.34 244 | curl -o /usr/local/bin/borg -L https://github.com/$ORG/$REPO/releases/download/${lastest_version}/borg-linux-glibc231 && chmod 755 /usr/local/bin/borg 245 | 246 | Logger "Installed borg $(get_version_borg)" "NOTICE" 247 | } 248 | 249 | function get_version_borg { 250 | echo "$(borg --version | awk '{print $2}')" 251 | } 252 | 253 | function install_borg_beta { 254 | Logger "Installing borg beta" "NOTICE" 255 | curl -L https://github.com/borgbackup/borg/releases/download/2.0.0b14/borg-linux-glibc231 -o /usr/local/bin/borg_beta && chmod 755 /usr/local/bin/borg_beta 256 | Logger "Installed borg_beta $(get_version_borg_beta)" "NOTICE" 257 | } 258 | 259 | function get_version_borg_beta { 260 | echo "$(borg_beta --version | awk '{print $2}')" 261 | } 262 | 263 | function setup_ssh_borg_server { 264 | echo "$(echo -n "command=\"cd ${TARGET_ROOT}/borg/data; borg serve --restrict-to-path ${TARGET_ROOT}/borg/data\",no-port-forwarding,no-x11-forwarding,no-agent-forwarding,no-pty,no-user-rc "; cat ${TARGET_ROOT}/borg/.ssh/authorized_keys)" > ${TARGET_ROOT}/borg/.ssh/authorized_keys 265 | } 266 | 267 | function setup_ssh_borg_beta_server { 268 | echo "$(echo -n "command=\"cd ${TARGET_ROOT}/borg_beta/data; borg_beta serve --restrict-to-path ${TARGET_ROOT}/borg_beta/data\",no-port-forwarding,no-x11-forwarding,no-agent-forwarding,no-pty,no-user-rc "; cat ${TARGET_ROOT}/borg_beta/.ssh/authorized_keys)" > ${TARGET_ROOT}/borg_beta/.ssh/authorized_keys 269 | } 270 | 271 | function init_borg_repository { 272 | local remotely="${1:-false}" 273 | 274 | Logger "Initializing borg repository. Remote: ${remotely}." "NOTICE" 275 | # -e repokey means AES-CTR-256 and HMAC-SHA256, see https://borgbackup.readthedocs.io/en/stable/usage/init.html) 276 | if [ "${remotely}" == true ]; then 277 | export BORG_REPO="$BORG_STABLE_REPO_REMOTE" 278 | borg init -e repokey --rsh "ssh -i ${SOURCE_USER_HOMEDIR}/.ssh/borg.key -p ${REMOTE_TARGET_SSH_PORT} -o StrictHostKeyChecking=accept-new" ${BORG_REPO} 279 | else 280 | export BORG_REPO="$BORG_STABLE_REPO_LOCAL" 281 | borg init -e repokey ${BORG_REPO} 282 | fi 283 | result=$? 284 | if [ "${result}" -ne 0 ]; then 285 | Logger "Failure with exit code $result" "CRITICAL" 286 | exit 125 287 | fi 288 | } 289 | 290 | function init_borg_beta_repository { 291 | local remotely="${1:-false}" 292 | 293 | Logger "Initializing borg_beta repository. Remote: ${remotely}." "NOTICE" 294 | # --encrpytion=repokey-aes-ocb was found using borg_beta benchmark cpu 295 | if [ "${remotely}" == true ]; then 296 | export BORG_REPO="$BORG_BETA_REPO_REMOTE" 297 | borg_beta --rsh "ssh -i ${SOURCE_USER_HOMEDIR}/.ssh/borg_beta.key -p ${REMOTE_TARGET_SSH_PORT} -o StrictHostKeyChecking=accept-new" repo-create --encryption=repokey-aes-ocb 298 | else 299 | export BORG_REPO="$BORG_BETA_REPO_LOCAL" 300 | borg_beta repo-create --encryption=repokey-aes-ocb 301 | fi 302 | result=$? 303 | if [ "${result}" -ne 0 ]; then 304 | Logger "Failure with exit code $result" "CRITICAL" 305 | exit 125 306 | fi 307 | } 308 | 309 | function clear_borg_repository { 310 | local remotely="${1:-false}" 311 | 312 | Logger "Clearing borg repository. Remote: ${remotely}." "NOTICE" 313 | # borg expects the data directory to already exist in order to serve it via borg --serve 314 | cmd="rm -rf \"${TARGET_ROOT:?}/borg/data\"; mkdir -p \"${TARGET_ROOT}/borg/data\"; if getent passwd | grep borg_user > /dev/null; then chown borg_user \"${TARGET_ROOT}/borg/data\"; fi" 315 | if [ "${remotely}" == true ]; then 316 | $REMOTE_SSH_RUNNER $cmd 317 | else 318 | eval "${cmd}" 319 | fi 320 | } 321 | 322 | function clear_borg_beta_repository { 323 | local remotely="${1:-false}" 324 | 325 | Logger "Clearing borg_beta repository. Remote: ${remotely}." "NOTICE" 326 | # borg expects the data directory to already exist in order to serve it via borg --serve 327 | cmd="rm -rf \"${TARGET_ROOT:?}/borg_beta/data\"; mkdir -p \"${TARGET_ROOT}/borg_beta/data\"; if getent passwd | grep borg_beta_user > /dev/null; then chown borg_beta_user \"${TARGET_ROOT}/borg_beta/data\"; fi" 328 | if [ "${remotely}" == true ]; then 329 | $REMOTE_SSH_RUNNER $cmd 330 | else 331 | eval "${cmd}" 332 | fi 333 | 334 | } 335 | 336 | function install_kopia { 337 | ORG=kopia 338 | REPO=kopia 339 | lastest_version=$(get_lastest_git_release $ORG $REPO) 340 | 341 | Logger "Installing kopia" "NOTICE" 342 | 343 | # Former kopia install instructions 344 | # rpm --import https://kopia.io/signing-key 345 | # cat < "${BACKUP_ROOT}/.duplicacy/filters" 576 | } 577 | 578 | function clear_duplicacy_repository { 579 | local remotely="${1:-false}" 580 | 581 | Logger "Clearing duplicacy repository. Remote: ${remotely}." "NOTICE" 582 | if [ "${remotely}" == true ]; then 583 | cmd="rm -rf \"${TARGET_ROOT:?}/duplicacy/data\" && mkdir -p \"${TARGET_ROOT}/duplicacy/data\" && chown duplicacy_user \"${TARGET_ROOT}/duplicacy/data\"" 584 | $REMOTE_SSH_RUNNER $cmd 585 | else 586 | cmd="rm -rf \"${TARGET_ROOT:?}/duplicacy/data\" && mkdir -p \"${TARGET_ROOT}/duplicacy/data\"" 587 | eval "${cmd}" 588 | fi 589 | # We also need to delete .duplicacy folder in source 590 | rm -rf "${BACKUP_ROOT}/.duplicacy" 591 | } 592 | 593 | function setup_git_dataset { 594 | #dnf install -y git # now installed in specific function 595 | # We'll assume that BACKUP_ROOT will be a git root, so we need to git clone in parent directory 596 | git_parent_dir="$(dirname ${BACKUP_ROOT:?})" 597 | [ ! -d "${git_parent_dir}" ] && mkdir -p "${git_parent_dir}" 598 | cd "${git_parent_dir}" || exit 127 599 | 600 | [ -d "${GIT_ROOT_DIRECTORY}" ] && rm -rf "${GIT_ROOT_DIRECTORY}" 601 | git clone ${GIT_DATASET_REPOSITORY} 602 | } 603 | 604 | function backup_bupstash { 605 | local remotely="${1}" 606 | local backup_id="${2}" 607 | 608 | Logger "Launching bupstash backup. Remote: ${remotely}." "NOTICE" 609 | if [ "${remotely}" == true ]; then 610 | export BUPSTASH_REPOSITORY_COMMAND=${BUPSTASH_REPOSITORY_COMMAND_REMOTE} 611 | unset BUPSTASH_REPOSITORY 612 | else 613 | export BUPSTASH_REPOSITORY=${BUPSTASH_REPOSITORY_LOCAL} 614 | unset BUPSTASH_REPOSITORY_COMMAND 615 | fi 616 | bupstash put --compression zstd:3 --exclude '.git' --exclude '.duplicacy' --print-file-actions --print-stats BACKUPID="${backup_id}" "${BACKUP_ROOT}/" >> /var/log/${PROGRAM}.bupstash_test.log 2>&1 617 | result=$? 618 | if [ "${result}" -ne 0 ]; then 619 | Logger "Failure with exit code $result" "CRITICAL" 620 | fi 621 | } 622 | 623 | function restore_bupstash { 624 | local remotely="${1}" 625 | local backup_id="${2}" 626 | 627 | Logger "Launching bupstash restore. Remote: ${remotely}." "NOTICE" 628 | if [ "${remotely}" == true ]; then 629 | export BUPSTASH_REPOSITORY_COMMAND=${BUPSTASH_REPOSITORY_COMMAND_REMOTE} 630 | unset BUPSTASH_REPOSITORY 631 | else 632 | export BUPSTASH_REPOSITORY=${BUPSTASH_REPOSITORY_LOCAL} 633 | unset BUPSTASH_REPOSITORY_COMMAND 634 | fi 635 | 636 | # Change store key by master key in order to be able to restore data 637 | export BUPSTASH_KEY="${SOURCE_USER_HOMEDIR}/bupstash.master.key" 638 | bupstash restore --into "${RESTORE_DIR}" BACKUPID="${backup_id}" 639 | result=$? 640 | if [ "${result}" -ne 0 ]; then 641 | Logger "Failure with exit code $result" "CRITICAL" 642 | fi 643 | export BUPSTASH_KEY="${SOURCE_USER_HOMEDIR}/bupstash.store.key" 644 | } 645 | 646 | function backup_borg { 647 | local remotely="${1}" 648 | local backup_id="${2}" 649 | 650 | Logger "Launching borg backup. Remote: ${remotely}." "NOTICE" 651 | if [ "${remotely}" == true ]; then 652 | export BORG_REPO="$BORG_STABLE_REPO_REMOTE" 653 | borg create --rsh "ssh -i ${SOURCE_USER_HOMEDIR}/.ssh/borg.key $SSH_OPTS -p ${REMOTE_TARGET_SSH_PORT}" --compression zstd,3 --exclude 're:\.git/.*$' --exclude 're:\.duplicacy/.*$' --stats --verbose ${BORG_REPO}::"${backup_id}" "${BACKUP_ROOT}/" >> /var/log/${PROGRAM}.borg_tests.log 2>&1 654 | else 655 | export BORG_REPO="$BORG_STABLE_REPO_LOCAL" 656 | borg create --compression zstd,3 --exclude 're:\.git/.*$' --exclude 're:\.duplicacy/.*$' --stats --verbose ${BORG_REPO}::"${backup_id}" "${BACKUP_ROOT}/" >> /var/log/${PROGRAM}.borg_tests.log 2>&1 657 | fi 658 | result=$? 659 | if [ "${result}" -ne 0 ]; then 660 | Logger "Failure with exit code $result" "CRITICAL" 661 | fi 662 | # We can check the exclusion patterns with borg create --list --dry-run --exclude ... 663 | } 664 | 665 | function restore_borg { 666 | local remotely="${1}" 667 | local backup_id="${2}" 668 | 669 | Logger "Launching borg restore. Remote: ${remotely}." "NOTICE" 670 | cd "${RESTORE_DIR}" || return 127 671 | # We'll use --noacls and --noxattrs to make sure we have same functionnality as others 672 | if [ "${remotely}" == true ]; then 673 | export BORG_REPO="$BORG_STABLE_REPO_REMOTE" 674 | borg extract --rsh "ssh -i ${SOURCE_USER_HOMEDIR}/.ssh/borg.key -p ${REMOTE_TARGET_SSH_PORT}" --noacls --noxattrs ${BORG_REPO}::"${backup_id}" >> /var/log/${PROGRAM}.borg_tests.log 2>&1 675 | else 676 | export BORG_REPO="$BORG_STABLE_REPO_LOCAL" 677 | borg extract --noacls --noxattrs ${BORG_REPO}::"${backup_id}" >> /var/log/${PROGRAM}.borg_tests.log 2>&1 678 | fi 679 | result=$? 680 | if [ "${result}" -ne 0 ]; then 681 | Logger "Failure with exit code $result" "CRITICAL" 682 | fi 683 | } 684 | 685 | function backup_borg_beta { 686 | local remotely="${1}" 687 | local backup_id="${2}" 688 | 689 | Logger "Launching borg_beta backup. Remote: ${remotely}." "NOTICE" 690 | if [ "${remotely}" == true ]; then 691 | export BORG_REPO="$BORG_BETA_REPO_REMOTE" 692 | borg_beta create --rsh "ssh -i ${SOURCE_USER_HOMEDIR}/.ssh/borg_beta.key $SSH_OPTS -p ${REMOTE_TARGET_SSH_PORT}" --compression zstd,3 --exclude 're:\.git/.*$' --exclude 're:\.duplicacy/.*$' --stats --verbose "${backup_id}" "${BACKUP_ROOT}/" >> /var/log/${PROGRAM}.borg_beta_tests.log 2>&1 693 | else 694 | export BORG_REPO="$BORG_BETA_REPO_LOCAL" 695 | borg_beta create --compression zstd,3 --exclude 're:\.git/.*$' --exclude 're:\.duplicacy/.*$' --stats --verbose "${backup_id}" "${BACKUP_ROOT}/" >> /var/log/${PROGRAM}.borg_beta_tests.log 2>&1 696 | fi 697 | result=$? 698 | if [ "${result}" -ne 0 ]; then 699 | Logger "Failure with exit code $result" "CRITICAL" 700 | fi 701 | # We can check the exclusion patterns with borg create --list --dry-run --exclude ... 702 | } 703 | 704 | function restore_borg_beta { 705 | local remotely="${1}" 706 | local backup_id="${2}" 707 | 708 | Logger "Launching borg_beta restore. Remote: ${remotely}." "NOTICE" 709 | cd "${RESTORE_DIR}" || return 127 710 | # We'll use --noacls and --noxattrs to make sure we have same functionnality as others 711 | if [ "${remotely}" == true ]; then 712 | export BORG_REPO="$BORG_BETA_REPO_REMOTE" 713 | borg_beta extract --rsh "ssh -i ${SOURCE_USER_HOMEDIR}/.ssh/borg_beta.key -p ${REMOTE_TARGET_SSH_PORT}" --noacls --noxattrs "${backup_id}" >> /var/log/${PROGRAM}.borg_beta_tests.log 2>&1 714 | else 715 | export BORG_REPO="$BORG_BETA_REPO_LOCAL" 716 | borg_beta extract --rsh "ssh -i ${SOURCE_USER_HOMEDIR}/.ssh/borg_beta.key -p ${REMOTE_TARGET_SSH_PORT}" --noacls --noxattrs "${backup_id}" >> /var/log/${PROGRAM}.borg_beta_tests.log 2>&1 717 | fi 718 | result=$? 719 | if [ "${result}" -ne 0 ]; then 720 | Logger "Failure with exit code $result" "CRITICAL" 721 | fi 722 | } 723 | 724 | function backup_kopia { 725 | local remotely="${1}" 726 | local backup_id="${2}" 727 | 728 | Logger "Launching kopia backup. Remote: ${remotely}." "NOTICE" 729 | 730 | if [ "${remotely}" == true ]; then 731 | if [ "${KOPIA_USE_HTTP}" == true ]; then 732 | kopia repository connect server --url=https://${REMOTE_TARGET_FQDN}:${KOPIA_HTTP_PORT} --server-cert-fingerprint=$(get_remote_certificate_fingerprint ${REMOTE_TARGET_FQDN} ${KOPIA_HTTP_PORT}) -p ${KOPIA_HTTP_PASSWORD} --override-username=${KOPIA_HTTP_USERNAME} --override-hostname=backup-bench-source 733 | export KOPIA_PASSWORD= # if not cleaned, kopia snapshot will fail 734 | else 735 | kopia repository connect sftp --path=${TARGET_ROOT}/kopia/data --host=${REMOTE_TARGET_FQDN} --port ${REMOTE_TARGET_SSH_PORT} --keyfile=${SOURCE_USER_HOMEDIR}/.ssh/kopia.key --username=kopia_user --known-hosts=${SOURCE_USER_HOMEDIR}/.ssh/known_hosts 736 | fi 737 | else 738 | kopia repository connect filesystem --path=${TARGET_ROOT}/kopia/data 739 | fi 740 | kopia snapshot create --parallel 8 --tags BACKUPID:"${backup_id}" "${BACKUP_ROOT}/" >> /var/log/${PROGRAM}.kopia_test.log 2>&1 741 | result=$? 742 | if [ "${result}" -ne 0 ]; then 743 | Logger "Failure with exit code $result" "CRITICAL" 744 | fi 745 | # We can check the exclusion patterns with kopia snapshot estimate 746 | 747 | } 748 | 749 | function restore_kopia { 750 | local remotely="${1}" 751 | local backup_id="${2}" 752 | 753 | Logger "Launching kopia restore. Remote: ${remotely}." "NOTICE" 754 | 755 | if [ "${remotely}" == true ]; then 756 | if [ "${KOPIA_USE_HTTP}" == true ]; then 757 | kopia repository connect server --url=https://${REMOTE_TARGET_FQDN}:${KOPIA_HTTP_PORT} --server-cert-fingerprint=$(get_remote_certificate_fingerprint ${REMOTE_TARGET_FQDN} ${KOPIA_HTTP_PORT}) -p ${KOPIA_HTTP_PASSWORD} --override-username=${KOPIA_HTTP_USERNAME} --override-hostname=backup-bench-source 758 | export KOPIA_PASSWORD= # if not cleaned, kopia restore will fail 759 | else 760 | kopia repository connect sftp --path=${TARGET_ROOT}/kopia/data --host=${REMOTE_TARGET_FQDN} --port ${REMOTE_TARGET_SSH_PORT} --keyfile=${SOURCE_USER_HOMEDIR}/.ssh/kopia.key --username=kopia_user --known-hosts=${SOURCE_USER_HOMEDIR}/.ssh/known_hosts 761 | fi 762 | else 763 | kopia repository connect filesystem --path=${TARGET_ROOT}/kopia/data 764 | fi 765 | id="$(kopia snapshot list --tags BACKUPID:${backup_id} | awk '{print $4}')" 766 | kopia restore --parallel 8 --skip-owners --skip-permissions ${id} "${RESTORE_DIR}" >> /var/log/${PROGRAM}.kopia_test.log 2>&1 767 | result=$? 768 | if [ "${result}" -ne 0 ]; then 769 | Logger "Failure with exit code $result" "CRITICAL" 770 | fi 771 | } 772 | 773 | function backup_restic { 774 | local remotely="${1}" 775 | local backup_id="${2}" 776 | 777 | Logger "Launching restic backup. Remote: ${remotely}." "NOTICE" 778 | 779 | if [ "${remotely}" == true ]; then 780 | if [ "${RESTIC_USE_HTTP}" == true ]; then 781 | restic --insecure-tls -r rest:https://${REMOTE_TARGET_FQDN}:${RESTIC_HTTP_PORT}/ backup --verbose --exclude=".git" --exclude=".duplicacy" --tag="${backup_id}" --compression=auto "${BACKUP_ROOT}/" >> /var/log/${PROGRAM}.restic_tests.log 2>&1 782 | else 783 | restic -r sftp::${TARGET_ROOT}/restic/data -o sftp.command="ssh restic_user@${REMOTE_TARGET_FQDN} -i ${SOURCE_USER_HOMEDIR}/.ssh/restic.key $SSH_OPTS -p ${REMOTE_TARGET_SSH_PORT} -s sftp" backup --verbose --exclude=".git" --exclude=".duplicacy" --tag="${backup_id}" --compression=auto "${BACKUP_ROOT}/" >> /var/log/${PROGRAM}.restic_tests.log 2>&1 784 | fi 785 | else 786 | restic -r ${TARGET_ROOT}/restic/data backup --verbose --exclude=".git" --exclude=".duplicacy" --tag="${backup_id}" --compression=auto "${BACKUP_ROOT}/" >> /var/log/${PROGRAM}.restic_tests.log 2>&1 787 | fi 788 | result=$? 789 | if [ "${result}" -ne 0 ]; then 790 | Logger "Failure with exit code $result" "CRITICAL" 791 | fi 792 | } 793 | 794 | function restore_restic { 795 | local remotely="${1}" 796 | local backup_id="${2}" 797 | 798 | Logger "Launching restic restore. Remote: ${remotely}." "NOTICE" 799 | 800 | if [ "${remotely}" == true ]; then 801 | if [ "${RESTIC_USE_HTTP}" == true ]; then 802 | id=$(restic --insecure-tls -r rest:https://${REMOTE_TARGET_FQDN}:${RESTIC_HTTP_PORT}/ snapshots | grep "${backup_id}" | awk '{print $1}') 803 | restic --insecure-tls -r rest:https://${REMOTE_TARGET_FQDN}:${RESTIC_HTTP_PORT}/ restore "$id" --target "${RESTORE_DIR}" >> /var/log/${PROGRAM}.restic_tests.log 2>&1 804 | else 805 | id=$(restic -r sftp::${TARGET_ROOT}/restic/data -o sftp.command="ssh restic_user@${REMOTE_TARGET_FQDN} -i ${SOURCE_USER_HOMEDIR}/.ssh/restic.key $SSH_OPTS -p ${REMOTE_TARGET_SSH_PORT} -s sftp" snapshots | grep "${backup_id}" | awk '{print $1}') 806 | restic -r sftp::${TARGET_ROOT}/restic/data -o sftp.command="ssh restic_user@${REMOTE_TARGET_FQDN} -i ${SOURCE_USER_HOMEDIR}/.ssh/restic.key $SSH_OPTS -p ${REMOTE_TARGET_SSH_PORT} -s sftp" restore "$id" --target "${RESTORE_DIR}" >> /var/log/${PROGRAM}.restic_tests.log 2>&1 807 | fi 808 | else 809 | id=$(restic -r ${TARGET_ROOT}/restic/data snapshots | grep "${backup_id}" | awk '{print $1}') 810 | restic -r ${TARGET_ROOT}/restic/data restore "$id" --target "${RESTORE_DIR}" >> /var/log/${PROGRAM}.restic_tests.log 2>&1 811 | fi 812 | result=$? 813 | if [ "${result}" -ne 0 ]; then 814 | Logger "Failure with exit code $result" "CRITICAL" 815 | fi 816 | } 817 | 818 | function backup_rustic { 819 | local remotely="${1}" 820 | local backup_id="${2}" 821 | 822 | Logger "Launching rustic backup. Remote: ${remotely}." "NOTICE" 823 | 824 | if [ "${remotely}" == true ]; then 825 | if [ "${RESTIC_USE_HTTP}" == true ]; then 826 | rustic --insecure-tls -r rest:https://${REMOTE_TARGET_FQDN}:${RUSTIC_HTTP_PORT}/ backup --glob="!.git" --glob="!.duplicacy" --tag="${backup_id}" "${BACKUP_ROOT}/" >> /var/log/${PROGRAM}.rustic_tests.log 2>&1 827 | else 828 | rustic -r sftp::${TARGET_ROOT}/rustic/data -o sftp.command="ssh rustic_user@${REMOTE_TARGET_FQDN} -i ${SOURCE_USER_HOMEDIR}/.ssh/rustic.key $SSH_OPTS -p ${REMOTE_TARGET_SSH_PORT} -s sftp" backup --verbose --glob="!.git" --glob="!.duplicacy" --tag="${backup_id}" "${BACKUP_ROOT}/" >> /var/log/${PROGRAM}.rustic_tests.log 2>&1 829 | fi 830 | else 831 | rustic -r ${TARGET_ROOT}/rustic/data backup --glob="!.git" --glob="!.duplicacy" --tag="${backup_id}" "${BACKUP_ROOT}/" >> /var/log/${PROGRAM}.rustic_tests.log 2>&1 832 | fi 833 | result=$? 834 | if [ "${result}" -ne 0 ]; then 835 | Logger "Failure with exit code $result" "CRITICAL" 836 | fi 837 | } 838 | 839 | function restore_rustic { 840 | local remotely="${1}" 841 | local backup_id="${2}" 842 | 843 | Logger "Launching rustic restore. Remote: ${remotely}." "NOTICE" 844 | 845 | if [ "${remotely}" == true ]; then 846 | if [ "${RUSTIC_USE_HTTP}" == true ]; then 847 | id=$(rustic --insecure-tls -r rest:https://${REMOTE_TARGET_FQDN}:${RUSTIC_HTTP_PORT}/ snapshots | grep "${backup_id}" | awk '{print $2}') 848 | rustic --insecure-tls -r rest:https://${REMOTE_TARGET_FQDN}:${RUSTIC_HTTP_PORT}/ restore "$id" "${RESTORE_DIR}" >> /var/log/${PROGRAM}.rustic_tests.log 2>&1 849 | else 850 | id=$(rustic -r sftp::${TARGET_ROOT}/rustic/data -o sftp.command="ssh rustic_user@${REMOTE_TARGET_FQDN} -i ${SOURCE_USER_HOMEDIR}/.ssh/rustic.key $SSH_OPTS -p ${REMOTE_TARGET_SSH_PORT} -s sftp" snapshots | grep "${backup_id}" | awk '{print $2}') 851 | rustic -r sftp::${TARGET_ROOT}/rustic/data -o sftp.command="ssh rustic_user@${REMOTE_TARGET_FQDN} -i ${SOURCE_USER_HOMEDIR}/.ssh/rustic.key $SSH_OPTS -p ${REMOTE_TARGET_SSH_PORT} -s sftp" restore "$id" "${RESTORE_DIR}" >> /var/log/${PROGRAM}.rustic_tests.log 2>&1 852 | fi 853 | else 854 | id=$(rustic -r ${TARGET_ROOT}/rustic/data snapshots | grep "${backup_id}" | awk '{print $2}') 855 | rustic -r ${TARGET_ROOT}/rustic/data restore "$id" "${RESTORE_DIR}" >> /var/log/${PROGRAM}.rustic_tests.log 2>&1 856 | fi 857 | result=$? 858 | if [ "${result}" -ne 0 ]; then 859 | Logger "Failure with exit code $result" "CRITICAL" 860 | fi 861 | } 862 | 863 | 864 | function backup_duplicacy { 865 | local remotely="${1}" 866 | local backup_id="${2}" 867 | 868 | cd "${BACKUP_ROOT}" || exit 124 869 | 870 | Logger "Launching duplicacy backup. Remote: ${remotely}." "NOTICE" 871 | 872 | # Added -threads 8 according to https://github.com/deajan/backup-bench/issues/14 873 | 874 | duplicacy backup -t "${backup_id}" --stats -threads 8 >> /var/log/${PROGRAM}.duplicacy_tests.log 2>&1 875 | result=$? 876 | if [ "${result}" -ne 0 ]; then 877 | Logger "Failure with exit code $result" "CRITICAL" 878 | fi 879 | } 880 | 881 | function restore_duplicacy { 882 | local remotely="${1}" 883 | local backup_id="${2}" 884 | 885 | Logger "Launching duplicacy restore. Remote: ${remotely}." "NOTICE" 886 | 887 | # duplicacy needs to init the repo (named someid here) to another directory so it can be restored 888 | if [ "${remotely}" == true ]; then 889 | cd "${RESTORE_DIR}" && duplicacy init -e remoteid sftp://duplicacy_user@${REMOTE_TARGET_FQDN}:${REMOTE_TARGET_SSH_PORT}/${TARGET_ROOT}/duplicacy/data 890 | else 891 | cd "${RESTORE_DIR}" && duplicacy init -e localid ${TARGET_ROOT}/duplicacy/data 892 | fi 893 | result=$? 894 | if [ "${result}" -ne 0 ]; then 895 | Logger "Failure with exit code $result" "CRITICAL" 896 | fi 897 | 898 | revision=$(duplicacy list | grep "${backup_id}" | awk '{print $4}') 899 | Logger "Using revision [${revision}]" "NOTICE" 900 | 901 | # Added -threads 8 according to https://github.com/deajan/backup-bench/issues/14 902 | 903 | duplicacy restore -r ${revision} -threads 8 >> /var/log/${PROGRAM}.duplicacy_tests.log 2>&1 904 | result=$? 905 | if [ "${result}" -ne 0 ]; then 906 | Logger "Failure with exit code $result" "CRITICAL" 907 | fi 908 | } 909 | 910 | 911 | function get_repo_sizes { 912 | local remotely="${1:-false}" 913 | 914 | CSV_SIZE="size(kb)," 915 | 916 | for backup_software in "${BACKUP_SOFTWARES[@]}"; do 917 | if [ "${remotely}" == true ]; then 918 | size=$($REMOTE_SSH_RUNNER du -cs "${TARGET_ROOT}/${backup_software}" | tail -n 1 | awk '{print $1}') 919 | else 920 | size=$(du -cs "${TARGET_ROOT}/${backup_software}" | tail -n 1 | awk '{print $1}') 921 | fi 922 | CSV_SIZE="${CSV_SIZE}${size}," 923 | Logger "Repo size for ${backup_software}: ${size} kb. Remote: ${remotely}." "NOTICE" 924 | done 925 | echo "${CSV_SIZE}" >> "${CSV_RESULT_FILE}" 926 | } 927 | 928 | function install_backup_programs { 929 | local is_remote="${1:-false}" 930 | 931 | # bupstash, borg and kopia need to be installed on both source and targets 932 | # restic and duplicity don't need to be installed on target 933 | # restic rest server only needs to be installed on target 934 | 935 | install_bupstash 936 | install_borg 937 | install_borg_beta 938 | install_kopia 939 | 940 | [ "$is_remote" == false ] && install_restic 941 | [ "$is_remote" == false ] && install_rustic 942 | [ "$is_remote" == true ] && install_restic_rest_server 943 | [ "$is_remote" == false ] && install_duplicacy 944 | } 945 | 946 | function setup_source { 947 | local remotely="${1:-false}" 948 | 949 | Logger "Setting up source server" "NOTICE" 950 | download_prerequisites ${NODEPS} 951 | 952 | install_backup_programs false 953 | 954 | if [ "${remotely}" == false ]; then 955 | Logger "Setting up local target" "NOTICE" 956 | setup_target_local_repos 957 | fi 958 | 959 | # Specific setup for bupstash where key is stored as file instead of env variable 960 | [ ! -f "${SOURCE_USER_HOMEDIR}/bupstash.master.key" ] && bupstash new-key -o ${SOURCE_USER_HOMEDIR}/bupstash.master.key 961 | [ ! -f "${SOURCE_USER_HOMEDIR}/bupstash.store.key" ] && bupstash new-sub-key -k ${SOURCE_USER_HOMEDIR}/bupstash.master.key --put --list -o ${SOURCE_USER_HOMEDIR}/bupstash.store.key 962 | 963 | } 964 | 965 | function setup_remote_target { 966 | local remotely="${1:-false}" # Has no use here obviously, but we'll keep it since remotely argument is passed 967 | 968 | Logger "Setting up remote target server" "NOTICE" 969 | 970 | setup_root_access 971 | download_prerequisites ${NODEPS} 972 | 973 | install_backup_programs true 974 | 975 | clear_users 976 | setup_target_remote_repos 977 | 978 | setup_ssh_bupstash_server 979 | 980 | setup_ssh_borg_server 981 | setup_ssh_borg_beta_server 982 | 983 | create_certificate https_backup-bench 984 | } 985 | 986 | function clear_repositories { 987 | local remotely="${1:-false}" 988 | 989 | Logger "Clearing all repositories from earlier data. Remote clean: $remotely". "NOTICE" 990 | for backup_software in "${BACKUP_SOFTWARES[@]}"; do 991 | clear_"${backup_software}"_repository "${remotely}" 992 | done 993 | Logger "Clearing done" "NOTICE" 994 | } 995 | 996 | function init_repositories { 997 | local remotely="${1:-false}" 998 | local git="${2:-false}" 999 | 1000 | # The only reason we need to setup our dataset before being able to init the backup repositories is because duplicacy needs an existing source dir to init it's repo... 1001 | [ "${git}" == true ] && setup_git_dataset 1002 | 1003 | if [ ! -d "${BACKUP_ROOT}" ]; then 1004 | Logger "Backup root ${BACKUP_ROOT} does not exist. Either create it and add test content, or use --git to initialize it." "CRITICAL" 1005 | exit 3 1006 | fi 1007 | 1008 | Logger "Initializing reposiories. Remote: ${remotely}." "NOTICE" 1009 | for backup_software in "${BACKUP_SOFTWARES[@]}"; do 1010 | init_"${backup_software}"_repository "${remotely}" 1011 | done 1012 | Logger "Initialization done." "NOTICE" 1013 | } 1014 | 1015 | function serve_http_targets { 1016 | [ ! -f "${TARGET_ROOT}/kopia/data/kopia.repository.f" ] && kopia repository create filesystem --path=${TARGET_ROOT}/kopia/data 1017 | cmd="kopia server start --address 0.0.0.0:${KOPIA_HTTP_PORT} --no-ui --tls-cert-file=\"${HOME}/https_backup-bench.crt\" --tls-key-file=\"${HOME}/https_backup-bench.key\"" 1018 | Logger "Running kopia server with following command:\n$cmd" "NOTICE" 1019 | eval $cmd & 1020 | pid=$! 1021 | # add acls for user 1022 | cmd="kopia server users add ${KOPIA_HTTP_USERNAME}@backup-bench-source --user-password=${KOPIA_HTTP_PASSWORD}" 1023 | eval $cmd 1024 | Logger "Adding kopia user woth following command:\n$cmd" "NOTICE" 1025 | # reload server 1026 | cmd="kopia server refresh --address https://localhost:${KOPIA_HTTP_PORT} --server-cert-fingerprint=$(get_certificate_fingerprint "${HOME}/https_backup-bench.crt") --server-control-username=${KOPIA_SERVER_CONTROL_USER} --server-control-password=${KOPIA_SERVER_CONTROL_PASSWORD}" 1027 | Logger "Running kopia refresh with following command:\n$cmd" "NOTICE" 1028 | sleep 2 # arbitrary wait time 1029 | eval $cmd & 1030 | Logger "Serving kopia on http port ${KOPIA_HTTP_PORT} using pid $pid." "NOTICE" 1031 | rest-server --no-auth --listen 0.0.0.0:${RESTIC_HTTP_PORT} --path ${TARGET_ROOT}/restic/data --tls --tls-cert="${HOME}/https_backup-bench.crt" --tls-key="${HOME}/https_backup-bench.key" & 1032 | pid=$! 1033 | Logger "Serving rest-serve for restic on http port ${RESTIC_HTTP_PORT} using pid $pid." "NOTICE" 1034 | Logger "Stop servers using $0 --stop-http-targets" "NOTICE" 1035 | 1036 | rest-server --no-auth --listen 0.0.0.0:${RUSTIC_HTTP_PORT} --path ${TARGET_ROOT}/rustic/data --tls --tls-cert="${HOME}/https_backup-bench.crt" --tls-key="${HOME}/https_backup-bench.key" & 1037 | pid=$! 1038 | Logger "Serving rest-serve for rustic on http port ${RUSTIC_HTTP_PORT} using pid $pid." "NOTICE" 1039 | Logger "Stop servers using $0 --stop-http-targets" "NOTICE" 1040 | echo "" # Just clear the line at the end 1041 | } 1042 | 1043 | function stop_serve_http_targets { 1044 | for i in $(pgrep kopia); do kill $i; done 1045 | for i in $(pgrep rest-server); do kill $i; done 1046 | } 1047 | 1048 | function benchmark_backup_standard { 1049 | local remotely="${1}" 1050 | local backup_id="${2:-defaultid}" 1051 | 1052 | CSV_BACKUP_EXEC_TIME="backup(s)," 1053 | 1054 | for backup_software in "${BACKUP_SOFTWARES[@]}"; do 1055 | CSV_HEADER="${CSV_HEADER}${backup_software}," 1056 | echo 3 > /proc/sys/vm/drop_caches # Make sure we drop caches (including zfs arc cache before every backup) 1057 | [ "${remotely}" == true ] && $REMOTE_SSH_RUNNER "echo 3 > /proc/sys/vm/drop_caches" 1058 | Logger "Starting backup bench of ${backup_software} name=${backup_id}" "NOTICE" 1059 | seconds_begin=$SECONDS 1060 | # Launch backup software from function "name"_backup as background so we keep control 1061 | backup_"${backup_software}" "${remotely}" "${backup_id}" & 1062 | ExecTasks "$!" "${backup_software}_bench" false 3600 36000 3600 36000 1063 | exec_time=$((SECONDS - seconds_begin)) 1064 | CSV_BACKUP_EXEC_TIME="${CSV_BACKUP_EXEC_TIME}${exec_time}," 1065 | Logger "It took ${exec_time} seconds to backup." "NOTICE" 1066 | done 1067 | 1068 | echo "$CSV_BACKUP_EXEC_TIME" >> "${CSV_RESULT_FILE}" 1069 | get_repo_sizes "${remotely}" 1070 | } 1071 | 1072 | function benchmark_backup_git { 1073 | local remotely="${1}" 1074 | 1075 | Logger "Running git dataset backup benchmarks. Remote: ${remotely}" "NOTICE" 1076 | 1077 | cd "${BACKUP_ROOT}" || exit 127 1078 | 1079 | # Backup that kernel 1080 | for tag in "${GIT_TAGS[@]}"; do 1081 | 1082 | # Thanks to duplicacy who tampers with backup root conntent by adding '.duplicacy'... we need to save .duplicacy directory before every git checkout in order not to loose the files 1083 | #alias cp=cp && cp -R "${BACKUP_ROOT}/.duplicacy" "/tmp/backup_bench.duplicacy" 1084 | # Make sure we always we checkout a specific kernel version so results are reproductible 1085 | git checkout "${tag}" 1086 | #alias cp=cp && cp -R "/tmp/backup_bench.duplicacy" "${BACKUP_ROOT}/.duplicacy" 1087 | benchmark_backup_standard "${remotely}" "bkp-${tag}" 1088 | done 1089 | } 1090 | 1091 | function benchmark_backup { 1092 | local remotely="${1}" 1093 | local git="${2:-false}" 1094 | local backup_id_timestamp="${3:-false}" 1095 | 1096 | echo "# $PROGRAM $PROGRAM_BUILD $(date) Remote: ${remotely}, Git: ${git}" >> "${CSV_RESULT_FILE}" 1097 | CSV_HEADER="," 1098 | 1099 | for backup_software in "${BACKUP_SOFTWARES[@]}"; do 1100 | CSV_HEADER="${CSV_HEADER}${backup_software} $(get_version_${backup_software})," 1101 | done 1102 | echo "${CSV_HEADER}" >> "${CSV_RESULT_FILE}" 1103 | 1104 | if [ "${git}" == true ]; then 1105 | benchmark_backup_git "${remotely}" 1106 | else 1107 | if [ "$backup_id_timestamp" == true ]; then 1108 | backup_id="$(date +"%Y-%m-%d-T%H-%M-%S")" 1109 | else 1110 | backup_id="defaultid" 1111 | fi 1112 | benchmark_backup_standard "${remotely}" "${backup_id}" 1113 | fi 1114 | } 1115 | 1116 | function benchmark_restore_standard { 1117 | local remotely="${1}" 1118 | local backup_id="${2:-defaultid}" 1119 | 1120 | CSV_RESTORE_EXEC_TIME="restoration(s)," 1121 | 1122 | # Restore last snapshot and compare with actual kernel 1123 | for backup_software in "${BACKUP_SOFTWARES[@]}"; do 1124 | echo 3 > /proc/sys/vm/drop_caches # Make sure we drop caches (including zfs arc cache before every backup) 1125 | [ "${remotely}" == true ] && $REMOTE_SSH_RUNNER "echo 3 > /proc/sys/vm/drop_caches" 1126 | 1127 | [ -d "${RESTORE_DIR}" ] && rm -rf "${RESTORE_DIR:?}" 1128 | mkdir -p "${RESTORE_DIR}" 1129 | 1130 | Logger "Starting restore bench of ${backup_software} name=${backup_id}" "NOTICE" 1131 | seconds_begin=$SECONDS 1132 | # Launch backup software from function "name"_restore as background so we keep control 1133 | restore_"${backup_software}" "${remotely}" "${backup_id}" & 1134 | ExecTasks "$!" "${backup_software}_restore" false 3600 18000 3600 18000 1135 | exec_time=$((SECONDS - seconds_begin)) 1136 | CSV_RESTORE_EXEC_TIME="${CSV_RESTORE_EXEC_TIME}${exec_time}," 1137 | Logger "It took ${exec_time} seconds to restore." "NOTICE" 1138 | 1139 | # Make sure restored version matches current version 1140 | Logger "Compare restored version to original directory" "NOTICE" 1141 | # borg and restic restore full paths, so we need to change restored path 1142 | if [ "${backup_software}" == "borg" ] || [ "${backup_software}" == "borg_beta" ] || [ "${backup_software}" == "restic" ] || [ "${backup_software}" == "rustic" ]; then 1143 | restored_path="${RESTORE_DIR}"/"${BACKUP_ROOT}/" 1144 | else 1145 | restored_path="${RESTORE_DIR}" 1146 | fi 1147 | diff -x .git -x .duplicacy -qr "${restored_path}" "${BACKUP_ROOT}/" 1148 | result=$? 1149 | if [ "${result}" -ne 0 ]; then 1150 | Logger "Failure with exit code $result for restore comparaison." "CRITICAL" 1151 | else 1152 | Logger "Restored files match source." "NOTICE" 1153 | fi 1154 | done 1155 | echo "$CSV_RESTORE_EXEC_TIME" >> "${CSV_RESULT_FILE}" 1156 | 1157 | } 1158 | 1159 | function benchmark_restore_git { 1160 | local remotely="${1}" 1161 | 1162 | Logger "Running git dataset restore Benchmarks. Remote: ${remotely}" "NOTICE" 1163 | 1164 | cd "${BACKUP_ROOT}/" || exit 127 1165 | git checkout "${GIT_TAGS[-1]}" 1166 | benchmark_restore_standard "${remotely}" "bkp-${GIT_TAGS[-1]}" 1167 | } 1168 | 1169 | function benchmark_restore { 1170 | local remotely="${1}" 1171 | local git="${2:-false}" 1172 | local backup_id_timestamp="${3:-false}" 1173 | 1174 | if [ "${git}" == true ]; then 1175 | benchmark_restore_git "${remotely}" 1176 | else 1177 | if [ "$backup_id_timestamp" == true ]; then 1178 | backup_id="$(date +"%Y-%m-%d-T%H-%M-%S")" 1179 | else 1180 | backup_id="defaultid" 1181 | fi 1182 | benchmark_restore_standard "${remotely}" "${backup_id}" 1183 | fi 1184 | } 1185 | 1186 | function benchmarks { 1187 | local remotely="${1}" 1188 | local git="${2:-false}" 1189 | local backup_id_timestamp="${3:-false}" 1190 | 1191 | benchmark_backup "${remotely}" "${git}" "${backup_id_timestamp}" 1192 | benchmark_restore "${remotely}" "${git}" "${backup_id_timestamp}" 1193 | } 1194 | 1195 | 1196 | function versions { 1197 | for backup_software in "${BACKUP_SOFTWARES[@]}"; do 1198 | version=$(get_version_${backup_software}) 1199 | echo "${backup_software} $version" 1200 | done 1201 | } 1202 | 1203 | function usage { 1204 | echo "$PROGRAM $PROGRAM_BUILD" 1205 | echo "$AUTHOR" 1206 | echo "" 1207 | echo "Please setup your config file (defaults to backup-bench.conf" 1208 | echo "Once you've setup the configuration, you may use it to initialize target, then source." 1209 | echo "After initialization, benchmarks may run" 1210 | echo "" 1211 | echo "--config=/path/to/file.conf Alternative configuration file" 1212 | echo "--setup-remote-target Install backup programs and setup SSH access (executed on target)" 1213 | echo "--setup-source Install backup programs and setup local (or remote with --remote) repositories (executed on source)" 1214 | echo "--init-repos Reinitialize local (or remote with --remote) repositories after clearing. Must be used with --git if multiple version benchmarks is used) (executed on source)" 1215 | echo "--serve-http-targets Launch http servers for kopia and restic manually" 1216 | echo "--stop-http-targets Stop http servers for kopia and restic" 1217 | echo "--benchmark-backup Run backup benchmarks using local (or remote with --remote) repositories" 1218 | echo "--benchmark-restore Run restore benchmarks using local (or remote with --remote) repositories, restores to local restore path" 1219 | echo "--benchmarks Run both backup and restore benchmark using local (or remote with --remote) repositories and local restore path" 1220 | echo "--all Clear, init and run bakcup with git dataset for both local and remote targets" 1221 | echo "" 1222 | echo "MODIFIERS" 1223 | echo "--git Use git dataset (multiple version benchmark)" 1224 | echo "--local Execute locally (works for --clear-repos, --init-repos, --benchmark*)" 1225 | echo "--remote Execute remotely (works for --clear-repos, --init-repos, --benchmark*)" 1226 | echo "--backup-id-timestamp Add a timestamp as backup id when doing using --git. If this option is disabled, backupid will be \"defaultid\". There cannot be multiple backups with the same id" 1227 | echo "" 1228 | echo "After some benchmarks, you might want to remove earlier data from repositories" 1229 | echo "--clear-repos Removes data from local (or remote with --remote) repositories" 1230 | echo "" 1231 | echo "DEBUG commands" 1232 | echo "--setup-root-access Manually setup root access (executed on target)" 1233 | echo "--no-deps Do not install dependencies. This requires you to have them installed manually" 1234 | echo "--install-backup-programs Locally install / upgrade backup programs into /usr/local/bin. If launched with --remote, it will install only remote target required programs" 1235 | echo "--versions Show versions of all installed backup programs" 1236 | exit 128 1237 | } 1238 | 1239 | ## SCRIPT ENTRY POINT 1240 | 1241 | self_setup 1242 | 1243 | if [ "$#" -eq 0 ] 1244 | then 1245 | usage 1246 | fi 1247 | 1248 | cmd="" 1249 | REMOTELY=false 1250 | USE_GIT_VERSIONS=false 1251 | ALL=false 1252 | CONFIG_FILE="backup-bench.conf" 1253 | NODEPS=false 1254 | BACKUP_ID_TIMESTAMP=false 1255 | 1256 | for i in "${@}"; do 1257 | case "$i" in 1258 | --config=*) 1259 | CONFIG_FILE="${i##*=}" 1260 | ;; 1261 | --setup-root-access) 1262 | cmd="setup_root_access" 1263 | ;; 1264 | --setup-source) 1265 | cmd="setup_source" 1266 | ;; 1267 | --setup-remote-target) 1268 | cmd="setup_remote_target" 1269 | ;; 1270 | --serve-http-targets) 1271 | cmd="serve_http_targets" 1272 | ;; 1273 | --stop-http-targets) 1274 | cmd="stop_serve_http_targets" 1275 | ;; 1276 | --benchmarks) 1277 | cmd="benchmarks" 1278 | ;; 1279 | --benchmark-backup) 1280 | cmd="benchmark_backup" 1281 | ;; 1282 | --benchmark-restore) 1283 | cmd="benchmark_restore" 1284 | ;; 1285 | --clear-repos) 1286 | cmd="clear_repositories" 1287 | ;; 1288 | --init-repos) 1289 | cmd="init_repositories" 1290 | ;; 1291 | --local) 1292 | REMOTELY=false 1293 | ;; 1294 | --remote) 1295 | REMOTELY=true 1296 | ;; 1297 | --git) 1298 | USE_GIT_VERSIONS=true 1299 | ;; 1300 | --backup-id-timestamp) 1301 | BACKUP_ID_TIMESTAMP=true 1302 | ;; 1303 | --no-deps) 1304 | NODEPS=true 1305 | ;; 1306 | --install-backup-programs) 1307 | cmd="install_backup_programs" 1308 | ;; 1309 | --all) 1310 | ALL=true 1311 | ;; 1312 | --versions) 1313 | cmd="versions" 1314 | ;; 1315 | *) 1316 | usage 1317 | ;; 1318 | esac 1319 | done 1320 | 1321 | # Load configuration file 1322 | Logger "Using configuration file ${CONFIG_FILE}" "NOTICE" 1323 | source "${CONFIG_FILE}" 1324 | 1325 | 1326 | if [ "${ALL}" == true ]; then 1327 | # prepare repos and run all tests locally and remotely 1328 | clear_repositories 1329 | init_repositories false true 1330 | benchmarks false true 1331 | clear_repositories true 1332 | init_repositories true rtue 1333 | benchmarks true true $BACKUP_ID_TIMESTAMP 1334 | else 1335 | full_cmd="$cmd $REMOTELY $USE_GIT_VERSIONS $BACKUP_ID_TIMESTAMP" 1336 | Logger "Running: ${full_cmd}" "DEBUG" 1337 | eval "$full_cmd" 1338 | fi 1339 | 1340 | CleanUp 1341 | --------------------------------------------------------------------------------