├── LICENSE ├── README.md └── proxborg /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 3-Clause License 2 | 3 | Copyright (c) 2020, Przemyslaw Kwiatkowski 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | 1. Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | 2. Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | 3. Neither the name of the copyright holder nor the names of its 17 | contributors may be used to endorse or promote products derived from 18 | this software without specific prior written permission. 19 | 20 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | NOTE: With release of Proxmox Backup Server this project became obsolete and is **not maintained** anymore. 2 | You may still use it, but I suggest PBS instead! PBS is superior in each field. :-) 3 | 4 | # PREREQUISITES 5 | 6 | This script should be run as root. 7 | 8 | This script requires jq and acl packages. Install them: 9 | 10 | apt install jq acl 11 | 12 | This script requires fuse-overlayfs. Version at least 1.0 is *recommended*. 13 | Unfortunately at the time of writing the newest version provided by Proxmox 14 | (or actually by Debian Buster) is 0.3. You may try it (apt install fuse-overlayfs), 15 | but better get newer package from Debian Bullseye. You may do it more or less 16 | this way: 17 | 18 | echo -e "# next release (important: remember to set priority to -1)\ndeb http://ftp.pl.debian.org/debian bullseye main" >/etc/apt/sources.list.d/debian-bullseye.list 19 | echo -e "Package: *\nPin: release n=bullseye\nPin-Priority: -1" >/etc/apt/preferences.d/00bullseye 20 | echo -e "Package: fuse-overlayfs\nPin: release n=bullseye\nPin-Priority: 700" >/etc/apt/preferences.d/99fuse-overlayfs 21 | apt update 22 | apt install fuse-overlayfs 23 | 24 | 25 | # GENERAL INFO 26 | 27 | This script automates backup of Proxmox VMs (both qemu and lxc) to a borg repository. 28 | 29 | Storing backups in a borg repo is an excellent solution, because of its deduplication and 30 | compression capabilities. Try it! :-) 31 | 32 | VERY IMPORTANT: 33 | THIS SCRIPT WORKS WITH *ZFS* ONLY! 34 | 35 | Proxmox server must be configured to use ZFS. Proxborg will not work with another storage. 36 | Proxmox server must be configured to use ZFS. Proxborg will not work with another storage. 37 | 38 | 39 | (Yes, that was twice.) 40 | 41 | 42 | # CONFIGURATION 43 | 44 | Edit the script and fill in available variables, or provide them in environment. 45 | At least you must set: 46 | 47 | export BORG_REPO='ssh://where/is/repo' 48 | 49 | Possibly you will need also other borg configuration variables, like 50 | BORG_PASSPHRASE, BORG_KEY_FILE, BORG_BASE_DIR or BORG_FILES_CACHE_TTL. 51 | See there: https://borgbackup.readthedocs.io/en/stable/usage/general.html 52 | 53 | 54 | # BASICS 55 | 56 | There are 2 basic modes of operation implemented: 57 | 1. You may simply store normal VM images into a borg repo. They are created by 58 | the standard backup tool (vzdump) and thus are 100% compatible with Proxmox. 59 | In order to restore them you simply extract the image from the repo and 60 | provide it to Proxmox via gui or cli. 61 | This is the only way to store backups of qemu VMs, but works also with lxc. 62 | In this method the deduplication works, but is not perfect, because the images 63 | consist of unstructured data (at least they look so from borg point of view). 64 | 2. The better way: You may dump the whole lxc directory tree into borg repository. 65 | Deduplication with this method is superior, because borg can track every 66 | single file from the container filesystem. 67 | Restoring the container is almost as easy as restoring a normal Proxmox backup 68 | (see below). 69 | 3. Extra: You may also archive any mounted zfs dataset to a borg repo. 70 | For example you may want to backup the hypervisor root filesystem 71 | or particular mount points of containers. 72 | 73 | 74 | # USAGE: 75 | 76 | proxborg [@]vmid|vmhostname[:[!]] | /local/path [...] 77 | 78 | 79 | 80 | # 81 | # HOWTO? 82 | # 83 | 84 | # How to backup standard vzdump image of a VM? (whatever - qemu or lxc) 85 | 86 | proxborg 100 tank @pool 87 | 88 | Meaning: 89 | - backup vm #100 90 | - backup vm named "tank". 91 | Note: The hostname must be unique! If more than one VM with the given name exist, none will be archived. 92 | - backup all VMs belonging to pool (note: pool names are case sensitive). 93 | 94 | 95 | 96 | # How to restore a qemu image? 97 | 98 | borg extract --stdout repo::archivename | qmrestore - 99 | 100 | Alternatively you may extract the image from the borg repository and restore it 101 | via Proxmox GUI. 102 | 103 | 104 | 105 | # How to restore an lxc image? 106 | 107 | Method 1: 108 | 109 | borg extract --stdout :: | pct restore - --rootfs rpool: [...] 110 | 111 | Note: when a container is restored via a pipe you must set the rootfs size manually. 112 | Also all mountpoints must be set via commandline (--mp) manually. (If not - everything 113 | will be extracted to rootfs.) 114 | 115 | Example: 116 | 117 | borg extract --stdout ::test1-100-2020_01_02-03_04_05 - | pct restore 120 - --rootfs rpool:3 --mp0 rpool:8,mp=/var/www --mp1 rpool:15,mp=/srv/vmail --unprivileged 1 118 | 119 | 120 | Method 2: 121 | 122 | You may mount borg archive and read the image from there. This way mounts will be 123 | restored automatically. 124 | 125 | borg mount :: /mnt/tmp 126 | pct restore /mnt/tmp/imagename.tar --storage rpool --unprivileged 1 127 | umount /mnt/tmp 128 | 129 | 130 | Alternatively, you may extract the image from the borg repository and restore it 131 | via Proxmox GUI. 132 | 133 | 134 | 135 | # 136 | # HERE IS THE MAGIC: 137 | # 138 | 139 | # How to backup lxc container THE BETTER WAY? 140 | 141 | Borg's deduplication is magic. But there is a problem: whenever you create a new 142 | vzdump-compatible image - it is different than the last one. This makes borg's 143 | job harder and reduces its deduplication success rate. That's why it's a good 144 | idea to provide borg with the real filesystem to be archived. This way borg can 145 | use metadata (like file size, ctime) to find more deduplicatable data. 146 | (And this is real magic!) 147 | 148 | This script can automatically dump a container root filesystem and all mount 149 | points directly into borg repository. The end result is 1:1 image of whole 150 | container directory tree. The container config is also archived. 151 | 152 | By the way: Did you know that separate archives in the same borg repository 153 | are all deduplicated together? (I told you - magic!) 154 | Imagine you have 20 containers with identical Debian base systems. Each base 155 | system takes 1GB. You make 20 standard vzdump backups. Each backup takes about 156 | 0.5GB after compression. So all backups take 20x0.5=10GB total. 157 | Now you switch to Borg: instead of keeping separate dumps - you put them 158 | together into one repository. Hocus-pocus-deduplication: The repository with 159 | backups of all 20 containers takes 0.5GB total! 160 | 161 | 162 | Usage: 163 | 164 | proxborg vmid:[!] 165 | ^ note colon here :-) 166 | 167 | Where vmid is the numerical id of the container or its hostname. (Note: Hostname must be 168 | unique! If more than one VM with the given name exist, none will be archived.) 169 | 170 | "!" means: dump all mount points, *including* the ones without "backup=1" flag in config. 171 | 172 | 173 | Example: 174 | 175 | proxborg tank:! 100: @pool: 176 | 177 | Meaning: 178 | Archive the following data: 179 | - container named "tank", including all mountpoints 180 | - contaner 100, excluding mountpoints without "backup=1" flag 181 | (This is how vzdump usually does its job.) 182 | - all VMs belonging to given pool (note: Pool names are case sensitive!) 183 | 184 | Note: Qemu VMs are always archived "the standard way" (as vzdump images), so 185 | "proxborg 100" and "proxborg 100:" mean exactly the same (if VM 100 is a qemu VM). 186 | 187 | 188 | 189 | # How to restore lxc container from borg repository? 190 | 191 | It's really easy: 192 | 193 | borg export-tar :: - | pct restore - --rootfs : --mp[n] :,mp= [--unprivileged 1 ...] 194 | 195 | Example: 196 | 197 | borg export-tar ::test1-100-2020_01_02-03_04_05 - | pct restore 120 - --rootfs rpool:3 --mp0 rpool:8,mp=/mnt/abc --mp1 rpool:8,mp=/mnt/xyz --unprivileged 1 198 | 199 | Make sure to set up mountpoints correctly! Use this command to see the 200 | necessary mountpoints configuration: 201 | 202 | borg extract --stdout :: etc/vzdump/pct.conf 203 | 204 | Read about ACLs below! 205 | 206 | 207 | # What to do when restore fails? 208 | 209 | Well... It shouldn't fail. :-) 210 | Content of "borg export-tar" is very similar to normal effect of vzdump, 211 | so most likely everything will work very well. But if not... 212 | First of all - remember: you have a full image of all files belonging to the 213 | container, as well as a copy of the old config. This is actually everything you 214 | need in order to revive it! 215 | 216 | Read the old config: 217 | 218 | borg extract --stdout :: etc/vzdump/pct.conf 219 | 220 | Using these information - create new identical container and restore files from archive. 221 | More or less something like this: 222 | 223 | cd /rpool/data/path/to/container/rootfs 224 | rm -rf * 225 | borg extract :: 226 | 227 | Consider mountpoints and permissions. Restoring as privileged container should be easier. 228 | Good luck! :-) 229 | 230 | # What about ACLs? 231 | 232 | THIS IS IMPORTANT if you have any mountpoint with ACLs enabled. 233 | You must read and UNDERSTAND this: 234 | 235 | The ACLs are correctly stored inside Borg repository (this is good). 236 | The "borg export-tar" command does not extract ACLs (this is bad). 237 | 238 | So in order to restore the ACLs you must manually extract the archive 239 | (or at least the affected subfolders) using "borg extract". 240 | 241 | I suggest you do something like this: 242 | 243 | borg export-tar --exclude path/with/acls/ :: - | pct restore [options as described above] 244 | 245 | Now create missing mountpoints (for example via Proxmox GUI) and extract missing subfolders from archive. 246 | Let's assume folder /srv/sambashare is to be restored. Do something like this: 247 | 248 | cd /tmp 249 | mkdir temporary 250 | mount -t tmpfs -o posixacl tmpfs temporary 251 | cd temporary 252 | mkdir sambashare 253 | # Note: the above folder name ("sambashare") MUST match the lowest-level folder name to be restored. 254 | mount --bind /rpool/data/path/to/container/mountpoint/with/acls/ sambashare 255 | borg extract --strip-components 1 :: srv/sambashare 256 | # ^ no slash here! 257 | # ^ here the depth of "sambashare" (number of slashes in "srv/sambashare") 258 | umount sambashare 259 | cd .. 260 | umount temporary 261 | cd .. 262 | rm -rf temporary 263 | 264 | The above procedure is so complicated in order to restore correctly the 265 | ACLs of mountpoint root. If it is not important you may simplify it to: 266 | 267 | cd /rpool/data/path/to/container/mountpoint/with/acls/ 268 | borg extract --strip-components 2 :: srv/sambashare/ 269 | # Note slash: ^ ^ 270 | 271 | 272 | If the container is unprivileged - the uid/gid shift must be considered. 273 | You may use for example shiftfs or fuse-overlayfs. It might be easier 274 | to simply restore the container as privileged, then backup via Proxmox GUI and 275 | restore again as unprivileged. :-) 276 | 277 | 278 | 279 | # Extra: How to backup an arbitrary zfs dataset? 280 | 281 | proxborg /path/to/dataset/you/want/to/backup 282 | 283 | This way you may backup for example root filesystem of the hypervisor or a particular 284 | mount point of a container. 285 | 286 | Important: Please note that file permissions are stored as-seen-by-hypervisor. 287 | It does not matter for privileged containers, but is important for mountpoints 288 | of unprivileged ones. 289 | 290 | Even more important: Backup is NOT recursive. Only ONE dataset will be dumped. 291 | If you want to archive children datasets too - archive them separately! 292 | 293 | 294 | 295 | # Q&A: 296 | 297 | - Why anyone would use Borg to store backups? 298 | 299 | Because of deduplication. :-) 300 | 301 | - Is this script safe? 302 | 303 | Well... It should be. But do your own tests. 304 | Remember: any untested backup solution is worthless! 305 | 306 | - Is it as safe as standard Proxmox backup tools? 307 | 308 | Of course it's not... but should work. Actually, the result of "borg export-tar" 309 | is equivalent to vzdump. You should notice no difference as long as you use stdin 310 | as restore source (as described above). 311 | 312 | - Will it work with non-ZFS storage? 313 | 314 | NO! It was designed to backup data stored on ZFS. It will also not work if you have mixed 315 | zfs and non-zfs storage in one container. 316 | "The standard way" (full VM images) might still work, but it has never been tested. 317 | 318 | 319 | 320 | # Hints: 321 | 322 | [Compression] 323 | 324 | Do first backup with high compression level (like "auto,lzma" or "auto,zstd,9"). 325 | It will take plenty of time (especially lzma is very slow), but thanks to deduplication 326 | never-changing data will stay in your repository forever with higher compression. 327 | Important: If you make first archives with lower compression and later want to 328 | switch to higher - create new repository! Otherwise the low-compressed data will stay 329 | in the repo forever. It's a feature of the deduplication: it writes only new data 330 | to repository, and compression is done on writes. Whatever has already been 331 | written stays in repository as-is. 332 | 333 | [Qemu images] 334 | 335 | It is generally a good idea to fill virtual disks with zeros from time to time. 336 | This will effect in smaller images, as zeros are compressible very well. :-) 337 | In Windows guests you may use sdelete.exe -z. 338 | In Linux consider using sfill. 339 | 340 | [Borg cache] 341 | 342 | In order to speed up backup process set BORG_FILES_CACHE_TTL accordingly. 343 | Generally it is good idea to set it to at least 2-3 times more than the number 344 | of VMs you are going store in repository. 345 | Read this: https://borgbackup.readthedocs.io/en/stable/faq.html#it-always-chunks-all-my-files-even-unchanged-ones 346 | 347 | [ACLs] 348 | 349 | Consider setting "backup=0" on all mountpoints with ACLs enabled and store 350 | these mountpoints in separate archives (might be in the same repository). 351 | This will make your life easier on restore. 352 | Note that separate backups of particular mountpoints will be done on different 353 | times, so your backup will not be "atomic". Depending on circumstances this 354 | might be bad or not-so-bad. :-) 355 | -------------------------------------------------------------------------------- /proxborg: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # proxborg 1.0.1a 3 | # Copyright (c) 2020, Przemyslaw Kwiatkowski 4 | # All rights reserved. 5 | # 6 | # Redistribution and use in source and binary forms, with or without 7 | # modification, are permitted provided that the following conditions are met: 8 | # 9 | # 1. Redistributions of source code must retain the above copyright notice, this 10 | # list of conditions and the following disclaimer. 11 | # 12 | # 2. Redistributions in binary form must reproduce the above copyright notice, 13 | # this list of conditions and the following disclaimer in the documentation 14 | # and/or other materials provided with the distribution. 15 | # 16 | # 3. Neither the name of the copyright holder nor the names of its 17 | # contributors may be used to endorse or promote products derived from 18 | # this software without specific prior written permission. 19 | # 20 | # THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 21 | # AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 22 | # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 23 | # DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 24 | # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 25 | # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 26 | # SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 27 | # CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 28 | # OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 29 | # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 30 | # 31 | 32 | ## CONFIGURATION 33 | 34 | # Edit here or provide values in environment. 35 | #export BORG_REPO='ssh://where/is/repo' 36 | #export BORG_PASSPHRASE=pass1234 37 | 38 | # You may want to add another borg configuration variables, for example 39 | # BORG_KEY_FILE, BORG_BASE_DIR or BORG_FILES_CACHE_TTL. 40 | # Remember to *export* them, as shown above! 41 | # See more: https://borgbackup.readthedocs.io/en/stable/usage/general.html 42 | 43 | 44 | # Exta borg options. This is good place to set compression level. 45 | # Consider adding options like --exclude-caches, --exclude-nodump 46 | # or --remote-ratelimit. 47 | # Add --progress if you want to see more info during backup process. 48 | # Add --list for list of all backed up files. 49 | # See more: https://borgbackup.readthedocs.io/en/stable/usage/create.html 50 | #BORG_OPTS="--compression auto,zstd" 51 | #BORG_OPTS="--compression auto,zstd,9" 52 | #BORG_OPTS="--progress --compression auto,lzma" 53 | #BORG_OPTS="--compression=lz4 --list --remote-ratelimit 9000" 54 | #BORG_OPTS="--compression=auto,lzma --exclude-caches --list" 55 | #BORG_OPTS="--compression=lz4 --exclude-caches --exclude-nodump --progress" 56 | #BORG_OPTS="--compression=lz4 --exclude-caches" 57 | #BORG_OPTS="--compression=lz4 --list --progress" 58 | 59 | # Set if you want to prune old archives automatically after successful backup. 60 | #PRUNE_OPTS="--keep-last=2 --keep-daily=7 --keep-weekly=6" 61 | #PRUNE_OPTS="--keep-last=2" 62 | 63 | # Set to "1" to enable pruning also when backup finished with warnings. 64 | #PRUNE_ON_WARNINGS=1 65 | 66 | # Stop processing if error occured. 67 | # If unset - try to go ahead with remaining tasks. 68 | #EXIT_ON_ERRORS=1 69 | 70 | # Set to "1" to include vm hostname in name of borg archive. 71 | #HOSTNAME_IN_ARCHIVENAME=1 72 | 73 | # Show borg repository information at the end. 74 | #SHOW_SUMMARY=1 75 | 76 | # Snaphot name to be used (usually no need to set this). 77 | #SNAPNAME="proxborg" 78 | 79 | # Extra vzdump options. 80 | # If you have custom settings in /etc/vzdump.conf - it is good place to supersede them. 81 | # Remove --quiet if you want to see vzdump messages. 82 | #VZDUMP_OPTS="--quiet --mailnotification failure --mailto root@localhost --script /bin/true" 83 | VZDUMP_OPTS="--quiet" 84 | 85 | # Sleep n seconds before destroying vm snapshot. (Might help to release filesystem locks.) 86 | #SLEEP_BEFORE_DELETE=10 87 | 88 | 89 | ## USER'S MANUAL 90 | # 91 | # Read more here: 92 | # https://github.com/michabbs/proxborg/blob/master/README.md 93 | 94 | # 95 | # Do not edit beyond this point. :-) 96 | 97 | echo "proxborg - Proxmox with BorgBackup made easy" >&2 98 | 99 | NECESSARY_EXECUTABLES="borg cat cut dd date echo getfacl grep head hostname jq pct printf pvesh pvesm qm realpath sed setfacl sleep sort tr vzdump xargs zfs" 100 | if ! which $NECESSARY_EXECUTABLES >/dev/null 2>&1; then 101 | echo "Missing binaries!" >&2 102 | echo "Make sure these tools are available: $NECESSARY_EXECUTABLES" >&2 103 | exit 255 104 | fi 105 | 106 | function help { 107 | cat >&2 < get vm image from vzdump 121 | : dump lxc container as full directory tree (good idea!) 122 | :! include also mountpoints without backup=1 flag 123 | 124 | localpath path relative to hypervisor filesystem 125 | EOF 126 | exit 255 127 | } 128 | 129 | function update_status() { 130 | [ $EXIT_STATUS -lt $1 ] && EXIT_STATUS=$1 131 | return $1 132 | } 133 | 134 | 135 | function set_snapsuffix() { 136 | # SNAP_SUFFIX=`date +%Y-%m-%dT%H:%M:%S` 137 | SNAP_SUFFIX=`date +%Y_%m_%d-%H_%M_%S` 138 | } 139 | 140 | 141 | function pipe_if_not_empty () { 142 | head=$(dd bs=1 count=1 2>/dev/null; echo a) 143 | head=${head%a} 144 | if [ "x$head" != x"" ]; then 145 | { printf %s "$head"; cat; } | "$@" 146 | fi 147 | } 148 | 149 | 150 | function do_backup() { 151 | local NAME 152 | local NAME_SUFFIX 153 | 154 | if ( echo "$PROCESSED_VMIDS"|grep -Fwq "$VMID" ); then 155 | echo "$VMID: vm already dumped, skipping" >&2 156 | return 0 157 | fi 158 | PROCESSED_VMIDS="$PROCESSED_VMIDS $VMID" 159 | 160 | if [ -n "$VMHOSTNAME" -a "$HOSTNAME_IN_ARCHIVENAME" = "1" ]; then NAME="$VMHOSTNAME-"; else NAME="vzdump-"; fi 161 | NAME="$NAME$VMTYPE-$VMID-" 162 | if [ "$VMTYPE" = "lxc" ]; then NAME_SUFFIX=".tar" 163 | else NAME_SUFFIX=".vma" 164 | fi 165 | echo -n "$VMID: dumping image - calling vzdump+borg [${NAME}${SNAP_SUFFIX}${NAME_SUFFIX}]... " >&2 166 | set -o pipefail 167 | vzdump $VMID --compress 0 --mode snapshot --dumpdir /tmp --stdout $VZDUMP_OPTS | pipe_if_not_empty borg create --stdin-name "${NAME}${SNAP_SUFFIX}${NAME_SUFFIX}" $BORG_OPTS "::${NAME}${SNAP_SUFFIX}${NAME_SUFFIX}" - 168 | if [ "$?" -eq "0" ]; then 169 | echo "done" 170 | PRUNE_LIST+=("$NAME") 171 | return 0 172 | fi 173 | echo "failed (ERROR)" >&2 174 | return 2 175 | } 176 | 177 | 178 | function backup_dataset() { 179 | # $1 = absolute path to mounted dataset without trailing slash 180 | local FSNAME 181 | local NAME= 182 | local ABORT=0 183 | 184 | echo -n "snapshotting $1... " >&2 185 | FSNAME=`zfs list -t filesystem -H|grep -P "\t$1\$"|cut -f1 -d$'\t'` 186 | if [ -z "$FSNAME" ]; then 187 | echo "dataset not found (ERROR)" >&2 188 | return 2 189 | fi 190 | if ( zfs snapshot "$FSNAME@$SNAPNAME" ); then 191 | echo "created $FSNAME@$SNAPNAME" >&2 192 | else 193 | echo "failed (ERROR)" >&2 194 | return 2 195 | fi 196 | 197 | if [ ! -d "$1/.zfs/snapshot/$SNAPNAME" ]; then 198 | echo "$1 unreadable (ERROR)" >&2 199 | ABORT=2 200 | # do not exit yet, snapshot still must be destroyed 201 | fi 202 | 203 | if [ "$ABORT" = "0" ]; then 204 | [ "$HOSTNAME_IN_ARCHIVENAME" = 1 ] && NAME=$MYHOSTNAME || NAME="dataset" 205 | NAME="$NAME`echo "$1"|sed -s 's/[^[:alnum:]-]/-/g'`--" 206 | # ^^ delimiter (make sure not to prune subfolders) 207 | echo -n "dumping dataset $1 - calling borg [${NAME}${SNAP_SUFFIX}]... " >&2 208 | ( 209 | cd "$1/.zfs/snapshot/$SNAPNAME" 210 | borg create --one-file-system $BORG_OPTS "::${NAME}${SNAP_SUFFIX}" . 211 | ) 212 | if [ "$?" -eq "0" ]; then 213 | echo "done" >&2 214 | PRUNE_LIST+=("$NAME") 215 | else 216 | echo "finished unclean (ERROR)" >&2 217 | ABORT=2 218 | fi 219 | fi 220 | 221 | echo -n "destroying snapshot $FSNAME@$SNAPNAME... " >&2 222 | if ( zfs destroy "$FSNAME@$SNAPNAME" ); then 223 | echo "done" >&2 224 | return $ABORT 225 | else 226 | echo "failed (WARNING)" >&2 227 | [ "$ABORT" -ge 1 ] && return $ABORT 228 | return 1 229 | fi 230 | } 231 | 232 | 233 | function find_vm() { 234 | local A 235 | local B 236 | VMHOSTNAME="" 237 | VMID="" 238 | VMTYPE="" 239 | VMCONFIG="" 240 | 241 | if [[ "${1}" =~ ^[0-9]+$ ]]; then 242 | # by numeric id 243 | # A=`pct list|grep "^${1} "|sed -e 's/ *$//g'|grep -oE '[^ ]+$'` 244 | # B=`qm list|sed -e 's/^ *//' -e 's/ \+/ /g'|grep "^${1} "|cut -d ' ' -f 2|tr "\n" " "|sed 's/ *$//g'` 245 | A=`pvesh get /nodes/$MYHOSTNAME/lxc --output-format json|jq -j ".[] | select(.vmid==\"$1\") | .name"` 246 | B=`pvesh get /nodes/$MYHOSTNAME/qemu --output-format json|jq -j ".[] | select(.vmid==\"$1\") | .name"` 247 | if [ -z "$A" -a -z "$B" ] ; then 248 | echo "${1}: vm id not found (WARNING)" >&2 249 | return 1 250 | fi 251 | if [ -n "$A" -a -n "$B" ] ; then 252 | # should never happen :-) 253 | echo "${1}: vm id ambigous - skipping (WARNING)" >&2 254 | return 1 255 | fi 256 | VMID="${1}" 257 | if [ -n "$A" ]; then 258 | VMHOSTNAME="$A" 259 | VMTYPE="lxc" 260 | else 261 | VMHOSTNAME="$B" 262 | VMTYPE="qemu" 263 | fi 264 | else 265 | # by hostname 266 | # A=`pct list|grep " ${1} *$"|cut -f 1 -d ' '|tr "\n" " "|sed -e 's/ $//'` 267 | # B=`qm list|sed -e 's/^ *//' -e 's/ \+/ /g'|cut -d ' ' -f 1,2|grep " ${1}$"|cut -d ' ' -f 1|tr "\n" " "|sed -e 's/ $//'` 268 | A=`pvesh get /nodes/$MYHOSTNAME/lxc --output-format json|jq -r ".[] | select(.name==\"$1\") | .vmid"|tr "\n" " "|sed -e 's/ $//'` 269 | B=`pvesh get /nodes/$MYHOSTNAME/qemu --output-format json|jq -r ".[] | select(.name==\"$1\") | .vmid"|tr "\n" " "|sed -e 's/ $//'` 270 | if [ -n "$A" -a -n "$B" ]; then 271 | echo "${1}: vm name ambigous - found ids: $A $B - skipping (WARNING)" >&2 272 | return 1 273 | fi 274 | if [ -z "$A" -a -z "$B" ]; then 275 | echo "${1}: vm name not found (WARNING)" >&2 276 | return 1 277 | fi 278 | # lxc? 279 | case "`echo $A|wc -w`" in 280 | "0") # qemu? 281 | ;; 282 | "1") VMHOSTNAME="${1}" 283 | VMTYPE="lxc" 284 | VMID="$A" 285 | echo "$VMHOSTNAME: id=$VMID" >&2 286 | return 0;; 287 | *) echo "${1}: vm name ambigous - found ids: $A - skipping (WARNING)" >&2 288 | return 1;; 289 | esac 290 | # qemu? 291 | case "`echo $B|wc -w`" in 292 | "0") # should never happen :-) 293 | echo "${1}: vm name not found (WARNING)" >&2 294 | return 1;; 295 | "1") VMHOSTNAME="${1}" 296 | VMTYPE="qemu" 297 | VMID="$B" 298 | echo "$VMHOSTNAME: id=$VMID" >&2 299 | return 0;; 300 | *) echo "${1}: vm name ambigous - found ids: $B - skipping (WARNING)" >&2 301 | return 1;; 302 | esac 303 | fi 304 | } 305 | 306 | 307 | function vm_snapshot() { 308 | echo -n "$VMID: snapshoting... " >&2 309 | case "$VMTYPE" in 310 | "lxc") if ( pct snapshot $VMID $SNAPNAME --description "borgprox snapshot $SNAP_SUFFIX" >/dev/null ); then 311 | echo "done" >&2 312 | return 0 313 | fi;; 314 | "qemu") if ( qm snapshot $VMID $SNAPNAME --description "borgprox snapshot $SNAP_SUFFIX" >/dev/null ); then 315 | echo "done" >&2 316 | return 0 317 | fi;; 318 | esac 319 | echo "error" >&2 320 | return 2 321 | } 322 | 323 | 324 | function vm_delsnapshot() { 325 | echo -n "$VMID: deleting snapshot... " >&2 326 | sleep "$SLEEP_BEFORE_DELETE" 327 | case "$VMTYPE" in 328 | "lxc") if ( pct delsnapshot $VMID $SNAPNAME >/dev/null ); then 329 | echo "done" >&2 330 | return 0 331 | fi;; 332 | "qemu") if ( qm delsnapshot $VMID $SNAPNAME >/dev/null ); then 333 | echo "done" >&2 334 | return 0 335 | fi;; 336 | esac 337 | echo "failed" >&2 338 | return 1 339 | } 340 | 341 | 342 | function get_mountpoints() { 343 | [ "$VMTYPE" == "lxc" ] || return 344 | local A 345 | local B 346 | local C 347 | #rootfs: 348 | A=`echo "$VMCONFIG"|grep "^rootfs: "|cut -f2 -d' '|cut -f1 -d,` 349 | if ( pvesm status -enabled -storage `echo $A|cut -f1 -d:` >/dev/null 2>&1 ); then 350 | B=`pvesm path $A` 351 | if [ -d "$B/.zfs/snapshot/$SNAPNAME" ]; then 352 | echo "/ $B 1" 353 | else 354 | echo "$VMID: invalid storage path: $B (skipping)" >&2 355 | fi 356 | else 357 | echo "$VMID: invalid storage: $A (skipping)" >&2 358 | fi 359 | #mountpoints: 360 | for A in `echo "$VMCONFIG"|grep -E '^mp[0-9]+: '|cut -d ' ' -f2`; do 361 | if ( pvesm status -enabled -storage `echo $A|cut -f1 -d:` >/dev/null 2>&1 ); then 362 | C=`echo "$A"|sed -s 's/^.*,mp=//'|cut -f1 -d,` 363 | if [ -z "$C" ]; then 364 | echo "$VMID: invaild mountpoint: $A (skipping)" >&2 365 | continue 366 | fi 367 | B=`echo $A|cut -d, -f1|xargs pvesm path` 368 | if ! [ -d "$B/.zfs/snapshot/$SNAPNAME" ]; then 369 | echo "$VMID: invalid storage path: $B (skipping)" >&2 370 | continue 371 | fi 372 | if echo "$A"|grep ",backup=1" >/dev/null 2>&1; then 373 | echo "$C $B 1" 374 | else 375 | echo "$C $B 0" 376 | fi 377 | else 378 | echo "$VMID: invalid storage: $A (skipping)" >&2 379 | continue 380 | fi 381 | done 382 | } 383 | 384 | 385 | function do_lxc_borgbackup() { 386 | local A 387 | local B 388 | local C 389 | local D 390 | local TMP_DIR 391 | local FUSE_MOUNTS= 392 | local FUSE_OPTS= 393 | local ABORT=0 394 | local NAME= 395 | local UPPER_DIR 396 | local WORK_DIR 397 | local VM_UNPRIVILEGED=0 398 | 399 | if ( echo "$PROCESSED_VMIDSB"|grep -Fwq "$VMID" ); then 400 | echo "$VMID: vm already dumped, skipping" >&2 401 | return 0 402 | fi 403 | if [ "$VMTYPE" != "lxc" ]; then 404 | echo "$VMID: not lxc container - skipping (WARNING)" >&2 405 | update_status 1 406 | return 1 407 | fi 408 | PROCESSED_VMIDSB="$PROCESSED_VMIDSB $VMID" 409 | 410 | TMP_DIR=$(mktemp -d -t proxborg-XXXXXXXXXX) 411 | if [ ! -d "$TMP_DIR" ]; then 412 | echo "$VMID: unable to create tmpdir. (ERROR)" >&2 413 | update_status 2 414 | return 2 415 | fi 416 | mount -t tmpfs -o posixacl tmpfs "$TMP_DIR" 417 | mkdir -p "$TMP_DIR/root" 418 | 419 | vm_snapshot || update_status $? || return $? 420 | VMCONFIG=`pct config $VMID --snapshot $SNAPNAME` 421 | VM_MOUNTPOINTS=`get_mountpoints| LC_ALL=C sort` 422 | 423 | if (echo "$VMCONFIG"|grep -q '^[[:space:]]*unprivileged:[[:space:]][[:space:]]*1[[:space:]]*$' ); then 424 | VM_UNPRIVILEGED=1 425 | FUSE_OPTS=",uidmapping=100000:0:65536,gidmapping=100000:0:65536" 426 | fi 427 | 428 | while read A B C; do 429 | [ "$C" = "1" ] || [ "$DUMP_ALL" = "1" ] || continue 430 | echo -n "$VMID: mounting $A... " >&2 431 | mkdir -p "$TMP_DIR/root/$A" 432 | UPPER_DIR=`mktemp -d -p $TMP_DIR` 433 | chown --reference="$B/.zfs/snapshot/$SNAPNAME/." $UPPER_DIR 434 | chmod --reference="$B/.zfs/snapshot/$SNAPNAME/." $UPPER_DIR 435 | getfacl "$B/.zfs/snapshot/$SNAPNAME/." 2>/dev/null| setfacl --set-file=- $UPPER_DIR 436 | WORK_DIR=`mktemp -d -p $TMP_DIR` 437 | if ( ! fuse-overlayfs -o "lowerdir=$B/.zfs/snapshot/$SNAPNAME,upperdir=$UPPER_DIR,workdir=$WORK_DIR$FUSE_OPTS" "$TMP_DIR/root/$A" 2>/dev/null ); then 438 | echo "failed" >&2 439 | update_status 2 440 | ABORT=1 441 | break 442 | fi 443 | FUSE_MOUNTS="$A $FUSE_MOUNTS" 444 | echo "done" >&2 445 | done < <(echo "$VM_MOUNTPOINTS") 446 | 447 | if [ "$ABORTED" != "1" -a ! -d "$TMP_DIR/root/etc/vzdump" ]; then 448 | mkdir -p "$TMP_DIR/root/etc/vzdump" 449 | # chown 65534:65534 "$TMP_DIR/root/etc/vzdump" 450 | fi 451 | 452 | if [ "$ABORTED" != "1" ]; then 453 | echo "$VMCONFIG"|grep -Ev '^(parent|snaptime|description): ' >"$TMP_DIR/root/etc/vzdump/pct.conf" 454 | chown 65534:65534 "$TMP_DIR/root/etc/vzdump/pct.conf" 455 | if [ -e "/etc/pve/firewall/$VMID.fw" ]; then 456 | cat "/etc/pve/firewall/$VMID.fw" >"$TMP_DIR/root/etc/vzdump/pct.fw" 457 | chown 65534:65534 "$TMP_DIR/root/etc/vzdump/pct.fw" 458 | fi 459 | fi 460 | 461 | if [ -n "$VMHOSTNAME" -a "$HOSTNAME_IN_ARCHIVENAME" = "1" ]; then NAME="$VMHOSTNAME-"; else NAME="dump-"; fi 462 | NAME="$NAME$VMID-" 463 | 464 | ( 465 | cd "$TMP_DIR/root" 466 | echo -n "$VMID: dumping directory tree - calling borg [${NAME}${SNAP_SUFFIX}]... " >&2 467 | borg create --numeric-owner $BORG_OPTS "::${NAME}${SNAP_SUFFIX}" ./ --exclude 'lost+found' --exclude './tmp/?*' --exclude './var/tmp/?*' --exclude './var/run/?*.pid' 468 | ) 469 | 470 | if [ "$?" -eq "0" ]; then 471 | echo "done" >&2 472 | PRUNE_LIST+=("$NAME") 473 | else 474 | echo "failed" >&2 475 | update_status 2 476 | ABORTED=1 477 | fi 478 | 479 | if [ -n "$FUSE_MOUNTS" ]; then 480 | echo -n "$VMID: unmounting... " >&2 481 | for A in $FUSE_MOUNTS; do 482 | umount "$TMP_DIR/root/$A" 483 | done 484 | echo "done" >&2 485 | fi 486 | 487 | VM_MOUNTPOINTS= 488 | vm_delsnapshot || update_status $? 489 | 490 | umount "$TMP_DIR" 491 | rm -rf "$TMP_DIR" 492 | return 0 493 | } 494 | 495 | 496 | function process_job() { 497 | local B 498 | local C 499 | local D 500 | local SFX 501 | local MOUNTPOINT 502 | [ "$EXIT_ON_ERRORS" = "1" -a "$EXIT_STATUS" -ge 2 ] && return 503 | 504 | set_snapsuffix 505 | if ( echo "$1"|grep -q "^@" ); then 506 | # backup pool 507 | B=`echo "$1"|sed -e 's/^@//'|cut -d: -f1` 508 | C=`pvesh get "/pools/$B" --output-format json 2>/dev/null | jq '.members[].vmid'` 509 | if [ -z "$C" ]; then 510 | echo "Pool '$B' does not exist (case sensitive!). (WARNING)" >&2 511 | update_status 1 512 | return 1 513 | fi 514 | echo "Processing pool $B..." >&2 515 | if ( echo "$1"|grep -q ":!$" ); then SFX=":!" 516 | elif ( echo "$1"|grep -q ":$" ); then SFX=":" 517 | else SFX="" 518 | fi 519 | for D in $C; do 520 | process_job "$D$SFX" 521 | done 522 | 523 | elif ( echo "$1"|grep -q ":" ); then 524 | find_vm `echo "$1"|cut -d: -f 1` || update_status $? || return $? 525 | if [ "$VMTYPE" = "lxc" ]; then 526 | # backup directory tree 527 | (echo "$1"|grep -q '!$') && DUMP_ALL=1 || DUMP_ALL=0 528 | do_lxc_borgbackup 529 | else 530 | # backup full vm 531 | do_backup || update_status $? || return $? 532 | fi 533 | 534 | elif ( echo "$1"|grep -q "/" ); then 535 | # backup local dataset 536 | MOUNTPOINT=`realpath -e "$1" 2>/dev/null` 537 | if [ -z "$MOUNTPOINT" ]; then 538 | echo "path not found: $1 (WARNING)" >&2 539 | update_status 1 540 | return 1 541 | fi 542 | if ( echo "$PROCESSED_MOUNTPOINTS"|grep -Fwq "$MOUNTPOINT" ); then 543 | echo "$MOUNTPOINT: already processed, skipping" >&2 544 | return 545 | fi 546 | PROCESSED_MOUNTPOINTS="$PROCESSED_MOUNTPOINTS $MOUNTPOINT" 547 | backup_dataset "$MOUNTPOINT" || update_status $? || return $? 548 | 549 | else 550 | # backup full vm 551 | find_vm "$1" || update_status $? || return $? 552 | do_backup || update_status $? || return $? 553 | fi 554 | } 555 | 556 | # 557 | # 558 | 559 | [ "$#" -eq 0 ] && help 560 | [ "$EXIT_ON_ERRORS" != "1" ] && EXIT_ON_ERRORS=0 561 | [ "$PRUNE_ON_WARNINGS" != "1" ] && PRUNE_ON_WARNINGS=0 562 | [ -z "$SNAPNAME" ] && SNAPNAME=`date +pb%s` 563 | [ -z "$SLEEP_BEFORE_DELETE" ] && SLEEP_BEFORE_DELETE=0 564 | MYHOSTNAME=`hostname` 565 | EXIT_STATUS=0 566 | PROCESSED_VMIDS= 567 | PROCESSED_VMIDSB= 568 | PROCESSED_MOUNTPOINTS= 569 | PRUNE_LIST=() 570 | 571 | for A in "$@"; do 572 | if [ "$EXIT_ON_ERRORS" = "1" -a "$EXIT_STATUS" -ge 2 ]; then 573 | echo "ABORTED" >&2 574 | break 575 | fi 576 | process_job `echo "$A"|head -n 1|cut -f 1 -d " "` 577 | done 578 | 579 | [ ${#PRUNE_LIST[*]} -eq 0 ] || [ -z "$PRUNE_OPTS" ] || [ "$EXIT_STATUS" -gt 1 ] || [ "$EXIT_STATUS" -eq 1 -a "$PRUNE_ON_WARNINGS" != "1" ] || { 580 | echo "Pruning old archives... " >&2 581 | for A in ${PRUNE_LIST[@]}; do 582 | borg prune --prefix "$A" $PRUNE_OPTS 583 | done 584 | } 585 | 586 | [ "$SHOW_SUMMARY" = "1" ] && borg info 587 | 588 | if [ "$EXIT_STATUS" -ge 2 ]; then echo "Finished with errors!" >&2 589 | elif [ "$EXIT_STATUS" = "1" ]; then echo "Finished with warnings." >&2 590 | else echo "Finished successfully." >&2 591 | fi 592 | exit $EXIT_STATUS 593 | --------------------------------------------------------------------------------