├── FreeBSD-armv6-10.2-RPI-B-ZFS-295681M.img.xz ├── README.md ├── backup.sh ├── keys ├── knox-ca ├── knox-login ├── knox-login-cert.pub └── knox-login.pub ├── patches ├── RPI-B-ZFS ├── RPI-B.patch ├── cpuvar.h.patch └── setup.sh.patch ├── pkg ├── autoconf-2.69.txz ├── autoconf-wrapper-20131203.txz ├── automake-1.15_1.txz ├── automake-wrapper-20131203.txz ├── ca_root_nss-3.22.txz ├── curl-7.46.0_1.txz ├── dejagnu-1.5.1_1.txz ├── expat-2.1.0_3.txz ├── expect-5.45_2.txz ├── fusefs-libs-2.9.5.txz ├── fusefs-sshfs-2.5.txz ├── gettext-runtime-0.19.6.txz ├── gettext-tools-0.19.6.txz ├── glib-2.44.1_3.txz ├── gmake-4.1_2.txz ├── gmake-lite-4.1_1.txz ├── help2man-1.43.3_1.txz ├── indexinfo-0.2.4.txz ├── libevent2-2.0.22_1.txz ├── libffi-3.2.1.txz ├── libiconv-1.14_9.txz ├── libtool-2.4.6.txz ├── m4-1.4.17_1,1.txz ├── mosh-1.2.5.txz ├── nmap-7.01.txz ├── p5-Locale-gettext-1.06.txz ├── pcre-8.37_4.txz ├── perl5-5.20.3_8.txz ├── pkg-1.6.3.txz ├── pkg-static ├── pkgconf-0.9.12_1.txz ├── protobuf-2.6.1.txz ├── python27-2.7.11_1.txz ├── python35-3.5.1_1.txz ├── rsync-3.1.2_1.txz ├── tcl86-8.6.4.txz ├── tmux-2.1_1.txz ├── tor-0.2.6.10_1.txz └── vim-lite-7.4.1030.txz └── top /FreeBSD-armv6-10.2-RPI-B-ZFS-295681M.img.xz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/FreeBSD-armv6-10.2-RPI-B-ZFS-295681M.img.xz -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | ZFS Remote Mirrors for Home Use 2 | =============================== 3 | **Update: Now with pre-built ZFS Raspberry Pi image! Jump to the Appendix for more information. In summary: Flash the image, change the keys, send your snapshots.** 4 | 5 | **ECC Memory Note: The requirement for ECC memory with ZFS is a little contentious, it's not needed for this use but see the second Appendix for more information.** 6 | 7 | Why pay a nebulous cloud provider to store copies of our boring, but nice to keep data? Old photographs, home videos, college papers, MP3s from Napster; we typically stick them somewhere and hope the storage doesn't rot. 8 | 9 | But we can do better. Magnetic storage is cheap; and our data is valuable. We don't need live synchronisation, cloud scaling, SLAs, NSAs, terms of service, lock-ins, buy-outs, up-sells, shut-downs, DoSs, fail whales, pay-us-or-we'll-deletes, or any of the noise that comes with using someone else's infrastructure. We'd just like a big drive that we can backup to, reliably, easily, and privately. 10 | 11 | How about an automatic, remote, encrypted, verifiable, incremental backup of all your data, for about 100 currency units in outlay, less if you have existing hardware, and no upkeep costs? How? 12 | 13 | * Computer hardware is cheap, an old laptop or a [SBC](https://en.wikipedia.org/wiki/Single-board_computer) is more than sufficient. 14 | * USB external storage is [laughably inexpensive](http://www.amazon.co.uk/s/keywords=external+hard+drive&ie=UTF8) ([35USD/TB](https://en.wikipedia.org/wiki/List_of_Storage_hierarchy_media_with_costs)). 15 | * Parents/Friends will allow you to keep the system in their homes. 16 | * Enterprise grade [security](https://en.wikipedia.org/wiki/OpenSSH) and [storage](https://en.wikipedia.org/wiki/ZFS) features are available in [free operating systems](https://en.wikipedia.org/wiki/FreeBSD). 17 | 18 | You can have: 19 | -------------- 20 | * A block level copy of your ZFS pool, replicating all datasets and snapshots. 21 | * Automated incremental backups so that only changed data is transferred. 22 | * Complete at-rest encryption, with the key resident in memory only during backup operations. 23 | * Hardened communication security, so your data is safe in-flight. 24 | * A minimalistic, locked down, low-power remote server requiring almost no maintenance. 25 | * All wrapped up as a single command, powered entirely by tools in the FreeBSD base system. 26 | 27 | You will need: 28 | -------------- 29 | * A FreeBSD 10.2 (or later) [supported](https://www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/archs.html) system, such as: 30 | * A [Raspberry Pi](https://en.wikipedia.org/wiki/Raspberry_Pi), [BeagleBoard](https://en.wikipedia.org/wiki/BeagleBoard) or another [FreeBSD ARM target](https://www.freebsd.org/platforms/arm.html), use a 4GB+ SD Card. 31 | * An old laptop, with around 512MB of memory, preferably something quiet. 32 | * A USB Hard Drive. 33 | * If using an SBC, a drive that comes with its own power supply is useful. 34 | * If using an old laptop, a drive powered directly by USB is probably better. 35 | * An Ethernet Internet connection in a location that is not your home. 36 | * Do not even consider Wi-Fi. Frustration abounds. 37 | * The ability to reach the system from the outside world. Potentially via: 38 | * [Port forwarding](http://portforward.com/) (most likely). 39 | * IPv6. 40 | * Overlay routing ([Tor](https://en.wikipedia.org/wiki/Tor_(anonymity_network)), [I2P](https://en.wikipedia.org/wiki/I2P), [cjdns](https://en.wikipedia.org/wiki/Cjdns)). 41 | 42 | I'm specifying a laptop/SBC as it's a lot more palatable a device to ask someone to let you keep in their home. You can of course use a desktop if you have sufficient charm. 43 | 44 | Please note, that ZFS is designed for serious, enterprise grade environments. Using it on older consumer hardware is doing an injustice to its many features, and the kernel will remind you of this on every boot. That said, it can be squeezed into this restricted use case, and you can have many of the benefits most important to a backup system. However, if you can supply more modern hardware, do. A system with ECC memory would be a wise investment also. Even if it is an older model. 45 | 46 | If you must go with "previously loved", hardware, do yourself a favour and run a memory test first. Faulty DIMMs are a heartbreak one only tolerates once. [MemTest86+](http://www.memtest.org/) is a good choice, but their ISOs don't work on USB drives, only CDs. You could mess about with *gpart* to create a [bootable partition manually](http://forum.canardpc.com/threads/28875-Linux-HOWTO-Boot-Memtest-on-USB-Drive?p=1396798&viewfull=1#post1396798) but the good folks at Canonical added *memtest* to the boot-screen of some Ubuntu installers, which can be dandily *dd*'d to a memory stick. I can attest to [14.04.2 LTS Server](http://releases.ubuntu.com/trusty/) having this. SBC users can try *[memtester](http://pyropus.ca/software/memtester/)*, but as it runs on top of a full kernel it can't test the memory the kernel occupies. 47 | 48 | Why not mirror the drives locally? 49 | ---------------------------------- 50 | **Pros:** 51 | * No remote system needed. 52 | * No network connection needed for backups, therefore much faster. 53 | * ZFS can auto-recover any corruption without missing a beat. 54 | 55 | **Cons:** 56 | * Devices are vulnerable to localised physical damage, e.g. fire, dogs, children. 57 | * Filesystems are susceptible to user stupidity as writes are synchronised live (snapshots help). 58 | 59 | Why not some another way? 60 | ------------------------- 61 | Some alternatives include [rsync](https://en.wikipedia.org/wiki/Rsync), [bup](https://bup.github.io/), [obnam](http://obnam.org/), [git-annex](https://git-annex.branchable.com/), [something else](https://wiki.archlinux.org/index.php/Backup_programs). 62 | 63 | **Pros:** 64 | * Easier to setup. 65 | * OS agnostic. 66 | * *Potentially* local only decryption. 67 | 68 | **Cons:** 69 | * No ZFS features, such as: 70 | * Free snapshots. 71 | * Data integrity verification. 72 | * Block compression. 73 | * Deltas based on actual modified blocks, not file modification timestamps. 74 | 75 | These solutions focus on backing up individual files, which I believe is a less useful system than a full mirror. They are certainly easier to setup though. 76 | 77 | It's worth noting that as we send the decryption key to the remote device, our method is somewhat less secure than a method that never allows the remote system to view plaintext data. If someone has physical access then it's possible for the key to leak through side-channel analysis or system subversion. 78 | 79 | The Solaris version of ZFS allows [per-dataset encryption](http://www.oracle.com/technetwork/articles/servers-storage-admin/solaris-zfs-encryption-2242161.html), which allows the pool to synchronise without exposing the plaintext data. This feature hasn't yet made it to FreeBSD, but for data that needs extra protection, we can approximate it (details further on). 80 | 81 | Threat Model 82 | ------------ 83 | Since nothing is really *secure*, just appropriately difficult, it's good to define what threats we'd like to defend against. This way we can spot when we've gone too far, or not far enough. 84 | 85 | * Data Loss - Made difficult by the existence of the remote backup and regular synchronisation. 86 | * Data Leaks - Made difficult by the at-rest encryption on the remote disk. 87 | * Server Breach (Digital) - Made difficult by judicious use of SSH keys. 88 | * Server Breach (Physical) - Made difficult by the front door lock of the home. 89 | 90 | This list doesn't cover any of the security of our main system of course, only the remote one. Presumably you have some sort of disk encryption in use already. 91 | 92 | The physical risk is the hardest to defend against. An attacker with physical access could potentially modify the system to capture the decryption key. Passwords are required for local logins, so the system would have to be shut down to modify it, unless they can exploit a faulty USB stack or similar alternative entry. 93 | 94 | To defend against physical threats, you could encrypt the OS drive of the system and require a USB key to boot it, that way if it was ever powered down it could not be surreptitiously modified and rebooted. However, any time it crashes, or suffers a power cut, you'd have to boot it manually, or request your kind host to do so. I feel that this level of defence is more trouble than it's worth. See the section *Extra Encryption* for an easier alternative. 95 | 96 | Secret Material 97 | -------------- 98 | Any good crypto system is defined by the secrets it requires. For this setup, we need two: 99 | 100 | * The password for the master SSH signing key trusted by the backup system. 101 | * The encryption key for the hard disk. 102 | 103 | Other secrets like the passwords for the root and the user account are less important. They will also not really be used beyond the initial setup stage, so I recommend choosing one single [strong password](https://xkcd.com/538/) and using it for all three cases. You can generate one like this: 104 | 105 | hugh@local$ echo $(strings /dev/random | sed -E '/^.{6}$/!d;/[[:space:]]|[[:punct:]]/d' | head -n 2 | tr -d '\n') 106 | 107 | brno7aaJKusz 108 | 109 | That reads any strings that emerge from the noise of the PRNG, discards any not six characters long, and any containing whitespace or punctuation. After two such strings have been emitted it puts them together and *echo* ensures there's a newline at the end. We could just look for a single twelve character string matching our needs but that takes a while, those monkeys can only type so fast. 110 | 111 | For future reference, all the commands I show you are written to run on a FreeBSD 10 system. GNU utilities often take different arguments so you might be able to translate with careful reference to the *man* pages. Also, I'll write the names of system components in *italics*. E.g. ZFS is the product, *zfs* is the command to control it. 112 | 113 | Server Setup 114 | ============ 115 | OS Installation 116 | --------------- 117 | Step one, is of course, to install FreeBSD. If this sounds daunting to you, take a look at the excellent [Handbook](https://www.freebsd.org/doc/handbook/). There's different ways to do this based on what platform you're using but here's a run down of some answers to the questions the installer will ask you on an i386/amd64 target. **Do not** connect your USB drive yet. 118 | 119 | * Choose *install* from the *Install / Shell / Live CD* dialogue. 120 | * Choose your desired keymap. (I use *UK ISO-8859-1*) 121 | * Name the machine (*knox* is a good name). 122 | * Deselect **all** optional system components (*doc, games, ports, src*) 123 | * Choose *Auto (UFS)* over the *entire disk*. Defaults are usually fine. 124 | * Set your **strong** root password. 125 | * Set up *IPv4* with *DHCP* unless you know better. 126 | * I don't bother with *IPv6* as Irish ISPs haven't heard of it. 127 | * Your clock is usually UTC, so say *yes*. 128 | * Choose your timezone. 129 | * **Disable all** services on boot. We'll configure them manually. 130 | * Do not add users now. We'll do it later. 131 | * Choose *Exit* from the final menu. 132 | * **YES**, you do want to enter a shell to make final modifications. 133 | 134 | 135 | Congratulations. You are standing in a root shell west of a fresh system, with a boarded network interface. [There is a small mailbox here](http://steel.lcc.gatech.edu/~marleigh/zork/transcript.html), but we'll soon disable it. 136 | 137 | Before we continue, a word about FreeBSD partitions. Old wisdom was to split out */usr*, */var* and sometimes other aspects of the directory hierarchy onto different partitions. This has a few benefits, the most obvious is that some branches, especially */var* tend to grow with system logs and other detritus. If these were allowed to grow unchecked, they might consume the entire disk which brings out edge-case complications. Splitting them off mitigates this, but with the system's specialised usage such growth isn't likely to be a problem, and it's nothing a quick check can't solve. 138 | 139 | There was another argument to be made for performance; different branches have different read/write profiles, and splitting them reduced fragmentation for the read-mostlies. This is still true, but not significant on flash media due to uniform seek times and not really worth the hassle on a dedicated system like this. More than a moment's system tuning is for high-performance servers, and if we were interested in that we wouldn't be using old hardware. Let's keep it simple. 140 | 141 | Finally, swap space used to be best placed at the edge of the platter (the last sectors) as the most sectors pass per rotation there. This reason goes out the window with flash, but it's still a good idea to put the swap at the end, it makes it easier to grow the partitions if we ever migrate to a larger card. 142 | 143 | Thankfully, the FreeBSD installer will choose all of the above tweaks by default if you use guided mode. 144 | 145 | Connecting 146 | ---------- 147 | Now we need to discover what IP address the DHCP server leased to the system. Ignore *lo0* as it's the loopback interface. Also note the interface name, *bfe0* in this case. Linux users may be used to *eth0* style naming, but in FreeBSD the interface is named after the driver it uses, a [Broadcom](https://www.freebsd.org/cgi/man.cgi?query=bfe&sektion=4) one in this case. 148 | 149 | # ifconfig 150 | 151 | bfe0: flags=8843 metric 0 mtu 1500 152 | options=80008 153 | ether 00:15:c5:00:00:00 154 | inet 192.168.1.23 netmask 0xffffff00 broadcast 192.168.1.255 155 | nd6 options=29 156 | media: Ethernet autoselect (100baseTX ) 157 | status: active 158 | 159 | With this in hand we can SSH into the machine; *sshd* will generate some host keys automatically at this point, but before we let that happen, let's shake that odd feeling we both have that there hasn't been enough entropy in this baby-faced system yet to yield secure random numbers. 160 | 161 | # dd if=/dev/random of=/dev/null bs=1m count=512 162 | 163 | Now we haven't really added entropy of course, but at least we're far enough down the PRNG stream to be hard to predict. Use a different count value if you like. 164 | 165 | Start OpenSSH. Note that this is being started in a *chroot* environment, so when we come to *ssh* in, we'll find ourselves within that same environment. 166 | 167 | # service sshd onestart 168 | 169 | Before we connect, we should first setup our **local** SSH configuration. 170 | 171 | hugh@local$ mkdir ~/.ssh 172 | hugh@local$ chmod 700 ~/.ssh 173 | hugh@local$ touch ~/.ssh/config 174 | hugh@local$ chmod 600 ~/.ssh/config 175 | 176 | Edit your **local** *~/.ssh/config* and insert the following: 177 | 178 | HashKnownHosts yes 179 | 180 | Host knox 181 | User root 182 | HostName 192.168.1.13 # We'll swap in the FQDN later 183 | 184 | Now we can simply use: 185 | 186 | hugh@local$ ssh knox 187 | root@knox# 188 | 189 | There'll be a prompt to accept the host key as it's the first time *ssh* has seen it. If you think you are currently the victim of a LAN scale MitM attack you can compare the displayed key to the one on the new system before accepting. You may also want to switch to a low-sodium diet. 190 | 191 | # ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub 192 | 193 | Now that we're in, we can use the copy/paste features of our terminal to make the config file editing a little more palatable. 194 | 195 | /etc/rc.conf 196 | ------------ 197 | 198 | We'll be editing several config files so I hope you know your *vi* key-bindings. If not, there's *ee*, which feels like a tricycle after *vi*, but I digress. **Replace** */etc/rc.conf* with the following. You'll have to supply your own network adapter name gleaned from *ifconfig*. 199 | 200 | hostname="knox" 201 | keymap="uk" # use 'us' for USA layouts. Full selection in /usr/share/vt/keymaps 202 | ifconfig_bfe0="DHCP" # Use your adapter's name, though the equivalent of this line may already be present 203 | 204 | sshd_enable="YES" 205 | ntpd_enable="YES" # keep the system regular 206 | ntpd_sync_on_start="YES" 207 | 208 | powerd_enable="YES" # keep power usage down 209 | performance_cx_lowest="C2" 210 | economy_cx_lowest="C2" 211 | 212 | sendmail_enable="NO" # no need for sendmail 213 | sendmail_submit_enable="NO" 214 | sendmail_outbound_enable="NO" 215 | sendmail_msp_queue_enable="NO" 216 | 217 | dumpdev="NO" # defaults to no on RELEASE but you may be on CURRENT 218 | #syslogd_enable="NO" # I think syslog should stay on, but you may feel otherwise 219 | 220 | Let's have *ntpd* synchronise the clock before we proceed: 221 | 222 | root@knox# service ntpd start 223 | 224 | /etc/crontab 225 | ------------ 226 | 227 | *cron* is the system command scheduler. *cron* really likes to send email, but since we've disabled *sendmail* this email will all pile up in */var/spool*. Add the following line to the top of */etc/crontab* to muzzle it. 228 | 229 | MAILTO="" 230 | 231 | /etc/fstab 232 | --------- 233 | This is the file system tab, which used to be a listing of all the file-systems the kernel should mount at boot time, though it's not as singular these days with the advent of removable devices and ZFS. Here's the contents of mine, note that the fields are separated by tabs, which is why that zero looks so lonesome. 234 | 235 | # Device Mountpoint FStype Options Dump Pass# 236 | /dev/ada0p2 / ufs rw,noatime 1 1 237 | /dev/ada0p3.eli none swap sw 0 0 238 | tmpfs /tmp tmpfs rw,mode=1777 0 0 239 | tmpfs /var/run tmpfs rw 0 0 240 | tmpfs /var/log tmpfs rw 0 0 241 | 242 | 243 | Your partition names may be different. I've enabled encrypted swap by appending the *.eli* suffix onto the device name of the swap partition. This ensures that no potentially sensitive memory is ever written out to disk in the clear. I've also added three *tmpfs* mounts; these are memory disks and are therefore cleared on reboot. By turning the */var/log* directory into a memory disk, we should be able to keep the main system drive spun down most of the time, reducing power and extending its life. Of course, this doesn't really matter on flash devices. The other two are really just used for storing tiny files that should be cleared between boots anyway, so it makes sense to hold them in memory. The *noatime* setting on the root partition tells the system not to bother updating timestamps when a file is read (but still when written), this should also help to keep the drives spun down. 244 | 245 | Be aware the using encrypted swap will prevent you from gathering automated crash dumps. This isn't a problem if your system isn't crashing, but if it starts to, switch back to regular swap (remove the *.eli* extension) and set *dumpdev="AUTO"* in */etc/rc.conf*. Then, after a panic, you can run *kgdb* against the kernel and the dump to isolate what function caused the issue. Hopefully you can forget this paragraph exists though. 246 | 247 | /etc/ttys 248 | --------- 249 | TTY stands for 'teletypewriter', which is an early method for interacting with systems of the UNIX era. This file controls the allocation of virtual TTYs. Here's my copy, you'll note that it's a lot smaller than the system default. Tabs are in use again. 250 | 251 | console none unknown off insecure 252 | ttyv1 "/usr/libexec/getty Pc" xterm on secure 253 | 254 | This disables all but one of the virtual consoles, leaving ttyv1, not v0 as the enabled one. This helps prevent the terrible temptation to login to ttyv0 and curse as various system messages overprint your session. The insecure option for *console* forces single user mode to require a password, which won't stop a serious physical attacker, but may deter a too-curious teenager. 255 | 256 | /boot/loader.conf 257 | ----------------- 258 | This controls the environment the kernel loads into. It's a mix of some power control options and ZFS tuning parameters that should improve performance and stability on low memory systems. ZFS is very fond of memory, and runs in the kernel address space, so by limiting it we avoid some nasty memory conditions. The values here are for a 512MB system, if you have more memory than that you might consider increasing them after researching what they do, but they'll work for now. You should probably omit the two *hint* lines on SBC systems, though they're likely harmless. Some deeper info on ZFS tuning is available [here](https://wiki.freebsd.org/ZFSTuningGuide). 259 | 260 | Note that this file isn't present by default. 261 | 262 | autoboot_delay=3 # speed up booting 263 | hint.p4tcc.0.disabled=1 # use only modern CPU throttling (x86 only) 264 | hint.acpi_throttle.0.disabled=1 # does your SBC use ACPI? RPi doesn't. 265 | zfs_load="YES" 266 | geom_eli_load="YES" 267 | vm.kmem_size="150M" # limit the kernel's memory usage 268 | vm.kmem_size_max="150M" 269 | vfs.zfs.arc_max="32M" # limit the ZFS cache size 270 | vfs.zfs.vdev.cache.size="16M" # limit per-device cache 271 | 272 | /etc/sshd/sshd_config 273 | --------------------- 274 | We're back to SSH now, and this is where I think things get really interesting. We're going to be using SSH's certificate authority features to authorise different keys. This means we give the system one key to love, honour, and obey, and then anything we sign with that key will work even if the system hasn't seen it before. Naturally, the CA key itself then should be protected with a password as good as the *root* password as it will essentially be another way of getting *root* access. Personally I think you should re-use the existing *root* password here but you may feel otherwise. 275 | 276 | hugh@local$ cd ~/.ssh 277 | hugh@local$ ssh-keygen -t ed25519 -f knox-ca 278 | hugh@local$ scp knox-ca.pub knox:/etc/ssh/knox-ca 279 | 280 | Now, **on knox**, **replace** the contents of */etc/ssh/sshd_config* with the following (the whole thing). This avoids crypto suspected to be [NSA/FVEYS vulnerable](http://www.spiegel.de/media/media-35515.pdf). Be aware that this machine will now only be contactable by recent versions of OpenSSH, which if you're running a recent OS will be fine. 281 | 282 | HostKey /etc/ssh/ssh_host_ed25519_key 283 | TrustedUserCAKeys /etc/ssh/knox-ca 284 | AllowUsers root 285 | PasswordAuthentication no 286 | PermitRootLogin without-password 287 | UseDNS no 288 | UsePAM no 289 | ChallengeResponseAuthentication no 290 | KexAlgorithms curve25519-sha256@libssh.org 291 | Ciphers chacha20-poly1305@openssh.com 292 | 293 | Then: 294 | 295 | # service sshd restart 296 | 297 | SSH Client 298 | ---------- 299 | Now we'll generate a login and sign it with the CA key, and we'll do it properly by using *ssh-agent*. This allows us the security of having password protected keys without the hassle of entering the password every time. We unlock the key once, add it to the agent, and it's available until we logout. *ssh-agent* operates as a wrapper around a shell so firstly we have to work out what shell you're using. 300 | 301 | hugh@local$ ssh-agent $(grep $(whoami) /etc/passwd | cut -d ':' -f 7) 302 | 303 | You can avoid that junk if you already know what shell you're using, *echo $0* or *echo $SHELL* can sometimes also contain the shell name, but not too reliably. 304 | 305 | Now we're in a sub-shell of the *ssh-agent* process, time to generate the new ID. 306 | 307 | hugh@local$ ssh-keygen -t ed25519 -f ~/.ssh/knox-shell 308 | 309 | Sign the key. The *-I* flag is just a comment. 310 | 311 | hugh@local$ ssh-keygen -s ~/.ssh/knox-ca -I knox-shell -n root ~/.ssh/knox-shell 312 | 313 | Now we tell *ssh* to use this key when connecting to *knox*. We can add some fanciness while we're at it. Edit your *~/.ssh/config*: 314 | 315 | HashKnownHosts yes 316 | ControlMaster auto 317 | ControlPath /tmp/ssh_mux_%h_%p_%r 318 | ControlPersist 30m 319 | 320 | Host knox 321 | User root 322 | HostName 192.168.1.13 # We'll swap in the FQDN later 323 | HostKeyAlgorithms ssh-ed25519 # Prevent downgrade attacks 324 | IdentityFile ~/.ssh/knox-shell 325 | 326 | The *Control* settings allow us to reuse connections which greatly speeds things up. Now that the key has its bona-fides (the *knox-shell-cert.pub* file), we should unlock it and use it to login. 327 | 328 | hugh@local$ ssh-add ~/.ssh/knox-shell 329 | hugh@local$ ssh knox 330 | 331 | If you get dropped into your shell without being asked for a password then all is well. For fun, let's log out and in again to see how snappy the persisted connection makes things: 332 | 333 | root@knox# exit 334 | hugh@local$ ssh knox 335 | root@knox# exit 336 | 337 | Splendid. 338 | 339 | Updates 340 | ------- 341 | FreeBSD provides binary updates for [Tier 1 architectures](https://www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/archs.html), i.e. i386 and amd64. You can check your architecture with *uname -p*. If you're not using one of those, (the RPi is ARMv6) you'll have to find [alternative](https://www.freebsd.org/doc/handbook/updating-upgrading.html) ways of keeping the system up to date (flashing a new image or doing a local buildworld). The system is pretty locked down though, so having it always have the latest software isn't really necessary. You can probably ignore this section for now. 342 | 343 | Tier 1 users should execute the following, they may take a few minutes to run. *fetch* will present a long list of files that will be updated, you can simply press '*q*' to exit this. 344 | 345 | root@knox# freebsd-update fetch install 346 | 347 | 348 | The pkg system 349 | ----------------------- 350 | *pkg*, too is tier 1 only, so look to the venerable [ports collection](https://www.freebsd.org/doc/handbook/ports-using.html) for package installations. I've heard tell of some [third party](https://forums.freebsd.org/threads/raspberry-pi-package-repository.48179/) pkg [repositories](https://www.raspberrypi.org/forums/viewtopic.php?f=85&t=30148). Though it usually better to build your own through *ports*, either locally or [on another system](https://www.freebsd.org/doc/handbook/ports-poudriere.html). If you don't feel you need any packages, then you can ignore this. 351 | 352 | The blessed ones can run the following; you'll be prompted to install *pkg*. 353 | 354 | root@knox# pkg update 355 | 356 | Install [any other packages](https://www.freebsd.org/doc/handbook/ports-finding-applications.html) you like at this point. 357 | 358 | At this point we can reboot the system. 359 | 360 | root@knox# reboot 361 | 362 | Log in again when it's back up and read through the boot messages to see all went well. '+G' tells *less* to start at the end of the file. 363 | 364 | hugh@local$ ssh knox 365 | root@knox# less +G /var/log/messages 366 | 367 | You may, for instance, see: 368 | 369 | Aug 2 00:53:21 knox root: /etc/rc: WARNING: failed to start powerd 370 | 371 | Which tells us that for whatever reason, *powerd* isn't able to function on this machine. SBCs may show this, as will VMs. If you see it, remove the *powerd* line from */etc/rc.conf* 372 | 373 | Be aware, that if you want to login to the machine physically, instead of via SSH, you must switch to the second console with \. \ will return you to the kernel messages screen. 374 | 375 | Disk Setup 376 | ========= 377 | GELI 378 | ---- 379 | Time to plug in the USB drive. Let's find out what *knox* is calling it. 380 | 381 | root@knox# camcontrol devlist 382 | 383 | at scbus0 target 0 lun 0 (ada0,pass0) 384 | at scbus4 target 0 lun 0 (pass1,cd0) 385 | at scbus5 target 0 lun 0 (da0,pass2) 386 | at scbus5 target 0 lun 1 (ses0,pass4) 387 | 388 | The first entry, *ada0* is *knox*'s internal hard drive, the second entry is the laptop's disc drive. The 2TB USB drive sits on *da0* with a monitoring interface we can disregard on *ses0*. Yours will likely be *da0* too. 389 | 390 | We should also work out what sector size the drive is using, though in all likelihood we're going to use 4KiB sectors anyway. Most large (2TiB+) drives will be using the [4KiB standard](https://en.wikipedia.org/wiki/Advanced_Format). 391 | 392 | root@knox# camcontrol identify da0 | grep size 393 | 394 | sector size logical 512, physical 4096, offset 0 395 | 396 | Yup, it's a 4KiB drive. Now we'll generate the encryption key for the [GELI](https://www.freebsd.org/cgi/man.cgi?geli(8)) full disk encryption (**locally**). The brackets in the below command are significant, they limit the duration of the *umask* change. 397 | 398 | hugh@local$ (umask 177; touch ~/.ssh/knox-geli-key) 399 | hugh@local$ dd if=/dev/random bs=1k count=1 > ~/.ssh/knox-geli-key 400 | 401 | Strictly, we shouldn't store that in *~/.ssh*, but it's as good a place as any. You'll have noticed that we're not using any password with this key, and since we can't back it up to the backup system (egg, chicken, etc.) we'll need to store it somewhere else. But while we might be happy to have it lying around unencrypted on our local system, where we can reasonably control physical access, we're better off encrypting it for storage on Dropbox or in an email to yourself or wherever makes sense as we don't know who might have access to those systems (presume everyone). You could also stick it on an old USB flash drive and put it in your sock drawer if you know what an [NSL](https://en.wikipedia.org/wiki/National_security_letter) is. 402 | 403 | hugh@local$ cd; tar -cf - .ssh | xz | openssl aes-128-cbc > key-backup.txz.aes 404 | 405 | Make sure you use a good password (maybe the same as you use for your login) and stick that *.aes* file somewhere safe (it also contains your SSH identity). Should you ever need to decrypt that file: 406 | 407 | hugh@local$ < key-backup.txz.aes openssl aes-128-cbc -d | tar -xf - 408 | hugh@local$ ls .ssh 409 | 410 | Let's get a quick measurement on the drive's normal speed before we activate the encryption. 411 | 412 | root@knox# dd if=/dev/zero of=/dev/da0 bs=1m count=100 413 | ... 414 | 104857600 bytes transferred in 15.057181 secs (6963960 bytes/sec) 415 | 416 | Send the key over to *knox*, this is only for the initial setup, it won't hold a copy of it. Also, since */tmp* is now a memory drive, we don't need to worry about anything as serious as [Guttmann](https://en.wikipedia.org/wiki/Gutmann_method) erasure. 417 | 418 | hugh@local$ scp ~/.ssh/knox-geli-key knox:/tmp 419 | 420 | The following command creates an AES-XTS block device with a 128 bit key. Other ciphers/lengths are available but the defaults are [pretty good](https://www.schneier.com/blog/archives/2009/07/another_new_aes.html). I feel we can skip *geli*'s integrity options as ZFS is going to handle any cases of incidental corruption and malicious corruption isn't really in our threat model. 421 | 422 | root@knox# geli init -s 4096 -PK /tmp/knox-geli-key /dev/da0 423 | root@knox# geli attach -pk /tmp/knox-geli-key /dev/da0 424 | 425 | The other *geli* option of note is the sector size. By forcing *geli* to use 4KiB sectors, and only writing to the *geli* overlay, we get the best performance from our drive. Though, given the low power nature of this system, we're unlikely to ever see the benefit due to slower links in the rest of the chain. Since *geli* encrypts per-sector, specifying a larger size also reduces it's workload versus the default 512 byte sectors. 426 | 427 | Note: There is a bug in older ARM versions of *geli* that may prevent you from attaching a drive you've just initialised. [See here for a workaround](https://github.com/hughobrien/zfs-remote-mirror/issues/6) 428 | 429 | Let's see how this encryption has affected our drive's speed: 430 | 431 | root@knox# dd if=/dev/zero of=/dev/da0.eli bs=1m count=100 432 | .. 433 | 104857600 bytes transferred in 17.759175 secs (5904418 bytes/sec) 434 | 435 | root@knox# echo "5904418 / 6963960" | bc -l 436 | 437 | .84785352012360783232 438 | 439 | Fifteen percent drop? Not too bad. Again, this was never going to be a high-performance system. 440 | 441 | ZFS 442 | --- 443 | Now that we have an encrypted substrate, we can hand it over to ZFS. The *zpool* command handles all things low-level in ZFS. I'm calling my pool [*wd*](https://en.wikipedia.org/wiki/Western_Digital). 444 | 445 | root@knox# zpool create -O atime=off -O compression=lz4 wd da0.eli 446 | 447 | Note that I specified the block device as *da0.eli* which is the overlay device exposed by *geli*. *atime* is access time, which logs when a file is accessed. We don't need this, and it hurts performance a little, so out it goes. *lz4* compression is extremely fast, to the point of being almost computationally free, and will let our already large drive go even further. Individual ZFS datasets can override these options later on but they make good defaults. I also have these options set on my local pool, but if your local pool differs then they will be overwritten when we send the filesystem. 448 | 449 | ZFS is all setup now (wasn't that easy? No partitioning or anything). Let's see what we have: 450 | 451 | root@knox# zpool status 452 | 453 | pool: wd 454 | state: ONLINE 455 | scan: none requested 456 | config: 457 | NAME STATE READ WRITE CKSUM 458 | wd ONLINE 0 0 0 459 | da0.eli ONLINE 0 0 0 460 | 461 | errors: No known data errors 462 | 463 | Wonderful, now we tear it all down. 464 | 465 | root@knox# zpool export wd 466 | root@knox# rm -P /tmp/knox-geli-key 467 | 468 | I securely erased the key anyway...couldn't help myself. Detach the *geli* and verify it's been removed: 469 | 470 | root@knox# geli detach /dev/da0 471 | root@knox# geli status 472 | 473 | You should only see references to the encrypted swap, probably on *ada0p3.eli*. No *da0.eli* in sight. What does *eli* stand for anyway? I haven't been able to [figure that out](https://forums.freebsd.org/threads/what-does-geli-stand-for-not-what-does-it-do.42546/). 474 | 475 | Datasets 476 | -------- 477 | ZFS datasets allow you to specify different attributes on different sets of data; whether to use compression, access control lists, quotas, etc. I find that I need precisely none of those features, preferring to treat my backup storage as one large dataset with sane properties. There's nothing stopping you from creating different datasets and synchronising them to the backup system, I just don't see the point for personal backups. Know also that it makes previewing the differences between snapshot more complex, as *zfs diff* cannot automatically recurse into dependent snapshots, it has to be done per dataset. This isn't really a big deal though. 478 | 479 | Plumbing 480 | ======== 481 | Drop the following script into *root*'s home directory, call it *zfs-receive.sh*. I would have preferred to invoke these commands remotely but after a lot of experimenting, triggering the script was the only way I found that properly detached the encrypted drive in the event of connection failure. So rest assured, you're protected against that. The *dd* at the start is to spin up the drive. 482 | 483 | #!/bin/sh 484 | 485 | dd if=/dev/da0 of=/dev/null bs=1m count=3 2>/dev/null 486 | geli attach -dpk /tmp/k /dev/da0 487 | zpool export -f wd 2>/dev/null 488 | zpool import -Nf wd 489 | zfs receive -Fu wd 490 | zpool export -f wd 491 | 492 | What's this you say? I promised that we wouldn't store the key on the backup server? [Behold! The Named Pipe](https://en.wikipedia.org/wiki/Named_pipe) 493 | 494 | Don't set any passwords on these two keys, we need them to be scriptable. 495 | 496 | hugh@local$ ssh-keygen -t ed25519 -f ~/.ssh/knox-fifo 497 | hugh@local$ ssh-keygen -t ed25519 -f ~/.ssh/knox-send 498 | 499 | Though these two keys are not password protected, but they are going to be completely restricted in what they can do. This allows us to use them in an automatic way, without the fear of them being abused. 500 | Now bless them. This will ask for the CA password. 501 | 502 | hugh@local$ ssh-keygen -s ~/.ssh/knox-ca -I knox-fifo -O clear -O force-command="mkfifo -m 600 /tmp/k; cat - > /tmp/k; rm -P /tmp/k" -n root ~/.ssh/knox-fifo.pub 503 | hugh@local$ ssh-keygen -s ~/.ssh/knox-ca -I knox-send -O clear -O force-command="/root/zfs-receive.sh" -n root ~/.ssh/knox-send.pub 504 | 505 | Terrified? Don't be. We're signing the keys we just created and specifying that if they are presented to the remote server, the only thing they can do is execute the described command. In the first case we create a [*fifo*](https://www.freebsd.org/cgi/man.cgi?query=mkfifo&sektion=1) on the */tmp* memory disk that we write to from *stdin*. This will block until someone reads from it, and that someone is the *zfs-receive.sh* script that we call next. Upon reading the *fifo* the key is transferred directly from our local system to the *geli* process and never touches the disk, or the RAM disk. 506 | 507 | And before you complain, that's not a [*useless use of cat*](https://image.slidesharecdn.com/youcodelikeasysadmin-141120122908-conversion-gate02/95/you-code-like-a-sysadmin-confessions-of-an-accidental-developer-10-638.jpg?cb=1416487010), it's required for *tcsh*. 508 | 509 | Let's add some shortnames for those keys in *~/.ssh/config*. 510 | 511 | Host knox-fifo 512 | User root 513 | HostName 192.168.1.13 514 | IdentityFile ~/.ssh/knox-fifo 515 | 516 | Host knox-send 517 | User root 518 | HostName 192.168.1.13 519 | IdentityFile ~/.ssh/knox-send 520 | 521 | Final Approach 522 | -------------- 523 | I trust you're quite excited at this point. Let's take a fresh snapshot of our local pool and send it. This will involve sending the entire dataset initially, which is likely a lot of data and is the reason we specified a local network address in *~/.ssh/config*. 524 | 525 | hugh@local$ snapname="$(date -u '+%FT%TZ')" 526 | hugh@local$ zfs snapshot -r "wd@$snapname" 527 | 528 | Snapshots will be given names of the form *'2015-07-24T16:14:10Z* ([ISO 8601 format)](https://en.wikipedia.org/wiki/ISO_8601). The time stamp is from UTC so it will probably be a few hours off from your local time. If you're convinced you'll never change timezone you could change this, but it's hardly an inconvenience. 529 | 530 | Let's make sure your *local* user has the requisite permissions to use *zfs*: 531 | 532 | root@local$ zfs allow -u hugh send,diff,snapshot,mount,hold wd 533 | 534 | Back to user level, and drum roll please... 535 | 536 | hugh@local$ ssh knox-fifo < ~/.ssh/knox-geli-key & 537 | hugh@local$ zfs send -Rev "wd@$snapname" | ssh knox-send 538 | 539 | Be prepared to wait...On the Raspberry Pi I get about 2.5MB/s, which is several times faster than my home upload speed, so not really a problem. [Here's](https://github.com/hughobrien/zfs-remote-mirror/blob/master/top) the output from *top* while the system is receiving a snapshot over the local network, you can see the load is roughly split between *geli* and *ssh*, indicating that the RPi's processor isn't so snappy for cryptographic operations. Interestingly ZFS is using fairly little resources, and as you can see the system memory is pretty empty. Perhaps we needn't have been so concerned with saving every byte. 540 | 541 | When it's done, your data will be safe, secure, and (soon to be) far away. Accessible to only someone with your SSH key and its password (or physical access) and readable only by someone with your geli key. 542 | 543 | Incremental Backups 544 | ------------------- 545 | All that was a lot of work, but we can automate the rest with a simple script. 546 | 547 | To take a snapshot at the current time: 548 | 549 | hugh@local$ ~/backup.sh snapshot 550 | 551 | To preview the changes between your latest snapshot and the latest one on the remote system: 552 | 553 | hugh@local$ ~/backup.sh preview 554 | 555 | To send those changes: 556 | 557 | hugh@local$ ~/backup.sh backup 558 | 559 | To snapshot and send without previewing: 560 | 561 | hugh@local$ ~/backup.sh snapback 562 | 563 | Do this once a day, week, whenever and your backups will always be fresh. Remember that ZFS snapshots are cheap (use *'zfs list -t snapshot'* to admire them) so feel free to make many. You might even consider adding it to your crontab: 564 | 565 | hugh@local$ crontab -e 566 | 567 | 59 17 * * * ~/backup.sh snapback 568 | 569 | Note that the way we send the snapshots will remove any data on the remote pool that isn't on the local pool - so don't store anything there manually. Store it in the local pool and let it propagate automatically. 570 | 571 | Also be aware, that there isn't currently support for resuming failed transfers, so they'll have to be restarted. With small, regular snapshots this shouldn't pose much of an issue, and it is an [upcoming feature](http://www.slideshare.net/MatthewAhrens/openzfs-send-and-receive). 572 | 573 | [Here's the whole script](https://github.com/hughobrien/zfs-remote-mirror/blob/master/backup.sh), save it as *~/backup.sh* on your local machine. 574 | 575 | hugh@local$ cd 576 | hugh@local$ fetch https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/master/backup.sh 577 | hugh@local$ chmod 744 backup.sh 578 | 579 | If you're still in the same shell that you ran the initial backup from, we can set the *remote state* file now. 580 | 581 | hugh@local$ echo "wd@$snapname" > .knox-last-sent 582 | 583 | Physically Placing Your Remote System 584 | ===================================== 585 | Now we can place the backup system in its permanent location. This is highly subjective but here are a few points to bear in mind: 586 | 587 | * Connect the system to the modem/gateway via Ethernet. Not Wi-Fi. 588 | * Try and place it somewhere out of the way, so it can gather dust in peace. 589 | * (If necessary, instruct the home-keeper not to dust it.) 590 | * Be respectful and use as few power sockets as possible. A simple [doubler](https://en.wikipedia.org/wiki/AC_power_plugs_and_sockets_-_British_and_related_types#/media/File:Doubler_and_tripler.jpg) can help here. 591 | * Use an extension cable to minimise the visible wires. Out of sight, out of mind. 592 | * Expect the system to crash or loose power, set it up to recover without intervention. 593 | * Configure the BIOS to ignore all boot sources but the root when booting. 594 | * To power up again if it looses power. 595 | * To auto-power on at a certain time in case the above fails. 596 | * Give the system a DHCP MAC reservation on the gateway, or a static IP if not possible. 597 | * Edit */etc/rc.conf* to set a static IP if you can't reserve an address through DHCP. 598 | * Set up port forwarding. Take port 22 if available, or map to port 22 and update your local ssh config file with *Port* details. 599 | * Set up a [Dynamic DNS](https://freedns.afraid.org/) name if the modem has a dynamic external IP. 600 | 601 | If it's not possible to use port forwarding, there are some alternatives for connecting to the system: 602 | 603 | * [Tor hidden service](https://www.torproject.org/docs/tor-hidden-service.html.en) and SSH via [torsocks](https://github.com/dgoulet/torsocks/). 604 | * IPv6. 605 | * SSH wizardry involving [autossh](http://www.harding.motd.ca/autossh/), proxy commands, and reverse tunnels. In this case your local system, or some mutually accessible proxy system must have a static IP. 606 | 607 | A Tor hidden service is probably the most reliable fall back, but it will be slow and it's not really in the spirit of Tor to use it in this way. One especially nice feature is that you can leave your remote system deep within a large network (e.g. your employer or University (**get permission**)) and as long as Tor can get out, you can get in. No need for a direct connection to the modem or DHCP reservations. Personally, I like to have a hidden service running even if I have port-forwarding working, as a fall back measure. e.g. if the modem were replaced or factory-reset, I could *tor* in and with SSH forwarding connect to the modem's WebUI and set up forwarding again. 608 | 609 | Once you're worked out your method, adjust your config file on your **local** machine to use the new Internet accessible name. 610 | 611 | Host knox 612 | User root 613 | ... 614 | HostName knox.tld.cc # FQDN for Internet routing 615 | #HostName 192.168.1.23 # Local address for initial backup 616 | #HostName http://idnxcnkne4qt76tg.onion/ # Tor hidden service 617 | 618 | # Remember to adjust the knox-fifo and knox-send entries also. 619 | # Or make an entry in /etc/hosts, but that's just another thing to manage. 620 | 621 | Care & Feeding 622 | ============== 623 | Upkeep 624 | ----- 625 | 626 | From time to time connect into the remote system and check the system logs and messages for anything suspicious. Also consider updating any installed packages and keeping up to date with errata patches. *pkg* and *freebsd-update* make this easy (on Tier 1 platforms). 627 | 628 | root@knox# less +G /var/log/messages 629 | root@knox# pkg upgrade 630 | root@knox# freebsd-update fetch install 631 | 632 | You will need to reboot if *freebsd-update* makes any changes. 633 | 634 | It's also sound practice to occasionally exercise the disks, both your local and the remote one with a *scrub* operation. The instructs ZFS to re-read every block on the disk and ensure that they checksum correctly. Any errors can be found will be logged and they probably signal that you should replace the disk. 635 | 636 | hugh@local$ ssh knox-fifo < ~/.ssh/knox-geli-key & 637 | hugh@local$ ssh knox-shell 638 | root@knox$ geli attach -dpk /tmp/k 639 | root@knox$ zpool scrub wd 640 | root@knox$ sleep 60; zpool status 641 | ..... 642 | # A considerable time later... 643 | ..... 644 | root@knox# zpool status # is it done? 645 | root@knox# zpool detach 646 | # geli detaches automatically here 647 | root@knox# rm -P /tmp/k 648 | 649 | *zpool status* will give you some information about the speed of the operation and an estimated time to completion. Be aware that your drive is unlocked in this state. 650 | 651 | Extra Encryption 652 | ---------------- 653 | If the thought of the decryption key for some sensitive data being automatically sent to a system outside of your immediate physical control concerns you, but you still want all the advantages of ZFS, you might consider adding an encrypted volume. 654 | 655 | This is a ZFS powered virtual storage device that we can layer GELI encryption on top of, using completely different key material but still stored in the ZFS pool such that it can be snapshotted, have only its changes transferred on backup and have the benefit of strong data integrity. 656 | 657 | root@local# mkdir /wd/vol; chown hugh /wd/vol 658 | root@local# zfs create -s -V 100G wd/vol 659 | root@local# geli init -s 4096 /dev/zvol/wd/vol 660 | root@local# geli attach -d /dev/zvol/wd/vol 661 | root@local# newfs -Ujn /dev/zvol/wd/vol.eli 662 | root@local# mount /dev/zvol/wd/vol.eli /wd/vol 663 | 664 | I suppose you could use ZFS instead of UFS on the new volume, if you have a [totem](https://www.google.ie/search?q=inception+totem&tbm=isch), but it's probably more trouble than it's worth. 665 | 666 | */wd/vol* is now available for secure storage. The *-s* flag to *zfs create* indicates a *sparse allocation*; the system won't reserve the full 100GiB and won't allocate any more data than you actually write to it. While 100GiB is the most it can hold, you can use ZFS properties to increase the volume size and then *growfs* if you ever hit that limit (*geli* may need to be informed of the resize too). 667 | 668 | When you've finished using the encrypted drive, unmount it. Remember not to have any shells active in the directory or they'll hold it open. Since we attached it with the *-d* flag, *geli* will automatically detach it. 669 | 670 | root@local# umount /wd/vol 671 | 672 | To mount the volume for further use: 673 | 674 | root@local# geli attach -d /dev/zvol/wd/vol 675 | root@local# mount /dev/zvol/wd/vol.eli /wd/vol 676 | 677 | You may wish to define some shell functions (using *sudo*, or scripts with *setuid*) to handle the repetitive attaching and detaching. The contents of *vol* will be included in any snapshots and will be sent to the remote system during *zfs send*. I recommend having the volume unmounted and detached before snapshotting though. 678 | 679 | Disaster Recovery 680 | ----------------- 681 | One day, one of the following things will happen: 682 | 683 | * A motor or sensor in your USB drive [will fail](https://www.youtube.com/watch?v=Nr-RsGX6Ywk). 684 | * The [GoldenEye](https://www.youtube.com/watch?v=HHFXthl5IJo) will be fired. 685 | * A power surge will blow the regulator on the old laptop. 686 | * Someone will knock the remote drive onto the ground while cleaning. 687 | * A drive head will turn masochistic. 688 | * [Cosmic rays](https://en.wikipedia.org/wiki/Soft_error#Cosmic_rays_creating_energetic_neutrons_and_protons) will flip your bits. 689 | * The drive containing the OS will fail. 690 | 691 | In all but the last two cases, we must consider the drive totally lost. It might happen to your local drive first, because it experiences more activity than the backup. It might happen to the remote drive first, because it lives in a room without central heating and is subject to wider temperature fluctuations. No matter how it happens, **it is going to happen**. 692 | 693 | **What shall we do when it does?** Simple. Buy a new drive. Set it up as above, create the first backup and then the incrementals as you've always done. Recycle the old drive. There's no need to worry about wiping it clean because everything is encrypted. It is enough to destroy whatever copies of the encryption key you have lying around. 694 | 695 | **I deleted a file by accident, can I recover it from a snapshot?** Naturally, you don't even need to access the remote system, snapshots are accessible through the hidden *.zfs* directory at the root of your pool. e.g. */wd/.zfs/snapshot/2015-07-24T23:35:18Z* 696 | 697 | **What if the backup computer dies, but the drive is okay?** Recycle the computer. Buy/liberate another one, install FreeBSD as above, then just connect the drive and carry on. 698 | 699 | **What about slow, creeping drive death, as opposed to total failure?** ZFS has your back. Take a look at '*zpool status*' every now and then on both machines (the remote will have to be attached of course). If you see any checksum errors, buy a new disk. Every so often, run '*zpool scrub*' on both disks to have ZFS read and verify every sector, then check the status and do what you need to do. Life is too short for bad hard disks, and 2TiB is a lot of data to loose. 700 | 701 | **My local disk failed, can I swap in my backup?** Probably. Use *geli* to attach it locally (with the key) and then use '*zpool import*'. Then buy a new drive and go through the motions again. 702 | 703 | **My local disk failed, but I can't physically access the remote one, what do I do?** You've got your SSH and GELI keys backed up somewhere, right? Use those to access the remote machine and pull down any files you need (mount the datasets with *'zfs mount -a'*). You *could* try a full backup onto a new disk over the Internet, but you'll be waiting a while, and your friendly server co-location administrators might be getting calls from their ISP. A better approach would be to buy a new drive, have it delivered to wherever the remote system lives and have someone connect it. Set it up as a second pool (use *geli*), do a local send/receive and once it holds a full copy politely ask that it be posted to you. Note: Some systems can't supply enough USB power for two high drain device like hard drives. If you're using a USB powered drive on the machine, connect the second drive through a powered hub or use one that has its own power. 704 | 705 | Further Reading 706 | ============== 707 | 708 | I would first recommend the handbook sections on [zpool](https://www.freebsd.org/doc/handbook/zfs-zpool.html) and [zfs](https://www.freebsd.org/doc/handbook/zfs-zfs.html), followed by their respective *man* pages. They're long, but you've already come this far. [geli](https://www.freebsd.org/cgi/man.cgi?geli(8)) too, is required reading. 709 | 710 | Here are some videos discussing ZFS: 711 | 712 | * [OpenZFS Overview](https://www.youtube.com/watch?v=RQlMDmnty80) 713 | * [OpenZFS Remote Replication](https://www.youtube.com/watch?v=RQlMDmnty80) 714 | 715 | Thank you for your attention, may you never need anything this guide helps prevent against. Please send details of any mistakes or areas for improvement to *obrien.hugh* at the Google mail system. 716 | 717 | Appendix - Raspberry Pi 718 | ======================= 719 | 720 | With some work, it's possible to use a [Raspberry Pi Model B](https://www.raspberrypi.org/products/model-b/), and likely any other supported SBC, as the remote system. I'll take you through how to prepare a system image for this, but for more trusting readers I'll provide a pre-made image file. 721 | 722 | The basic process is: 723 | * Download the source for our chosen branch. 724 | * Download the [crochet](https://github.com/freebsd/crochet) image building tool. 725 | * Apply customisations. 726 | * Build image. 727 | * Customise built image. 728 | * Boot RPi. 729 | 730 | I'm working with FreeBSD 10.2, which is the production branch at the time of writing. There are some [pre-made images](https://ftp.heanet.ie/pub/FreeBSD/releases/arm/armv6/ISO-IMAGES/10.2/) provided by the foundation but they're of 10.2-RELEASE, 10.2-STABLE has advanced a little since then. And since we're going to all the bother of building our own images, it makes sense to get the best available code. Also, rather crucially, these images do not support ZFS. 731 | 732 | The [FreeBSD Release Engineering](https://www.freebsd.org/doc/en/articles/releng/) process is worth [reading up on](https://www.freebsd.org/doc/en/articles/releng/release-proc.html). In a nutshell, most development occurs in *current*, also called *head*, which changes all the time, breaking and being repaired as new code is added. Every now and then the release team cleave off a section from current and call it a *stable* branch, such as 10. Code from current which is considered to be good stuff gets brought down into the latest stable branch. At some point the team branches off the stable branch to make a *feature* branch, such as 10.0. This branch stops getting the new toys added to the stable branch and the team focus on making everything in it work well together. When they're satisfied they *tag* a certain state of the feature branch as a *release*. Then they go to all the work of building the images and documentation for this release, make a big announcement, and we get to download and play with, for example, FreeBSD-10.0-RELEASE. 733 | 734 | Of course, not everything is always rosy with a release, sometime minor bug fixes or security patches come out afterwards, known as *errata*. These make it into the feature branch, but since the release has already been tagged and distributed, it doesn't change. In an ideal world, we'd always run the most recent code from a feature branch. However this would mean each user would have to track the branch themselves and rebuild as necessary. Since most people use the RELEASE images (as recommended), the team also put out binary patches for the main supported architectures to allow people on a RELEASE to change just the necessary files, without compiling anything, bringing them to an equivalent state as if they were running a system compiled from the latest feature branch. This is provided by *freebsd-update*, for [supported platforms](https://www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/archs.html). 735 | 736 | I mention all of this, to answer the seemingly simple question, of what source branch should we download and compile for our Raspberry Pi? The Pi is an ARMv6 board, and thus isn't provided with binary updates. So if we want errata fixes, we have to get them ourselves. Here is the [current list of branches](https://www.freebsd.org/releng/): 737 | 738 | * head - no, too unstable. 739 | * stable/10 - currently working towards 10.3, so not quite stable. 740 | * releng/10.2 - the latest (at the time of writing) feature branch with all known errata applied. 741 | * releng/10.1 - an older 10 feature branch. 742 | * releng/10.0 - as above. 743 | * stable/9 - 9 isn't getting much love right now, but it's still supported. 744 | * releng/9.3 - the last feature branch for 9, will still get errata fixes if necessary. 745 | * ... 746 | * stable/8 - 8 is no longer supported, but it's still there for all the world to see. Doesn't support the RPi. 747 | * ... 748 | * stable/2.2 - FreeBSD is indeed old. 749 | 750 | So, since we're building our own image, and compiling all our own code, we want the latest errata fixes from the latest feature branch. That's *releng/10.2*. Let's get the code. Depending on what version of FreeBSD you're currently running, you may already have an earlier version of this code in */usr/src*, but it's cleaner if we grab a fresh copy. 751 | 752 | hugh@local$ mkdir ~/knox 753 | hugh@local$ svnlite co https://svn.freebsd.org/base/releng/10.2 knox/src 754 | 755 | *svn* is the short command name for [Subversion](https://subversion.apache.org/), the source code management system the FreeBSD project uses *'co* is shorthand for the *checkout* subcommand, which has a special meaning within Subversion, but for our purposes just think of it as *download*). However, Subversion is a somewhat large package, so in the name of minimalism, FreeBSD ships *svnlite* instead, which is just fine for our needs. The last argument is the destination folder. 756 | 757 | This checkout process will take some time. On my system the process will occasionally fail due to network issues, leaving the source directory in an inconsistent state. If this happens, I recommend you delete the whole directory (*rm -rf ~/knox/src*) and try again. 758 | 759 | If it succeeds, you'll get a message similar to: 760 | 761 | Checked out revision 295681 762 | 763 | Now we'll get the [*crochet*](https://github.com/freebsd/crochet) build tool, which delightfully does almost all the hard work of image building for us. The package is maintained on GitHub, which is a little unusual for FreeBSD. If you have *git*, or indeed *git-lite* installed on your system you can get the code with: 764 | 765 | hugh@local$ git clone https://github.com/freebsd/crochet.git 766 | hugh@local$ cd crochet 767 | 768 | If you don't have *git*, GitHub provide zipped archives: 769 | 770 | hugh@local$ fetch https://github.com/freebsd/crochet/archive/master.zip 771 | hugh@local$ unzip master.zip 772 | hugh@local$ mv crochet-master crochet 773 | hugh@local$ cd crochet 774 | 775 | Crochet operates around a central build script, called *config.sh*. There's a sample file in this directory, which I recommend you take a look at, but for expediency, simply create a new file in this directory called *config.sh* with the following contents: 776 | 777 | board_setup RaspberryPi 778 | option ImageSize 3900mb 779 | KERNCONF=RPI-B-ZFS 780 | FREEBSD_SRC=/home/hugh/knox/src 781 | 782 | I'm using a 4GB card, and leaving about 10% of the space unused so the internal chip can handle bad sectors more easily. The formula for ImageSize is n x 1024 x 0.9, where n is the number of Gigabytes on your card. 783 | 784 | The *KERNCONF* is the specification of how to build the kernel for the Raspberry Pi. There's an existing config file in *~/knox/src/sys/arm/conf/RPI-B* that I've modified as by default it doesn't come with, or support ZFS. Here are the modifications: 785 | 786 | * Change 'ident' line from RPI-B to RPI-B-ZFS. 787 | * Add 'options KSTACK_PAGES=6' as required for ZFS (it needs extra memory). 788 | * Add 'opensolaris' and 'zfs' to the MODULES_EXTRA makeoptions, to trigger the build of ZFS. 789 | 790 | I've also removed the following modules, as I don't feel them necessary for this use case and as the RPi is so memory constrained, every byte helps. Some of these options are explained in more detail in the [Developer's Handbook](https://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-options.html). 791 | * INET6 - Support for IPv6, you may want to leave this in. 792 | * SCTP - Stream Control Transmission Protocol, like an optimised TCP, not much use here. 793 | * UFS_DIRHASH - A speed/memory trade-off in the wrong direction for us. 794 | * QUOTA - Quota supports not relevant as we're the only human user of this system. 795 | * NFSCL - Network File System, no need for this at all. 796 | * NFSLOCKD - As above. 797 | * NFS_ROOT - As above. 798 | * KTRACE - Kernel debugging, not needed unless you're developing on this system. 799 | * KBD_INSTALL_CDEV - This system won't have a keyboard (KBD) so not necessary. 800 | * BREAK_TO_DEBUGGER - For developers. 801 | * ALT_BREAK_TO_DEBUGGER - For developers. 802 | * KDB - Debugging. 803 | * DDB - Debugging. 804 | * INVARIANTS - Kernel self-tests, they'll slow us down on an already slow system. 805 | * INVARIANT_SUPPORT - As above. 806 | * gpioled - Driver to control [PWM](https://en.wikipedia.org/wiki/Pulse-width_modulation) on the RPi LEDs, not for us. 807 | * makeoptions DEBUG=-g - Don't build debug symbols, again, we're not developing on this. 808 | 809 | This is, of course, somewhat cumbersome to do manually, so you can grab the config file directly from here: 810 | 811 | hugh@$local$ cd ~/knox/src/sys/arm/conf 812 | hugh@$local$ fetch https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/master/patches/RPI-B-ZFS 813 | 814 | Verify that I'm not sneaking anything in with the following command: 815 | 816 | hugh@local$ diff --side-by-side --suppress-common-lines RPI-B RPI-B-ZFS 817 | 818 | There's one tweak to make to crochet before we build. Since we're being so security conscious, it make sense to use encrypted swap, on the off chance that some of our data might get paged out of memory. There's a possibility that the key-material might even be swapped out, so if it's going to be written to the SD card, let's make sure it's not readable afterwards. 819 | 820 | To make it easy to enable encrypted swap, we're going to direct crochet to create an extra partition in the image. Edit the file *~/knox/crochet/board/RaspberryPi/setup.sh* and find the function *raspberry_pi_partition_image ( )*. 821 | Above the line *disk_ufs_create* add *disk_ufs_create 3000m*. [Here's a patch showing the change](https://github.com/hughobrien/zfs-remote-mirror/blob/master/patches/setup.sh.patch). 822 | 823 | If you're using a card size other than 4GB you should tweak that 3000 figure, it's specifying the size of the root partition on the image. The next call to *disk_ufs_create* will use up all remaining space in the image, which for the 4GB case is about 700MB, plenty for swap. Bear in mind that this explicit specification of partition size will conflict with the *Growfs* option that crochet normally uses, though we've excluded it from our config file. 824 | 825 | Just one last change, there's a DEFINE statement in the *opensolaris* code that causes some build issues, thankfully it's not needed so we can simply delete it. Edit the file *~/knox/src/sys/cddl/compat/opensolaris/sys/cpuvar.h* and delete the line *#define>cpu_id>-cpuid*, it's on line 50 of the file at the time of writing. [Here's the patch](https://github.com/hughobrien/zfs-remote-mirror/blob/master/patches/cpuvar.h.patch). 826 | 827 | With all this done, we can kick off the build. It needs to run as root as it will mount and unmount some virtual file-systems as it goes. We also need the Raspberry Pi version of *uboot* installed, which will be automatically placed into the image. 828 | 829 | root@local# pkg install u-boot-rpi 830 | root@local# cd /home/hugh/knox/crochet 831 | root@local# nice -n 20 ./crochet.sh -c config.sh 832 | 833 | This will take some time. *nice* puts the task at a lower priority so as not to interfere with the rest of the system. 834 | 835 | Once it's finished, you'll have a 4GB image that's ready to be put on the SD card. But not so fast, we can do most of our post-installation configuration changes on the image itself, so that it boots up fully ready. To do this, we first mount the image as a memory device. 836 | 837 | root@local# mdconfig work/FreeBSD-armv6-10.2-RPI-B-ZFS-295681M.img # note the name this returns, it will probably be md0. 838 | root@local# mount /dev/md0s2a /mnt 839 | 840 | Create the */mnt/boot/loader.conf* file: 841 | 842 | autoboot_delay=3 843 | zfs_load="YES" 844 | vm.kmem_size="180M" 845 | vm.kmem_size_max="180M" 846 | vfs.zfs.arc_max="24M" 847 | 848 | This causes the ZFS module to load with the kernel, sets the kernel memory size at a hard 180MB, which should be large enough for ZFS's needs, and restricts the size of the [ARC](https://en.wikipedia.org/wiki/Adaptive_replacement_cache), leaving more of that 180MB for the meat of ZFS. 849 | 850 | Now edit */mnt/etc/fstab*. 851 | 852 | /dev/mmcsd0s1 /boot/msdos msdosfs ro 0 0 853 | /dev/mmcsd0s2a / ufs ro 1 1 854 | /dev/mmcsd0s3.eli none swap sw 0 0 855 | tmpfs /tmp tmpfs rw,mode=1777 0 0 856 | tmpfs /var tmpfs rw 0 0 857 | 858 | First thing to note here, is that we're using read-only mounts. The RPi will not make any writes to the file-system unless we explicitly mount it in read-write mode. The benefit of this, is that the system can always be reset to a stable state by switching it off and on again, and also that if it were to lose power during normal operation, it wouldn't find itself in an inconsistent state on reboot. 859 | 860 | We're also directing the system to use the third partition (the one we edited the crochet setup file to create) as a swap device, but the addition of '.eli' causes it to be automatically encrypted with a one-time key at boot. Lastly, we're going to use a memory backed file system for the system's working directories. This means they're cleared on every reboot, and won't end up filling up the disk if they grow too much. Since we've added a swap partition of about 700MB, the contents of these memory disks are easily swapped out (unlike the kernel), so we're not likely to hit memory issues. A good trade off I think. 861 | 862 | Now here's */mnt/etc/rc.conf*: 863 | 864 | hostname="knox" 865 | keymap="uk" 866 | ifconfig_ue0="DHCP" 867 | sshd_enable="YES" 868 | 869 | ntpd_enable="YES" 870 | ntpd_sync_on_start="YES" 871 | 872 | powerd_enable="YES" 873 | powerd_flags="-M 800 -m 300 -n hiadaptive -p 3000" 874 | 875 | sendmail_enable="NO" 876 | sendmail_submit_enable="NO" 877 | sendmail_outbound_enable="NO" 878 | sendmail_msp_queue_enable="NO" 879 | 880 | dumpdev="NO" 881 | syslogd_enable="NO" 882 | cron_enable="NO" 883 | 884 | Transatlantic sorts may wish to change the keymap to 'us'. Details of these choices are presented further up in this document, the only difference being that I also disable cron here. 885 | 886 | We'll only ever access this system remotely, so it makes no sense for it to have terminal emulators hanging around in the background, this will also make local attacks more difficult. Replace *mnt/etc/ttys* with an empty file: 887 | 888 | root@local# echo > /mnt/etc/ttys 889 | 890 | Here's */mnt/etc/ssh/sshd_config*, this is detailed earlier in the document. 891 | 892 | HostKey /etc/ssh/ssh_host_ed25519_key 893 | TrustedUserCAKeys /etc/ssh/knox-ca 894 | AllowUsers root 895 | PasswordAuthentication no 896 | PermitRootLogin without-password 897 | UseDNS no 898 | UsePAM no 899 | ChallengeResponseAuthentication no 900 | KexAlgorithms curve25519-sha256@libssh.org 901 | Ciphers chacha20-poly1305@openssh.com 902 | 903 | Now some SSH tasks. Install the fingerprint of the signing key, and generate the host key for the device while we're at it. 904 | 905 | hugh@local$ ssh-keygen -t ed25519 -f ~/.ssh/knox-ca 906 | root@local# cp /usr/home/hugh/.ssh/knox-ca.pub /mnt/etc/ssh/knox-ca 907 | root@local# ssh-keygen -t ed25519 -f /mnt/etc/ssh/ssh_host_ed25519_key # press when prompted for a passphrase 908 | 909 | Note the key fingerprint generated from the above. Lastly, we should set a nameserver and add some entropy. 910 | 911 | root@local# echo "nameserver 8.8.4.4" > /etc/resolv.conf 912 | root@local# dd if=/dev/random of=/mnt/entropy bs=4k count=1 913 | 914 | Last thing is to put the *zfs-receive* script into the *root* user's home folder. Edit */mnt/root/zfs-receive.sh*: 915 | 916 | #!/bin/sh 917 | 918 | geli attach -dpk /tmp/k /dev/da0 919 | zpool import wd 920 | zfs receive -Fu wd 921 | zpool export wd 922 | 923 | Then set the permissions: 924 | 925 | root@local# chmod 744 /mnt/root/zfs-receive.sh 926 | 927 | All done. Let's unmount and write the image. Insert the SD card into the building system and take a look at '*dmesg | tail*' to see what device name it gets. Mine is *mmcsd0*. 928 | 929 | root@local# umount /mnt 930 | root@local# mdconfig -du 0 # where 0 is from the name it gave you, here md0 931 | root@local# dd if=/home/hugh/knox/crochet/work/FreeBSD-armv6-10.2-RPI-B-ZFS-295681M.img of=/dev/mmcsd0 bs=1m 932 | root@local# sync 933 | 934 | You can check the progress of the *dd* operation by typing *ctrl-t*. When it's done, put the SD card in your RPi and boot it up. To connect, we'll need to find out what IP address it's been assigned. Sometimes home routers have a 'connected devices' page that shows the active DHCP leases, if not we can do a quick scan for open SSH ports on the local network. (You may need to install *nmap*). 935 | 936 | hugh@local$ nmap -Pn -p ssh --open / # probably 192.168.1.0/24 937 | 938 | Once you've found the new addition, connect in using a key signed by the *knox-ca* key, as detailed in the main section. Then, go back to the start of this guide, filling in all the blanks. 939 | 940 | One last thing, because ARMv6 isn't a Tier 1 supported architecture, there aren't any binary packages provided by the FreeBSD Foundation. Thankfully, FreeBSD is all about the source code, and the famous [Ports](https://www.freebsd.org/ports/) tree makes it easy to compile your own packages for whatever architecture you have a compiler for. Unfortunately...the RPi is very slow at compiling packages. Being a patient man, I've compiled a few myself that I find useful to use on this system, but I stress that none of these are necessary for the ZFS backup features - the base system has everything needed for that. RPi packages are available [here](https://github.com/hughobrien/zfs-remote-mirror/tree/master/pkg). If you do decide to build some ports, bear in mind that ports tree from *portsnap* is approximately 900MB in size, before you begin to compile anything. [Poudriere](https://www.freebsd.org/doc/handbook/ports-poudriere.html) is an alternative that makes cross-compilation (building on your local machine for the RPi) easier, but I found it as easy to just wait for the RPi. 941 | 942 | To use these packages, grab the *pkg-static* binary from the folder and use that to install *pkg* properly. 943 | 944 | root@knox# ./pkg-static add pkg-1.6.3.txz 945 | 946 | I should also note, that much to my surprise, my simple 1A USB power supply is able to both power the RPi, and the 2TB USB powered drive I attached to it, no powered hub needed - though this may be putting some strain on the RPi's linear regulators. To my further surprise, the choice of USB cable that connects the power supply to the RPi is also significant, lower quality ones seem less able to carry higher amounts of current. 947 | 948 | Congratulations on making it to the end, as a reward, here's a pre-made RPi image file containing almost all of the above modifications. You'll have to install your own CA key, but otherwise it should speed things up quite a bit. Why didn't I mention this earlier? Think how much more you now know! Though this is sized for 4GB cards, it will also work with larger ones, but the extra space won't accessible. 949 | 950 | It's easiest to flash the image directly, then connect in and make the necessary changes, you should also verify the checksum matches the one shown below. 951 | 952 | hugh@local$ prefix="https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/master" 953 | hugh@local$ fetch $prefix/FreeBSD-armv6-10.2-RPI-B-ZFS-295681M.img.xz 954 | hugh@local$ sha256 FreeBSD-armv6-10.2-RPI-B-ZFS-295681M.img.xz 955 | SHA256 (FreeBSD-armv6-10.2-RPI-B-ZFS-295681M.img.xz) = 338490fb8985d0778a5de743b1e8f4b5cac9b30ba3b398e6ffa49937c8a56137 956 | root@local# xzcat FreeBSD-armv6-10.2-RPI-B-ZFS-295681M.img.xz | dd of=/dev/mmcsd0 bs=1m 957 | # transfer to RPi, give it about 30 seconds to boot. 958 | hugh@local$ fetch $prefix/keys/knox-login 959 | hugh@local$ fetch $prefix/keys/knox-login-cert.pub 960 | hugh@local$ chmod 600 knox-login 961 | hugh@local$ ssh -i knox-login root@192.168.1.13 # replace with your assigned IP 962 | 963 | ED25519 key fingerprint will be 8f:a5:94:2e:b5:a2:97:93:ed:71:2c:67:de:a1:32:9c 964 | 965 | Now that you're in, you should replace the ssh host key and the ssh certificate authority key with your own (or else I'll be able to login). These were explained above. You'll have to mount the root file-system as read-write beforehand though: 966 | 967 | root@knox# mount -o rw / 968 | root@knox# cd /etc/ssh/ 969 | root@knox# rm ssh_host_ed25519_key ssh_host_ed25519_key.pub 970 | root@knox# ssh-keygen -t ed25519 -f ssh_host_ed25519_key # don't use any password 971 | # replace /etc/ssh/knox-ca with the public key of your chosen CA key 972 | root@knox# tzsetup # might as well set the timezone now 973 | # maybe edit /etc/rc.conf as you desire, e.g. the keymap 974 | root@knox# mkdir /wd # or whatever you're going to call your pool 975 | root@knox# echo "nameserver 8.8.4.4" > /etc/resolv.conf # Select your nameserver 976 | root@knox reboot 977 | 978 | Now create and sign a login key as described in the main guide. When the system is back up and you attempt to reconnect, you'll get an SSH error about 'remote host identification changed', as indeed it has. Edit your *~/.ssh/known_hosts* file to remove the offending entry (probably the last line in the file). 979 | 980 | With that done, go back to the start of this guide and fill in any missing steps. 981 | 982 | As a final note, it's common to overclock the Raspberry Pi using the */boot/msdos/config.txt* file, simply add the following lines for a moderate speed increase: (More info [here](http://elinux.org/RPiconfig#Overclocking).) 983 | 984 | arm_freq=850 985 | core_freq=300 986 | sdram_freq=400 987 | 988 | However, any overclocking increases the potential for errors or system faults. Increasing the chip frequencies also increases the power draw which puts extra load on the power supply, which in my case is also powering the USB HDD. The benefit of the overclocking is about a 25% CPU boost, the drawback is considerable. I suggest you play it safe and ignore the temptation. 989 | 990 | In the *config.txt* file there's also a line to adjust the memory split between the on-board GPU and the CPU. *u-boot* will default to 32MB, which I suggest you leave it alone. Lowering to 16M in my experience caused boot issues. 991 | 992 | Appendix - ECC Memory 993 | ==================== 994 | It's been pointed out that if ZFS detects a bad checksum while reading, it will always mark the disk as the problem, even if the real fault is a bad memory module. If ZFS then attempts to *correct* it, based on some online redundancy, that correction too may pass through the bad memory thus actually corrupting it. 995 | 996 | A *scrub* operation, which is normally used to check the disks, might end up funnelling all the data through a bad memory module. However, there are a few factors that mitigate this potential disaster: 997 | 998 | * ZFS will take the pool offline if it detects too may errors, thus reducing the fallout. 999 | * ZFS will only attempt to auto-correct the data if it has some redundancy information such as provided by the *copies* parameter or by parity data. We use neither in this setup, though you may optionally enable *copies* on a per-dataset basis. 1000 | 1001 | [Here is a paper](https://research.cs.wisc.edu/adsl/Publications/zfs-corruption-fast10.pdf) which analysed the result of many different types of memory corruption on a running ZFS system. The general conclusion is that while ZFS is not immune to memory issues, more often than not it will crash when it encounters them. Without performing a similar analysis for simpler file-systems, we cannot definitively say whether ZFS handles memory issues better or worse than its contemporaries. 1002 | 1003 | As a general rule, use ECC memory if possible. Though I suspect that if your data were so critical as to require it, you would be using a more dedicated, specialised backup solution. 1004 | 1005 | Here's some more debate on the matter: 1006 | 1007 | * [Ars Technica](http://arstechnica.com/civis/viewtopic.php?f=2&t=1235679&p=26303271#p26303271) 1008 | * [freenas](https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-ram-and-zfs.15449/) 1009 | * [brianmoses](http://blog.brianmoses.net/2014/03/why-i-chose-non-ecc-ram-for-my-freenas.html) 1010 | * [JRS Systems](http://jrs-s.net/2015/02/03/will-zfs-and-non-ecc-ram-kill-your-data/) 1011 | * [Louwrentius] (http://louwrentius.com/please-use-zfs-with-ecc-memory.html) 1012 | 1013 | In short, since this is a remote backup drive, we will always have two copies of our data. Should the backup system fail for any reason, memory or otherwise, we can easily get another one and recreate the backup. It is always better to have two independent copies of your data on systems without ECC memory than one copy on a system with it. And since non-ECC systems are easier to come by, I think it's the better move. 1014 | -------------------------------------------------------------------------------- /backup.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # Hugh O'Brien 2016 - obrien.hugh@gmail.com 3 | 4 | # pipe-viewer isn't technically needed, but it is nice for progress info 5 | [ -z $(which pv) ] && echo "pv not found" && exit 6 | 7 | last_sent_file=~/.knox-last-sent 8 | [ ! -f "$last_sent_file" ] && touch "$last_sent_file" 9 | 10 | latest_remote="$(cat "$last_sent_file")" 11 | [ -z "$latest_remote" ] && echo "remote state unknown. Set it in "$last_sent_file"" 12 | 13 | # This is split out as we need to recalculate it if we make a new snapshot 14 | update_latest_local() { 15 | latest_local="$(zfs list -H -d1 -t snapshot \ 16 | | grep -e '-[0-9][0-9]T[0-9][0-9]:' \ 17 | | cut -f1 \ 18 | | sort \ 19 | | tail -n 1)" 20 | } 21 | update_latest_local 22 | 23 | # pv will only take integer size arguments, so convert them here 24 | calculate_size() { 25 | size=$(zfs send -RevnI "$latest_remote" "$latest_local" 2>&1 \ 26 | | tail -n 1 \ 27 | | sed -E 's/.* ([0-9]*)/\1/') 28 | 29 | unit=$(echo "$size" | sed -E 's/.*([A-Z,a-z])/\1/') 30 | unitless_size=$(echo "$size" | tr -d "$unit") 31 | 32 | case "$unit" in 33 | (K) exponent=10 ;; 34 | (M) exponent=20 ;; 35 | (G) exponent=30 ;; 36 | (T) exponent=40 ;; 37 | esac 38 | 39 | integer_size=$(echo "$unitless_size * 2^$exponent" | bc) 40 | truncated_integer_size=$(echo "$integer_size" \ 41 | | sed -E 's/([0-9].*)\..*/\1/') 42 | 43 | # last test to make sure the data is good, else pv will fail 44 | case "$truncated_integer_size" in 45 | ''|*[!0-9]*) truncated_integer_size=0 ;; 46 | esac 47 | 48 | } 49 | 50 | print_size() { 51 | calculate_size 52 | echo "Estimated Size: $size" 53 | } 54 | 55 | snapshot() { 56 | zfs snapshot -r "wd@$(date -u '+%FT%TZ')" 57 | update_latest_local 58 | } 59 | 60 | send_incremental_snapshot() { 61 | calculate_size 62 | ssh -q knox-fifo < ~/.ssh/knox-geli-key & 63 | sleep 3 64 | zfs send -ReI "$latest_remote" "$latest_local" \ 65 | | pv --size "$truncated_integer_size" \ 66 | | ssh -q knox-send 67 | } 68 | 69 | preview() { 70 | zfs diff "$latest_remote" "$latest_local" | less 71 | } 72 | 73 | backup() { 74 | send_incremental_snapshot && echo "$latest_local" > "$last_sent_file" 75 | } 76 | 77 | case "$1" in 78 | backup) print_size; backup ;; 79 | preview) preview ;; 80 | snapshot) snapshot; print_size ;; 81 | snapback) snapshot; print_size; backup ;; 82 | *) echo "Latest Local: "$latest_local"" 83 | echo "Latest Remote: "$latest_remote"" 84 | print_size; echo 85 | echo "Commands are: snapshot, preview, backup, snapback" 86 | esac 87 | -------------------------------------------------------------------------------- /keys/knox-ca: -------------------------------------------------------------------------------- 1 | -----BEGIN OPENSSH PRIVATE KEY----- 2 | b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW 3 | QyNTUxOQAAACBUJjf+M5YyOftIyClHLIeFuJfFBP4rtghFZIFBiI7T+AAAAJDpU9NV6VPT 4 | VQAAAAtzc2gtZWQyNTUxOQAAACBUJjf+M5YyOftIyClHLIeFuJfFBP4rtghFZIFBiI7T+A 5 | AAAECcm47MvjgT5eOfsyLMo6hoDc9WHL0JNEzhtVDpPWTCG1QmN/4zljI5+0jIKUcsh4W4 6 | l8UE/iu2CEVkgUGIjtP4AAAAB2tub3gtY2EBAgMEBQY= 7 | -----END OPENSSH PRIVATE KEY----- 8 | -------------------------------------------------------------------------------- /keys/knox-login: -------------------------------------------------------------------------------- 1 | -----BEGIN OPENSSH PRIVATE KEY----- 2 | b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW 3 | QyNTUxOQAAACD6Og/fiur9+0v05EAhNk0+c2wDbbmExzLSPrAIeQWXrAAAAJCWHLlQlhy5 4 | UAAAAAtzc2gtZWQyNTUxOQAAACD6Og/fiur9+0v05EAhNk0+c2wDbbmExzLSPrAIeQWXrA 5 | AAAEBROHmiyqYvnx5SKmO45zYO9m5wZH6MK7l+uo3/u6o6Dfo6D9+K6v37S/TkQCE2TT5z 6 | bANtuYTHMtI+sAh5BZesAAAADWh1Z2hAbWFqb3J0b20= 7 | -----END OPENSSH PRIVATE KEY----- 8 | -------------------------------------------------------------------------------- /keys/knox-login-cert.pub: -------------------------------------------------------------------------------- 1 | ssh-ed25519-cert-v01@openssh.com AAAAIHNzaC1lZDI1NTE5LWNlcnQtdjAxQG9wZW5zc2guY29tAAAAIM74OP3lcBPL3uVLxr1p9uPJNvdJMm5w7y6b63T11tswAAAAIPo6D9+K6v37S/TkQCE2TT5zbANtuYTHMtI+sAh5BZesAAAAAAAAAAAAAAABAAAACmtub3gtbG9naW4AAAAIAAAABHJvb3QAAAAAAAAAAP//////////AAAAAAAAAIIAAAAVcGVybWl0LVgxMS1mb3J3YXJkaW5nAAAAAAAAABdwZXJtaXQtYWdlbnQtZm9yd2FyZGluZwAAAAAAAAAWcGVybWl0LXBvcnQtZm9yd2FyZGluZwAAAAAAAAAKcGVybWl0LXB0eQAAAAAAAAAOcGVybWl0LXVzZXItcmMAAAAAAAAAAAAAADMAAAALc3NoLWVkMjU1MTkAAAAgVCY3/jOWMjn7SMgpRyyHhbiXxQT+K7YIRWSBQYiO0/gAAABTAAAAC3NzaC1lZDI1NTE5AAAAQN71Jl7f/T9Ihgh9AB7401hqyyOu/BWPGknMTEP56yGalLj3mrYaFqrboX3313duyFWA+xUEJYVjOXsP/4TkxQ0= knox-login.pub 2 | -------------------------------------------------------------------------------- /keys/knox-login.pub: -------------------------------------------------------------------------------- 1 | ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPo6D9+K6v37S/TkQCE2TT5zbANtuYTHMtI+sAh5BZes 2 | -------------------------------------------------------------------------------- /patches/RPI-B-ZFS: -------------------------------------------------------------------------------- 1 | # 2 | # RPI-B -- Custom configuration for the Raspberry Pi 3 | # 4 | # For more information on this file, please read the config(5) manual page, 5 | # and/or the handbook section on Kernel Configuration Files: 6 | # 7 | # http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html 8 | # 9 | # The handbook is also available locally in /usr/share/doc/handbook 10 | # if you've installed the doc distribution, otherwise always see the 11 | # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the 12 | # latest information. 13 | # 14 | # An exhaustive list of options and more detailed explanations of the 15 | # device lines is also present in the ../../conf/NOTES and NOTES files. 16 | # If you are in doubt as to the purpose or necessity of a line, check first 17 | # in NOTES. 18 | # 19 | # $FreeBSD: releng/10.2/sys/arm/conf/RPI-B 285132 2015-07-04 14:50:32Z gjb $ 20 | 21 | ident RPI-B-ZFS 22 | 23 | include "../broadcom/bcm2835/std.rpi" 24 | 25 | options KSTACK_PAGES=6 # required by ZFS 26 | options HZ=100 27 | options SCHED_4BSD # 4BSD scheduler 28 | options PREEMPTION # Enable kernel thread preemption 29 | options INET # InterNETworking 30 | #options INET6 # IPv6 communications protocols 31 | #options SCTP # Stream Control Transmission Protocol 32 | options FFS # Berkeley Fast Filesystem 33 | options SOFTUPDATES # Enable FFS soft updates support 34 | options UFS_ACL # Support for access control lists 35 | #options UFS_DIRHASH # Improve performance on big directories 36 | options UFS_GJOURNAL # Enable gjournal-based UFS journaling 37 | #options QUOTA # Enable disk quotas for UFS 38 | #options NFSCL # New Network Filesystem Client 39 | #options NFSLOCKD # Network Lock Manager 40 | #options NFS_ROOT # NFS usable as /, requires NFSCL 41 | options MSDOSFS # MSDOS Filesystem 42 | options CD9660 # ISO 9660 Filesystem 43 | options PROCFS # Process filesystem (requires PSEUDOFS) 44 | options PSEUDOFS # Pseudo-filesystem framework 45 | options TMPFS # Efficient memory filesystem 46 | options GEOM_PART_GPT # GUID Partition Tables 47 | options GEOM_PART_BSD # BSD partition scheme 48 | options GEOM_PART_MBR # MBR partition scheme 49 | options GEOM_LABEL # Provides labelization 50 | options COMPAT_43 # Compatible with BSD 4.3 [KEEP THIS!] 51 | options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI 52 | #options KTRACE # ktrace(1) support 53 | options SYSVSHM # SYSV-style shared memory 54 | options SYSVMSG # SYSV-style message queues 55 | options SYSVSEM # SYSV-style semaphores 56 | options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions 57 | #options KBD_INSTALL_CDEV # install a CDEV entry in /dev 58 | options FREEBSD_BOOT_LOADER # Process metadata passed from loader(8) 59 | options VFP # Enable floating point hardware support 60 | 61 | # Debugging for use in -current 62 | #makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols 63 | #options BREAK_TO_DEBUGGER 64 | #options ALT_BREAK_TO_DEBUGGER 65 | #options VERBOSE_SYSINIT # Enable verbose sysinit messages 66 | #options KDB # Enable kernel debugger support 67 | # For minimum debugger support (stable branch) use: 68 | #options KDB_TRACE # Print a stack trace for a panic 69 | # For full debugger support use this instead: 70 | #options DDB # Enable the kernel debugger 71 | #options INVARIANTS # Enable calls of extra sanity checking 72 | #options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS 73 | #options WITNESS # Enable checks to detect deadlocks and cycles 74 | #options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed 75 | #options DIAGNOSTIC 76 | 77 | # NFS root from boopt/dhcp 78 | #options BOOTP 79 | #options BOOTP_NFSROOT 80 | #options BOOTP_COMPAT 81 | #options BOOTP_NFSV3 82 | #options BOOTP_WIRED_TO=ue0 83 | 84 | #options ROOTDEVNAME=\"ufs:mmcsd0s2\" 85 | 86 | device bpf 87 | device loop 88 | device ether 89 | device uart 90 | device pty 91 | device snp 92 | device pl011 93 | 94 | # Comment following lines for boot console on serial port 95 | device vt 96 | device kbdmux 97 | device ukbd 98 | 99 | device sdhci 100 | device mmc 101 | device mmcsd 102 | 103 | device gpio 104 | #device gpioled 105 | 106 | # I2C 107 | device iic 108 | device iicbus 109 | device bcm2835_bsc 110 | 111 | #options KDB 112 | #options DDB # Enable the kernel debugger 113 | #options INVARIANTS # Enable calls of extra sanity checking 114 | #options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS 115 | 116 | device md 117 | device random # Entropy device 118 | 119 | # USB support 120 | device usb 121 | options USB_DEBUG 122 | device dwcotg # DWC OTG controller 123 | 124 | # USB storage support 125 | device scbus 126 | device da 127 | device umass 128 | 129 | # USB ethernet support 130 | device smcphy 131 | device mii 132 | device smsc 133 | 134 | # SPI 135 | device spibus 136 | device bcm2835_spi 137 | 138 | # Flattened Device Tree 139 | options FDT # Configure using FDT/DTB data 140 | # Note: DTB is normally loaded and modified by RPi boot loader, then 141 | # handed to kernel via U-Boot and ubldr. 142 | #options FDT_DTB_STATIC 143 | #makeoptions FDT_DTS_FILE=rpi.dts 144 | makeoptions MODULES_EXTRA="dtb/rpi opensolaris zfs" # ZFS not built by default 145 | -------------------------------------------------------------------------------- /patches/RPI-B.patch: -------------------------------------------------------------------------------- 1 | --- /home/hugh/knox/src/sys/arm/conf/RPI-B 2016-02-09 18:56:16.856914000 +0000 2 | +++ /home/hugh/knox/src/sys/arm/conf/RPI-B-ZFS 2015-12-19 12:43:26.000000000 +0000 3 | @@ -18,25 +18,26 @@ 4 | # 5 | # $FreeBSD: releng/10.2/sys/arm/conf/RPI-B 285132 2015-07-04 14:50:32Z gjb $ 6 | 7 | -ident RPI-B 8 | +ident RPI-B-ZFS 9 | 10 | include "../broadcom/bcm2835/std.rpi" 11 | 12 | +options KSTACK_PAGES=6 # required by ZFS 13 | options HZ=100 14 | options SCHED_4BSD # 4BSD scheduler 15 | options PREEMPTION # Enable kernel thread preemption 16 | options INET # InterNETworking 17 | -options INET6 # IPv6 communications protocols 18 | -options SCTP # Stream Control Transmission Protocol 19 | +#options INET6 # IPv6 communications protocols 20 | +#options SCTP # Stream Control Transmission Protocol 21 | options FFS # Berkeley Fast Filesystem 22 | options SOFTUPDATES # Enable FFS soft updates support 23 | options UFS_ACL # Support for access control lists 24 | -options UFS_DIRHASH # Improve performance on big directories 25 | +#options UFS_DIRHASH # Improve performance on big directories 26 | options UFS_GJOURNAL # Enable gjournal-based UFS journaling 27 | -options QUOTA # Enable disk quotas for UFS 28 | -options NFSCL # New Network Filesystem Client 29 | -options NFSLOCKD # Network Lock Manager 30 | -options NFS_ROOT # NFS usable as /, requires NFSCL 31 | +#options QUOTA # Enable disk quotas for UFS 32 | +#options NFSCL # New Network Filesystem Client 33 | +#options NFSLOCKD # Network Lock Manager 34 | +#options NFS_ROOT # NFS usable as /, requires NFSCL 35 | options MSDOSFS # MSDOS Filesystem 36 | options CD9660 # ISO 9660 Filesystem 37 | options PROCFS # Process filesystem (requires PSEUDOFS) 38 | @@ -48,27 +49,27 @@ 39 | options GEOM_LABEL # Provides labelization 40 | options COMPAT_43 # Compatible with BSD 4.3 [KEEP THIS!] 41 | options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI 42 | -options KTRACE # ktrace(1) support 43 | +#options KTRACE # ktrace(1) support 44 | options SYSVSHM # SYSV-style shared memory 45 | options SYSVMSG # SYSV-style message queues 46 | options SYSVSEM # SYSV-style semaphores 47 | options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions 48 | -options KBD_INSTALL_CDEV # install a CDEV entry in /dev 49 | +#options KBD_INSTALL_CDEV # install a CDEV entry in /dev 50 | options FREEBSD_BOOT_LOADER # Process metadata passed from loader(8) 51 | options VFP # Enable floating point hardware support 52 | 53 | # Debugging for use in -current 54 | -makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols 55 | -options BREAK_TO_DEBUGGER 56 | -options ALT_BREAK_TO_DEBUGGER 57 | +#makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols 58 | +#options BREAK_TO_DEBUGGER 59 | +#options ALT_BREAK_TO_DEBUGGER 60 | #options VERBOSE_SYSINIT # Enable verbose sysinit messages 61 | -options KDB # Enable kernel debugger support 62 | +#options KDB # Enable kernel debugger support 63 | # For minimum debugger support (stable branch) use: 64 | #options KDB_TRACE # Print a stack trace for a panic 65 | # For full debugger support use this instead: 66 | -options DDB # Enable the kernel debugger 67 | -options INVARIANTS # Enable calls of extra sanity checking 68 | -options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS 69 | +#options DDB # Enable the kernel debugger 70 | +#options INVARIANTS # Enable calls of extra sanity checking 71 | +#options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS 72 | #options WITNESS # Enable checks to detect deadlocks and cycles 73 | #options WITNESS_SKIPSPIN # Don't run witness on spinlocks for speed 74 | #options DIAGNOSTIC 75 | @@ -100,15 +101,15 @@ 76 | device mmcsd 77 | 78 | device gpio 79 | -device gpioled 80 | +#device gpioled 81 | 82 | # I2C 83 | device iic 84 | device iicbus 85 | device bcm2835_bsc 86 | 87 | -options KDB 88 | -options DDB # Enable the kernel debugger 89 | +#options KDB 90 | +#options DDB # Enable the kernel debugger 91 | #options INVARIANTS # Enable calls of extra sanity checking 92 | #options INVARIANT_SUPPORT # Extra sanity checks of internal structures, required by INVARIANTS 93 | 94 | @@ -140,4 +141,4 @@ 95 | # handed to kernel via U-Boot and ubldr. 96 | #options FDT_DTB_STATIC 97 | #makeoptions FDT_DTS_FILE=rpi.dts 98 | -makeoptions MODULES_EXTRA=dtb/rpi 99 | +makeoptions MODULES_EXTRA="dtb/rpi opensolaris zfs" # ZFS not built by default 100 | -------------------------------------------------------------------------------- /patches/cpuvar.h.patch: -------------------------------------------------------------------------------- 1 | --- cpuvar.h 2016-02-10 10:25:12.157361000 +0000 2 | +++ src/sys/cddl/compat/opensolaris/sys/cpuvar.h 2016-02-10 09:14:06.047361000 +0000 3 | @@ -47,7 +47,6 @@ 4 | 5 | /* Some code may choose to redefine this if pcpu_t would be more useful. */ 6 | #define cpu_t solaris_cpu_t 7 | -#define cpu_id cpui 8 | 9 | extern solaris_cpu_t solaris_cpu[]; 10 | 11 | -------------------------------------------------------------------------------- /patches/setup.sh.patch: -------------------------------------------------------------------------------- 1 | --- setup.sh 2016-02-10 10:19:58.159262000 +0000 2 | +++ /home/hugh/knox/crochet/board/RaspberryPi/setup.sh 2016-02-10 10:09:04.606991000 +0000 3 | @@ -40,6 +40,7 @@ 4 | disk_partition_mbr 5 | # Raspberry Pi boot loaders require FAT16, so this must be at least 17m 6 | disk_fat_create 17m 16 7 | + disk_ufs_create 3000m 8 | disk_ufs_create 9 | } 10 | 11 | -------------------------------------------------------------------------------- /pkg/autoconf-2.69.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/autoconf-2.69.txz -------------------------------------------------------------------------------- /pkg/autoconf-wrapper-20131203.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/autoconf-wrapper-20131203.txz -------------------------------------------------------------------------------- /pkg/automake-1.15_1.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/automake-1.15_1.txz -------------------------------------------------------------------------------- /pkg/automake-wrapper-20131203.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/automake-wrapper-20131203.txz -------------------------------------------------------------------------------- /pkg/ca_root_nss-3.22.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/ca_root_nss-3.22.txz -------------------------------------------------------------------------------- /pkg/curl-7.46.0_1.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/curl-7.46.0_1.txz -------------------------------------------------------------------------------- /pkg/dejagnu-1.5.1_1.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/dejagnu-1.5.1_1.txz -------------------------------------------------------------------------------- /pkg/expat-2.1.0_3.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/expat-2.1.0_3.txz -------------------------------------------------------------------------------- /pkg/expect-5.45_2.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/expect-5.45_2.txz -------------------------------------------------------------------------------- /pkg/fusefs-libs-2.9.5.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/fusefs-libs-2.9.5.txz -------------------------------------------------------------------------------- /pkg/fusefs-sshfs-2.5.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/fusefs-sshfs-2.5.txz -------------------------------------------------------------------------------- /pkg/gettext-runtime-0.19.6.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/gettext-runtime-0.19.6.txz -------------------------------------------------------------------------------- /pkg/gettext-tools-0.19.6.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/gettext-tools-0.19.6.txz -------------------------------------------------------------------------------- /pkg/glib-2.44.1_3.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/glib-2.44.1_3.txz -------------------------------------------------------------------------------- /pkg/gmake-4.1_2.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/gmake-4.1_2.txz -------------------------------------------------------------------------------- /pkg/gmake-lite-4.1_1.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/gmake-lite-4.1_1.txz -------------------------------------------------------------------------------- /pkg/help2man-1.43.3_1.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/help2man-1.43.3_1.txz -------------------------------------------------------------------------------- /pkg/indexinfo-0.2.4.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/indexinfo-0.2.4.txz -------------------------------------------------------------------------------- /pkg/libevent2-2.0.22_1.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/libevent2-2.0.22_1.txz -------------------------------------------------------------------------------- /pkg/libffi-3.2.1.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/libffi-3.2.1.txz -------------------------------------------------------------------------------- /pkg/libiconv-1.14_9.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/libiconv-1.14_9.txz -------------------------------------------------------------------------------- /pkg/libtool-2.4.6.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/libtool-2.4.6.txz -------------------------------------------------------------------------------- /pkg/m4-1.4.17_1,1.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/m4-1.4.17_1,1.txz -------------------------------------------------------------------------------- /pkg/mosh-1.2.5.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/mosh-1.2.5.txz -------------------------------------------------------------------------------- /pkg/nmap-7.01.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/nmap-7.01.txz -------------------------------------------------------------------------------- /pkg/p5-Locale-gettext-1.06.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/p5-Locale-gettext-1.06.txz -------------------------------------------------------------------------------- /pkg/pcre-8.37_4.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/pcre-8.37_4.txz -------------------------------------------------------------------------------- /pkg/perl5-5.20.3_8.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/perl5-5.20.3_8.txz -------------------------------------------------------------------------------- /pkg/pkg-1.6.3.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/pkg-1.6.3.txz -------------------------------------------------------------------------------- /pkg/pkg-static: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/pkg-static -------------------------------------------------------------------------------- /pkg/pkgconf-0.9.12_1.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/pkgconf-0.9.12_1.txz -------------------------------------------------------------------------------- /pkg/protobuf-2.6.1.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/protobuf-2.6.1.txz -------------------------------------------------------------------------------- /pkg/python27-2.7.11_1.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/python27-2.7.11_1.txz -------------------------------------------------------------------------------- /pkg/python35-3.5.1_1.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/python35-3.5.1_1.txz -------------------------------------------------------------------------------- /pkg/rsync-3.1.2_1.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/rsync-3.1.2_1.txz -------------------------------------------------------------------------------- /pkg/tcl86-8.6.4.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/tcl86-8.6.4.txz -------------------------------------------------------------------------------- /pkg/tmux-2.1_1.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/tmux-2.1_1.txz -------------------------------------------------------------------------------- /pkg/tor-0.2.6.10_1.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/tor-0.2.6.10_1.txz -------------------------------------------------------------------------------- /pkg/vim-lite-7.4.1030.txz: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/hughobrien/zfs-remote-mirror/9174fadbe9a7f9b060c85f31691f3ed3a5adf2d4/pkg/vim-lite-7.4.1030.txz -------------------------------------------------------------------------------- /top: -------------------------------------------------------------------------------- 1 | last pid: 722; load averages: 2.48, 2.42, 2.37 up 0+01:07:58 17:48:03 2 | 36 processes: 4 running, 31 sleeping, 1 waiting 3 | 4 | Mem: 10M Active, 10M Inact, 93M Wired, 13M Buf, 349M Free 5 | ARC: 24M Total, 18M MRU, 6544K Anon, 53K Header, 94K Other 6 | Swap: 702M Total, 702M Free 7 | 8 | 9 | PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAND 10 | 674 root 1 63 - 0K 6584K RUN 11:54 31.40% [g_eli[0] da0] 11 | 667 root 1 62 0 18648K 7588K RUN 11:25 25.54% sshd: root@notty (sshd) 12 | 14 root 5 -68 - 0K 6680K - 6:51 14.75% [usb] 13 | 677 root 1 -8 0 11780K 3104K piperd 1:39 2.83% zfs receive -Fu wd 14 | 0 root 139 -8 0 0K 9872K - 3:28 2.54% [kernel] 15 | 11 root 24 -84 - 0K 7136K WAIT 1:21 1.46% [intr] 16 | 13 root 1 -16 - 0K 6584K - 0:34 0.68% [rand_harvestq] 17 | 12 root 3 -8 - 0K 6632K - 0:18 0.00% [geom] 18 | 3 root 6 -16 - 0K 6712K tx->tx 0:12 0.00% [zfskern] 19 | 2 root 2 -8 - 0K 6608K - 0:09 0.00% [cam] 20 | 504 root 2 40 0 13352K 13288K usem 0:05 0.00% /usr/sbin/ntpd -g -c /etc/ntp.conf -p 21 | 16 root 1 -16 - 0K 6584K - 0:02 0.00% [schedcpu] 22 | 618 root 1 40 0 18648K 6244K select 0:02 0.00% sshd: root@pts/0 (sshd) 23 | 296 root 1 40 0 9336K 1060K select 0:01 0.00% /sbin/devd 24 | 53 root 1 40 - 0K 6584K geli:w 0:00 0.00% [g_eli[0] mmcsd0s3] 25 | 17 root 1 8 0 10856K 2184K wait 0:00 0.00% sh /etc/rc autoboot 26 | 620 root 1 16 0 11332K 3268K pause 0:00 0.00% -csh (csh) 27 | 507 root 1 40 0 10368K 976K select 0:00 0.00% /usr/sbin/powerd -M 800 -m 300 -n hiad 28 | 29 | --------------------------------------------------------------------------------