├── containers ├── img │ ├── moverTuning.png │ └── site-rules.png ├── MariaDbAndMySql.md ├── ombi.md ├── sonarrRadarrLidarr.md ├── postgres.md └── nextcloud.md ├── general ├── DockerImageOnZFS.md ├── commonIssues.md ├── WhyZfsOnUnRAID.md └── helpfulCommands.md └── README.md /containers/img/moverTuning.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/teambvd/UnRAID-Performance-Compendium/HEAD/containers/img/moverTuning.png -------------------------------------------------------------------------------- /containers/img/site-rules.png: -------------------------------------------------------------------------------- https://raw.githubusercontent.com/teambvd/UnRAID-Performance-Compendium/HEAD/containers/img/site-rules.png -------------------------------------------------------------------------------- /containers/MariaDbAndMySql.md: -------------------------------------------------------------------------------- 1 | ## MariaDB / MySQL 2 | 3 | ### THIS IS STILL UNDER CONSTRUCTION!!! I've probably only got this 30% completed; there's no harm in using what's in here, just note, it's not "done" yet. 4 | 5 | ### Assumptions 6 | 7 | * All zpools on host included redundancy (i.e no unmirrored stripes, so raidz/mirror) 8 | * Based on the [LSIO MariaDB](https://github.com/linuxserver/docker-mariadb) container (applies to any maria/mysql, just the location of config files may differ otherwise) 9 | * (and of course, that you've enough system memory to allow for such additional 10 | 11 | #### Host changes 12 | 13 | * Edit your go file (nano /boot/config/go) adding: 14 | 15 | ```bash 16 | echo 1 /sys/module/zfs/parameters/zfs_txg_timeout 17 | ``` 18 | 19 | (note assumptions section: all zpools should be redundant setting the above) 20 | 21 | #### Creating ZFS dataset tuned for MySQL/Maria #### 22 | 23 | ```bash 24 | zfs create wd/dock/mariadb -o recordsize=16K -o logbias=throughput -o primarycache=metadata -o atime=off -o xattr=sa -o compression=zstd-3 25 | ``` 26 | 27 | [**Explanation**](https://shatteredsilicon.net/blog/2020/06/05/mysql-mariadb-innodb-on-zfs/) 28 | * `Recordsize` - InnoDB writes in 16KB chunks 29 | * `Logbias` - recommendation from oracle re: [databases on ZFS](https://docs.oracle.com/cd/E19253-01/819-5461/givdo/index.html) 30 | * `Primarycache` - the DB does it's own caching ('buffer pool'), so we should only cache filesystem metadata, not 'everything' 31 | * `aTime` - updating metadata every time a file is accessed is, in nearly all cases, absurd. Should be disabled as a general rule in ZFS unless you've a very specific need for it 32 | * `xAttr` - If you're on Linux, you should frankly have this configured at the pool level as it should be set for all (otherwise you'll likely encounter severe issues if you ever do anything like NFS or SMB access, at a minimum) 33 | * `Compression` - zstd-3 is 'cheap' enough (resource-wise) that I feel it should be the standard when running any modern-ish CPU (at least for server use... there are caveats for workstations etc) 34 | 35 | #### Creating your container #### 36 | 37 | Example env variables: 38 | 39 | ```yaml 40 | mariadb: 41 | container_name: mariadb 42 | image: mariadb 43 | ports: 44 | - 3306:3306 45 | - 3307:3307 46 | volumes: 47 | - /opt/mariadb:/var/lib/mysql 48 | environment: 49 | MYSQL_ROOT_PASSWORD: yourrootpasswordhere 50 | MYSQL_USER: homeassistant 51 | MYSQL_PASSWORD: yourstrongpasswordhere 52 | MYSQL_DATABASE: homeassistant 53 | restart: on-failure 54 | ``` 55 | 56 | #### MariaDB Tuning that's specific to use on ZFS 57 | 58 | **_Explain each: why we're disabling various maria data 'safety' features, etc_** 59 | 60 | ```bash 61 | innodb_doublewrite=0 62 | innodb_checksum_algorithm=crc32 63 | innodb_flush_neighbors=0 64 | innodb_use_native_aio=0 65 | innodb_use_atomic_writes=0 66 | innodb-compression-algorithm=none 67 | ``` 68 | 69 | #### Tuning MariaDB Generally 70 | 71 | **_Run diff on currently running custom.cnf against base my.cnf, explain where necessary, and note that user shouldn't blindly copy/pasta from "some guy on the internet" without confirming with their own research_** 72 | -------------------------------------------------------------------------------- /general/DockerImageOnZFS.md: -------------------------------------------------------------------------------- 1 | ## Run your docker image on ZFS using zvols 2 | 3 | I recommend zvols (equivalent to virtual disks) over filesets for the docker image data for a variety of reasons - it'll be more performant at reduced resource cost (more efficient), integrates more smoothly with unraid (imo - less differences from the norm at the very least), and several others. Only thing you lose by doing so is the ability to individually snapshot specific containers; but containers are supposed to be disposable, with everything needing persistence mapped to volumes anyway, so what's the loss there right? 4 | 5 | Lets get started: 6 | ```bash 7 | # creating a 40GB zvol named 'imageLocation' nested under wd/docker 8 | zfs create -V 40G wd/docker/imageLocation 9 | 10 | # now partition the disk as GPT 11 | gdisk /dev/wd/docker/imageLocation 12 | 13 | # Check the physical block size with the 'p' option, then 'o' to create a new partition, then 'w' to apply and exit 14 | Command (? for help): p 15 | Disk /dev/wd/docker/imageLocation: 83886080 sectors, 40960.0 MiB 16 | Sector size (logical/physical): 512/8192 bytes... 17 | ... 18 | Command (? for help): o 19 | This option deletes all partitions and creates a new protective MBR. 20 | Proceed? (Y/N): Y 21 | 22 | Command (? for help): w 23 | 24 | Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING 25 | PARTITIONS!! 26 | 27 | Do you want to proceed? (Y/N): Y 28 | OK; writing new GUID partition table (GPT) to /dev/wd/docker/imageLocation. 29 | The operation has completed successfully. 30 | 31 | # You should now see a new zvol created listed both directly in dev as zd0, as well as several other places - ex: 32 | ls /dev/zvol/wd/portvols/ 33 | zvoltest@ 34 | 35 | 36 | # Now format as xfs, while optionally specifying the blocksize a small tunable change optimizing this filesystem for our docker image data - the default is 4K, but some SSD's may be 8k native. While can technically choose btrfs (among others), xfs is far more performant for high IOPs workloads. The '-f' option is also available must be before '-q' to force mkfs to ignore warnings 37 | mkfs.xfs -b size=8k -d cowextsize=64 -q /dev/zd0 38 | 39 | # Create a place to mount this disk - recommend you use somwhere under /mnt 40 | sudo mkdir /mnt/dockerImage 41 | 42 | # Finally, use configure 43 | mount /dev/zd0 /mnt/dockerImage 44 | ``` 45 | 46 | ## Common issues 47 | 48 | ### Cannot Set Blocksize 49 | 50 | Should you see an error like the below even after confirming your disks physical size: 51 | ```bash 52 | mkfs.xfs: error - cannot set blocksize 8192 on block device /dev/tank/imageLocation: Invalid argument 53 | ``` 54 | 55 | This means your zpool's ashift value was set to something smaller than the blocksize you're trying to configure. The common recommendation of "everyone should use `ashift=12`" means this will be a frequent occurrence (hopefully people actually check their physical block sizes before blindly following some random guy on the internet, right? ...RIGHT!?) 56 | 57 | If that's the case, no big deal (and not a massive loss really for our use case here), just set it to 4k instead (for `ashift=12` at least). 58 | 59 | ### mkfs fails with "appears to contain an existing filesystem" 60 | 61 | You forgot the `-f` option. Use this carefully - if you have no other zvols, no big concern, but otherwise, triple-check before proceeding. 62 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # UnRAID + ZFS + This = Performance Bliss! (...Hopefully) 2 | 3 | **A collection of documentation I've written up surrounding UnRAID's use while running the ZFS filesystem** 4 | 5 | My 9-5 is essentially "this is slow, make fast" for a data management/security company, and since basically every application out there has **some** persistent data to be managed, some OS/hardware to run it, and some way to connect to it, I end up knowing just a liiiiittle bit (enough to cause problems at least!) about a whole lot. When I started homelabbing, I tended to apply the same methodologies gained from that time at work. 6 | 7 | Performance lives on a continuum of "This straight up doesn't work" on down to "I can't find anything to complain about", with what's actually technically possible being somewhere in the neighborhood of "It's good enough that I probably wouldn't complain"; as applications have to run on a variety of hardware, it's almost never possible to set something up as a 'one size fits all' from a performance perspective. Performance tuning, then, is simply working towards the goal of configuring that application towards achieving or exceeding that "wouldn't complain" point while running in your specific infrastructure. 8 | 9 | This will never be exhaustive, conclusive, or even finished (lol)... But with any luck, it'll give you what you need to get started. This will grow as I find time to convert my servers' config modification log/notes to something usable by more than just myself hehehe: 10 | 11 | ### Index 12 | * General 13 | * [Why would we want ZFS on UnRAID? What can we do with it?](https://github.com/teambvd/UnRAID-Performance-Compendium/blob/main/general/WhyZfsOnUnRAID.md) 14 | * [SR-IOV on UnRAID](https://forums.unraid.net/topic/103323-how-to-using-sr-iov-in-unraid-with-1gb10gb40gb-network-interface-cards-nics/) 15 | * [Common issues/questions/general information related to ZFS on UnRAID](https://github.com/teambvd/UnRAID-Performance-Compendium/blob/main/general/commonIssues.md) 16 | * [Hosting the Docker Image on ZFS](https://github.com/teambvd/UnRAID-Performance-Compendium/blob/main/general/DockerImageOnZFS.md) 17 | * [NFS - to be documented] 18 | * [Virtual Machines] 19 | * [Setting up various tools and scripts for monitoring and improved server mgmt quality of life] 20 | * [Installed tools and apps outside the ecosystem, and integrating them into UnRAID (cleanly)] 21 | * Container Specific 22 | * [Ombi](https://github.com/teambvd/unraid-zfs-docs/blob/main/containers/ombi.md) 23 | * [Sonarr/Radarr/Lidarr - Anything with SQLite](https://github.com/teambvd/unraid-zfs-docs/blob/main/containers/sonarrRadarrLidarr.md) 24 | * [Nextcloud - (in progress)](https://github.com/teambvd/unraid_docs-ZFS_and_Containers/blob/main/containers/nextcloud.md) 25 | * [Postgres](https://github.com/teambvd/unraid-zfs-docs/blob/main/containers/postgres.md) 26 | * [MariaDB - (in progress)](https://github.com/teambvd/UnRAID-Performance-Compendium/blob/main/containers/MariaDbAndMySql.md) 27 | * [OpenLDAP] 28 | * [LAM - LDAP Account Manager] 29 | * [Authentik] 30 | * [PWM] 31 | * [ElasticSearch] 32 | * [InfluxDB/Telegraf/Prometheus/Grafana] 33 | * [ELK stack using Elastic from above] 34 | * [Home Assistant] 35 | * [Frigate] 36 | * [Doube take] 37 | * [Compreface] 38 | * [Deepstack] 39 | * [MQTT] 40 | * Assorted stuff (that doesn't fit anywhere else) 41 | * [SMB on UnRAID](https://forums.unraid.net/topic/97165-smb-performance-tuning/) 42 | * [Compiled commands reference](https://github.com/teambvd/UnRAID-Performance-Compendium/blob/main/general/helpfulCommands.md) 43 | * 44 | 45 | -------------------------------------------------------------------------------- /containers/ombi.md: -------------------------------------------------------------------------------- 1 | # ***Ombi on ZFS*** 2 | 3 | ### !!! FYI - I've moved everything over to Jellyseer at this point as the UI is **FAR** more responsive than Ombi - even with postgres+ombi, it's easily orders of magnitude more efficient. This doc is remaining for reference, but may grow outdated as time moves on !!! 4 | 5 | ## "Ombi's UI just doesn't seem to work anymore..." 6 | 7 | Many have noticed that over time and as your media library continues to grow, you eventually get to the point that it just seems Ombi's no longer fit for purpose. Browsing the UI is fine, but then when your users try to make any new requests, they just never get saved, or perhaps parts of it get saved (e.g. your manually added sonarr 'custom parameters', but not the request approval itself), and so on. 8 | 9 | It's this that starts people searching for either an alternative application, or if they're up for troubleshooting, to finding [this page](https://docs.ombi.app/guides/migrating-databases/) which walks them through converting everything over to MariaDB... 10 | 11 | Don't get me wrong, MySQL is great and all, but that's like proposing the use of a telescoping crane usually reserved for mining work to move a box that's just a little too heavy to lift by hand. 12 | 13 | ## Figuring out the default config to get a baseline 14 | 15 | Let's take a look at our persistent data directory (usually appdata for UnRAID users) to see what we have on hand: 16 | 17 | ```bash 18 | root@server:/ombi# ll 19 | total 1.8M 20 | drwxrw-rw- 7 nobody users 10 Apr 17 10:58 ./ 21 | drwxrw-rw- 48 nobody users 48 Apr 16 21:38 ../ 22 | drwxr-xr-x 3 nobody users 3 Nov 17 10:20 .aspnet/ 23 | drwxr-xr-x 3 nobody users 3 Nov 17 13:39 .dotnet/ 24 | drwxr-xr-x 2 nobody users 33 Apr 17 04:23 Logs/ 25 | -rw-r--r-- 1 nobody users 1.3M Apr 17 06:00 Ombi.db 26 | -rw-r--r-- 1 nobody users 3.4M Apr 17 10:15 OmbiExternal.db 27 | -rw-r--r-- 1 nobody users 24K Mar 1 13:21 OmbiSettings.db 28 | drwxr-xr-x 2 root root 2 Jun 24 2021 custom-cont-init.d/ 29 | drwxr-xr-x 2 root root 2 Jun 24 2021 custom-services.d/ 30 | ``` 31 | 32 | Three separate DB's (information on their individual uses can be found [here](https://docs.ombi.app/info/faq/#database-uses)) - checking their config: 33 | 34 | ```sql 35 | root@server:/ombi# sqlite3 Ombi.db 'pragma journal_mode;' 36 | delete 37 | root@server:/ombi# sqlite3 Ombi.db 'pragma synchronous;' 38 | 2 39 | root@server:/ombi# sqlite3 Ombi.db 'pragma page_size;' 40 | 4096 41 | ``` 42 | 43 | Delete is the absolute slowest of the options, and we're doing this as a fully synchronous operation, each action waiting on the outcome of the last to be committed fully, each time we do *anything* in the application other than browsing pages. For reference, here's the application's workflow for newly added requests: 44 | 45 | 46 | 47 | Now imagine you're requesting an entire series of episodes, 10 seasons with 20 episodes each. Each one of those episodes has to be tracked individually in the DB, and we've only got 4KB of memory to work with - looking at the above workflow, you can imagine how many database actions this could add up to... Each time that 4KB fills, we have to commit, each time we have an episode to add to the db we have to commit, and every time we complete, we have to delete the 'log' that said we had a commit to do. 48 | 49 | It's no wonder the database completely locks up once you've got multiple users and thousands of items in your media library! 50 | 51 | ## You don't need a crane, just a dolly 52 | 53 | SQLite's more than capable of handling this as long as we give it the proper resources it needs to do so. 54 | 55 | We'll use the WAL (Write-Ahead Log) logging mechanism instead of TRUNCATE as it's more performant than both `truncate` and `delete` and we don't really care about the additional files being generated additional files generated as we would with WAL, as well as bumping the page size to max to minimize our number of unnecessary writes: 56 | 57 | ```sql 58 | # Do this for each of the three .db files 59 | sqlite3 Ombi.db 'pragma page_size=65536; pragma journal_mode=wal; VACUUM;' 60 | ``` 61 | 62 | (this setting doesn't stick and can be ignored for now) - Just like with Sonarr, we change sync to 'most' as we depend on ZFS for db consistency - helps lower the number of blocking ops on the database: 63 | ```shell 64 | sqlite3 Ombi.db 'pragma synchronous = NORMAL;' 65 | ``` 66 | 67 | ## Filesystem settings recommendations 68 | See [Arr's on ZFS](https://github.com/teambvd/unraid-zfs-docs/main/containers/sonarrRadarrLidarr.md) 69 | -------------------------------------------------------------------------------- /containers/sonarrRadarrLidarr.md: -------------------------------------------------------------------------------- 1 | # ***Sonarr / Lidarr / Radarr on ZFS*** 2 | 3 | ## Reference information 4 | 5 | (My 'workloads'): 6 | 7 | > * Sonarr - `259MB` - 31,689 Episodes 8 | > * Radarr - `115MB` - 2460 Movies 9 | > * Lidarr - `422MB` - 34,581 Songs 10 | 11 | 12 | ## Filesystem settings recommendations: 13 | 14 | * Recordsize - `64K` - This'll make more sense when we get to tuning the DB 15 | * Compression - `zstd-3` - I'd used `lz4` since opensolaris was released, but z-std has made me a believer - 2.57 compressratio, no noticeable performance differnce (on a modern CPU at least). Standard recommendation for all non-media datasets imo. 16 | * Primarycache - `none` - the DB does it's own caching, so this would be duplicating resources, wasting them; normally you'd set this to `metadata` for a database, but sqlite's so small, there's limited gains (if any) from doing so, especially on NVME; again, more on this later 17 | * atime - `off` - another 'do this for everything' setting... except, actually for everything, media or otherwise 18 | * logbias - `throughput` - again, databases 19 | * xattr - `sa` - recommended for all ZoL datasets 20 | 21 | Note: These settings (in general) should be set at creation time. Modifying a setting doesn't change existing data and only applies to newly created data placed within that dataset. If data already exists, it's best to migrate that data to a newly created dataset with the appropriate settings. **DO NOT** make changes to an existing dataset while the data is being actively accessed (unless you know what you're doing). 22 | 23 | ## Application tuning 24 | 25 | The fileset's been configured this way forever, and things had been humming along swimmingly. As times gone on and my library has grown, I've noticed my Sonarr instance had been getting incrementally slower over time. It used to be that when I first loaded the UI, it'd show up within ~2-3 seconds, it'd now take twice that or more. 26 | 27 | The biggest tell though was when I'd go to the History tab, usually to try to sort out what'd happened with a given series (e.g. an existing series was for some reason updated with a quality I didn't expect, or maybe an incorrect episode was pulled, etc.). It'd take ~10+ seconds to load the first set of 20 most recent activities, then close to that for each subsequent page. 28 | 29 | ### Finding the Cause 30 | 31 | I won't bore you too much with the details, but a trace revealed that the majority of the time spent was in waiting on a query response from the database, sqlite. Looking at the database size: 32 | ```bash 33 | 259M Mar 23 06:59 sonarr.db 34 | ``` 35 | 36 | Fairly large for an SQLite DB. Vacuum brought no tangible benefit either. Next I checked the page size: 37 | ```bash 38 | sqlite3 sonarr.db 'pragma page_size;' 39 | 4096 40 | ``` 41 | 42 | We're only able to act on 4KB of transaction data at a time? Bingo. 43 | 44 | ### The Solution 45 | 46 | All the 'arr's use an sqlite backend - you'll need to acquire sqlite3 from the the Dev Pack, if not already installed. First, we need to verify the data base is healthy - **MAKE SURE YOUR CONTAINER IS STOPPED BEFORE PROCEEDING FURTHER**: 47 | ```bash 48 | sqlite3 sonarr.db 'pragma integrity_check;' 49 | ``` 50 | 51 | This can take awhile, so be patient. If this returns anything other than `OK`, then stop here and either work on restoring from backup or repairing the DB. 52 | 53 | Without boring you to death (hopefully), SQLite essentially treats the page_size as the upper bound for it's DB cache. You can instead set the 'cache_size', but this setting is lost each time you disconnect from the DB (e.g. container reboot), so that's kinda worthless for our needs. We're setting our to the maximum, `64KB`: 54 | ```bash 55 | sqlite3 sonarr.db 'pragma page_size=65536; pragma journal_mode=truncate; VACUUM;' 56 | ``` 57 | 58 | This might take a bit, depending on the size of your sonarr.db and transaction log files. Once it's done, we need to re-enable `WAL`: 59 | ```bash 60 | sqlite3 sonarr.db 'pragma journal_mode=WAL; 61 | ``` 62 | 63 | 64 | 65 | ***WORK IN PROGRESS / TBD - has to be done on a per connection basis and isn't persistent, which means this requires container modification*** 66 | 67 | Most people don't 'notice' slowness related to commits to the DB, but if you've ever seen the UI sort of spaz out whenever you've added some new series with 18 seasons at 20+ episodes per, you've encountered it: 68 | 69 | ```shell 70 | sqlite3 sonarr.db 'pragma synchronous;' 71 | 2 72 | ``` 73 | 74 | The default setting uses both [WAL](https://sqlite.org/wal.html) (Write-Ahead Log) as well as full sync, which double-commits transactions; this helps protect the database from inconsistency in the event of an unexpected failure at the OS level and can also slow down reads. Many OS's/filesystems (cough **windows** cough) treat in-memory data as disposable, neglecting the fact that a file may be a database with in-flight writes inside. 75 | 76 | Given ZFS already checksums and syncs data itself (whole other topic), I'm a little less concerned about the database protecting itself and simply rely on the filesystem for this - if the DB becomes corrupted on ZFS, I've got bigger problems to worry about than rebuilding sonarr/radarr/lidarr. We can set this to 'normal' mode, for single commit operation: 77 | 78 | ```shell 79 | sqlite3 sonarr.db 'pragma synchronous = Normal;' 80 | ``` 81 | 82 | 83 | 84 | References: 85 | * [SQLite Page Sizes](https://www.sqlite.org/pragma.html#pragma_page_size) 86 | * [WAL vs Journal in SQLite](https://www.sqlite.org/wal.html) 87 | * [Previously, on Sonarr Dev](https://github.com/Sonarr/Sonarr/issues/2080#issuecomment-318859070) 88 | -------------------------------------------------------------------------------- /general/commonIssues.md: -------------------------------------------------------------------------------- 1 | ## Common Issues 2 | 3 | **Table of Contents** 4 | - [Game stutters when using ZFS on UnRAID](#game-stutters-when-using-zfs-on-unraid) 5 | - [My zpool is busy but I cant tell why](#my-zpool-is-busy-but-i-cant-tell-why) 6 | * [Customizing UnRAIDs CLI/shell, and running commands natively](#customizing-unraids-cli-and-running-commands-natively) 7 | - [Unable to modify filesets status](#unable-to-modify-filesets-status) 8 | * [Sledgehammer approach](#sledgehammer-approach) 9 | * [Dataset is busy](#dataset-is-busy) 10 | * [Dealing with PID Whackamole](#dealing-with-pid-whackamole) 11 | 12 | ### Game stutters when using ZFS on UnRAID 13 | 14 | In addition to all the '[normal](https://www.youtube.com/watch?v=miYUGWq6l24)' things you should [go through](https://www.youtube.com/watch?v=A2dkrFKPOyw) in order to ensure a [low latency gaming](https://www.youtube.com/watch?v=QlTVANDndpM) experience on UnRAID, you need to take some additional steps to attempt to further restrict the opportunities for the kernel to use the CPU's you're attempting to isolate for VM usage. Unfortunately ZFS [doesn't properly respect the isolcpus](https://github.com/openzfs/zfs/issues/8908) value alone, so some further steps are necessary. 15 | 16 | For reference, I've copied my `syslinux.cfg` configuration below, where my intention is to isolate the last 4 cores of a 16 core CPU: 17 | 18 | ```bash 19 | label Unraid OS 20 | menu default 21 | kernel /bzimage 22 | append initrd=/bzroot nomodeset mitigations=off amd_iommu=on isolcpus=12-15,28-31 nohz_full=12-15,28-31 rcu_nocbs=12-15,28-31 23 | ``` 24 | 25 | This was all that was needed for my threadripper 3955wx; looking at the CPU topology, I'm restricting all cores of a single CCX: 26 | 27 | ![3955wx](https://www.servethehome.com/wp-content/uploads/2021/04/AMD-Threadripper-Pro-3955WX-Topology.jpg) 28 | 29 | You can see a really great video on how this topology image is generated in [SpaceInvader's video](https://www.youtube.com/watch?v=14dvDX17GH0), but if you don't have a multi-cpu or multi-chiplet CPU (which you could tell by generating the topology image as in the video), then you may need to take take some [additional steps](https://github.com/openzfs/zfs/issues/8908#issuecomment-1066046587). 30 | 31 | ### My zpool is busy but I cant tell why 32 | 33 | There's a tool for this now, created by the same person who created sanoid/syncoid (Jim Salter), called `ioztat` (think '[iostat for zfs](https://github.com/jimsalterjrs/ioztat)'). It's not available through the nerd/dev packs (it's a relatively recent-sh creation), and may well continue to be updated anyway, so we'll install directly from git. 34 | 35 | You'll likely want to a fileset which we can use to store the tool - for myself, I have all little git tools in one fileset, so did something like this: 36 | ```bash 37 | zfs create wd/utility/git -o compression=zstd-3 -o xattr=sa -o recordsize=32K 38 | 39 | # now we're going to pull down the code into our fileset 40 | cd /mnt/wd/utility/git 41 | git clone https://github.com/jimsalterjrs/ioztat 42 | 43 | # you should see it listed there now 44 | ls -lhaFr 45 | drwx------ 3 root root 8 Mar 14 11:29 ioztat/ 46 | 47 | # don't forget to make it executable 48 | chmod +x ioztat/ioztat 49 | ``` 50 | 51 | Now, you can just execute the command, but only currently from the directory it's saved in. I've a bunch of random tools installed, and there's no way I'd remember all of these. One of the bigger annoyances I've had with unraid is how complicated it is to customize in a way that survives a reboot (recompile the kernel to install a tool a few KB in size? Ugh). We could just link them to our go file or something, but if you're like me, your go file is messy enough as it is... 52 | 53 | #### Customizing UnRAIDs CLI and running commands natively 54 | 55 | What I've done instead is to create files (placing them all inside of a folder called 'scripts', which I also put in the 'utility' fileset) that contain all my cli/shell 'customizations', then have unraid's User Scripts run through them at first array start. While they don't * Technically * persist across reboots, this method sufficiently works around it (annoying, but solvable): 56 | ```bash 57 | # create the files 58 | nano /mnt/wd/utility/scripts/createSymlinks.sh 59 | 60 | # create your symlinks - here's a small subset of mine - 61 | 62 | # # Shell profiles 63 | ln -sf /mnt/wd/utility/git/.tmux/.tmux.conf /root/.tmux.conf 64 | cp /mnt/wd/utility/git/.tmux/.tmux.conf.local /root/.tmux.conf.local 65 | ln -sf /mnt/wd/utility/bvd-home/.bash_profile /root/.bash_profile 66 | # # Bad ideas 67 | ln -sf /mnt/wd/dock/free-ipa /var/lib/ipa-data 68 | # # CLI tools and Apps 69 | ln -sf /mnt/wd/utility/git/ioztat/ioztat /usr/bin/ioztat 70 | ln -sf /mnt/wd/utility/git/btop/btop /usr/bin/btop 71 | ``` 72 | 73 | I actually have multiple files here, each one being 'grouped' - if it's just linking a tool so it's available from the shell, that's one file, another is for linking a few privileged containers I've got running to where they 'should' be where they installed on bare metal, things like that. This way, if I screw something up and dont find out till I reboot some months later, it'll make figuring out **which** thing I screwed up (and where) far easier. 74 | 75 | ### Unable to modify filesets status 76 | 77 | This happens when **something** is using either the dataset itself (at the filesystem level), or some of the contents within it. Common causes include: 78 | 79 | * Application using fileset contents 80 | * Network access (mounted via SMB/NFS/iSCSI) 81 | * If attempting deletion: 82 | * The filesystem is currently `mounted` by your host (unraid) 83 | * Snapshots currently exist for the fileset 84 | 85 | #### Sledgehammer approach 86 | 87 | For unmount issues, you can bypass the userspace restrictions by going directly to ZFS and issuing a rename; it's important to note that this could impact any applications using it, only recommended when you're either certain there's no active access, or if you plan on deleting the fileset anyway after unmounting 88 | 89 | ```bash 90 | zfs rename tank/filesetName tank/newFilesetName 91 | zfs destroy tank/newFilesetName 92 | 93 | # be cognizant of potential impact on nested filesets and their mountpoints, such as tank/datasetName/datsetChileName 94 | ``` 95 | 96 | #### Dataset is busy 97 | 98 | * This usually due to a process (PID) using that fileset or some part of it's contents. You can verify whether the below example, where I have my pool `wd` mounted to `mnt` and am trying to figure out what's using my `nginx` container: 99 | ```bash 100 | root@milkyway:~# ps -auxe | grep /mnt/wd/dock/nginx 101 | root 63910 0.0 0.0 3912 2004 pts/0 S+ 12:19 0:00 grep /mnt/wd/dock/nginx 102 | ``` 103 | 104 | * You then simply kill that PID: 105 | ```bash 106 | kill -9 63910 107 | ``` 108 | 109 | #### Dealing with PID Whackamole 110 | 111 | If a new PID is spawned almost right after you've killed one for the same process, you've likely got some kind of automation running - 112 | * **Autosnapshot tools** - sanoid/syncoid/auto-snapshot.sh, anything that automatically handles snapshot management and is currently configured on the system can cause this. If you have it, kill *that* process (e.g. sanoid) first, then retry. 113 | * As an alternative, edit the config (such as your `sanoid.conf`) so it's no longer set to snapshot that fileset, restart the tool (sanoid here), and try again 114 | * **intermittent commands** - you can try `umount -l /mnt/path` to do a `lazy` unmount; pretty commonly needed for a variety of reasons. 115 | -------------------------------------------------------------------------------- /general/WhyZfsOnUnRAID.md: -------------------------------------------------------------------------------- 1 | # The case for ZFS on UnRAID 2 | 3 | - [Whats UnRAIDs purpose](#whats-unraids-purpose) 4 | - [What are we running on our servers](#what-are-we-running-on-our-servers) 5 | - [How UnRAID tries to address the performance gap](#how-unraid-tries-to-address-the-performance-gap) 6 | - [ZFS to the Rescue](#zfs-to-the-rescue) 7 | * [Individualized performance characteristics](#individualized-performance-characteristics) 8 | * [Snapshot and backup strategies unique to the application](#snapshot-and-backup-strategies-unique-to-the-application) 9 | * [Clone your data for multiple instances of one container/VM](#clone-your-data-for-multiple-instances-of-one-container-vm) 10 | - [A word of caution](#a-word-of-caution) 11 | 12 | ## Whats UnRAIDs purpose 13 | 14 | There are many ways to think of 'storage performance', but one of the most common is described here by Nexenta: 15 | ![Nexenta](https://blogdotnexentadotcom.files.wordpress.com/2016/06/pic2.png?w=487&h=268) 16 | 17 | The idea here being that you can place yourself at any one point within the triangle, but that means you're not getting 'the maximum of any one of the three'. You're instead receiving some sort of balanced approach, a compromise really, that matches your specific needs. I think the problem UnRAID is trying to solve isn't **quite** a match for this way of thinking. Instead, we can think of the 'Good vs Fast vs Cheap' model: 18 | 19 | ![GoodFastCheap](https://larrycuban.files.wordpress.com/2015/06/good-fast-cheap.jpg) 20 | 21 | Any NAS's goals are to store as much data as possible. So for our purposes we can equate the above to: 22 | 23 | * Good = Durable/Reliable 24 | * Cheap = Running Costs 25 | * Fast = IO performance 26 | 27 | With the array, we've firmly chosen low running costs and durability over performance - if you want to have a hundred terabytes of data, all immediately available (so no tape), and with some semblance of redundancy, you'd be hard put to find a better option out there. 28 | 29 | ## What are we running on our servers 30 | 31 | Each application you run on your server will likely have a unique set of workload requirement - some (extremely limited) examples: 32 | 33 | | Workload | Example | IO Pattern | 34 | |--- |--- |--- | 35 | | Bulk Media | Movies/Music/etc | All sequential, large block, write once / read many | 36 | | Web Apps | Grafana/Gaps/Dashboards | Writes infrequently, various sizes, random reads often | 37 | | DBs | Postgres/MariaDB/SQLite | Reads and writes frequently, small block IO | 38 | 39 | We've got the bulk media part covered by the array, but everything else? The array's not terribly well suited for it. The array is comprised of all HDD's (assuming you're following LimeTech's recommendations - which you should!), and our IO operations per second are essentially limited to that of a single drive. 40 | 41 | ## How UnRAID tries to address the performance gap 42 | 43 | As anyone who's ever tried to run virtual machines or containers directly on the array will likely attest, the UnRAID array just doesn't have the juice to power even a couple moderately needy containers. 44 | 45 | As you all are likely aware, this is where the cache pool comes in. Your only options here are a single disk XFS pool, or BTRFS if you wish to pool multiple drives (either for additional performance, or redundancy). We want our storage to have durability built in and allow for disk failures, leaving BTRFS the only option (at least as of August 2022). 46 | 47 | Don't get me wrong, BTRFS is a fantastic filesystem (in my opinion at least) - the internet is full of folks who'll parrot other's noting that "[BTRFS hasn't solved the write-hole problem](https://arstechnica.com/gadgets/2021/09/examining-btrfs-linuxs-perpetually-half-finished-filesystem/)", and that for that reason alone, is just completely inferior to other technologies. I disagree, but that's not really the point in this case. 48 | 49 | The problem with this is that BTRFS just isn't as performant for the random IO loads enthusiasts (that's us) often find we need from that cache. You'll find that even [Josef Bacik](https://www.linkedin.com/in/josef-bacik-83508a7/), a Facebook employee (the largest known consumer of BTRFS) and one of the most prolific BTRFS developers out there, has essentially stated "[We'll likely never move our mysql databases to BTRFS](https://youtu.be/U7gXR2L05IU?t=3032)" - I could've swore he'd said 'just don't run databases on BTRFS', though maybe that was in a different talk (that's since been redacted?). The maximum latency values experienced with BTRFS are just too high when compared to alternatives, as much as 5x or more when compared to XFS (which is Facebook's choice). 50 | 51 | ## ZFS to the Rescue 52 | 53 | This is where the flexibility+performance of ZFS can help us. Just like with BTRFS, each application can have it's own fileset (command line only in UnRAID), allowing you to customize how the storage behaves to the specific applications needs. But ZFS's caching system means it doesn't have the same kind of penalties (nor to the same extent) that one experiences when running those workloads on BTRFS. 54 | 55 | ### Individualized performance characteristics 56 | 57 | Have an application that only reads once upon startup (e.g. pulls it's config and loads into memory) and doesn't do any further filesystem activity until the next restart? You can configure a higher level of compression on it so it takes up as little space as possible, setting the fileset to not cache at the filesystem level at all so those resources stay free for other applications which need them. 58 | 59 | Maybe you're running Nextcloud, Paperless, or TandoorRecipes, applications that require a backing MariaDB/Postgres/(etc) database in order to be performant at scale? Well you can then ensure that the block/record sizes for your chosen DB's fileset match up with the page size of that database, specify what kind of data within that fileset will be cached, and how that cache operates to best squeeze every drop of performance your storage is capable of providing that database. 60 | 61 | ### Snapshot and backup strategies unique to the application 62 | 63 | As you've each application within it's own dedicated filesystem, you're able to be as granular as you wish (or not) with your backups. Choose to snapshot your highest change-rate data every 6 hours for a week, keep weekly snapshots for a couple months, and automatically replicate them off-site, while you only take weekly backups of those which mostly just read config data. 64 | 65 | If you like pure command-line/cron, you can use something like [zap](https://github.com/Jehops/zap), where if you prefer a config driven option there are tools like [Sanoid/Syncoid](https://github.com/jimsalterjrs/sanoid) or [zfs-auto-snapshot](https://github.com/zfsonlinux/zfs-auto-snapshot) - any of which allows you full control over what snapshots get taken when, how often, how long they're kept, and where to send them for backup (if desired). You set it up once, and your backups are just automated from then on out (but please don't forget to test them occasionally!) 66 | 67 | In the worst case scenario where your server goes completely down for any reason, you can simply start up the services on that backup server and limit your downtime whilst you work away on recovering whatever caused the failure to your main systems (hopefully it's not a flooded basement, ugh.) 68 | 69 | ### Clone your data for multiple instances of one container/VM 70 | 71 | If you've ever had a home assistant upgrade go sideways, don't worry about it anymore - just create a clone of the fileset containing your VM/app data, give it it's own unique test network in UnRAID, and upgrade away. If it goes sideways? Simply destroy the clone, leaving your 'production' systems running while you research the failure. 72 | 73 | Perhaps you've got a need to run a temporary machine for a guest in the house, but don't feel like setting up one from scratch, and **REALLY** don't want to risk their wreaking havoc on an install you've spent (years) perfecting? Just clone the VM's data, let them get their grubby paws all over it, and when they're done littering in your playground, throw it away - your source machine is still safe and sound. 74 | 75 | ## A word of caution 76 | 77 | ZFS, while powerful, isn't for everyone - you'll need not only have the desire/drive to do some of your own background research and learning, but just as importantly, the **TIME** to invest in that learning. It's almost comically easy to have a much worse experience with zfs than alternatives by simply copy/pasting something seen online, while not fully understanding the implications of what that exact command syntax was meant to do. 78 | 79 | The possibilities here are virtually endless - both for making your life easier, and for seeing massive performance gains. None of that, however, is worth the potential for lossing that data. As long as one's willing to put forth the time and effort into it though, the world's your oyster! 80 | -------------------------------------------------------------------------------- /general/helpfulCommands.md: -------------------------------------------------------------------------------- 1 | ## Table of Contents: 2 | 3 | - [Plex related](#plex-related) 4 | - [Unraid host specific](#unraid-host-specific) 5 | * [Restart the UnRAID UI from cli](#restart-the-unraid-ui-from-cli) 6 | * [Get total number of inotify user watches used](#get-total-number-of-inotify-user-watches-used) 7 | + [List all current inotify users and totals](#list-all-current-inotify-users-and-totals) 8 | * [Start or stop the array from the CLI](#start-or-stop-the-array-from-the-cli) 9 | * [Spin down all drives in the array](#spin-down-all-drives-in-the-array) 10 | - [General Linux stuff](#general-linux-stuff) 11 | * [List top 10 biggest files in all sub-dirs of current directory](#list-top-10-biggest-files-in-all-sub-dirs-of-current-directory) 12 | * [Bulk Downloading from Archive.org](#bulk-downloading-from-archivedotorg) 13 | * [Remove the first 7 characters from all files in a dir](#remove-the-first-7-characters-from-all-files-in-a-dir) 14 | * [Get a breakdown of file sizes for a given directory including subdirs](#get-a-breakdown-of-file-sizes-for-a-given-directory-including-subdirs) 15 | - [Personal tools stuff](#personal-tools-stuff) 16 | * [Chrome to table generation](#make-a-table-showing-links-displayed-on-page-) 17 | * [Fix time machine stuck in stopping](#kill-bvdmbp-time-machine-locks-so-it-can-pick-back-up) 18 | 19 | 20 | ## Plex related 21 | 22 | - Increase memory allocation to the container (docker extra params) 23 | ``--shm-size=2048m` 24 | - Location of database 25 | ``/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db` 26 | - Stopping plex service (LSIO container) - required prior to running repair, undertaken within the container (`docker exec -it plex /bin/bash`) 27 | ```bash 28 | cd /run/service/ 29 | s6-svc -d svc-plex 30 | ``` 31 | - Check DB integrity (must use `Plex SQLite`, not standard `sqlite3` package) - clean run should show result `ok` 32 | ```bash 33 | cd "/config/Library/Application Support/Plex Media Server/Plug-in Support/Databases" 34 | 35 | /usr/lib/plexmediaserver/Plex\ SQLite com.plexapp.plugins.library.db "PRAGMA integrity_check;" 36 | ``` 37 | - Run repair on the DB 38 | ```bash 39 | "/usr/lib/plexmediaserver/Plex\ SQLite" com.plexapp.plugins.library.db ".output recover.out" ".recover" 40 | ``` 41 | 42 | ## Unraid host specific 43 | 44 | Stuff that's specific to unraid as opposed to generalized linux stuff 45 | 46 | ### Restart the UnRAID UI from cli 47 | 48 | * If the GUI fails to load in your browser, becomes slow over time, etc 49 | ```bash 50 | /etc/rc.d/rc.nginx reload 51 | ``` 52 | 53 | ### Get total number of inotify user watches used 54 | 55 | * Figure out if you need to increase the max watchers settings in tips and tweaks 56 | ```bash 57 | find /proc/*/fd -lname anon_inode:inotify | cut -d/ -f3 | awk '{s+=$1} END {print s}' 58 | ``` 59 | 60 | #### List all current inotify users and totals 61 | 62 | * This is a more detailed version of the above, listing what application/process is using watchers, how many for each individual process individually, so you can scope in more specifically when troubleshooting 63 | ```bash 64 | find /proc/*/fd -lname anon_inode:inotify | cut -d/ -f3 | xargs -I '{}' -- ps --no-headers -o '%p %U %c' -p '{}' | uniq -c | sort -nr 65 | ``` 66 | 67 | ### Start or stop the array from the CLI 68 | 69 | ```bash 70 | #start 71 | wget -qO /dev/null http://localhost:$(lsof -nPc emhttp | grep -Po 'TCP[^\d]*\K\d+')/update.htm?cmdStart=Start 72 | 73 | #stop 74 | wget -qO /dev/null http://localhost:$(lsof -nPc emhttp | grep -Po 'TCP[^\d]*\K\d+')/update.htm?cmdStop=Stop 75 | ``` 76 | 77 | ### Spin up a drive manually 78 | 79 | * Manually spinning down a drive 80 | ```bash 81 | #where 'X' equals your drive letter 82 | sdspin sdX up 83 | ``` 84 | 85 | ### Spin down all drives in the array 86 | 87 | * You can choose to forcefully spin down all drives. If there's an active session, no worries - they may experience a temporary interruption, but the drive needed will automatically spin back up: 88 | ```bash 89 | for dev in /dev/sd?; do /usr/local/sbin/emcmd cmdSpindown="$(grep -zoP "(?<=name=\")[a-z0-9]+(?=\"\ndevice=\"${dev: -3})" /var/local/emhttp/disks.ini | tr -d '\0')"; done 90 | ``` 91 | 92 | ## General Linux stuff 93 | 94 | (that works just the same on UnRAID) 95 | 96 | ### List top 10 biggest files in all sub-dirs of current directory 97 | 98 | ```bash 99 | find $PWD -type f -printf '%s %p\n' | sort -nr | head -10 100 | ``` 101 | 102 | ### Bulk Downloading from archiveDOTorg 103 | 104 | * Download files from archiveDOTorg based on listed output (e.g. get all iso, zip, and 7z files in this case). This option is typically fine if you're not working with the Internet Archive a great deal, and only occasionally pull the odd thing down every once in a while: 105 | ```bash 106 | wget -A iso,zip,7z -m -p -E -k -K -np https://archive.org/download/GCUSRVZ-Arquivista 107 | ``` 108 | 109 | * If you're doing this any kind of frequently though, make it easy on yourself and install the [Internet Archive's command line tool](https://archive.org/developers/internetarchive/installation.html#binaries) (direct link to download page for their binaries). Once saved onto the server (wherever it's going to live permanently), use the same method described in [this section](https://github.com/teambvd/UnRAID-Performance-Compendium/blob/main/general/commonIssues.md#customizing-unraids-cli-and-running-commands-natively) of the 'Common Issues' page to make it act as a 'native' unraid CLI command. 110 | * Now that you've 'installed' (sorta... semantics!) the `ia` cli tool, you can download at your leisure. Their [reference section](https://archive.org/developers/internetarchive/cli.html#) has some good starting points, but just to give you an idea of the power you have available... Lets just say you decided you want every PS2 game ever released in the U.S. - you search the archive, and it seems like [it's in 3 parts](https://archive.org/search.php?query=title%3A%28Playstation+2%29+AND+creator%3A%28AlvRo%29&sort=titleSorter). The links look like: 111 | `https://archive.org/details/ps2usaredump1` 112 | 113 | * You *could* just type in three separate terminal windows 114 | ```bash 115 | ia download gamecollectionlinks 116 | # hey, that was fast! 117 | ia download ps2usaredump1 118 | # now wait a long time 119 | ia download ps2usaredump1_20200816_1458 120 | # ugh, wait some more 121 | ia download httpsarchive.orgdetailsps2usaredump3 122 | # great, now maybe my great grand-kids will get to enjoy em at least 123 | ``` 124 | 125 | * Or instead, you could install the `parallel` utility from the `nerd pack`, and make the server do all the work for you at once (albeit, without the pretty progress bars, if you care about that kind of thing...). **Be sure you're already in the directory you want to download the data to before executing of course!** 126 | ```bash 127 | /path/to/ia search "ps2usaredump1" --itemlist | parallel '/path/to/ia download {} --checksum --retries 10' 128 | ``` 129 | * You're now downloading all three at once, and in a single window - hope you've got enough space on your cache drive(s)! Do note, you will likely need to specify the full actual path of the ia tool regardless of however you've made the tool available to be called directly (where we just called `ia download` above), as the `parallel` command doesn't run ia directly as your account/user. To explain the command (see links above for full description plz) 130 | * At first we have to search for the "ps2usaredump1" collection (from the URL above), then listing its items. If you ran the command on it's own, it'd show 131 | ```bash 132 | gamecollectionlinks 133 | httpsarchive.orgdetailsps2usaredump3 134 | ps2usaredump1 135 | ps2usaredump1_20200816_1458 136 | ``` 137 | * Now that we have our list, we call parallel to 'do this for each' (e.g. insert each of the 4 lines output in place of the `{}`). The `--checksum` argument looks for any existing files of the same name, compares the two, and only downloads if the local copy doesn't match the archive. Adding `--retries 10` does just what it sounds like; if the connection times out, try again (x10). These are both especially helpful if you're on a typical North American ISP that tends to like to randomly drop offline for absolutely no reason whatsoever. 138 | 139 | 140 | ### Remove the first 7 characters from all files in a dir 141 | 142 | `for f in *; do mv "$f" "${f:7}"; done` 143 | 144 | ### Get a breakdown of file sizes for a given directory including subdirs 145 | 146 | * Note - this can take a **very** long time to complete if you've got a ton of tiny little files 147 | ```bash 148 | find /mnt/whatever/directory -type f -print0 | xargs -0 ls -l | awk '{ n=int(log($5)/log(2)); if (n<10) { n=10; } size[n]++ } END { for (i in size) printf("%d %d\n", 2^i, size[i]) }' | sort -n | awk 'function human(x) { x[1]/=1024; if (x[1]>=1024) { x[2]++; human(x) } } { a[1]=$1; a[2]=0; human(a); printf("%3d%s: %6d\n", a[1],substr("kMGTEPYZ",a[2]+1,1),$2) }' 149 | ``` 150 | * Would recommend doing this in a `tmux` pane or `screen` session, just so it survives any potential disconnect 151 | 152 | ## Personal tools stuff 153 | 154 | Probably not useful to anyone else 155 | 156 | ### Make a table showing links displayed on page: 157 | ```html 158 | var x = document.querySelectorAll("a"); 159 | var myarray = [] 160 | for (var i=0; i'+myarray[i][1]+''; 170 | }; 171 | 172 | var w = window.open(""); 173 | w.document.write(table); 174 | } 175 | make_table() 176 | ``` 177 | 178 | ### Kill BVDMBP time machine locks so it can pick back up 179 | 180 | ```bash 181 | kill -9 $(smbstatus | grep BVDMBP | grep timemac | cut -d' ' -f1 | uniq) 182 | ``` 183 | -------------------------------------------------------------------------------- /containers/postgres.md: -------------------------------------------------------------------------------- 1 | # ***Postgres on ZFS*** 2 | 3 | ## **Table of Contents** 4 | 5 | - [Primer](#primer) 6 | * [My usecase](#my-usecase) 7 | - [General Recommendations](#general-recommendations) 8 | - [Configuring Dataset and OS](#configuring-dataset-and-os) 9 | - [Configuring Postgres](#configuring-postgres) 10 | * [A word on Autovacuum](#a-word-on-autovacuum) 11 | - [Backups](#backups) 12 | * [Via pgAdmin](#via-pgadmin) 13 | * [Via CLI - psql](#via-cli---psql) 14 | * [Taking advantage of some ZFS](#taking-advantage-of-some-zfs) 15 | + [Sanoid Automated Snapshots](#sanoid-automated-snapshots) 16 | + [Pre and Post scripts](#pre-and-post-scripts) 17 | - [For when youre ready to go pro](#for-when-youre-ready-to-go-pro) 18 | - [References](#references) 19 | 20 | ## Primer 21 | 22 | I got started with Postgres long after ZFS, with my introduction being [this talk](https://www.youtube.com/watch?v=T_1Zo4m4v_M) from one of the engineers over at Joyent. If you really wanna nerd out, and have 40 minutes to burn, it's worth a watch - some of the below comes directly from that talk. Another good place to look is [this article](https://www.2ndquadrant.com/en/blog/pg-phriday-postgres-zfs/), which walks you through initially poking around with both ZFS and Postgres. 23 | 24 | I've tried to give what I feel are some sane config values, along with just enough justification on them to explain without going **too** deep into it; this should hopefully provide the minimum detail necessary to allow for your own additional testing/tuning to best match your specific scenario. Please understand, every person's deployment is unique, and simply copying my config values and running with them may have unexpected results! 25 | 26 | ### My usecase 27 | 28 | What I use postgres for, the DB sizes, and any workload information I can think of that might allow someone to equate my settings and usage to their own. DB Size equals DB + Temp files (as reported by pgadmin): 29 | 30 | 31 | > * Nextcloud - `6` users - `1.1m` files - `1991MB` DB size 32 | > * Paperless - `2` users - `1438` files - `14MB` DB size 33 | > * OnlyOffice - Nextcloud connected only - `8MB` DB size 34 | > * NextcloudTest - `2` users - `2.35m` files - `3.05GB` DB size - a random local test instance for perf tuning tests prior to putting into 'production' 35 | 36 | 37 | ## General Recommendations 38 | 39 | There are several broad strokes I take with regards to any database implementation 40 | 41 | * If optional (e.g. built in sqlite, json, whatever), **do I really need a full production database** (PostgreSQL, MariaDB), or will the built in option suit the need? For example, I run vaultwarden (~bitwarden), ombi, all the 'arr's, NginxProxyManager, and several others... but none of them really *need* anything more than sqlite provides, at least for the vast majority of users; with proper tuning of sqlite, it can be made to handle more than most give it credit for. 42 | * The thing to think about here is "how many actions do I foresee occurring at once?". For a password manager, I'm mostly just reading data, with the occasional add or change. 43 | * NginxProxyManager will rarely have a configuration change, and most of it's 'db activities' will simply be for (bi-monthly) cert updates or the occasional new proxy host. 44 | * An exception to this **might** be Lidarr (once it [eventually gets psql support](https://github.com/Lidarr/Lidarr/pull/2625)), namely due to how many unique records there end up being (at one per song, even a medium sized collection can get 'metadata heavy'). 45 | * Simply "SSD" storage isn't really good enough for the best level of performance. **SATA introduces significant latency penalties that don't exist for PCIe based storage** (NVME/Optane). Always put your databases on storage that's the most performant you can afford for the best possible outcome. 46 | * The amount of latency that can be added simply at the protocol layer (SATA) is more than enough to turn a snappy, responsive application UI into one that feels sluggish/dated 47 | * **Snapshots alone are not enough to 'always' ensure the DB is up to date at that time** - you need to take special precautions when backing up DBs, as they operate much like ZFS does. For ZFS, you have the ZIL; for DBs, this is called a WAL (write ahead log), another sort of 'abstraction layer' meant to improve performance (though designed without the robustness of ZFS in mind). 48 | * **Mo Memory = MO BETTAH** - but you're running ZFS right? So you already knew that. 49 | * Specific to Postgres, **Get familiar with [pgAdmin](https://www.pgadmin.org/)** - while I typically only use it for reporting, it can be helpful for TONS of situations. It'll report statistics for individual databases (size, number of current connections, etc), allow you to take backups via GUI if you prefer, and is just a fantastic way to visualize what's happening 'beneath the covers' of your application. 50 | 51 | ## Configuring Dataset and OS 52 | 53 | * In your BIOS, disable NUMA, and ensure memory interleaving is enabled - [ref](http://rhaas.blogspot.com/2014/06/linux-disables-vmzonereclaimmode-by.html) - (NOT transparent huge pages) 54 | 55 | * Refrain from using swap (on unraid, just don't install the swap plugin) 56 | 57 | * Configure the dataset where your postgres data will reside with properties tuned specifically for pg's read/write behavior: 58 | 59 | ```bash 60 | zfs set compression=zstd-3 recordsize=16K atime=off logbias=throughput primarycache=metadata redundantmetadata=most zpool/dataset 61 | ``` 62 | 63 | As with SQLite, we use the same compression and logbias settings, but this time we're setting the primarycache to `metadata`. This means we're caching the filesystem pointers that say 'this is where the data is', but not the data itself. 64 | 65 | **Note**, the `redundantmetadata` setting should only be modified in a pool with redundancy (mirror, raidz, etc), and where `checksums` have not been disabled (it's on by default). 66 | 67 | * Edit your go file (nano /boot/config/go) adding: 68 | 69 | ```bash 70 | echo 1 /sys/module/zfs/parameters/zfs_txg_timeout 71 | ``` 72 | 73 | The above is a global parameter (see [github docs](https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Module%20Parameters.html#zfs-txg-timeout))- even if the pool you plan to put your DB on is redundant, should you have any other non-redundant pools, recommend avoiding this setting. While it'll likely help performance, in combination with the other settings we're using (both above and below), there's an increased risk (however small) of data loss in certain circumestances. 74 | 75 |
76 | 77 |

78 | NOTE

79 |

80 | All of the settings mentioned your ZFS dataset configuration should be set prior to putting any data on them. If data already exists, you'll need to create a new dataset and migrate the data to that dataset in order to fully enjoy their benefits. 81 |

82 |

83 |
84 | 85 | ## Configuring Postgres 86 | * ZFS (and BTRFS for that matter) are both copy-on-write filesystems, while postgres does a version of this itself as well by default (to allow for the typically journaled filesystems it's installed on); disabling this behavior removes excessive writes, increasing performance: 87 | ```sql 88 | ALTER SYSTEM SET full_page_writes=off; 89 | ALTER SYSTEM SET synchronous_commit=off; 90 | CHECKPOINT 91 | ``` 92 | 93 | * Edit `postgresql.conf` to better take advantage of that TB of memory you bought - some of this is (strictly speaking) unnecessary for most of us, as if you have compression enabled, the databases tend to be pretty small (more on that below): 94 | 95 | ```sql 96 | shared_buffers = 2048MB 97 | huge_pages = try 98 | work_mem = 64MB 99 | maintenance_work_mem = 64MB 100 | max_stack_depth = 16MB 101 | max_files_per_process = 8000 102 | effective_io_concurrency = 200 103 | maintenance_io_concurrency = 200 104 | max_worker_processes = 24 105 | max_parallel_maintenance_workers = 8 106 | max_parallel_workers = 24 107 | random_page_cost = 1.1 108 | max_wal_size = 1GB 109 | min_wal_size = 80MB 110 | checkpoint_completion_target = 0.9 111 | ``` 112 | 113 | Shared buffers is effectively the 'total memory allocation' for PG (among all it's processes). Work mem is the max memory that may be used by an individual worker, so it's important to keep that shared buffers value in mind when setting this. In the above, I've allocated `2GB` to PG; we have `24` workers and `8` maint workers, giving us `32 * 64MB = 2048MB`. With autovacuum disabled, maint workers effectively just carry out indexing tasks, so we don't need a great many of them (more on autovacuum below). Keep an eye on your DB sizes (as reported by postgres, NOT the filesystem) - if you're allocating memory beyond what your DB's total sizes are, you're simply wasting resources! 114 | 115 | On the other side, if you're allocating less memory than the number of workers you've set up can ask for, you'll start throwing errors around memory reservations and allocation. Check out postgres's [official documentation](https://www.postgresql.org/docs/current/runtime-config-resource.html#RUNTIME-CONFIG-RESOURCE-MEMORY), and read through the config options carefully before making your changes. 116 | 117 | The IO concurrency setting of 200 allows for us to take better advantage of all those IOPs of our NVME; we could theoretically increase this even further, but for home use, there's a point of diminishing returns... And if you're using SATA SSD's, there's a greater chance that increasing this too high would result in excess CPU utilization due to iowait. The `200` value should be safe for even the most basic NVME, while SATA might be better off with a `100` setting, depending on the device's performance characteristics. 118 | 119 | For reference, the above settings have proven highly performant for myself, and I currently have DBs for Nextcloud, OnlyOffice, Paperless, and Gitea (there's vaultwarden as well, but it's needs can be suitably met by SQLite) 120 | 121 | * Now restart postgres: 122 | 123 | ```bash 124 | psql restart 125 | ``` 126 | 127 |
128 | 129 |

130 | NOTE

131 |

132 | Unlike the ZFS settings, all of the settings mentioned for postgresql.conf can be changed even after the DB has been created and still reap the benefits. If you're not getting the performance you desire, you can test and tweak at will; simply stop whatever applications are using PG, make the change to the config file, then restart the PG container (and start the application container back up as well following). There is unfortunately no 'one size fits all', but the above can be a healthy starting point. Tune, test, evaluate - rinse and repeat! 133 |

134 |

135 |
136 | 137 | ### A word on Autovacuum 138 | 139 | Many will recommend also uncommenting the `autovacuum = on` line within `postgresql.conf`. For truly "production" deployments, this is a must. However, in smaller scale deployments (such as my own), I prefer to manually vacuum the databases myself. This has two benefits: 140 | 141 | 1. I know exactly when the DBs are going to be vacuumed, and can plan accordingly. Vacuum's can have a significant performance impact, even if properly tuned (more on that in a moment), and my overly cautious nature means I always take a backup (pgdump) prior to manually running vacuum. If 'something goes wrong', I have a known good point in time to restore from. 142 | 2. Autovacuum, like everything else, has yet more tuning variables to be concerned with. Improper tuning can mean trying to track down intermittent performance issues indefinitely, only to (hopefully) eventually find that 'oh... it was running a vacuum'. 143 | 144 | I do all my maintenance at once on databases, usually as part of a scheduled outage where I'm already undertaking other operations (e.g. nextcloud upgrade, disk replacement, or quarterly {if I remember...} server reboot). It's just 'cleaner', at least for me. 145 | 146 | ## Backups 147 | 148 | There are two 'standard' options when it comes to backups; using pgAdmin, and via commandline using 'psql'. In addition, there are countless other methods/tools as well, and we'll touch on < however many I do >: 149 | 150 | ### Via pgAdmin 151 | 1. Stop the application which is using the DB 152 | 2. Log into pgAdmin and connect to the database instance (your server's container) 153 | 3. Right-click on the DB to be backed up and select... wait for it... 'Backup' 154 | 4. Give it a memorable name, selecting `Tar` as the type, and (typically) `UTF-8` for the encoding type 155 | 5. Under the Data/Objects table, ensure all three options are selected (`Pre`, `Data`, and `Post`), and click `Backup`. 156 | 157 | By default these are saved in pgAdmin's appdata directory under your user account, similar to: 158 | 159 | ```bash 160 | /mnt/user/appdata/pgadmin/storage/email.address_domain/yourBackupName.tar 161 | ``` 162 | 163 | For a more detailed description (including pretty pitchaz!), see pgAdmin's [official documentation](https://www.pgadmin.org/docs/pgadmin4/development/backup_dialog.html) 164 | 165 | ### Via CLI - psql 166 | 1. Once again, stop the application using the DB 167 | 2. Open your postgres container's console by either right-clicking and selecting `console`, or alternatively from your terminal: 168 | ```bash 169 | docker exec -ti containerName bash 170 | ``` 171 | 3. Create a backup directory in your persistent storage location, and log in with a user that has rights to the DB(s) 172 | ```bash 173 | / # mkdir /var/lib/postgresql/data/backups 174 | / # psql -U usernameA 175 | usernameA=# 176 | ``` 177 | 4. Create your backup using `pg_dump`: 178 | ```bash 179 | usernameA=> pg_dump -E UTF-8 -F t myDatabaseName > /var/lib/postgresql/data/backups/myDatabaseNameAndDate.tar 180 | ``` 181 | 182 | The dumps are placed wherever your appdata is located at for postgres, and available for restore/testing. If all of your applications are down for something of a 'global maintenance', you can use pg_dumpall (though I typically don't recommend it for it's lack of granularity as it tends to cause confusion for some): 183 | 184 | ```bash 185 | usernameA=> pg_dumpall > allDBs-date.tar 186 | ``` 187 | 188 | ### Taking advantage of some ZFS 189 | 190 | ***WORK IN PROGRESS / TBD - I really need to finish documenting this!*** 191 | 192 | We're going to use ZFS snapshots for our backups. Since we're using WAL in combination with zfs, this *could* be considered optional by some - the ZIL keeps the ZFS filesystem layer consistent, and the WAL keeps the DB application layer consistent. Paranoia, however, is healthy when it comes to data loss prevention. 193 | 194 | The only difference between the above and below is that they're not necessarily 'both consistent at the same time'. Because of this, when the DB starts up, it may start in recovery mode - it replays the logs from WAL as needed. 195 | 196 | #### Sanoid Automated Snapshots 197 | 198 | #### Pre and Post scripts 199 | 200 | In sanoid.conf, use the below to let pg know you intend to start and stop a backup - recoveries are faster this way (not implemented in my setup, felt too much like 'work'): 201 | 202 | * **Pre-script** - 203 | 204 | ``` sql 205 | pg_start_backup('backup_name') 206 | ``` 207 | 208 | * **Post-script** - 209 | 210 | ``` sql 211 | pg_stop_backup 212 | ``` 213 | 214 | ## For when youre ready to go pro 215 | 216 | You almost certainly don't need these, but at least from my work experience, these are the best in the business - allowing for point-in-time recovery (e.g. your backup was taken at noon, you can recover to any individual second prior, such as 11:54 and 32 seconds), highly configurable backups from multiple local/cloud locations, and so on: 217 | 218 | * [PMM](https://www.percona.com/software/database-tools/percona-monitoring-and-management) - If there were a product that was "Grafana + GoAccess, but for databases", they'd call it Percona Monitoring and Management. This is the kind of tool folks like the Nextcloud dev team would use to optimize the queries used by the application. 219 | * [PGHero](https://github.com/ankane/pghero) - Think "PMM, but if I don't run a fortune 500 company". I run it [at home](https://github.com/ankane/pghero/blob/master/guides/Docker.md). 220 | * [pg_probackup](https://github.com/wal-g/wal-g) - Dedicated highly configurable and excellent cataloging of backups, making efficient work of understanding what your backup history and recoverability points are. Believe it or not, sometimes just *finding* the right backup to recover from (or even knowing what your options are) is the hardest part, especially when managing thousands of databases across 10s/100s of instances, and this makes it wickid simple. 221 | * [WAL-G](https://github.com/wal-g/wal-g) - Biggest win for wal-g is that it's the one tool I'm aware of that does (those things mentioned above), **AND** can handle Postgres, Maria/MySQL, Microsoft's SQLServer, and more recently even Mongo and Redis (crazy eh!?). Really, if you're in the devops world and haven't tried WAL-G, you owe it to yourself to give it a shot. 222 | * [Metabase](https://github.com/metabase/metabase) - asdfa 223 | * [Chartbrew](https://github.com/chartbrew/chartbrew) - Like metabase, just 'a bit less'. Fewer crazy features, fewer options, resulting in fewer resources used by the application. Great for the small/medium business market! 224 | 225 | ## References 226 | [PSQL tuner](https://github.com/jfcoz/postgresqltuner) - Dockerfile designed around finding issues and some areas for optimization 227 | 228 | [pgTune](https://pgtune.leopard.in.ua/#/) - Input your containers resource allocations (memory, CPU, storage type) and use case, helps provide a starting point for tuning `postgresql.conf` 229 | 230 | [pgConfig](https://www.pgconfig.org/) - More tunable output than pgTune, but based on the same underlying code/work. The real value here lies not in the 'recommended tuning' output it provides (which is just average, as all 'general' recommendations must be without having a deeper understanding of the workload), but instead the explanations given for what the config variables specific impacts are when you select their dropdown 231 | 232 | -------------------------------------------------------------------------------- /containers/nextcloud.md: -------------------------------------------------------------------------------- 1 | # Nextcloud - Performance Optimized 2 | 3 | This doc goes through the various performance-related tips and tweaks I've compiled for my own implementation over the course of the last several years. It's not intended as a one-stop-shop for 'how to install nextcloud', but instead more of a 'what to do once you've gotten your nextcloud up and running' to make it run more efficiently. 4 | 5 | ## Table of Contents 6 | 7 | + [Assumes use of the following containers](#assumes-use-of-the-following-containers) 8 | - [Additional Notes](#additional-notes) 9 | + [Databases](#databases) 10 | + [Tuning the Nextcloud container](#tuning-the-nextcloud-container) 11 | - [Global PHP parameters](#global-php-parameters) 12 | - [Preview generation related](#preview-generation-related) 13 | + [Making sure pregeneration serves its purpose in UnRAID](#making-sure-pregeneration-serves-its-purpose-in-unraid) 14 | + [Generally helpful configuration options](#generally-helpful-configuration-options) 15 | - [SSL on LAN for secured local access](#ssl-on-lan-for-secured-local-access) 16 | - [The Deck app is dumb](#the-deck-app-is-dumb) 17 | - [Create previews for numerous additional filetypes](#create-previews-for-numerous-additional-filetypes) 18 | - [HTTP2 in NginxProxyManager](#http2-enablement) 19 | - [Some notes on Cloudflare](#some-notes-on-cloudflare) 20 | + [Bypassing site rules](#bypassing-site-rules) 21 | 22 | ### Assumes use of the following containers 23 | * LSIO nextcloud with php8 24 | * xternet onlyoffice image (manually updated from within container - I know, terrible, I'm lazy I guess :-\ ) 25 | * redis (`official alpine`) 26 | * postgres 15 (`official alpine`) - However pg16 should be sufficiently stable at current if deploying from scratch 27 | * and NginxProxyManager (`official`) as double proxy 28 | * (if already using rabbitmq, for something like homeassistant or w/e, you can use that same instance for onlyoffice here too. Extra credit or some crap, brown noser.) 29 | 30 | #### Additional Notes 31 | 32 | * All containers on same custom network, including NginxProxyManager (can have multiple networks attached if necessary - easiest using something like portainer for managing more complex docker network deployments) 33 | * Container names are used for DNS within that custom network, and are named: nextcloud, redis, postgres, onlyoffice, nginx 34 | 35 | ### Databases 36 | 37 | * Postgres user and db creation: 38 | ```sql 39 | CREATE USER nc WITH PASSWORD 'MakeThisOneGoodAndSTRONK'; 40 | CREATE DATABASE nc TEMPLATE template0 ENCODING 'UNICODE'; 41 | ALTER DATABASE nc OWNER TO nc; 42 | GRANT ALL PRIVILEGES ON DATABASE nc TO nc; 43 | ``` 44 | * Create one for `onlyoffice` as well, so the docs container doesn't have to spin up some postgres 12 container or some such nonsense, same for `redis` (but no additional redis config necessary). This helps to reduce excess CPU consumption. See [here](https://github.com/ONLYOFFICE/Docker-DocumentServer#available-configuration-parameters) for variables to add to the onlyoffice container to allow use of existing postgres/redis containers. 45 | 46 | * Performance recommendations related to DB's - Add the below to 'go' file - see [here](https://docs.actian.com/vector/4.2/index.html#page/User%2FOS_Settings.htm%23) for details/calculations - 47 | ```bash 48 | ### (see Vector documentation) - re DB performance 49 | ## allow overcommit of mem for virtualization 50 | echo 1 > /proc/sys/vm/overcommit_memory 51 | ## required for Elasticsearch 52 | sysctl -w vm.max_map_count=393216 53 | ``` 54 | 55 | * [Redis](https://github.com/redis/redis) - needed for high performace backend as well as onlyoffice, and hugely helpful for systems with either high user counts, or where users may be syncing a massive number of files; this assumes you've created a custom docker network which both nextcloud and redis share, and that the redis container is simply named 'redis' - edit config.php to look like: 56 | ```php 57 | $CONFIG = array ( 58 | 'memcache.local' => '\\OC\\Memcache\\APCu', 59 | 'memcache.distributed' => '\\OC\\Memcache\\Redis', 60 | 'memcache.locking' => '\\OC\\Memcache\\Redis', 61 | 'redis' => 62 | array ( 63 | 'host' => 'redis', 64 | 'port' => 6379, 65 | ), 66 | ``` 67 | 68 | ### Tuning the Nextcloud container 69 | 70 | Nearly all our nextcloud 'application-specific' tuning options are PHP related, though some are to the 'global' PHP instance, while others are for directed to applications coded in PHP which use that PHP instance. 71 | 72 | #### Global PHP parameters 73 | 74 | You can find explanations of these tunable options (PHP calls them 'directives') and what their impacts are can be found in the [official PHP documentation](https://www.php.net/manual/en/ini.core.php): 75 | 76 | * Set your nextcloud's limits - input and execution time must be increased as well (as this is the max time it'll allow an action to take, and 60G can't be downloaded in 60 seconds, set to 2 hours here) "nano /mnt/wd/dock/nextcloud/php/php-local.ini" 77 | ```php 78 | memory_limit = 8192M 79 | upload_max_filesize = 60G 80 | post_max_size = 60G 81 | max_input_time = 7200 82 | max_execution_time = 7200 83 | ``` 84 | * For the execution and input times here, you should try to do some rough math to deduce 'with my max file size, and my available upload bandwidth, how long would it realistically take to complete the upload?' Taking the above as an example - for a `60GB` file to upload using a 30Mb/s pipe takes roughly 4 hours and 20 mins. Building in some buffer room for other small sync operations in between and rounding up, 5 hours is chosen (our `7200` above, converting to time in seconds). You can use a [bandwidth calculator](https://www.calculator.net/bandwidth-calculator.html) to make things easier on you. 85 | 86 | * Increase the number of PHP processes allowed, adding the below to to the bottom of - /mnt/wd/dock/nextcloud/php/www2.conf 87 | ```php 88 | pm = dynamic 89 | pm.max_children = 120 90 | pm.start_servers = 12 91 | pm.min_spare_servers = 6 92 | pm.max_spare_servers = 24 93 | ``` 94 | * Be aware that these settings are reliant upon your [DB's configuration](https://github.com/teambvd/UnRAID-Performance-Compendium/blob/main/containers/postgres.md) for proper function; for instance, if you've set `pm.max_children` to `100`, but your database's max connections allowed is 50, you're going to see php reporting errors spinning up another server/child process if it tries to request a connection beyond the DB's allowed settings. 95 | 96 | * install pdlib at startup for facial recognition plugin - "nano /mnt/wd/dock/nextcloud/custom-cont-init.d/pdlib" 97 | ```bash 98 | #!/bin/bash 99 | echo "**** installing pdlib ***" 100 | apk add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing php8-pdlib 101 | apk add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing dlib 102 | ``` 103 | 104 | #### Preview generation related 105 | 106 | Highly customizable; the more options you add here, the more preview types you'll generate, though this is at the cost of more storage for preview. One should note, whether and to what degree this helps page load performance will be highly dependent upon the storage which contains these preview files, and as such, recommend they're kept on low latency / high performance data storage. 107 | 108 | * Below pre-gen's thumbnails, but not the 'clicked' images (when you open the image, this preview is generated on demand), and limits the on demand generated size to 2048x2048 109 | * From the cli, configure the pre-generator 110 | ```bash 111 | s6-setuidgid abc php8 -f /config/www/nextcloud/occ config:app:set --value="32 64 1024" previewgenerator squareSizes 112 | s6-setuidgid abc php8 -f /config/www/nextcloud/occ config:app:set --value="64 128 1024" previewgenerator widthSizes 113 | s6-setuidgid abc php8 -f /config/www/nextcloud/occ config:app:set --value="64 256 1024" previewgenerator heightSizes 114 | ``` 115 | * Then limit on demand generation size, adding below to the bottom - "nano /mnt/wd/dock/nextcloud/config/config.php" 116 | ```php 117 | 'preview_max_x' => '2048', 118 | 'preview_max_y' => '2048', 119 | 'jpeg_quality' => '60', 120 | ``` 121 | * Or alternatively, from the cli 122 | ```python 123 | s6-setuidgid abc php8 -f /config/www/nextcloud/occ config:app:set preview preview_max_y --value="2048" 124 | s6-setuidgid abc php8 -f /config/www/nextcloud/occ config:app:set preview preview_max_x --value="2048" 125 | s6-setuidgid abc php8 -f /config/www/nextcloud/occ config:app:set preview jpeg_quality --value="60" 126 | ``` 127 | * Set up automated preview generation and facial recognition hourly for face, 3 times/hour for previews - "nano /mnt/wd/dock/nextcloud/crontabs/root" 128 | ```bash 129 | */20 * * * * s6-setuidgid abc php8 -f /config/www/nextcloud/occ preview:pre-generate -vvv 130 | */20 * * * * s6-setuidgid abc php8 -f /config/www/nextcloud/occ documentserver:flush 131 | 0 * * * * s6-setuidgid abc php8 -f /config/www/nextcloud/occ face:background_job -t 35 132 | ``` 133 | 134 |
135 | 136 |

137 | NOTE

138 |

139 | Using 's6-setuidgid' is typically preferred as this pretty well 'always works', as it sets the config as the specific (occ) user would from the command line, 'impersonating' them, but transparently doing so. Modifying the config files directly may not always work, depending on the install, as one config file may be overridden by another during init. Just depends on the specific container. 140 |

141 |

142 |
143 | 144 | *Refer to the [s6 Overlay Github](https://github.com/just-containers/s6-overlay) for further information on how s6 works* 145 | 146 | ##### **Making sure pregeneration serves its purpose in UnRAID** 147 | 148 | Preview pre-generation's entire goal is to ensure that, when you click to load a page, there's no waiting around for that pages contents to load other than that which is caused by the upload bandwidth of your server. If, like me, you've a boatload of files in your nextcloud, you may well be using the UnRAID array to host your files (if so, recommend the Integrity plugin from community apps), likely with the share you've mapped to nextcloud's "/data" folder set to `cache=yes`. 149 | 150 | The problem here is that nextcloud's appdata (where it puts all data related to items installed from the nextcloud Hub) is stored in this same share - with `cache=yes`, all your previews will inevitably end up on the (slow) array once mover runs. We want **at least** the pre-generated images on the cache; we don't really care about the longevity of these files, so even if you've only a single disk cache pool, we can keep them there. Should the worst occur and your cache disk dies, we can always re-generate the previews. 151 | 152 | * First ensure the [CA mover tuning](https://forums.unraid.net/topic/70783-plugin-mover-tuning/) plugin is installed. This can then be configured at `https://UnraidIpAddress/Plugins/Scheduler`, then the `Mover Tuning` tab. 153 | * Now we need to create a file noting what to ignore - on your flash boot device is fine, I've chosen to put it on my zpool with the rest of my scripts and such: 154 | ```bash 155 | nano /mnt/user/scriptLocation/moverTuningExclusions.txt 156 | 157 | # now we're telling Mover Tuning what to leave alone - put just this line into the file and save/exit: 158 | /mnt/user/nxstore/appdata_ocbgah4z0nhr/preview 159 | ``` 160 | * We'll now set up mover tuning to ignore moving anything within our preview dir using `Ignore files listed inside of a text file` and specifying the file we just created: 161 | ![MoverTuning](https://github.com/teambvd/UnRAID-Performance-Compendium/blob/main/containers/img/moverTuning.png) 162 | 163 | **IMPORTANT** 164 | You can set mover tuning to leave the entire appdata directory on the cache, but should **ONLY** do this if you've a redundant cache pool (mirrored, raid5/6) as not all data within the appdata dir is 'expendable' in the same way the pre-generated previews are. 165 | 166 | ### Generally helpful configuration options 167 | 168 | There are some 'quality of life' customizations I've found over my time running owncloud, then eventually nextcloud after the split, which have been significantly useful: 169 | 170 | #### SSL on LAN for secured local access 171 | 172 | * Set up user script to ensure that SSL certs are copied from NginxProxyManager to nextcloud, I've mine kicking off daily at 0305 early each morning (`custom sched, '5 3 * * *'`) 173 | ```bash 174 | ## copying key and cert for nextcloud 175 | cp /mnt/wd/compose/nginx/letsencrypt/live/npm-30/privkey.pem /mnt/wd/compose/nextcloud/keys/privkey.pem 176 | cp /mnt/wd/compose/nginx/letsencrypt/live/npm-30/cert.pem /mnt/wd/compose/nextcloud/keys/cert.pem 177 | ``` 178 | 179 | #### The Deck app is dumb 180 | 181 | * Make your login actually go to your files instead of the dashboard - for home users utilizing nextcloud mainly for it's initial design purpose (cloud storage / file sync and share), the dashboard 'Deck' app is... Sorta worthless. You can instead make nextcloud default to showing your files (or whatever you prefer) immediately after logging in to save you that extra click each time! Add the following to config.php - this can be set to any app (e.g. 'photos', 'circles', 'calendar', etc): 182 | 183 | ```php 184 | 'defaultapp' => 'files', 185 | ``` 186 | 187 | #### Create previews for numerous additional filetypes 188 | 189 | This can be resource intensive at first when setting up, but once it's churned through all your existing files, the additional load should be minimal (if even noticeable): 190 | 191 | * Allow previews to be generated for files other than pictures - 192 | * List of potential preview candidates are as follows: 193 | ```php 194 | OC\Preview\BMP 195 | OC\Preview\GIF 196 | OC\Preview\JPEG 197 | OC\Preview\MarkDown 198 | OC\Preview\MP3 199 | OC\Preview\PNG 200 | OC\Preview\TXT 201 | OC\Preview\XBitmap 202 | OC\Preview\OpenDocument 203 | OC\Preview\Krita 204 | OC\Preview\Illustrator 205 | OC\Preview\HEIC 206 | OC\Preview\Movie 207 | OC\Preview\MSOffice2003 208 | OC\Preview\MSOffice2007 209 | OC\Preview\MSOfficeDoc 210 | OC\Preview\PDF 211 | OC\Preview\Photoshop 212 | OC\Preview\Postscript 213 | OC\Preview\StarOffice 214 | OC\Preview\SVG 215 | OC\Preview\TIFF 216 | OC\Preview\Font 217 | ``` 218 | * Defaults only to a select few (images mostly, but notably excluding HEIC, which is super annoying if you've any iOS users)... But you can modify the included providers. Note, if you specify ANY providers, then none of the defaults are selected, meaning f you're going to add one, then you need to specify EACH of the ones you want previews for (e.g. setting just the HEIC provider means that nothing other than HEIC will get previews, so ensure you've listed each one you need). 219 | 220 | However, if you specify any providers, only those explicitly specified will be used (so you can't add just one then expect the defaults to be called as well); example (add to config.php): 221 | ```php 222 | 'enable_previews' => true, 223 | 'enabledPreviewProviders' => 224 | array ( 225 | 1 => 'OC\\Preview\\HEIC', 226 | 2 => 'OC\\Preview\\PDF', 227 | 3 => 'OC\\Preview\\XBitmap', 228 | 4 => 'OC\\Preview\\PNG', 229 | 5 => 'OC\\Preview\\Image', 230 | 6 => 'OC\\Preview\\Photoshop', 231 | 7 => 'OC\\Preview\\TIFF', 232 | 8 => 'OC\\Preview\\SVG', 233 | 9 => 'OC\\Preview\\BMP', 234 | 10 => 'OC\\Preview\\GIF', 235 | 11 => 'OC\\Preview\\JPEG', 236 | ), 237 | ``` 238 | * Lastly, finally enable your configured stuff from above (docker terminal for nextcloud): 239 | ```php 240 | s6-setuidgid abc php8 -f /config/www/nextcloud/occ preview:generate-all -vvv 241 | ``` 242 | 243 | #### HTTP2 Enablement 244 | 245 | This is enabled by default in NginxProxyManager, but worth it to confirm the HTTP/2 switch is enabled, just in case - external connectivity on modern browsers will likely throw a warning if this isn't set to enabled regardless though, so you've probably already got this covered. 246 | 247 | #### Some notes on Cloudflare 248 | 249 | While I won't go in to the full setup/config here (tons of great guides on that available elsewhere), and it's definitely a fantastic tool for both protecting yourself, as well as speeding up (most) websites, there are various features and functionality CF enables / makes available which can cause some severe issues with Nextcloud. Some examples include the Auto-Minify functions for CSS and Javascript, caching content (counterintuitive, I know!), and some compression types (which don't always behave properly with PHP). 250 | 251 | We don't want CF caching anything from Nextcloud for a couple reasons: 252 | * We already generate lower resolution previews ourselves (or should at least - see above). Having CF cache already tiny files has limited (if any) benefit, with the possible down side of rendering out-of-date content (which can result in weird rendering issues) 253 | * From a security perspective, it's usually not a great idea to have someone else hosting your images, which is effectively what's happening here, except in this case, you're not able to encrypt them from view as you would with a backup. Using CF caching for your private and possibly sensitive files increases your datas surface area. 254 | 255 | ##### Bypassing site rules 256 | 257 | In order to keep the benefits of various CF tools / components (brotli compression, caching for static asset sites, IP protection, etc) while still hosting Nextcloud under the same domain, we can set up a page rule in order to have the site we're hosting Nextcloud under bypass those potentially problematic settings. 258 | 259 | Under "Rules" -> "Page Rules", we'll create a new rule specifying our Nextcloud location (e.g. nextcloud.mydomain.com), and choosing to both bypass caching as well as "Disable Performance" (more like "Disable Suffering"): 260 | ![SiteRules](https://github.com/teambvd/UnRAID-Performance-Compendium/blob/main/containers/img/site-rules.png?raw=true) 261 | 262 | ******* 263 | 264 | ******* 265 | 266 | ******* 267 | 268 | 269 | ***** EXTRA / BONUS ***** 270 | 271 | * Rename and set all your photos to the datetime that you actually took them (makes them show up in the order you took them in web interface) - requires installation of 'exiftool' (see Nerd Pack): 272 | 273 | * Create a script dir and script (as the 'nobody' user to avoid perms issues): 274 | 275 | sudo -u nobody mkdir /mnt/user/appdata/nextcloud/scripts 276 | sudo -u nobody nano /mnt/user/appdata/nextcloud/scripts/photo-name-to-create-date.sh 277 | 278 | * Paste in the following: 279 | 280 | #!/bin/sh 281 | 282 | albumdir=$1 283 | 284 | # comment out the next two lines if running outside of nextcloud 285 | nextclouddir="/config/var/www/nextcloud" 286 | nextclouduser="" 287 | 288 | echo "Changing modified date to date photo was taken..." 289 | exiftool "-filemodifydate 308 | 309 | 310 | * config.php 311 | 312 | 'tempdirectory' => '/tmp/nextcloudtemp', 313 | 'localstorage.allowsymlinks' => false, 314 | 'filesystem_check_changes' => 0, 315 | 'simpleSignUpLink.shown' => false, 316 | 317 | * Nginx Proxy Manager - Advanced / nextcloud - 318 | 319 | location ^~ /push/ { 320 | proxy_pass http://:7867/; 321 | proxy_http_version 1.1; 322 | proxy_set_header Upgrade $http_upgrade; 323 | proxy_set_header Connection "Upgrade"; 324 | proxy_set_header Host $host; 325 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 326 | } 327 | 328 | proxy_max_temp_file_size 16384m; 329 | client_max_body_size 0; 330 | 331 | 332 | 333 | 334 | * Allow for implementation of high performance backend for nextcloud - duplicate your container 335 | 336 | # Rule added in order to allow implementation of HPB 337 | location = /push/ { 338 | proxy_pass http://:7867/; 339 | proxy_http_version 1.1; 340 | proxy_set_header Upgrade $http_upgrade; 341 | proxy_set_header Connection "Upgrade"; 342 | proxy_set_header Host $host; 343 | proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; 344 | 345 | --------------------------------------------------------------------------------