├── .gitignore ├── LICENSE ├── README.md ├── install.sh ├── log2zram ├── log2zram.conf ├── log2zram.logrotate ├── log2zram.service └── uninstall.sh /.gitignore: -------------------------------------------------------------------------------- 1 | /overlayfs-tools 2 | -------------------------------------------------------------------------------- /LICENSE: -------------------------------------------------------------------------------- 1 | The MIT License (MIT) 2 | 3 | Copyright (c) 2016, Azlux 4 | 5 | Permission is hereby granted, free of charge, to any person obtaining a copy 6 | of this software and associated documentation files (the "Software"), to deal 7 | in the Software without restriction, including without limitation the rights 8 | to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 9 | copies of the Software, and to permit persons to whom the Software is 10 | furnished to do so, subject to the following conditions: 11 | 12 | The above copyright notice and this permission notice shall be included in all 13 | copies or substantial portions of the Software. 14 | 15 | THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 16 | IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 17 | FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 18 | AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 19 | LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 20 | OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 21 | SOFTWARE. 22 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | Not supported anymore look at https://github.com/StuartIanNaylor/zram-config as does the same and also further zram config if required 2 | 3 | # Log2Zram 4 | 5 | Usefull for IoT / maker projects for reducing SD, Nand and Emmc block wear via log operations. 6 | Uses Zram to minimise precious memory footprint and extremely infrequent write outs. 7 | 8 | Can not be used for mission critical logging applications where a system crash and log loss is unaceptable. 9 | If the extremely unlikely event of a system crash is not a major concern then L2Z can massively reduce log block wear whilst maintaining and extremely tiny memory footprint. Further resilience can be added by the use of a watchdog routine to force stop. 10 | 11 | Uses an OverlayFS lower bind mount of /var/log to /opt/zram and upper zram to create fast startups and extremely small memory requirements. 12 | On stop uses https://github.com/kmxz/overlayfs-tools merge tool to merge from volatile ram to persistant. 13 | Many thanks kmxz for providing one of the only overlayfs tools available. 14 | 15 | Works well in conjunction with https://github.com/StuartIanNaylor/zramdrive and https://github.com/StuartIanNaylor/zram-swap-config 16 | _____ 17 | ## Menu 18 | 1. [Install](#install) 19 | 2. [Config](#config) 20 | 3. [It is working ?](#it-is-working) 21 | 4. [Uninstall](#uninstall-) 22 | 23 | ## Install 24 | sudo apt-get install git 25 | git clone https://github.com/StuartIanNaylor/log2zram 26 | cd log2zram 27 | sudo sh install.sh 28 | 29 | 30 | ## Customize 31 | #### variables : 32 | In the file `/etc/log2zram.conf` `sudo nano /etc/log2zram.conf` to edit: 33 | ``` 34 | # Size for the zram memory used, it defines the mem_limit for the zram device. 35 | # The default is 20M and is basically enough for minimal applications. 36 | # Because aplications can often vary in logging frequency this may have to be tweaked to suit application . 37 | SIZE=20M 38 | 39 | # COMP_ALG this is any compression algorithm listed in /proc/crypto 40 | # lz4 is fastest with lightest load but deflate (zlib) and Zstandard (zstd) give far better compression ratios 41 | # lzo is very close to lz4 and may with some binaries have better optimisation 42 | # COMP_ALG=lz4 for speed or deflate for compression, lzo or zlib if optimisation or availabilty is a problem 43 | COMP_ALG=lz4 44 | 45 | # LOG_DISK_SIZE is the uncompressed disk size. Note zram uses about 0.1% of the size of the disk when not in use 46 | # LOG_DISK_SIZE is expected compression ratio of alg chosen multiplied by log SIZE where 300% is an approx good level. 47 | # lzo/lz4=2.1:1 compression ratio zlib=2.7:1 zstandard=2.9:1 48 | # Really a guestimate of a bit bigger than compression ratio whilst minimising 0.1% mem usage of disk size 49 | LOG_DISK_SIZE=60M 50 | 51 | # mke2fs & mount drive options set for max write perf in volatile ram 52 | # https://manpages.debian.org/jessie/e2fsprogs/mke2fs.8.en.html 53 | # https://manpages.debian.org/stretch/mount/mount.8.en.html 54 | MKFS_OPTS="-O ^has_journal" 55 | MNT_OPTS="-o rw,noatime,async,nosuid,noexec,nodev,nobarrier,nodelalloc" 56 | 57 | # Zram & mount directories defaults can be changed if wished 58 | ZDIR=/opt/zram 59 | HDD_LOG=/opt/zram/hdd.log 60 | ``` 61 | 62 | ### It is working? 63 | ``` 64 | pi@raspberrypi:~ $ df -h 65 | Filesystem Size Used Avail Use% Mounted on 66 | /dev/root 15G 1.2G 13G 9% / 67 | devtmpfs 460M 0 460M 0% /dev 68 | tmpfs 464M 0 464M 0% /dev/shm 69 | tmpfs 464M 6.2M 458M 2% /run 70 | tmpfs 5.0M 4.0K 5.0M 1% /run/lock 71 | tmpfs 464M 0 464M 0% /sys/fs/cgroup 72 | /dev/mmcblk0p1 44M 22M 22M 50% /boot 73 | /dev/zram0 59M 920K 54M 2% /opt/zram/zram0 74 | overlay0 59M 920K 54M 2% /var/log 75 | tmpfs 93M 0 93M 0% /run/user/1000 76 | ``` 77 | ``` 78 | pi@raspberrypi:~ $ zramctl 79 | NAME ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT 80 | /dev/zram0 lz4 60M 1004K 212.3K 424K 4 /opt/zram/zram0 81 | ``` 82 | ``` 83 | pi@raspberrypi:~ $ ls /usr/local/share/log2zram/log 84 | log2zram.log 85 | ``` 86 | 87 | | Compressor name | Ratio | Compression | Decompress. | 88 | |------------------------|----------|-------------|-------------| 89 | |zstd 1.3.4 -1 | 2.877 | 470 MB/s | 1380 MB/s | 90 | |zlib 1.2.11 -1 | 2.743 | 110 MB/s | 400 MB/s | 91 | |brotli 1.0.2 -0 | 2.701 | 410 MB/s | 430 MB/s | 92 | |quicklz 1.5.0 -1 | 2.238 | 550 MB/s | 710 MB/s | 93 | |lzo1x 2.09 -1 | 2.108 | 650 MB/s | 830 MB/s | 94 | |lz4 1.8.1 | 2.101 | 750 MB/s | 3700 MB/s | 95 | |snappy 1.1.4 | 2.091 | 530 MB/s | 1800 MB/s | 96 | |lzf 3.6 -1 | 2.077 | 400 MB/s | 860 MB/s | 97 | 98 | Zstd & Zlib are great for text compression where ratios of up to 3.3 can be obtained. Generally 230% for LZO/LZ4 LOG_DISK_SIZE over the Mem_Limit of Size should be OK, with Zlib/Zstd maybe even up to 350% can be expected. 99 | Most logrotate schedules compress after the second stored log (log.1) and the ratio between uncompressed and compressed log can have much effect. So it dependent on your logging setup with LZO/LZ4 ranging from 210-250% ZSTD/ZLIB 290-350% for the same Mem_Limit Size. 100 | Generally these are minimum compression rates but how much whitespace and zeroes does a file contain? 101 | 102 | ## Uninstall 103 | ``` 104 | sudo sh /usr/local/bin/log2zram/uninstall.sh 105 | ``` 106 | 107 | ## Git Branches & Update 108 | From the command line, enter `cd ` so that you can enter commands for your repository. 109 | Enter `git add --all` at the command line to add the files or changes to the repository 110 | Enter `git commit -m ''` at the command line to commit new files/changes to the local repository. For the , you can enter anything that describes the changes you are committing. 111 | Enter `git push` at the command line to copy your files from your local repository to remote. 112 | -------------------------------------------------------------------------------- /install.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | systemctl -q is-active log2zram && { echo "ERROR: log2zram service is still running. Please run \"sudo service log2zram stop\" to stop it and uninstall"; exit 1; } 4 | [ "$(id -u)" -eq 0 ] || { echo "You need to be ROOT (sudo can be used)"; exit 1; } 5 | [ -d /usr/local/bin/log2zram ] && { echo "Log2Zram is already installed, uninstall first"; exit 1; } 6 | 7 | apt-get install libattr1-dev -y 8 | git clone https://github.com/kmxz/overlayfs-tools 9 | cd overlayfs-tools 10 | make 11 | cd .. 12 | 13 | mkdir -p /usr/local/share/log2zram/ 14 | # log2zram install 15 | install -m 644 log2zram.service /etc/systemd/system/log2zram.service 16 | install -m 755 log2zram /usr/local/bin/log2zram 17 | install -m 644 log2zram.conf /etc/log2zram.conf 18 | install -m 644 uninstall.sh /usr/local/share/log2zram/uninstall.sh 19 | install -m 644 log2zram.logrotate /etc/logrotate.d/00_log2zram 20 | mkdir -p /usr/local/lib/log2zram/ 21 | install -m 755 overlayfs-tools/overlay /usr/local/lib/log2zram/overlay 22 | mkdir -p /usr/local/share/log2zram/log 23 | systemctl enable log2zram 24 | 25 | -------------------------------------------------------------------------------- /log2zram: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | . /etc/log2zram.conf 4 | 5 | 6 | ZRAM_LOG=/var/log 7 | ZSHARE=/usr/local/share/log2zram 8 | ZLOG=${ZSHARE}/log/log2zram.log 9 | 10 | createZramLogDrive () { 11 | # Check Zram Class created 12 | if modprobe --verbose --first-time zram >>${ZLOG} 2>&1 13 | then 14 | DEV_NUM='0' 15 | else 16 | DEV_NUM=$(cat /sys/class/zram-control/hot_add) 17 | fi 18 | echo ${COMP_ALG} > /sys/block/zram${DEV_NUM}/comp_algorithm 19 | echo ${LOG_DISK_SIZE} > /sys/block/zram${DEV_NUM}/disksize 20 | echo ${SIZE} > /sys/block/zram${DEV_NUM}/mem_limit 21 | mke2fs -v -t ext4 $MKFS_OPTS /dev/zram${DEV_NUM} >>${ZLOG} 2>&1 || return 1 22 | } 23 | 24 | mergeOverlay () { 25 | echo "overlay lowerdir=${HDD_LOG},upperdir=${ZDIR}/zram${DEV_NUM}/upper" >>${ZLOG} 26 | cd /usr/local/lib/log2zram/ 27 | ./overlay merge -l "${HDD_LOG}" -u "${ZDIR}/zram${DEV_NUM}/upper" >>${ZLOG} 2>&1 || return 1 28 | sh *.sh >>${ZLOG} 2>&1 || return 1 29 | rm -v *.sh >>${ZLOG} 2>&1 || return 1 30 | } 31 | 32 | case "$1" in 33 | start) 34 | echo "log2zram start $(date +%Y-%m-%d-%H:%M:%S)" >>${ZLOG} 35 | invoke-rc.d rsyslog stop >>${ZLOG} 2>&1 36 | mkdir -p $HDD_LOG >>${ZLOG} 2>&1 37 | mount --verbose --bind $ZRAM_LOG/ $HDD_LOG/ >>${ZLOG} 2>&1 || exit 1 38 | mount --verbose --make-private $HDD_LOG/ >>${ZLOG} 2>&1 39 | createZramLogDrive >>${ZLOG} 2>&1 || exit 1 40 | mkdir -vp ${ZDIR}/zram${DEV_NUM} >>${ZLOG} 2>&1 41 | mount --verbose --types ext4 $MNT_OPTS /dev/zram${DEV_NUM} ${ZDIR}/zram${DEV_NUM}/ >>${ZLOG} 2>&1 || exit 1 42 | mkdir -vp ${ZDIR}/zram${DEV_NUM}/upper ${ZDIR}/zram${DEV_NUM}/workdir ${ZRAM_LOG} >>${ZLOG} 2>&1 43 | mount --verbose --types overlay -o lowerdir=${HDD_LOG},upperdir=${ZDIR}/zram${DEV_NUM}/upper,workdir=${ZDIR}/zram${DEV_NUM}/workdir overlay${DEV_NUM} ${ZRAM_LOG} >>${ZLOG} 2>&1 || exit 1 44 | echo "/zram${DEV_NUM}" > ${ZSHARE}/zram-device-list 45 | invoke-rc.d rsyslog restart >>${ZLOG} 2>&1 46 | journalctl --flush >>${ZLOG} 2>&1 47 | exit 0 48 | ;; 49 | 50 | stop) 51 | echo "log2zram stop $(date +%Y-%m-%d-%H:%M:%S)" >>${ZLOG} 52 | ZRAM_DEV=$(cat ${ZSHARE}/zram-device-list) 53 | DEV_NUM=$(echo "$ZRAM_DEV" | tr -dc '0-9') 54 | invoke-rc.d rsyslog stop >>${ZLOG} 2>&1 55 | if umount --verbose ${ZRAM_LOG}/ >>${ZLOG} 2>&1 56 | then 57 | echo "umount ${ZRAM_LOG}/" >>${ZLOG} 58 | else 59 | sleep .1 60 | invoke-rc.d rsyslog stop >>${ZLOG} 2>&1 61 | umount --verbose -l ${ZRAM_LOG}/ >>${ZLOG} 2>&1 62 | fi 63 | mergeOverlay >>${ZLOG} 2>&1 || exit 1 64 | if umount --verbose ${ZDIR}${ZRAM_DEV}/ >>${ZLOG} 2>&1 65 | then 66 | echo "umount ${ZDIR}${ZRAM_DEV}/" >>${ZLOG} 67 | else 68 | umount --verbose -l ${ZDIR}{ZRAM_DEV}/ >>${ZLOG} 2>&1 69 | fi 70 | if umount --verbose $HDD_LOG/ >>${ZLOG} 2>&1 71 | then 72 | echo "umount $HDD_LOG/" >>${ZLOG} 73 | else 74 | sleep .1 75 | invoke-rc.d rsyslog stop >>${ZLOG} 2>&1 76 | umount --verbose -l $HDD_LOG/ >>${ZLOG} 2>&1 77 | fi 78 | if [ "$DEV_NUM" = "0" ] 79 | then 80 | rmmod zram 81 | else 82 | echo "$DEV_NUM" > /sys/class/zram-control/hot_remove 83 | fi 84 | invoke-rc.d rsyslog restart >>${ZLOG} 2>&1 85 | exit 0 86 | ;; 87 | 88 | *) 89 | echo "Usage: log2ram {start|stop}" >&2 90 | exit 1 91 | esac 92 | -------------------------------------------------------------------------------- /log2zram.conf: -------------------------------------------------------------------------------- 1 | # Size for the zram memory used, it defines the mem_limit for the zram device. 2 | # The default is 20M and is basically enough for minimal applications. 3 | # Because aplications can often vary in logging frequency this may have to be tweaked to suit application . 4 | SIZE=20M 5 | 6 | # COMP_ALG this is any compression algorithm listed in /proc/crypto 7 | # lz4 is fastest with lightest load but deflate (zlib) and Zstandard (zstd) give far better compression ratios 8 | # lzo is very close to lz4 and may with some binaries have better optimisation 9 | # COMP_ALG=lz4 for speed or deflate for compression, lzo or zlib if optimisation or availabilty is a problem 10 | COMP_ALG=lz4 11 | 12 | # LOG_DISK_SIZE is the uncompressed disk size. Note zram uses about 0.1% of the size of the disk when not in use 13 | # LOG_DISK_SIZE is expected compression ratio of alg chosen multiplied by log SIZE where 300% is an approx good level. 14 | # lzo/lz4=2.1:1 compression ratio zlib=2.7:1 zstandard=2.9:1 15 | # Really a guestimate of a bit bigger than compression ratio whilst minimising 0.1% mem usage of disk size 16 | LOG_DISK_SIZE=60M 17 | 18 | # mke2fs & mount drive options set for max write perf in volatile ram 19 | # https://manpages.debian.org/jessie/e2fsprogs/mke2fs.8.en.html 20 | # https://manpages.debian.org/stretch/mount/mount.8.en.html 21 | #MKFS_OPTS="-O ^has_journal" 22 | MKFS_OPTS="" 23 | #MNT_OPTS="-o rw,noatime,async,nosuid,noexec,nodev,nobarrier,nodelalloc" 24 | MNT_OPTS="-o defaults,noatime" 25 | # Zram & mount directories defaults can be changed if wished 26 | ZDIR=/opt/zram 27 | HDD_LOG=/opt/zram/hdd.log 28 | 29 | 30 | 31 | -------------------------------------------------------------------------------- /log2zram.logrotate: -------------------------------------------------------------------------------- 1 | olddir /opt/zram/log.old 2 | createolddir 755 root root 3 | renamecopy 4 | 5 | /usr/local/share/log2zram/log/log2zram.log 6 | { 7 | rotate 7 8 | daily 9 | missingok 10 | notifempty 11 | delaycompress 12 | compress 13 | } -------------------------------------------------------------------------------- /log2zram.service: -------------------------------------------------------------------------------- 1 | [Unit] 2 | Description=log2zram 3 | After=local-fs.target 4 | Before=reboot.target halt.target zram-swap-config.service zramdrive.service 5 | 6 | [Service] 7 | Type=oneshot 8 | ExecStart=/usr/local/bin/log2zram start 9 | ExecStop=/usr/local/bin/log2zram stop 10 | TimeoutSec=240 11 | RemainAfterExit=yes 12 | 13 | [Install] 14 | WantedBy=basic.target 15 | 16 | -------------------------------------------------------------------------------- /uninstall.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | 3 | if [ "$(id -u)" -eq 0 ] 4 | then 5 | systemctl stop log2zram 6 | systemctl disable log2zram 7 | rm /etc/systemd/system/log2zram.service 8 | rm /usr/local/bin/log2zram 9 | rm /etc/log2zram.conf 10 | rm /etc/logrotate.d/00_log2zram 11 | echo "Log2Zram is uninstalled, removing the uninstaller in progress" 12 | rm -rf /usr/local/share/log2zram 13 | rm -rf /usr/local/lib/log2zram 14 | echo "##### Reboot isn't needed #####" 15 | else 16 | echo "You need to be ROOT (sudo can be used)" 17 | fi 18 | --------------------------------------------------------------------------------