├── LICENSE ├── README.md └── microbhyve-from-jail.txt /LICENSE: -------------------------------------------------------------------------------- 1 | BSD 2-Clause License 2 | 3 | Copyright (c) 2022, CBSD Project 4 | All rights reserved. 5 | 6 | Redistribution and use in source and binary forms, with or without 7 | modification, are permitted provided that the following conditions are met: 8 | 9 | 1. Redistributions of source code must retain the above copyright notice, this 10 | list of conditions and the following disclaimer. 11 | 12 | 2. Redistributions in binary form must reproduce the above copyright notice, 13 | this list of conditions and the following disclaimer in the documentation 14 | and/or other materials provided with the distribution. 15 | 16 | THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" 17 | AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 18 | IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 19 | DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 20 | FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 21 | DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 22 | SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 23 | CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, 24 | OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 25 | OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. 26 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # Microbhyve: generate a bootable working minimal installation of FreeBSD for bhyve VM 2 | 3 | ## Introduction 4 | 5 | This article describes how to generate a minimalistic image of the [FreeBSD OS](https://www.freebsd.org) (architecture: amd64) for the [bhyve](https://en.wikipedia.org/wiki/Bhyve) virtual machine using the `jail2iso` script of the [CBSD project](https://github.com/cbsd/cbsd) as an example. As a result of these works, we get a **12MB** distribution kit and a working network stack with the ability to remotely access via SSH, as well as a set of elementary Unix utilities. The consumption of RAM by such an installation in multi-user mode does not exceed **80MB**: 6 | 7 | ``` 8 | rctl -hu process:20349 |grep memoryuse 9 | memoryuse=78M 10 | vmemoryuse=569M 11 | ``` 12 | ![microtop](https://user-images.githubusercontent.com/926409/168559735-b07c9c92-afd9-4376-b841-dd8503a8804a.png) 13 | 14 | Why is it needed and to whom: 15 | 16 | - can be used as firmware when creating embedded products and services; 17 | - research activities, studying how FreeBSD boot works; 18 | 19 | For example, the initial microcontainer image with SSH access is used in the [MyBee project](https://github.com/myb-project/guide) when creating a [Kubernetes](https://kubernetes.io/) cluster. In this case, the [FreeBSD jail](https://docs.freebsd.org/en/books/handbook/jails/) is created as a jump-host environment for accessing a Kubernetes cluster, which has utilities for working with K8S: `kubectl`, `helm`, `k9s`. Here we go a little further and use CBSD `jail2iso` script to get the VM image. At the same time, the minimalism of the kernel is achieved by eliminating all options and drivers that are not required when working as a VM. Basically, only the network stack and the virtio driver remain in the kernel, which allows you to get a **2MB** kernel. 20 | 21 | ## Preparation 22 | 23 | We assume that we are getting a "bhyve" (and jail) image of the same version of FreeBSD we are working on. For the purposes of this article, this is FreeBSD 13.1-RELEASE. If you are using a different version, use the appropriate numbering instead of **13.1**. 24 | 25 | There are many ways to get a custom lightweight FreeBSD environment. One option is to use KNOBS (the *WITH/WITHOUT_* option in [src.conf](https://man.freebsd.org/src.conf/5), which can be used to control which components to include (or exclude) when building. Another option is to have a full installation of the base system and only copy the files we need. We will use the second option because it allows us to minimize the environment as much as possible - we will control each file. 26 | 27 | To do this, create an empty CBSD container in rootfs called micro1 with RW rights: 28 | 29 | > cbsd jcreate jname=micro1 baserw=1 ver=empty applytpl=0 30 | 31 | Now in the *~cbsd/jails-data/micro1-data* directory (which is the rootfs for the created container), we need to create an environment hierarchy that will allow us to start the container, the SSH service and provide authorization/login. For the `jail2iso` script, CBSD has the `copy-binlib` script, which copies the necessary files by index and, if it is an executable file, libraries for it (search through the [ldd](https://man.freebsd.org/ldd/1) utility). 32 | 33 | If you want to relax and get the environment as soon as possible, skip this chapter, as the work described below is already part of CBSD. For the even more impatient, this repository contains the script that you can just run. 34 | 35 | Generating such an index is an exploratory activity: you can know by heart all the files involved and needed to boot FreeBSD (in which case you are an incredible FreeBSD hacker!), or, as the author of this article, find it in practice. To do this, you can use any file access tracking methods, such as DTRACE. But we will consider two other options. The first is to track access to files by the "atime" attribute of the file system. To do this, let's create a full-fledged CBSD container: 36 | 37 | > cbsd jcreate jname=test1 runasap=1 baserw=1 38 | 39 | Since we are working on ZFS, a separate named dataset is created for the container: 40 | 41 | ``` 42 | zfs list | grep test1 43 | jails/test1 632M 550G 632M /usr/jails/jails-data/test1-data 44 | ``` 45 | 46 | Enable the atime attribute for jails/test1: 47 | 48 | > zfs set atime=on jails/test1 49 | 50 | Now that you have configured the SSH service and adding user for login inside the container, you can restart the container and login via SSH. After that, the [find](https://man.freebsd.org/find/1) command will show files that "participated" in this activity (pay attention to the `-atime -5m` modifier - it shows the changed time for the last 5 minutes, so the interval between entering the jail and running `find` command should be no more than these values: 51 | 52 | ``` 53 | find ~cbsd/jails-data/test1-data -type f -atime -5m -print 54 | /usr/jails/jails-data/test1-data/var/run/sshd.pid 55 | /usr/jails/jails-data/test1-data/var/run/motd 56 | /usr/jails/jails-data/test1-data/var/run/ld-elf.so.hints 57 | /usr/jails/jails-data/test1-data/var/run/utx.active 58 | /usr/jails/jails-data/test1-data/var/log/utx.lastlogin 59 | /usr/jails/jails-data/test1-data/root/.cshrc 60 | /usr/jails/jails-data/test1-data/root/.history 61 | /usr/jails/jails-data/test1-data/root/.login 62 | /usr/jails/jails-data/test1-data/lib/libcrypto.so.111 63 | /usr/jails/jails-data/test1-data/bin/pwd 64 | ... 65 | ``` 66 | 67 | Obviously, these files must be present in our 'gold' micro-container. 68 | 69 | or **second method**: Another convenient option for monitoring the required files is [AUDITD](https://man.freebsd.org/auditd/8). Set up AUDIT to monitor files on the host system: 70 | 71 | ``` 72 | cat > /etc/security/audit_control < service auditd onestart 85 | 86 | Relogin to host and restart the `test1` container (AUDITD does not log events in sessions started **before** AUDITD started). Now, through the [praudit](https://man.freebsd.org/praudit/1) utility, you will be able to see all the files that are being read and written. Filter the output to get files related to the `test1` container only: 87 | 88 | > praudit /dev/auditpipe | grep ^path| grep test1 89 | 90 | ## Copy file structure to micro-container 91 | 92 | Using the CBSD `copy-binlib` script and the already existing index file for a minimal FreeBSD + SSH environment in the `test1` container: 93 | 94 | ``` 95 | cbsd copy-binlib basedir=/ chaselibs=1 dstdir=/usr/jails/jails-data/micro1-data filelist=/usr/local/cbsd/share/FreeBSD-microbhyve.txt.xz 96 | ``` 97 | 98 | Note: make sure */usr/jails* is the correct path for the CBSD environment in your installation. 99 | 100 | ## Micro (gold) container customization 101 | 102 | With a minimal structure, we can make a number of necessary settings, such as allowing SSH access for the root user, configuring the network, and so on. We will configure the virtual machine network through the */etc/rc.local* file - are you striving for a minimal structure? ;-) so don't copy the entire */etc/rc.d* directory: 103 | 104 | ``` 105 | cbsd sysrc jname=micro1 \ 106 | sshd_flags="-oUseDNS=no -oPermitRootLogin=yes" \ 107 | root_rw_mount="YES" \ 108 | sshd_enable=YES \ 109 | rc_startmsgs="YES" 110 | 111 | cat > /usr/jails/jails-data/micro1-data/etc/rc.local < cbsd jail2iso name=BHYVE jname=micro1 dstdir=/tmp media=bhyve freesize=4m efi=1 ver=native 163 | 164 | As a result, a small image **/tmp/micro1-13.1_amd64.img** will be generated: 165 | 166 | ``` 167 | du -sh /tmp/micro1-13.1_amd64.img 168 | 12M /tmp/micro1-13.1_amd64.img 169 | ``` 170 | 171 | which you can check immediately via the `bhyve` startup script: 172 | 173 | > sh /usr/share/examples/bhyve/vmrun.sh -d /tmp/micro1-13.1_amd64.img micro1 174 | 175 | However, it is better to create a CBSD virtual machine to set up the network and so on. To do this, stop the running VM (if started): 176 | 177 | ``` 178 | ps axfwww | grep "bhyve: micro1" |grep -v grep 179 | kill 180 | ``` 181 | 182 | Create VM. Here we can create a disk of any size - we will overwrite it anyway: 183 | 184 | ``` 185 | cbsd bcreate jname=micro vm_os_type=freebsd vm_os_profile=FreeBSD-x64-13.1 imgtype=md imgsize=1g ip4_addr=10.0.100.10 186 | ``` 187 | 188 | Let's overwrite the disk created by the `bcreate` with a `micro` file from CBSD `jail2iso`: 189 | 190 | > cp -a /tmp/micro1-13.1_amd64.img ~cbsd/jails-data/micro-data/dsk1.vhd 191 | 192 | Now we can start the VM and by calling `ssh root@10.0.100.10` you can log into the VM using the password of the 'root' user from your host system, because we copied the files from the host system ( do you remember `cbsd copy-binlib basedir=/` ? ). 193 | 194 | ## Errata 195 | 196 | The kernel config file is really minimal. You can view its contents: *~cbsd/etc/defaults/FreeBSD-kernel-BHYVE-amd64-13.1*. 197 | 198 | If you open the VNC console on bhyve terminal, apart from the boot menu, you will not see the boot process and the console, since the devices responsible for outputting information in UEFI mode are also **turned off**. You can rebuild the kernel by uncommenting the entries: 199 | 200 | ``` 201 | device vt 202 | device vt_efifb 203 | ``` 204 | 205 | 206 | :bangbang: | :warning: Attention! Do not edit the *~cbsd/etc/defaults/FreeBSD-kernel-BHYVE-amd64-13.1* file, instead copy it or create your own 'FreeBSD-kernel-XXXX-amd64-13.1' file in the *~cbsd/etc/* directory because the files in *~cbsd/etc/defaults* is overwritten by `cbsd initenv`. 207 | :---: | :--- 208 | 209 | ## Afterword, instead of conclusion 210 | 211 | [CBSD](https://github.com/cbsd/cbsd) is not only a jail and VM management tool, but also a framework that includes various auxiliary utilities for working with jail and VM, since they reuse the lots of common code of basic functions and maintaining them separately from CBSD is inappropriate. One of these scripts is `jail2iso`, with which you can get various bonuses from **CBSD**. For example, we all know such LiveCD distributions as [NomadBSD](https://nomadbsd.org/), [GhostBSD](https://ghostbsd.org/), [helloSystem](https://github.com/helloSystem/hello), etc. Using the CBSD `jail2iso` script, you can easily create your own LiveCD distributions based on FreeBSD, preparing and checking the settings and services in the jail container in advance. In a similar way, you can generate the most stripped-down FreeBSD distribution, where there is only your service and nothing more. 212 | 213 | Since we used compression only for the kernel, but not rootfs, it is possible to get an image size of up to **5MB** without changing the file structure (for example, with specialized busybox-like projects) by using `uzip/geom_uzip`, but more on that in another article. 214 | 215 | _ 216 | - **About FreeBSD OS**: Freely distributed under the most liberal BSD license, a general-purpose OS, the code of which is not affiliated with any companies; 217 | - **About the CBSD project**: Founded in 2013, the project aims to create a framework for working with FreeBSD virtual environments to facilitate the creation of cluster and cloud solutions based on FreeBSD; 218 | - **About the 'jail2iso' script**: the script was written and included in **CBSD** since 2014, the purpose of the script is to convert the jail of the FreeBSD operating system into a bootable media: ISO, memstick or virtual machine image. It is convenient for creating custom liveCD builds of FreeBSD - you can pre-configure and test all services in jail and make sure that they are correct and working, get a working ISO image or VM image with one command; 219 | - **About bhyve**: a type 2 hypervisor included with FreeBSD that can run modern operating systems; 220 | 221 | ## Demo 222 | 223 | [![asciicast](https://asciinema.org/a/494721.svg)](https://asciinema.org/a/494721) 224 | -------------------------------------------------------------------------------- /microbhyve-from-jail.txt: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | # HowTo/Sample how to create micro (~11 MB) FreeBSD bhyve guest 3 | 4 | # use 'micro1' as origin jail 5 | jname="micro1" 6 | myip="10.0.100.10/24" 7 | mygw="10.0.100.1" 8 | 9 | . /etc/rc.conf 10 | 11 | if [ -z "${cbsd_workdir}" ]; then 12 | echo "no such cbsd_workdir in /etc/rc.conf" 13 | exit 1 14 | fi 15 | 16 | my_ver=$( /usr/local/cbsd/misc/elf_tables --freebsdver /bin/sh ) 17 | echo "Build bhyve image for FreeBSD ${my_ver} via jail: ${jname}" 18 | echo "network settings: IP: ${myip}, GW: ${mygw}" 19 | echo 20 | 21 | set -o errexit 22 | set -o xtrace 23 | 24 | # remove old jail 25 | cbsd jstatus jname=${jname} || cbsd jremove jname=${jname} 26 | 27 | cbsd jcreate jname=${jname} baserw=1 ver=empty applytpl=0 28 | cbsd copy-binlib basedir=/ chaselibs=1 dstdir=${cbsd_workdir}/jails-data/${jname}-data filelist=/usr/local/cbsd/share/FreeBSD-microbhyve.txt.xz 29 | 30 | cbsd sysrc jname=${jname} \ 31 | sshd_flags="-oUseDNS=no -oPermitRootLogin=yes" \ 32 | root_rw_mount="YES" \ 33 | sshd_enable=YES \ 34 | ifconfig_vtnet0="inet 10.0.100.10/24 up" \ 35 | defaultrouter="10.0.100.1" \ 36 | rc_startmsgs="YES" 37 | 38 | # strip debug info 39 | find ${cbsd_workdir}/jails-data/${jname}-data/ -type f -perm +111 -exec strip {} \; 40 | 41 | # unset autoboot_delay via custom loader.conf: 42 | cat > ${cbsd_workdir}/jails-system/${jname}/loader.conf < ${cbsd_workdir}/jails-data/${jname}-data/etc/rc.local <