├── README.md ├── bkq.sh ├── launch-qemu-buildroot.sh ├── launch-qemu.sh └── setup_dev.sh /README.md: -------------------------------------------------------------------------------- 1 | # QEMU with kernel modules 2 | 3 | Using QEMU is nice, however, using kernel modules together with QEMU is usually 4 | not very convenient. 5 | 6 | Some possible options are: 7 | 1) Build everything as built-in instead of as kernel modules. 8 | However, sometimes you actually need to test with kernel modules, so this does 9 | not always work. 10 | 2) Put all the kernel modules in the initramfs. 11 | However, the contents of the initramfs is usually no longer accessible after 12 | boot. (You could modify your initscript to bind mount your initramfs before 13 | performing the chroot, but even then, this would not allow us to update the 14 | kernel modules while the VM is running.) 15 | 3) Use virtio-blk and mkfs.ext2 on a file, and put all the kernel modules in 16 | that file/filesystem. 17 | However, this also does not allow us to update the kernel modules while the VM 18 | is running. 19 | 4) Use VirtioFS. 20 | However, VirtioFS requires a separate daemon. 21 | (VirtioFS is supposed to be faster than VirtFS, but for simply loading kernel 22 | modules, performance is not the only thing we care about.) 23 | 5) Use VirtFS (Plan 9 folder sharing over Virtio). 24 | VirtFS support is built-in to QEMU and requires no extra daemon. 25 | 26 | This guide will explain how to use VirtFS (Plan 9 folder sharing over Virtio). 27 | 28 | Install QEMU. 29 | On Ubuntu/Debian: 30 | ``` 31 | sudo apt install qemu-system-x86 qemu-utils 32 | ``` 33 | 34 | or on Fedora/RHEL: 35 | ``` 36 | sudo dnf install qemu-system-x86 qemu-img 37 | ``` 38 | 39 | Create a new directory were you will keep the new code, e.g.: 40 | ``` 41 | mkdir ~/src 42 | ``` 43 | 44 | Clone and build the kernel that you will run inside QEMU: 45 | ``` 46 | cd ~/src 47 | git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git 48 | cd linux 49 | cp /boot/config-$(uname -r) .config 50 | make olddefconfig 51 | ./scripts/config --enable CONFIG_BTRFS_FS 52 | ./scripts/config --enable CONFIG_BTRFS_FS_POSIX_ACL 53 | ./scripts/config --enable CONFIG_PSI 54 | ./scripts/config --enable CONFIG_MEMCG 55 | ./scripts/config --enable CONFIG_CRYPTO_LZO 56 | ./scripts/config --enable CONFIG_ZRAM 57 | ./scripts/config --enable CONFIG_ZRAM_DEF_COMP_LZORLE 58 | ./scripts/config --enable CONFIG_ISO9660_FS 59 | ./scripts/config --enable CONFIG_VFAT_FS 60 | ./scripts/config --enable CONFIG_NET_9P 61 | ./scripts/config --enable CONFIG_NET_9P_VIRTIO 62 | ./scripts/config --enable CONFIG_9P_FS 63 | ./scripts/config --enable CONFIG_9P_FS_POSIX_ACL 64 | ./scripts/config --enable CONFIG_VIRTIO 65 | ./scripts/config --enable CONFIG_VIRTIO_PCI 66 | ./scripts/config --enable CONFIG_PCI 67 | ./scripts/config --enable CONFIG_VIRTIO_BLK 68 | ./scripts/config --enable CONFIG_VIRTIO_NET 69 | make olddefconfig 70 | make -j$(nproc) 71 | ``` 72 | 73 | Create a new directory were you will keep the QEMU scripts and data, e.g.: 74 | ``` 75 | mkdir ~/qemu-data 76 | ``` 77 | 78 | Get a QEMU friendly rootfs image (this guide uses Fedora) using: 79 | ``` 80 | cd ~/qemu-data 81 | wget https://download.fedoraproject.org/pub/fedora/linux/releases/40/Cloud/x86_64/images/Fedora-Cloud-Base-UEFI-UKI.x86_64-40-1.14.qcow2 82 | ``` 83 | 84 | Create a new image that will be used as our rootfs, based on the Fedora image that we just downloaded. 85 | It will not modify the Fedora image we just downloaded. Differences will be saved in fedora-rootfs-overlay.img. 86 | It will not take up 128 GB, it will simply allow it to grow up to that size, since this new file will automatically 87 | increase in size when we install new packages. 88 | ``` 89 | qemu-img create -f qcow2 -b Fedora-Cloud-Base-UEFI-UKI.x86_64-40-1.14.qcow2 -F qcow2 fedora-rootfs-overlay.img 128G 90 | ``` 91 | 92 | Install cloud-localds on Ubuntu/Debian: 93 | ``` 94 | sudo apt install cloud-image-utils genisoimage 95 | ``` 96 | 97 | or on Fedora/RHEL: 98 | ``` 99 | sudo dnf install cloud-utils genisoimage 100 | ``` 101 | 102 | We need to tell Fedora to create a user that you can use inside your new image. 103 | This is done using cloud-config. The username will be the same as your current 104 | user ($USER). There will be no password for this user, the only way to login 105 | will be via SSH. Start by saving the value of your public SSH key into a 106 | variable: 107 | ``` 108 | export PUB_KEY=$(cat ~/.ssh/id_ed25519.pub) 109 | ``` 110 | 111 | Create the user-data file by pasting the following into a terminal and press enter: 112 | ``` 113 | cat >user-data <> /etc/fstab' 181 | ``` 182 | 183 | The **x-initrd.mount** mount option is interpreted by systemd, in order to mount 184 | the share early, before running systemd-udev-trigger.service (which will 185 | coldplug all devices). 186 | 187 | The **context="system_u:object_r:modules_object_t:s0"** mount option is needed 188 | such that the SELinux label will be set correctly. Without this option SELinux 189 | will not allow you to load kernel modules from the share, and would result in 190 | something like the following being printed in the journal (which can be dumped 191 | using e.g. **journalctl -b**): 192 | ``` 193 | audit[570]: AVC avc: denied { module_load } for pid=570 comm="(udev-worker)" path="/usr/lib/modules/6.10.0-rc3/kernel/drivers/ata/libahci.ko" dev="9p" ino=33011355 scontext=system_u:system_r:udev_t:s0-s0:c0.c1023 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=system permissive=0 194 | ``` 195 | 196 | After modifying fstab, reboot the QEMU machine: 197 | ``` 198 | sudo reboot 199 | ``` 200 | 201 | Log on to your QEMU machine again: 202 | ``` 203 | ssh -A -p 10222 localhost 204 | ``` 205 | 206 | While logged on to your QEMU machine, list the loaded modules: 207 | ``` 208 | lsmod 209 | ``` 210 | 211 | If everything is working, the VirtFS share (containing the kernel modules) 212 | should have been mounted before udev coldplugged all devices (before 213 | systemd-udev-trigger.service), and lsmod should show a bunch of modules that 214 | have been automatically loaded, e.g. something like: 215 | ``` 216 | Module Size Used by 217 | intel_rapl_msr 20480 0 218 | ppdev 24576 0 219 | intel_rapl_common 57344 1 intel_rapl_msr 220 | pktcdvd 65536 0 221 | kvm_amd 217088 0 222 | ccp 180224 1 kvm_amd 223 | kvm 1441792 1 kvm_amd 224 | crct10dif_pclmul 12288 1 225 | crc32_pclmul 12288 0 226 | crc32c_intel 16384 0 227 | polyval_clmulni 12288 0 228 | polyval_generic 12288 1 polyval_clmulni 229 | ghash_clmulni_intel 16384 0 230 | bochs 20480 0 231 | sha512_ssse3 53248 0 232 | drm_vram_helper 28672 1 bochs 233 | drm_ttm_helper 12288 2 bochs,drm_vram_helper 234 | parport_pc 53248 0 235 | sha256_ssse3 36864 0 236 | ttm 114688 2 drm_vram_helper,drm_ttm_helper 237 | sha1_ssse3 32768 0 238 | parport 81920 2 parport_pc,ppdev 239 | floppy 163840 0 240 | i2c_piix4 40960 0 241 | pcspkr 12288 0 242 | joydev 32768 0 243 | qemu_fw_cfg 20480 0 244 | serio_raw 20480 0 245 | ata_generic 12288 0 246 | pata_acpi 12288 0 247 | msr 12288 0 248 | loop 45056 0 249 | fuse 229376 1 250 | ``` 251 | 252 | If you need to modify a kernel module, you do not need to restart your QEMU 253 | machine, you can simply edit the kernel source and then run: 254 | ``` 255 | ./bkq.sh 256 | ``` 257 | 258 | This is of course assuming that your changes did not change the ABI between the 259 | kernel and your module. (In case of an ABI change, you need to reboot your VM.) 260 | 261 | Then on your QEMU machine, run: 262 | ``` 263 | sudo rmmod 264 | sudo modprobe 265 | ``` 266 | 267 | and enjoy being able to quickly test new changes without rebooting your VM. 268 | 269 |
270 | 271 | NOTE: The bkq.sh script sets LOCALVERSION= in order to avoid silly suffixes 272 | being added to the kernel modules directory (e.g. -dirty or +). Without this, 273 | the running kernel version string (e.g. 6.10.0-rc4) would not match the kernel 274 | version string of your build (e.g. 6.10.0-rc4-dirty), which means that you 275 | would not be able rmmod + modprobe without rebooting your VM. 276 | 277 | This means that if you invoke **make** directly without having LOCALVERSION 278 | set to the empty string, make will rebuild all kernel modules. Therefore, either 279 | always build using bkq.sh, or if invoking make directly, make sure that the 280 | environment variable LOCALVERSION is set to the empty string, e.g. 281 | **make LOCALVERSION=**, in order to avoid recompiling modules needlessly. 282 | -------------------------------------------------------------------------------- /bkq.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh -e 2 | 3 | # Build Kernel for QEMU (bkq) 4 | 5 | if [ ! -e ".git" ]; then 6 | echo "you are not standing in a git tree or a git worktree" 7 | echo "" 8 | echo "while standing in your kernel directory:" 9 | echo "" 10 | echo "usage: bqk.sh" 11 | exit 12 | fi 13 | 14 | # remove tmp files 15 | # virtio-9p does cache things, but if a file is removed + added, the client 16 | # will see the new file, e.g. if a rebuild is done without rebooting the VM 17 | rm -rf tmp-modules/lib/modules/* 18 | 19 | # set LOCALVERSION to the empty string, to avoid scripts/setlocalversion from 20 | # appending -dirty or + to the kernel version string. If we do not do this, the 21 | # kernel modules (which uses the kernel version string) might get installed to a 22 | # directory that does not match the kernel version string of the running kernel. 23 | make -j$(nproc) LOCALVERSION= 24 | make -j$(nproc) -s modules_install INSTALL_MOD_PATH=tmp-modules INSTALL_MOD_STRIP=1 LOCALVERSION= 25 | -------------------------------------------------------------------------------- /launch-qemu-buildroot.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | QEMU_BINARY=qemu-system-x86_64 4 | KERNEL=arch/x86/boot/bzImage 5 | ROOTFS=~/qemu-data/buildroot-rootfs.ext2 6 | QEMU_GUEST_SSH_FWD_PORT=10222 7 | RAM=4G 8 | 9 | # Change this to contain the PCI address of the device you want to pass through 10 | # to QEMU. It has to be the same PCI address as you specified in setup_dev.sh 11 | PCI_BDF=0000:00:17.0 12 | 13 | # do not change this 14 | MODULES=tmp-modules/lib/modules 15 | 16 | if [ ! -f "$KERNEL" ]; then 17 | echo "you are not standing in a kernel tree" 18 | exit 19 | fi 20 | 21 | exec $QEMU_BINARY -m $RAM -cpu host -smp $(nproc) -enable-kvm -nographic \ 22 | -kernel $KERNEL -drive file=$ROOTFS,if=virtio,format=raw \ 23 | -fsdev local,id=test_dev,path=$MODULES,security_model=mapped,multidevs=remap \ 24 | -device virtio-9p-pci,fsdev=test_dev,mount_tag=tag_modules \ 25 | -append "rootwait root=/dev/vda console=tty1 console=ttyS0" \ 26 | -net nic,model=virtio -net user,hostfwd=tcp::$QEMU_GUEST_SSH_FWD_PORT-:22 \ 27 | -drive driver=null-co,read-zeroes=on,latency-ns=50000000,if=none,id=disk \ 28 | -device ich9-ahci,id=ahci -device ide-hd,drive=disk,bus=ahci.0 29 | #-device vfio-pci,host=$PCI_BDF 30 | -------------------------------------------------------------------------------- /launch-qemu.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | QEMU_BINARY=qemu-system-x86_64 4 | KERNEL=arch/x86/boot/bzImage 5 | ROOTFS=~/qemu-data/fedora-rootfs-overlay.img 6 | USER_DATA=~/qemu-data/user-data.img 7 | QEMU_GUEST_SSH_FWD_PORT=10222 8 | RAM=4G 9 | 10 | # do not change this 11 | MODULES=tmp-modules/lib/modules 12 | 13 | if [ ! -f "$KERNEL" ]; then 14 | echo "you are not standing in a kernel tree" 15 | exit 16 | fi 17 | 18 | exec $QEMU_BINARY -m $RAM -cpu host -smp $(nproc) -enable-kvm -nographic \ 19 | -drive file=$ROOTFS,format=qcow2,if=virtio \ 20 | -drive file=$USER_DATA,format=raw,if=virtio \ 21 | -fsdev local,id=test_dev,path=$MODULES,security_model=mapped,multidevs=remap \ 22 | -device virtio-9p-pci,fsdev=test_dev,mount_tag=tag_modules \ 23 | -kernel $KERNEL \ 24 | -append "console=ttyS0 root=/dev/vda3 rootflags=subvol=root net.ifnames=0 nokaslr oops=panic panic=0" \ 25 | -net nic,model=virtio -net user,hostfwd=tcp::$QEMU_GUEST_SSH_FWD_PORT-:22 \ 26 | -drive driver=null-co,read-zeroes=on,latency-ns=50000000,if=none,id=disk \ 27 | -device ich9-ahci,id=ahci -device ide-hd,drive=disk,bus=ahci.0 28 | -------------------------------------------------------------------------------- /setup_dev.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | 3 | # Change this to contain the PCI address of the device you want to pass through 4 | # to QEMU. In this example, we will use: 5 | # 0000:00:17.0 SATA controller: Intel Corporation C620 Series Chipset Family SATA Controller [AHCI mode] (rev 0a) 6 | # Therefore, the variable will be set to: 0000:00:17.0 7 | PCI_BDF=0000:00:17.0 8 | 9 | sudo modprobe vfio-pci 10 | 11 | bind_driver() { 12 | local drv="$1" 13 | for DEV in $(ls /sys/bus/pci/devices/$PCI_BDF/iommu_group/devices) ; do 14 | if [ $(( 0x$(setpci -s $DEV HEADER_TYPE) & 0x7f )) -eq 0 ]; then 15 | sudo sh -c "echo $drv > /sys/bus/pci/devices/$DEV/driver_override" 16 | sudo sh -c "echo $DEV > /sys/bus/pci/devices/$DEV/driver/unbind" 17 | sudo sh -c "echo $DEV > /sys/bus/pci/drivers_probe" 18 | fi 19 | done 20 | } 21 | 22 | if [ $# -eq 1 ] && [ $1 = reset ]; then 23 | bind_driver "" 24 | else 25 | bind_driver "vfio-pci" 26 | group=$(readlink /sys/bus/pci/devices/$PCI_BDF/iommu_group) 27 | group_nbr=$(printf '%s' $group | sed 's,.*/iommu_groups/,,') 28 | sudo chown $USER:$USER /dev/vfio/$group_nbr 29 | fi 30 | --------------------------------------------------------------------------------