├── README.md └── start-qemu.sh /README.md: -------------------------------------------------------------------------------- 1 | # Linux Kernel Debugging 2 | How to create a setup for linux kernel debugging using buildroot, qemu and gdb. 3 | 4 | # Part 1: Compile linux kernel and rootfs 5 | We are going to compile linux kernel and rootfs using buildroot. 6 | Buildroot supplies all the toolchain which needed for automate the process of compiling linux kernel and rootfs. 7 | Buildroot was created for creating linux embedded/minimal systems. 8 | However, if your purpose is developing or debugging the linux kernel its really good solution. 9 | 10 | First, Clone buildroot repository (latest version): 11 | 12 | 1. cd ~/workspace 13 | 2. git clone https://github.com/buildroot/buildroot.git 14 | 3. cd buildroot 15 | 16 | ## Config buildroot 17 | Now we need to configure buildroot in order to build every packages with debug symbols. 18 | In order to be able to ssh to the vm we'll add the openssh package. 19 | 20 | 4. make qemu_x86_64_defconfig // Generate buildroot default config 21 | 5. make menuconfig 22 | 23 | 24 | | Option | Corresponding config symbols | 25 | | ------ | -------------------------------------------------------------------------------------------------------------------------| 26 | | In Build options, toggle “build packages with debugging symbols” | `BR2_ENABLE_DEBUG` | 27 | | In System configuration, untoggle "Run a getty (login prompt) after boot" | `BR2_TARGET_GENERIC_GETTY` | 28 | | In System configuration, enter root password | `BR2_TARGET_GENERIC_ROOT_PASSWD` | 29 | | In Target packages, Networking applications, toggle "openssh" | `BR2_PACKAGE_OPENSSH` | 30 | | In Filesystem images, change to ext4 root filesystem | `BR2_TARGET_ROOTFS_EXT2` & `BR2_TARGET_ROOTFS_EXT2_4` | 31 | 32 | > The path to the options may change between buildroot versions, if an option is missing validate the symbols 33 | > are set appropriately using `cat .config | grep ` from buildroot's folder 34 | 35 | #### Notes: 36 | If you want to tell buildroot to download and compile antoher version of linux kernel: 37 | * In Toolchain, change “linux version” to 38 | * In Toolchain, change “Custom kernel version headers series” to 39 | * In Kernel, change “Kernel version" to 40 | 41 | ## Config linux kernel 42 | Now we are going to configure linux kernel in order to compile it with debug symbols. 43 | Before opening the menuconfig it will trigger buildroot to download linux kernel source code. 44 | 45 | 6. make linux-menuconfig 46 | 47 | | Option | Corresponding config symbols | 48 | | ------ | -------------------------------------------------------------------------------------------------------------------------| 49 | | In “Kernel hacking”, toggle “Kernel debugging” | `CONFIG_DEBUG_KERNEL` | 50 | | In “Kernel hacking/Compile-time checks and compiler options”, toggle “Compile the kernel with debug info” | `DEBUG_INFO` | 51 | | In “Kernel hacking”, toggle “Compile the kernel with frame pointers” | `FRAME_POINTER` | 52 | 53 | ## Compile linux kernel and rootfs 54 | Now lets compile everything: 55 | 56 | 7. make -j8 57 | 58 | Important files: 59 | 60 | * output/build/linux- contains the downloaded kernel source code 61 | * output/images/bzImage is the compressed kernel image 62 | * output/images/rootfs.ext4 is the rootfs 63 | * output/build/linux-/vmlinux is the raw kernel image 64 | 65 | # Part 2: Debugging linux kernel using qemu and gdb 66 | 67 | After we compiled the linux kernel and rootfs we can debug it. 68 | Our emulator will be qemu because qemu is a really lightweight emulator that can be easily configured to run almost anything and qemu works fine with kvm which improves the performance. 69 | 70 | 8. First, we'll enable ssh connections by changing `sshd_config` 71 | ``` 72 | sudo -i 73 | cd /path/to/buildroot/output/images 74 | mkdir /mnt/dbg_kernel_fs 75 | mount rootfs.ext2 /mnt/dbg_kernel_fs 76 | echo "PermitRootLogin yes" >> /mnt/dbg_kernel_fs/etc/ssh/sshd_config 77 | umount /mnt/dbg_kernel_fs 78 | rmdir /mnt/dbg_kernel_fs 79 | exit 80 | ``` 81 | 82 | Now we'll convert our raw rootfs to qemu format which will enable us to create snapshots later on. 83 | 84 | 9. 85 | ``` 86 | cd ./output/images 87 | qemu-img convert -f raw -O qcow2 rootfs.ext2 rootfs.qcow2 88 | ``` 89 | 90 | > Note: rootfs.ext4 is just a symlink to rootfs.ext2 91 | 92 | ## Launch the vm 93 | 94 | Copy/Replace `start-qemu.sh` from this repo into buildroot/output/images. 95 | This shell script runs qemu with customized flags explained below: 96 | 97 | * -monitor unix:qemu-monitor-socket,server,nowait -> creates a socket file named `qemu-monitor-socket` to which we'll connect with socat for the qemu monitoring 98 | * -enable-kvm -> kvm is a virtualization solution for linux which use hardware virtualization extensions, we will use it in order to improve the vm performance 99 | * -cpu host -> use host cpu, we will use it in order to improve the vm performence 100 | * -s -> qemu will open a gdbserver on TCP port 1234 101 | * -m 2048 -> amount of memory of the vm (2mb in our example) 102 | * -hda -> path to the root filesystem image in our case the rootfs 103 | * -append -> send command line arguments to the linux kernel 104 | * -net nic,model=virtio -> connect a network interface 105 | * -net user,hostfwd=tcp::5555-:22 -> forwards tcp traffic from host port 5555 to guest port 22 106 | which allows us to use ssh. 107 | 108 | 109 | Now we can launch our vm: 110 | 111 | 11. ./start-qemu.sh 112 | 113 | #### SSH to guest 114 | `ssh root@localhost -p 5555` 115 | 116 | 117 | ## Snapshots 118 | In order to take snapshots we'll connect to the qemu monitor 119 | 11. socat stdio,echo=0,icanon=0 unix-connect:qemu-monitor-socket 120 | 121 | saving and loading snapshots can be done in the following manner respectively: 122 | 123 | * savevm 124 | 125 | * loadvm 126 | 127 | ## Start the debugging session 128 | Now we are going to attach to our vm and the debug the kernel, we will also use our symbols to the kernel. 129 | 130 | 12. cd output/build/linux-{version} 131 | 13. gdb ./vmlinux 132 | 14. target remote :1234 133 | 134 | And now you got a kernel debugging session. 135 | 136 | **DONE!!!** 137 | -------------------------------------------------------------------------------- /start-qemu.sh: -------------------------------------------------------------------------------- 1 | #!/bin/sh 2 | ( 3 | BINARIES_DIR="${0%/*}/" 4 | cd ${BINARIES_DIR} 5 | 6 | if [ "${1}" = "serial-only" ]; then 7 | EXTRA_ARGS='-nographic' 8 | else 9 | EXTRA_ARGS='-serial stdio' 10 | fi 11 | 12 | export PATH="../host/bin:${PATH}" 13 | exec qemu-system-x86_64 -s -monitor unix:qemu-monitor-socket,server,nowait -enable-kvm -cpu host -m 2048 -M pc -kernel bzImage -hda rootfs.qcow2 -append "rootwait root=/dev/sda console=tty1 console=ttyS0rw nokaslr" -net nic,model=virtio -net user,hostfwd=tcp::5555-:22 ${EXTRA_ARGS} 14 | ) 15 | --------------------------------------------------------------------------------