├── README.md ├── example-revert.sh └── example-start.sh /README.md: -------------------------------------------------------------------------------- 1 | # Single GPU Passthrough on Linux 2 | This guide is to help people through the process of using GPU Passthrough via libvirt/virt-manager on systems that only have one GPU. 3 | 4 | ## Special Thanks to: 5 | ### [The Passthrough post](https://passthroughpo.st) 6 | For hosting news and information about VFIO passthrough, and for the libvirt/qemu hook helper in this guide. 7 | 8 | ### andre-ritcher 9 | For providing the vfio-pci-bind tool. A tool that is no longer used in this guide, but was previously used and he still deserves thanks. 10 | 11 | ### Matoking 12 | For the Nvidia ROM patcher. Making passing the boot gpu to the VM without GPU bios problems. 13 | Patching the rom is no longer required but I never would have written this guide without the original work so I'm keeping them here. 14 | 15 | ### Sporif 16 | For diagnosing, developing, and testing methods to successfully rebind the EFI-Framebuffer when passing the video card back to the host OS. 17 | 18 | ### droidman 19 | For instructions on manually editing the vBIOS hex for use with VFIO passthrough 20 | 21 | ### [Yuri Alek](https://gitlab.com/YuriAlek/vfio) 22 | A guide that is no doubt better than mine. Learning a few things from his implementation that can help me out a bit. This guide depends on libvirt at the base where as his has implementations that do not. 23 | 24 | #### So many other people and organizations I need to thank. If feel your name should be here, please contact me. Credit where credit is due is very important to me, and to making the Linux community a better place. 25 | 26 | ## Contents 27 | 28 | 1. [Disclaimer](#disclaimer) 29 | 2. [Background](#background) 30 | 3. [Advantages](#advantages) 31 | 4. [Disadvantages](#disadvantages) 32 | 3. [Prerequisites](#prerequisites) and [Assumptions](#assumptions) 33 | 4. [Procedure](#procedure) 34 | 35 | # Disclaimer 36 | You are completely responsible for your hardware and software. This guide makes no guarentees that the process will work for you, or will not void your waranty on various parts or break your computer in some way. Everything from here on out is at your own risk. 37 | 38 | # Background 39 | Historically, VFIO passthrough has been built on a very specific model. I.E. 40 | 41 | * 2 GPUs, 1 for the host, and one for the VM 42 | * 2 monitors *OR* a monitor with 2 inputs *OR* a KVM switch 43 | 44 | I personally, as well as some of you out there, might not have those things available. Maybe You've got a Mini-ITX build with no iGPU. Or maybe you're poor like me, and can't shell out for new computer components without some financial planning before hand. 45 | 46 | Whatever your reason is. VFIO is still possible. But with caveats. Here's some advantages and disadvantages of this model. 47 | 48 | This setup model is a lot like dual booting, without actually rebooting. 49 | 50 | # Advantages 51 | * As already stated, this model only requires one GPU 52 | * The ability to switch back and forth between different OSes with FULL use of a discrete graphics processor (Linux on Host with full GPU, Windows 10 Guest with Full GPU, MacOS guest with full GPU) 53 | * Bragging rights 54 | * Could be faster than dual booting (this depends on your system) 55 | * Using virtual disk images (like qcow) gives you management of snapshots, making breaking your guest os easy to recover from. 56 | 57 | # Disadvantages 58 | * Can only use one OS at a time. 59 | - Once the VM is running, it's basically like running that as your main OS. You will be logged out of your user on the host, but will be unable to manage the host locally at all. You can still use ssh/vnc/xrdp to manage the host. 60 | * There are still some quirks (I need your help to iron these out!) 61 | * Using virtual disk images could be a performance hit 62 | - You can still use raw partitions/lvm/pass through raw disks, but loose the more robust snapshot and management features 63 | * If you DO have a second video card, solutions like looking-glass are WAYYY more convenient and need active testing and development. 64 | * All VMs must be run as root. There are security considerations to be made there. This model requires a level of risk acceptance. 65 | 66 | For my personal use case. This model is worth it to me and it might be for you too! 67 | 68 | # Prerequisites and Assumptions 69 | 70 | ## Assumptions 71 | This guide is going to assume a few things 72 | 73 | 1. You have a system capable of VFIO passthrough. I.E. a processors that supports IOMMU, sane IOMMU groups, and etc. 74 | 2. I am going to start in a place where you have a working libvirt config, or qemu script, that boots a guest OS without PCI devices passed through. 75 | 76 | I am not going to cover the basic setup of VFIO passthrough here. There are a lot of guides out there that cover the process from beginning to end. 77 | 78 | What I will say is that using the [Arch Wiki][arch_wiki] is your best bet. 79 | 80 | Follow the instructions found [here][arch_wiki] 81 | 82 | [arch_wiki]: https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF 83 | 84 | **Skip the Isolating the GPU section** We are not going to do that in this method as we still want the host to have access to it. I will cover this again in the procedure section. 85 | 86 | ## Prerequisites 87 | 88 | 1. A working Libvirt VM or Qemu script for your guest OS. 89 | 2. IOMMU enabled and Sane IOMMU groups 90 | 3. The Following Tools 91 | * (If using Libvirt) [The Libvirt Hook Helper](https://passthroughpo.st/simple-per-vm-libvirt-hooks-with-the-vfio-tools-hook-helper/) 92 | * (Optional) Another machine to SSH/VNC to your host with for testing might be useful 93 | 94 | With all this ready. Let's move on to how to actually do this. 95 | 96 | # Procedure 97 | 98 | ## Setting up Libvirt hooks 99 | 100 | Using libvirt hooks will allow us to automatically run scripts before the VM is started and after the VM has stopped. 101 | 102 | Using the instructions [here](https://passthroughpo.st/simple-per-vm-libvirt-hooks-with-the-vfio-tools-hook-helper/) to install the base scripts, you'll find a directory structure that now looks like this: 103 | 104 | ``` 105 | /etc/libvirt/hooks 106 | ├── qemu <- The script that does the magic 107 | └── qemu.d 108 | └── {VM Name} 109 | ├── prepare 110 | │   └── begin 111 | │   └── start.sh 112 | └── release 113 | └── end 114 | └── revert.sh 115 | ``` 116 | 117 | Anything in the directory ````/etc/libvirt/hooks/qemu.d/{VM Name}/prepare/begin```` will run when starting your VM 118 | 119 | Anything in the directory ````/etc/libvirt/hooks/qemu.d/{VM Name}/release/end```` will run when your VM is stopped 120 | 121 | ### Libvirt Hook Scripts] 122 | #### Do not copy my scripts. Use them as a template, but write your own. 123 | 124 | I've made my start script ```/etc/libvirt/hooks/qemu.d/{VMName}/prepare/begin/start.sh``` 125 | 126 | 127 | ### Start Script 128 | ``` 129 | #!/bin/bash 130 | # Helpful to read output when debugging 131 | set -x 132 | 133 | # Stop display manager 134 | systemctl stop display-manager.service 135 | ## Uncomment the following line if you use GDM 136 | #killall gdm-x-session 137 | 138 | # Unbind VTconsoles 139 | echo 0 > /sys/class/vtconsole/vtcon0/bind 140 | echo 0 > /sys/class/vtconsole/vtcon1/bind 141 | 142 | # Unbind EFI-Framebuffer 143 | echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind 144 | 145 | # Avoid a Race condition by waiting 2 seconds. This can be calibrated to be shorter or longer if required for your system 146 | sleep 2 147 | 148 | # Unbind the GPU from display driver 149 | virsh nodedev-detach pci_0000_0c_00_0 150 | virsh nodedev-detach pci_0000_0c_00_1 151 | 152 | # Load VFIO Kernel Module 153 | modprobe vfio-pci 154 | ``` 155 | NOTE: Gnome/GDM users. You have to uncommment the line ````killall gdm-x-session```` in order for the script to work properly. Killing GDM does not destroy all users sessions like other display managers do. 156 | 157 | 158 | ### VM Stop script 159 | My stop script is ```/etc/libvirt/hooks/qemu.d/{VMName}/release/end/revert.sh``` 160 | ``` 161 | #!/bin/bash 162 | set -x 163 | 164 | # Re-Bind GPU to Nvidia Driver 165 | virsh nodedev-reattach pci_0000_0c_00_1 166 | virsh nodedev-reattach pci_0000_0c_00_0 167 | 168 | # Reload nvidia modules 169 | modprobe nvidia 170 | modprobe nvidia_modeset 171 | modprobe nvidia_uvm 172 | modprobe nvidia_drm 173 | 174 | # Rebind VT consoles 175 | echo 1 > /sys/class/vtconsole/vtcon0/bind 176 | # Some machines might have more than 1 virtual console. Add a line for each corresponding VTConsole 177 | #echo 1 > /sys/class/vtconsole/vtcon1/bind 178 | 179 | nvidia-xconfig --query-gpu-info > /dev/null 2>&1 180 | echo "efi-framebuffer.0" > /sys/bus/platform/drivers/efi-framebuffer/bind 181 | 182 | # Restart Display Manager 183 | systemctl start display-manager.service 184 | 185 | ``` 186 | 187 | # Running the VM 188 | When running the VM, the scripts should now automatically stop your display manager, unbind your GPU from all drivers currently using it and pass control over the libvirt. Libvirt handles binding the card to VFIO-PCI automatically. 189 | 190 | When the VM is stopped, Libvirt will also handle removing the card from VFIO-PCI. The stop script will then rebind the card to Nvidia and SHOULD rebind your vtconsoles and EFI-Framebuffer. 191 | 192 | # Troubleshooting 193 | First of all. If you ask for help, then tell me you skipped some step... I'm gonna be a little annoyed. So before moving on to troubleshooting, and DEFINATELY before asking for help, make sure you've follwed ALL of the steps of this guide. They are all here for a reason. 194 | 195 | ## Logs 196 | Logs can be found under /var/log/libvirt/qemu/[VM name].log 197 | 198 | ## Common issues 199 | ### Black Screen on VM Activation 200 | 1. Make sure you've removed the Spice Video and QXL video adapter on the VM 201 | 2. It can be extremely helpful to SSH into the host to check if scripts have executed properly, and that the VM is running. Try these in this order. 202 | 1. SSH into the host, and manually run the start script. If the start script runs properly, the host monitors should go completely black, and the terminal should return you to the prompt. 203 | 2. If all goes well there, try running the vm manually using `sudo virsh start {vmname}` 204 | 3. If there is a problem here, typically the command will hang. That would signify a problem with the VM libvirt configuration. 205 | 4. If you are returned to the prompt, check if the vm is in a running state by using `sudo virsh list` 206 | 5. If it's running fine, and you've made sure that you are not having the issue in step 1 and 2, yell at me in the issue tracker or reddit 207 | 208 | ### Audio 209 | Check out the ArchWIKI entry for tips on audio. I've used both Pulseaudio Passthrough but am currently using a Scream IVSHMEM device on the VM. 210 | 211 | ## NOTE 212 | Either of these will require a user systemd service. You can keep user systemd services running by enabling linger for your user account like so: 213 | `sudo loginctl enable-linger {username}` 214 | This will keep services running even when your account is not logged in. I do not know the security implications of this. My assumption is that it's not a great idea, but oh well. 215 | 216 | # Tips and Tricks 217 | ## Personal Touches 218 | Here's a few things I do to make managing the host easier. 219 | 220 | 1. Start a VNC server on the host in the start script 221 | 2. Set pulseaudio volume to 100% 222 | 3. Anything you want the host to do upon VM activation. 223 | 224 | 225 | # Let me know what works and what doesnt! 226 | Let me know your success and failure stories. 227 | 228 | 229 | #### [Fuel my coffee addiction or help me test new hardware](https://www.paypal.com/donate?business=87AQBT5TGFRJS&item_name=Github+Testing¤cy_code=USD) 230 | #### Always appreciated, never required. 231 | 232 | -------------------------------------------------------------------------------- /example-revert.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | set -x 3 | 4 | # Re-Bind GPU to Nvidia Driver 5 | virsh nodedev-reattach pci_0000_0c_00_1 6 | virsh nodedev-reattach pci_0000_0c_00_0 7 | 8 | # Reload nvidia modules 9 | modprobe nvidia 10 | modprobe nvidia_modeset 11 | modprobe nvidia_uvm 12 | modprobe nvidia_drm 13 | 14 | # Rebind VT consoles 15 | echo 1 > /sys/class/vtconsole/vtcon0/bind 16 | # Some machines might have more than 1 virtual console. Add a line for each corresponding VTConsole 17 | #echo 1 > /sys/class/vtconsole/vtcon1/bind 18 | 19 | nvidia-xconfig --query-gpu-info > /dev/null 2>&1 20 | echo "efi-framebuffer.0" > /sys/bus/platform/drivers/efi-framebuffer/bind 21 | 22 | # Restart Display Manager 23 | systemctl start display-manager.service 24 | -------------------------------------------------------------------------------- /example-start.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Helpful to read output when debugging 3 | set -x 4 | 5 | # Stop display manager 6 | systemctl stop display-manager.service 7 | ## Uncomment the following line if you use GDM 8 | #killall gdm-x-session 9 | 10 | # Unbind VTconsoles 11 | echo 0 > /sys/class/vtconsole/vtcon0/bind 12 | echo 0 > /sys/class/vtconsole/vtcon1/bind 13 | 14 | # Unbind EFI-Framebuffer 15 | echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind 16 | 17 | # Avoid a Race condition by waiting 2 seconds. This can be calibrated to be shorter or longer if required for your system 18 | sleep 2 19 | 20 | # Unbind the GPU from display driver 21 | virsh nodedev-detach pci_0000_0c_00_0 22 | virsh nodedev-detach pci_0000_0c_00_1 23 | --------------------------------------------------------------------------------