├── stop.sh ├── start.sh ├── README.md └── macOS.xml /stop.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Helpful to read output when debugging 3 | set -x 4 | 5 | # Load the config file with our environmental variables 6 | source "/etc/libvirt/hooks/kvm.conf" 7 | 8 | # Unload all the vfio modules 9 | modprobe -r vfio_pci 10 | modprobe -r vfio_iommu_type1 11 | modprobe -r vfio 12 | 13 | # Reattach the gpu 14 | virsh nodedev-reattach $VIRSH_GPU_VIDEO 15 | virsh nodedev-reattach $VIRSH_GPU_AUDIO 16 | 17 | # Load all Radeon drivers 18 | 19 | modprobe amdgpu 20 | modprobe gpu_sched 21 | modprobe ttm 22 | modprobe drm_kms_helper 23 | modprobe i2c_algo_bit 24 | modprobe drm 25 | modprobe snd_hda_intel 26 | 27 | #Start you display manager 28 | systemctl start sddm.service 29 | -------------------------------------------------------------------------------- /start.sh: -------------------------------------------------------------------------------- 1 | #!/bin/bash 2 | # Helpful to read output when debugging 3 | set -x 4 | 5 | # Load the config file with our environmental variables 6 | source "/etc/libvirt/hooks/kvm.conf" 7 | 8 | # Stop your display manager. If youre on kde it ll be sddm.service. Gnome users should use killall gdm-x-session instead 9 | systemctl stop sddm.service 10 | pulse_pid=$(pgrep -u YOURUSERNAME pulseaudio) 11 | pipewire_pid=$(pgrep -u YOURUSERNAME pipewire-media) 12 | kill $pulse_pid 13 | kill $pipewire_pid 14 | 15 | # Unbind VTconsoles 16 | echo 0 > /sys/class/vtconsole/vtcon0/bind 17 | echo 0 > /sys/class/vtconsole/vtcon1/bind 18 | 19 | 20 | # Avoid a race condition by waiting a couple of seconds. This can be calibrated to be shorter or longer if required for your system 21 | sleep 4 22 | 23 | # Unload all Radeon drivers 24 | 25 | modprobe -r amdgpu 26 | #modprobe -r gpu_sched 27 | #modprobe -r ttm 28 | #modprobe -r drm_kms_helper 29 | #modprobe -r i2c_algo_bit 30 | #modprobe -r drm 31 | #modprobe -r snd_hda_intel 32 | 33 | # Unbind the GPU from display driver 34 | virsh nodedev-detach $VIRSH_GPU_VIDEO 35 | virsh nodedev-detach $VIRSH_GPU_AUDIO 36 | 37 | # Load VFIO kernel module 38 | modprobe vfio 39 | modprobe vfio_pci 40 | modprobe vfio_iommu_type1 41 | -------------------------------------------------------------------------------- /README.md: -------------------------------------------------------------------------------- 1 | # vfio-single-amdgpu-passthrough 2 | This repo is a tutorial for single amd gpu passthrough to various qemu VMs. 3 | After reading some great turorials by [Maagu Karuri](https://gitlab.com/Karuri/vfio) and [Joe Knockenhauer](https://github.com/joeknock90/Single-GPU-Passthrough) I wanted to try it myself, unfortunately there were no instructions for amd gpus, just warnings about the dreaded reset bug. 4 | 5 | So, this will be my first attempt at contributing back to this community of enthusiasts and tinkers. 6 | 7 | This description will be changed and beautified, but untill then, lets get to the improtant stuff. 8 | 9 | Considering that you already seen the other tutroials and have an ideea about what goes on, i'll just show why it didn't work for me, and after that I'll show you the final working version of start.sh. Please keep in mind that, the gpu has already been added in the vm xml, and vfio has also been enabled, but that's it, no nomode 10 | set or efifb:off was used. 11 | 12 | ```sh 13 | #!/bin/bash 14 | # Helpful to read output when debugging 15 | set -x 16 | 17 | # Load the config file with our environmental variables 18 | source "/etc/libvirt/hooks/kvm.conf" 19 | 20 | # Stop your display manager 21 | systemctl stop lightdm.service 22 | 23 | # Unbind VTconsoles 24 | echo 0 > /sys/class/vtconsole/vtcon0/bind 25 | echo 0 > /sys/class/vtconsole/vtcon1/bind 26 | 27 | 28 | 29 | # Avoid a race condition by waiting a couple of seconds. This can be calibrated to be shorter or longer if required for your system 30 | sleep 5 31 | 32 | # Unload all Nvidia drivers 33 | modprobe -r nvidia_drm 34 | modprobe -r nvidia_modeset 35 | modprobe -r drm_kms_helper 36 | modprobe -r nvidia 37 | modprobe -r i2c_nvidia_gpu 38 | modprobe -r drm 39 | 40 | # Unbind the GPU from display driver 41 | virsh nodedev-detach $VIRSH_GPU_VIDEO 42 | virsh nodedev-detach $VIRSH_GPU_AUDIO 43 | 44 | 45 | # Load VFIO kernel module 46 | modprobe vfio 47 | modprobe vfio_pci 48 | modprobe vfio_iommu_type1 49 | ``` 50 | DO NOT USE THIS FOR YOUR START.SH , use the start.sh uploaded by me! 51 | 52 | The reason why it locks up, is because we are basically cutting the gpu short without letting the driver know that it will end, with NVidia it works because the driver knows how to reset itself, amd's doesn't 53 | 54 | The working version will be attached as start.sh , but just to explain a bit, this is what I did, and the reason behind it. 55 | 56 | First things first, after killing the display manager, we have to kill pulseAudio and pipewire, we get the process id for both of them using pgrep, and then we kill them. 57 | We do this because our gpu is used by amdgpu(as a display adapter) and by snd_hda_intel(as an audio output for hdmi), this can be checked using the following command: 58 | 59 | lspci -v: 60 | ``` 61 | 09:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] (rev e7) (prog-if 00 [VGA controller]) 62 | Subsystem: XFX Pine Group Inc. Radeon RX 580 63 | Flags: bus master, fast devsel, latency 0, IRQ 95, IOMMU group 23 64 | Memory at d0000000 (64-bit, prefetchable) [size=256M] 65 | Memory at e0000000 (64-bit, prefetchable) [size=2M] 66 | I/O ports at e000 [size=256] 67 | Kernel driver in use: amdgpu <------- HERE WE CAN SEE THE MODULE\DRIVER 68 | Kernel modules: amdgpu 69 | 70 | 09:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] 71 | Subsystem: XFX Pine Group Inc. Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] 72 | Flags: bus master, fast devsel, latency 0, IRQ 108, IOMMU group 23 73 | Memory at fce60000 (64-bit, non-prefetchable) [size=16K] 74 | Kernel driver in use: snd_hda_intel 75 | Kernel modules: snd_hda_intel <----- HERE WE CAN SEE THE OTHER MODULE\DRIVER 76 | 77 | ``` 78 | 79 | Next we unbind the VTConsoles without unbinding the EFI-Framebuffer, because if we would do that , we would get a segmentation fault. 80 | Instead, now that the display manager and pulseaudio are off ,we can unload all the modules(amdgpu and snd_hda_intel) that use the gpu(both the video and audio part) 81 | 82 | After that is done, we now detach the gpu using virsh, and we load the vfio modules and thats it, if you added the gpu to you VM and you configured it to use q35 efi instead of bios, everything will work :D . 83 | One last trick would be to always remove the monitor(spice or vnc) and vgpu of the vm(qxl or virtio), by doing so, you will see the bios/efi output as well as the final os on your passed gpu. 84 | 85 | As of 27/1/2021 I have also added my stop.sh script, if my start.sh script(which I also updated today) work for you, the stop.sh should work as well. 86 | 87 | Again, huge thanks to Maagu Karuri annd Joe Knockenhauer for there work and dedication. 88 | 89 | #BONUS: [vendor-reset](https://github.com/gnif/vendor-reset) , this will allow you to always(or almost always) be able to have a clean release of the gpu when you stop a vm. 90 | As far as I'm concerned, it works great for my card, some say it does not work for everyone, for me it does. 91 | Some operating systems are almost guaranteed to do a clean shutdown, but others like macOS are not 92 | And of course, by using vendor-reset, you can safely kill(virsh destroy) any stuck vm, without the risk of losing the usability of the card until a new reset. 93 | -------------------------------------------------------------------------------- /macOS.xml: -------------------------------------------------------------------------------- 1 | 2 | macOS 3 | macOS 4 | 12582912 5 | 12582912 6 | 7 | 8 | 9 | 8 10 | 2 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | hvm 26 | /home/cosmin/Downloads/OSX-KVM/OVMF_CODE.fd 27 | /home/cosmin/Downloads/OSX-KVM/OVMF_VARS-1024x768.fd 28 | 29 | 30 | 31 | 32 | 33 | 34 | qemu64 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | destroy 43 | restart 44 | restart 45 | 46 | /usr/bin/qemu-system-x86_64 47 | 48 | 49 | 50 | 51 | 52 |
53 | 54 | 55 | 56 | 57 | 58 |
59 | 60 | 61 |
62 | 63 | 64 | 65 | 66 | 67 |
68 | 69 | 70 | 71 | 72 |
73 | 74 | 75 | 76 | 77 |
78 | 79 | 80 | 81 | 82 |
83 | 84 | 85 | 86 | 87 |
88 | 89 | 90 | 91 | 92 |
93 | 94 | 95 | 96 | 97 |
98 | 99 | 100 |
101 | 102 | 103 |
104 | 105 | 106 | 107 |
108 | 109 | 110 | 111 |
112 | 113 | 114 | 115 |
116 | 117 | 118 | 119 | 120 | 121 |
122 | 123 | 124 | 125 | 126 | 127 | 128 | 129 | 130 | 131 | 132 | 133 |
134 | 135 | 136 |
137 | 138 | 139 | 140 |
141 | 142 | 143 | 144 | 145 |
146 | 147 |
148 | 149 | 150 | 151 |
152 | 153 |
154 | 155 | 156 | 157 |
158 | 159 |
160 | 161 | 162 | 163 |
164 | 165 |
166 | 167 | 168 | 169 |
170 | 171 |
172 | 173 | 174 | 175 | 176 | 177 |
178 | 179 |
180 | 181 | 182 | 183 | 184 | 185 |
186 | 187 |
188 | 189 | 190 | 191 | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | --------------------------------------------------------------------------------