I've been following along since part 0 and I'm continually impressed by the quality of these videos. You could have just given us a set of instructions to follow, but you went over the top and gave us a comprehensive guide on what's happening and why, supported by beginner-friendly explanations. Keep up the great work, I'm excited for your content and the future of your channel!
That's so awesome to hear! Part of the joy of Linux is getting to learn what happens under the hood. Very happy I could share that and very happy my explanations made sense :)
This is probably the best tutorial video series I have ever seen on youtube! Great work! The way you calmly and clearly explain every step and the possible alternatives on other distros is amazing.
This is an amazingly detailed guide. I've done a few single gpu passthrough systems on Ubuntu and Arch based distros, but switched to Fedora recently, and your videos have helped me achieve the same here as well! Thanks a ton! Subscribed!
I can't believe this works! Props to you man! Simply the best tutorial on this topic 👍 Had some troubles setting up in existing fedora 37 (dracut part produced errors I couldn't fix) then try fresh install and it worked! Although now I can't change the resolution since I basically just depend on my processor's abysmal gpu power 😂. Gonna try single gpu passthrough. You deserve much more subs for how good these videos are.
For those on ubuntu linux the grub file is in /etc/default/grub and you should run sudo update-grub to update grub and install dracut and the new command is now dractu -f --no-kernel --kver "uname -r" at least this is what I did on my system when following this.
i am on linux mint, and when i run dracut after installing, i get this error: Can't write to /boot/efi/09c7bf9099104344a6982459b4191d0a/uname -r: Directory /boot/efi/09c7bf9099104344a6982459b4191d0a/uname -r does not exist or is not accessible. Any ideas where to look for an answer?
hey this is so super cool i did notice ultramarine 40 is so far the only distro i can follow line by line and im sooooo a newb but at the 2:37 the part of the command groupinstall is actally seperate on mine took me some digging but i got it workin thank you and im so following along.
Thanks for the comprehensive guide man. Still lack the hardware for this but I'll definitely save the video for future reference. Will you be talking about CPU pinning in the next part? I was looking up other tutorials and there was one that said it improves performance but the cores would be inaccessible by the host even with the VM disabled. I'm hoping there's a way around that because I still do CPU-heavy tasks on the host OS.
Thanks for the comment! I'll don't think I'll do CPU pinning in the next video because you can get quality performance without it. But maybe in the future for a "performance tweaks" video I bet you'll be able to set it up in a way that balances performance both inside the VM and outside From what I understand "CPU pinning" is telling the VM to use specific cores. Then "cpu isolation" is when you use those grub commands to tell the host to ignore those cores (even when the VM is off, like you described) It is a performance boost inside the VM because the host will never borrow those cores from the VM In practice, I haven't needed that extra performance boost. But you can turn it on and off with a reboot. You could also have multiple grub configs so that you can easily choose between "Linux with all CPUs" or "Linux with CPU isolation" when you boot
God reading how nice of a dude in the comments making feel bad I keep having to stop, f new fedora lmao messing with this masterpeice of a series! 8:34 now baby having g edit helps a lot)
hi, there is a short section on the dracut.conf.d from 8:34 in that is not clearly visible on the screen and i ran: which is what it looks like but got this: What should this command be? The one preceding this was also unclear on the screen. I read:
Hey thanks for asking! I think I know what the issue is There is a very subtle difference between the backtick character (`) and the single quote (') You want to surround the part of the command with the backtick character so it looks like this dracut -f --kver `uname -r`
Great video! would be nice to see a video surrounding using qemu hooks to use pci passthrough while being able to use the gpu being passed through whenever the vm is off
Glad to hear it and good to know! If I get a chance to work on laptops, i'll likely be using this guide as my starting point lantian.pub/en/article/modify-computer/laptop-muxed-nvidia-passthrough.lantian/
@@NoahLorano I recently ran through the process on Ubuntu to do single gpu pass through. But I haven't been able to find the time to write it up or film it. Hopefully soon
5:50 how would i do this for systemd? I'm on Pop OS and from what I'm reading (with my very limited knowledge) grub isn't used any more, or at least when entering "sudo gedit /etc/sysconfig/grub" it's empty
Looks like the way to add "kernel parameters" (aka "Linux command line arguments") with systemd-boot on arch is under "options" in a file at /boot/loader/entries/arch.conf wiki.archlinux.org/title/Kernel_parameters#systemd-boot Might be a similar path, file, and process on Pop!OS
@@BlandManStudios Funny enough I saw that site before I saw your video as I was getting confused by a different guide on reddit. Arch seems to have "/boot/loader/loader.conf" but Pop OS doesn't (used the gnome file manager to look at the folder, couple of things with "initrd" or "vmlinuz" in the name but I don't think that's what I'm looking for) (I'm gonna leave this long comment here to remind myself of what I did because I'm gonna forget by tomorrow, and for anyone else on Pop OS 22.04 if interested) A bit more searching and found out Pop uses "kernelstub". I entered "cat /proc/cmdline" and got a few things to pop up similar to 6:55 (quiet splash amd_iommu=on). I found a website that Firefox warned me wasn't safe (I don't know why) that said: "To make the change persistent after reboot, edit /etc/kernelstub/configuration and run kernelstub" Entering "sudo nano /etc/kernelstub/configuration" brought up a file with some kernel options and a user section with "amd_iommu=on". I'm going to add everything from 6:55 into this part and hope for the best. My primary GPU is a 6600xt and the VM will be using an old Vega 56, so anything with Vega in the name is getting added to the list ("pci-stub.ids=1002:687f,1002:aaf8" no spaces between stub ids, no idea if it should be a separate line or bundled together) There may be a new (potential) problem. There's an IOMMU for "VGA compatible controller" and "Audio device" with (Vega 10 listed in the name) for video and audio, but also "PCI bridge". I'm gonna skip adding these If I don't respond back to this comment in a few days, assume that I've broken my desktop
@@GoingMenthol Did you ever figure this out, or did your computer explode? I am also on Pop OS and want to know if it is safe to do what you did, or if I should turn back.
@@0Noobslayer0 Computer crashes when i try to run virtual machine but didn't blow up, I've only just noticed I added a 5 port PCI USB hub to the list of PCI IDs. I can remove it from /etc/kernelstub/configuration using "sudo nano" to edit the file, but using "cat /proc/cmdline" or "sudo dmesg | grep -i IOMMU" shows that the wrong PCI ID is still being listed. I tried "sudo nano /proc/cmdline" to edit it but I'm getting an error that doesn't let me save changes (apparently it's not a real file) What I'm gonna try now is "sudo kernelstub" to confirm changes and attempt to reboot to the boot menu (hold space on boot up I think) to see if there's anything I can edit from there. Will see how that goes UPDATE: restart the desktop, hold space bar, let go of space bar when you get to the recovery options listed and press "e", edit the boot parameters (turns out I cannot spell and there were a few typos) and hit enter. Boot to desktop and open terminal, enter "sudo kernelstub" to confirm changes to kernel in case it doesn't do that already follow part 3 VFIO tutorial that BlandManStudios made. If you get an error with something like "pci controller with index 0 must be pci-root" and you cannot delete the "controller PCIe"when trying to begin installation, go to the "overview" section on Virtual Machine Manager, click the XML tab, search for something like "controller type=pci" (for me there were a bunch of the same thing) and replace with "", which now lets me boot to the virtual machine I can now boot the virtual machine, will update again once windows drivers are installed and will attempt to add the GPU
UPDATE AGAIN: installed windows, installed all available drivers, added the Vega 56 PCI devices to VMM, as soon as it tries to install AMD GPU drivers via windows update it blue screens, trying to install from AMD's own drivers causes a blue screen. Now the VM is a black screen when I power it up, but works fine when the Vega GPU and HDMI Audio are removed. @BlandManStudios I don't suppose you know what I should be checking for? Is there something in the XML I need to change?
As of kernel 6.2, the vfio_virqfd functionality has been folded into the base vfio module. If you also have another driver loaded this way for early modesetting (such as nouveau, radeon, amdgpu, i915, etc.), all of the aforementioned VFIO modules must precede it. If you are modesetting the nvidia driver, the vfio-pci.ids must be embedded in the initramfs image. If given via kernel arguments, they will be read too late to take effect. Follow the instructions in #Binding vfio-pci via device ID for adding the ids to a modprobe conf file.
Per the docs I tried using: force_drivers+=" vfio_pci vfio vfio_iommu_type1 " .. but I still see the device kernel driver as: nvidia after reboot ...not sure where to go from here
Do you really need root privilegs to run virt-manager on Fedora? I definitely don't need to do this on Debian based distros... Maybe just adding your user to the libvirt group can let you launch it just fine without sudo? I'm pretty sure that Debian adds you automatically to that group. Maybe Fedora just... doesn't? Also there's another driver category that could f-up things: EFI framebuffer drivers. Especially if you boot with CSM (compatibility support module) disabled. I personally had to also pass video=efifb:off to the kernel to make things work on my machine. This is typically not something that a new user encounters though. Especially since CSM is enabled by default on most motherboards.
So far I have needed it to run virt-manager or "virsh edit " But I just checked and you're right. I just needed to add my user to the libvirt group and it works passwordlessly. Definitely a better solution than my sudo hack. Thanks! And thanks for the info on CSM
Are you using the backtick character? Single quote ' Double quote " Backtick ` On the keyboard it is over by the escape key with ~ unix.stackexchange.com/questions/27428/what-does-backquote-backtick-mean-in-commands#27432
The pci id that is written into the grub file, is that only the GPU which will be passed to the guest in the VM or does it also include the GPU used by the host?
Wait, so my linux host won't *ever* have access to my gaming GPU? This is kind of a deal breaker, as I game a lot on linux too natively or using proton. I thought I'd pass it *only* when my windows VM was running. Is this possible too? Has anyone done this?
Mercy, I have been trying to follow along, but I have recently discovered a new level of OCD. Where if I have an inkling of doubt... I wipe my drives, (backups too) and restart with a fresh install. I''m on 3rd install of Ubuntu.
@@NextToken this is relatable "Oops I typed this command wrong, better refilm the entire tutorial" I bet you'll get through it. Trust yourself and recognize there will be bugs to work out and get through, but once you squash them all you can start over fresh to create the "perfect install" with all your newfound knowledge/experience
is it normal that when i run sudo gedit it gives me this:** (gedit:3224): WARNING **: 18:13:21.293: AT-SPI: Could not obtain desktop path or name ** (gedit:3224): WARNING **: 18:13:21.298: atk-bridge: GetRegisteredEvents returned message with unknown signature ** (gedit:3224): WARNING **: 18:13:21.298: atk-bridge: get_device_events_reply: unknown signature ** (gedit:3224): WARNING **: 18:13:21.298: atk-bridge: get_device_events_reply: unknown signature
Yup that's normal. I'm pretty sure the warning comes from using a graphical application (gedit) with sudo If you're familiar with nano or vim, a command-line text editor won't give those warnings
I've finally got lspci -nnk to show using vfio-pci instead of pci-stub. However, I still can't get the vm to display out to the monitor (host on display port and vm's gpu on hdmi).
@@BlandManStudios Yup. But it only covers Windows VM. I want it for Linux too. I know that it makes no sense, but that's what my goal is. I want to be able to boot into my main linux, shut it down, and boot into Windows without ever having to hit the reset button. I've already figured out how to add boot from the Linux storage device bare metal as it were.
@@andromydous if virt-manager is started and "connected" I think its already running so you don't need to start it You can run this to know for sure sudo systemctl status libvirtd
@@BlandManStudios So after some searching, though not finding an exact answer, it seems that it's a video port issue. When I run lspci -nnk in my PopOS vm, it shows that the gpu was indeed passed through and is using the amdgpu drivers.
i get an error during boot, can anyone please help me how to fix it? I turned on virtualization in BIOS and disabled secure boot. Google search suggests disabling vt-d [drm] VT-d active for gfx access
VT-d does need to be enabled Have you been able to verify (by finding product spec or other people with the same hardware) that your MOBO and CPU support IOMMU/VFIO
@@BlandManStudios Thanks for the reply i really appreciate this! Yes, it does support virtualisation. My CPU is i7-13700K and motherboard is MSI Z690-A. In fact, I event managed to get it working on Manjaro, but there was another issue and i decided to get through your complete Fedora tutorial. But it doesn't seem to work on Fedora :( I did everything exactly like you showed in your guide. It does boot if i remove grub options related to virtualisation, but doesn't boot with them. I haven't tried reinstalling Fedora yet though, maybe it will help
so im following the tutorial word for word and when I got to the end my GPU audio driver was in pci-stub but the actually card was still in amd-gpu. is there something I missed?
There was a typo in the video that might be causing your problem The command line argument is supposed to be "rd.driver.pre=vfio-pci" and not rd.driver.pre=vfio.pci" which I accidentally put on the screen in the beginning Also, I haven't done this with an AMD GPU (I've only used Nvidia) so maybe you need to blacklist the "amd-gpu" driver
I had a problem in part3 of video and I come back here. I did some research but without success... I got " Please ensure all devices within the IOMMU_GROUP are bound to their vfio bus driver". Does I need to put every single item that I present in the IMMO GROUP to vFio? In this video you doesnt show in the LSPCI as group, but my group has a lot of things that doesnt make any sense... I read that I could ignore the PCI Bridge's but I stil receive the same error... In my IOMMU Group I have 7 itens 1- PCI Bridge E3-1200 x16 Controller 1- PCI Bridge E3-1200 x8 Controller 1- PCI Bridge AMD Pcie Bridge 1- PCI Bridge AMD Pcie Bridge (just the id change) 1- VGA AMD VEGA 10 XL 1- Audio device AMD 1- Non volatile memory controller (x16 to m2) Checking anothers computers just mine has a lot of stuff in the same group...
Great question! Usually my recommendation is to pass everything in the GPU's IOMMU group into the VM (I assume cuz this error you're seeing) I don't think you'll be able to do that because if you pass in both your m.2 drives you won't be able to run an OS on the host (if the m.2 drive has your host OS and is in the same group as the GPU) Maybe you could try the GPU in a different PCI slot to see if that moves it to its own group? Or if it's within the return period maybe try another motherboard? Other than that you could do some research on the vfio subreddit on the "ACS Patch". I haven't used it but I think it is a trick people use to get around bad IOMMU grouping
I chose dual-GPU for this tutorial series because it's easier to setup and easier to use. The host OS stays on and has a monitor attached it, so you do have the ability to use them both at the same time But you're right, single-GPU is possible. You'd just need a script that kills the Xserver on the host and unbinds the GPU when the VM starts
@@killertigergaming7395 it would be like you start up your computer in Linux and you have graphics and you can navigate windows Then when you want to start your VM you would run a special script that turns off everything graphical, then gives the GPU to the VM, then starts the VM and you would see the VM graphics That's is the way I think single GPU passthrough work work Does that make more sense?
Hey whatsup BlandMan! I followed the tutorial thru and thru but I only have 1 GPU and I want to do this project. So basically when I finished with this video (I did everything) and rebooted my PC. The resolution started looking terrible, I would assume that it is because my GPU is being used by the config that we did. But is there a way to config it, in such a way that it only starts using my GPU when I start running the VM? Thank You in advance!
Yes it is possible. It's called "single GPU passthrough." Its a little annoying that you have to kill the graphics on the host to start using the guest, but it is possible I followed this tutorial github.com/joeknock90/Single-GPU-Passthrough Then filmed this demo th-cam.com/video/p3eJNp3OmrA/w-d-xo.html But I'm still working on my video guide, so hopefully that's out soon
@@BlandManStudios One last thing though BlandMan, when I'm done using the VM and turn it off, my gpu doesn't re-attach to my Linux. So my monitors don't get any signal and I have to hard reset. By any chance You have a discord server where I can seek out help on this?
That's a really good question. Get some acceleration in the VM and save the good GPU for the superior OS haha I haven't tried it, but you'd need two things for it to work 1. Host graphics need to show up as PCI device in "lspci" 2. Host graphics needs to be in it's own IOMMU group (or pass everything that is in the same group or find a way to hack around it)
That's a really good question. Get some acceleration in the VM and save the good GPU for the superior OS haha I haven't tried it, but you'd need two things for it to work Host graphics need to show up as PCI device in "lspci" Host graphics needs to be in it's own IOMMU group (or pass everything that is in the same group or find a way to hack around it)
That's a really good question. Get some acceleration in the VM and save the good GPU for the superior OS haha I haven't tried it, but you'd need two things for it to work Host graphics need to show up as PCI device in "lspci" Host graphics needs to be in it's own IOMMU group (or pass everything that is in the same group or find a way to hack around it)
not working for me, tested 20/04/22. after generating the new grub config and rebooting my fedora wont boot up anymore. do I have to blacklist the nautilus driver? can you make a tutorial how to run fedore on onboard graphics and use main gpu for passthrough only?
Hmmm, I'm not sure what the problem is without more info. Did you do this on a fresh install? It's possible if you already had the Nvidia drivers installed and you disabled the GPU (using the arguments I gave in the video) now your computer has no GPU driver available What onboard graphics (card and driver) is it supposed to be using when you have the GPU set aside for passthrough?
@@BlandManStudios thx for response. I fixed it by using my onboard graphics for fedora and nvidia for passthrough. Good job on the series. I've noticed you edited some mistakes away like when you added the virtio iso as sata storage instead of cd drive. This took me an hour to figure out cause i followed everything as you showed. It would have been great if mentioned that you changed it to cd drive instead of editing it away. Anyway great job and thank you 😀
I've been following along since part 0 and I'm continually impressed by the quality of these videos. You could have just given us a set of instructions to follow, but you went over the top and gave us a comprehensive guide on what's happening and why, supported by beginner-friendly explanations. Keep up the great work, I'm excited for your content and the future of your channel!
That's so awesome to hear!
Part of the joy of Linux is getting to learn what happens under the hood. Very happy I could share that and very happy my explanations made sense :)
This is probably the best tutorial video series I have ever seen on youtube! Great work! The way you calmly and clearly explain every step and the possible alternatives on other distros is amazing.
Heck yes! Mission accomplished! Thanks for the message and letting me know which parts are the most helpful!
This is an amazingly detailed guide. I've done a few single gpu passthrough systems on Ubuntu and Arch based distros, but switched to Fedora recently, and your videos have helped me achieve the same here as well! Thanks a ton! Subscribed!
So glad I could help! Thanks for the sub!
This tutorial is such high quality, and extremely thorough, big big thanks
I can't believe this works! Props to you man! Simply the best tutorial on this topic 👍
Had some troubles setting up in existing fedora 37 (dracut part produced errors I couldn't fix) then try fresh install and it worked!
Although now I can't change the resolution since I basically just depend on my processor's abysmal gpu power 😂. Gonna try single gpu passthrough.
You deserve much more subs for how good these videos are.
For those on ubuntu linux the grub file is in /etc/default/grub and you should run sudo update-grub to update grub and install dracut and the new command is now dractu -f --no-kernel --kver "uname -r" at least this is what I did on my system when following this.
THANK YOU!!!!
thanks alot!
i am on linux mint, and when i run dracut after installing, i get this error: Can't write to /boot/efi/09c7bf9099104344a6982459b4191d0a/uname -r: Directory /boot/efi/09c7bf9099104344a6982459b4191d0a/uname -r does not exist or is not accessible. Any ideas where to look for an answer?
N/M, I just realized reading other comments, I used "uname -r" instead of `uname -r`. Once I changed it, it worked! thanks
hey this is so super cool i did notice ultramarine 40 is so far the only distro i can follow line by line and im sooooo a newb but at the 2:37 the part of the command groupinstall is actally seperate on mine took me some digging but i got it workin thank you and im so following along.
@@bootinnj good to know that part needs an update! Thanks! And nice job following along, its a complicated set of tutorials!
@@BlandManStudios thank you for taking the time to show this and explain it
Great idea to throw commands at the beginning for more advance users.
Thanks! It turned out to be helpful for me when I reset my computer from fresh OS install haha
Thanks for the comprehensive guide man. Still lack the hardware for this but I'll definitely save the video for future reference.
Will you be talking about CPU pinning in the next part? I was looking up other tutorials and there was one that said it improves performance but the cores would be inaccessible by the host even with the VM disabled. I'm hoping there's a way around that because I still do CPU-heavy tasks on the host OS.
Thanks for the comment! I'll don't think I'll do CPU pinning in the next video because you can get quality performance without it. But maybe in the future for a "performance tweaks" video
I bet you'll be able to set it up in a way that balances performance both inside the VM and outside
From what I understand "CPU pinning" is telling the VM to use specific cores. Then "cpu isolation" is when you use those grub commands to tell the host to ignore those cores (even when the VM is off, like you described)
It is a performance boost inside the VM because the host will never borrow those cores from the VM
In practice, I haven't needed that extra performance boost. But you can turn it on and off with a reboot. You could also have multiple grub configs so that you can easily choose between "Linux with all CPUs" or "Linux with CPU isolation" when you boot
@@BlandManStudios sweet. Thanks for the really detailed reply man. You rock.
Using eos for this, and it all works. Thank you very much.
God reading how nice of a dude in the comments making feel bad I keep having to stop, f new fedora lmao messing with this masterpeice of a series! 8:34 now baby having g edit helps a lot)
hi, there is a short section on the dracut.conf.d from 8:34 in that is not clearly visible on the screen and i ran: which is what it looks like but got this:
What should this command be? The one preceding this was also unclear on the screen. I read:
Hey thanks for asking! I think I know what the issue is
There is a very subtle difference between the backtick character (`) and the single quote (')
You want to surround the part of the command with the backtick character so it looks like this
dracut -f --kver `uname -r`
Thanks for the question and the awnswer really helps a lot
Great video! would be nice to see a video surrounding using qemu hooks to use pci passthrough while being able to use the gpu being passed through whenever the vm is off
Thanks! I totally agree! I hadn't thought about qemu hooks, but maybe that is part of the answer for a simple setup that can pass and unpass the GPU
Man I love the way you takle this kinda tricky subject and It'd be really cool if you made a guide for laptops 😜
Glad to hear it and good to know!
If I get a chance to work on laptops, i'll likely be using this guide as my starting point
lantian.pub/en/article/modify-computer/laptop-muxed-nvidia-passthrough.lantian/
This is a great video series. Any chance you can do one for ubuntu? Certain commands are just different
@@NoahLorano I recently ran through the process on Ubuntu to do single gpu pass through. But I haven't been able to find the time to write it up or film it. Hopefully soon
i have all these lists of devices but i dont know the ones that i need to passthrough in the grub loader
00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7b48
Kernel driver in use: skl_uncore
Kernel modules: ie31200_edac
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 07)
Kernel driver in use: pcieport
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7b48
00:14.0 USB controller: Intel Corporation 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7b48
Kernel driver in use: xhci_hcd
00:14.2 Signal processing controller: Intel Corporation 200 Series PCH Thermal Subsystem
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7b48
00:16.0 Communication controller: Intel Corporation 200 Series PCH CSME HECI #1
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7b48
Kernel driver in use: mei_me
Kernel modules: mei_me
00:17.0 SATA controller: Intel Corporation 200 Series PCH SATA controller [AHCI mode]
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7b48
Kernel driver in use: ahci
00:1b.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #17 (rev f0)
Kernel driver in use: pcieport
00:1b.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #21 (rev f0)
Kernel driver in use: pcieport
00:1c.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #1 (rev f0)
Kernel driver in use: pcieport
00:1c.3 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #4 (rev f0)
Kernel driver in use: pcieport
00:1f.0 ISA bridge: Intel Corporation Z370 Chipset LPC/eSPI Controller
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7b48
00:1f.2 Memory controller: Intel Corporation 200 Series/Z370 Chipset Family Power Management Controller
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7b48
00:1f.3 Audio device: Intel Corporation 200 Series PCH HD Audio
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 9b48
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
00:1f.4 SMBus: Intel Corporation 200 Series/Z370 Chipset Family SMBus Controller
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7b48
Kernel driver in use: i801_smbus
Kernel modules: i2c_i801
01:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c4)
Kernel driver in use: pcieport
02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch
Kernel driver in use: pcieport
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 [Radeon RX 5600 OEM/5600 XT / 5700/5700 XT] (rev c4)
Subsystem: ASUSTeK Computer Inc. Device 0583
Kernel driver in use: amdgpu
Kernel modules: amdgpu
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 HDMI Audio
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
05:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 550 640SP / RX 560/560X] (rev cf)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 8a91
Kernel driver in use: amdgpu
Kernel modules: amdgpu
05:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X]
Subsystem: Micro-Star International Co., Ltd. [MSI] Device aae0
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 7b48
Kernel driver in use: r8169
Kernel modules: r8169
i want to passthrough my rx 5700 wich one of these would i need to include in the grub
It looks like 03:00.0 and 03:00.1 are your RX5700
@@BlandManStudios so i would not need to passthrough another device
@@NetFelixOfficiel yup, it looks like those two will get you graphics in the VM
@@BlandManStudios thank you so much for all your help
5:50 how would i do this for systemd? I'm on Pop OS and from what I'm reading (with my very limited knowledge) grub isn't used any more, or at least when entering "sudo gedit /etc/sysconfig/grub" it's empty
Looks like the way to add "kernel parameters" (aka "Linux command line arguments") with systemd-boot on arch is under "options" in a file at /boot/loader/entries/arch.conf
wiki.archlinux.org/title/Kernel_parameters#systemd-boot
Might be a similar path, file, and process on Pop!OS
@@BlandManStudios Funny enough I saw that site before I saw your video as I was getting confused by a different guide on reddit. Arch seems to have "/boot/loader/loader.conf" but Pop OS doesn't (used the gnome file manager to look at the folder, couple of things with "initrd" or "vmlinuz" in the name but I don't think that's what I'm looking for)
(I'm gonna leave this long comment here to remind myself of what I did because I'm gonna forget by tomorrow, and for anyone else on Pop OS 22.04 if interested)
A bit more searching and found out Pop uses "kernelstub". I entered "cat /proc/cmdline" and got a few things to pop up similar to 6:55 (quiet splash amd_iommu=on). I found a website that Firefox warned me wasn't safe (I don't know why) that said:
"To make the change persistent after reboot, edit /etc/kernelstub/configuration and run kernelstub"
Entering "sudo nano /etc/kernelstub/configuration" brought up a file with some kernel options and a user section with "amd_iommu=on". I'm going to add everything from 6:55 into this part and hope for the best. My primary GPU is a 6600xt and the VM will be using an old Vega 56, so anything with Vega in the name is getting added to the list ("pci-stub.ids=1002:687f,1002:aaf8" no spaces between stub ids, no idea if it should be a separate line or bundled together)
There may be a new (potential) problem. There's an IOMMU for "VGA compatible controller" and "Audio device" with (Vega 10 listed in the name) for video and audio, but also "PCI bridge". I'm gonna skip adding these
If I don't respond back to this comment in a few days, assume that I've broken my desktop
@@GoingMenthol Did you ever figure this out, or did your computer explode? I am also on Pop OS and want to know if it is safe to do what you did, or if I should turn back.
@@0Noobslayer0 Computer crashes when i try to run virtual machine but didn't blow up, I've only just noticed I added a 5 port PCI USB hub to the list of PCI IDs. I can remove it from /etc/kernelstub/configuration using "sudo nano" to edit the file, but using "cat /proc/cmdline" or "sudo dmesg | grep -i IOMMU" shows that the wrong PCI ID is still being listed. I tried "sudo nano /proc/cmdline" to edit it but I'm getting an error that doesn't let me save changes (apparently it's not a real file)
What I'm gonna try now is "sudo kernelstub" to confirm changes and attempt to reboot to the boot menu (hold space on boot up I think) to see if there's anything I can edit from there. Will see how that goes
UPDATE: restart the desktop, hold space bar, let go of space bar when you get to the recovery options listed and press "e", edit the boot parameters (turns out I cannot spell and there were a few typos) and hit enter. Boot to desktop and open terminal, enter "sudo kernelstub" to confirm changes to kernel in case it doesn't do that already
follow part 3 VFIO tutorial that BlandManStudios made. If you get an error with something like "pci controller with index 0 must be pci-root" and you cannot delete the "controller PCIe"when trying to begin installation, go to the "overview" section on Virtual Machine Manager, click the XML tab, search for something like "controller type=pci" (for me there were a bunch of the same thing) and replace with "", which now lets me boot to the virtual machine
I can now boot the virtual machine, will update again once windows drivers are installed and will attempt to add the GPU
UPDATE AGAIN: installed windows, installed all available drivers, added the Vega 56 PCI devices to VMM, as soon as it tries to install AMD GPU drivers via windows update it blue screens, trying to install from AMD's own drivers causes a blue screen. Now the VM is a black screen when I power it up, but works fine when the Vega GPU and HDMI Audio are removed. @BlandManStudios I don't suppose you know what I should be checking for? Is there something in the XML I need to change?
awesome
Thanks!
Trying this on Ubuntu 24.04 .. for the dracut step. the vfio_virqfd config fails, googling leads me to arch wiki stating:
As of kernel 6.2, the vfio_virqfd functionality has been folded into the base vfio module.
If you also have another driver loaded this way for early modesetting (such as nouveau, radeon, amdgpu, i915, etc.), all of the aforementioned VFIO modules must precede it.
If you are modesetting the nvidia driver, the vfio-pci.ids must be embedded in the initramfs image. If given via kernel arguments, they will be read too late to take effect. Follow the instructions in #Binding vfio-pci via device ID for adding the ids to a modprobe conf file.
Per the docs I tried using: force_drivers+=" vfio_pci vfio vfio_iommu_type1 " .. but I still see the device kernel driver as: nvidia after reboot ...not sure where to go from here
Also tried: /etc/modprobe.d/vfio.conf
softdep nvidia pre: vfio-pci
Do you really need root privilegs to run virt-manager on Fedora? I definitely don't need to do this on Debian based distros... Maybe just adding your user to the libvirt group can let you launch it just fine without sudo? I'm pretty sure that Debian adds you automatically to that group. Maybe Fedora just... doesn't?
Also there's another driver category that could f-up things: EFI framebuffer drivers. Especially if you boot with CSM (compatibility support module) disabled. I personally had to also pass video=efifb:off to the kernel to make things work on my machine. This is typically not something that a new user encounters though. Especially since CSM is enabled by default on most motherboards.
So far I have needed it to run virt-manager or "virsh edit "
But I just checked and you're right. I just needed to add my user to the libvirt group and it works passwordlessly. Definitely a better solution than my sudo hack. Thanks!
And thanks for the info on CSM
Description has been updated with the info on the libvirt group. Thanks!
Any ideas why the command of 8:39 report this? Cannot find module directory /lib/modules/uname -r/
And --no-kernel was not specified
Are you using the backtick character?
Single quote '
Double quote "
Backtick `
On the keyboard it is over by the escape key with ~
unix.stackexchange.com/questions/27428/what-does-backquote-backtick-mean-in-commands#27432
@@BlandManStudios bingo! That was it.
@@manuel.arriola yay!
The pci id that is written into the grub file, is that only the GPU which will be passed to the guest in the VM or does it also include the GPU used by the host?
Just the one on the guest goes in the grub file
Using force_drivers instead of add_drivers will ensure that the drivers are tried to be loaded early via modprobe.
Good tip, I didn't know that
Wait, so my linux host won't *ever* have access to my gaming GPU? This is kind of a deal breaker, as I game a lot on linux too natively or using proton. I thought I'd pass it *only* when my windows VM was running. Is this possible too? Has anyone done this?
It is possible to make a script that binds and unbinds the gpu from a VM while the system is running
@@AbdallahRabbea-c2o Interesting, do you have any examples on github?
@@Voshchronos this channel has a laptop tutorial for VFIO with a cheat sheet in description. It also works for desktop and it should cover everything.
@@AbdallahRabbea-c2o ooh, nice. thanks, brother!
Mercy, I have been trying to follow along, but I have recently discovered a new level of OCD. Where if I have an inkling of doubt... I wipe my drives, (backups too) and restart with a fresh install. I''m on 3rd install of Ubuntu.
@@NextToken this is relatable
"Oops I typed this command wrong, better refilm the entire tutorial"
I bet you'll get through it. Trust yourself and recognize there will be bugs to work out and get through, but once you squash them all you can start over fresh to create the "perfect install" with all your newfound knowledge/experience
is it normal that when i run sudo gedit it gives me this:** (gedit:3224): WARNING **: 18:13:21.293: AT-SPI: Could not obtain desktop path or name
** (gedit:3224): WARNING **: 18:13:21.298: atk-bridge: GetRegisteredEvents returned message with unknown signature
** (gedit:3224): WARNING **: 18:13:21.298: atk-bridge: get_device_events_reply: unknown signature
** (gedit:3224): WARNING **: 18:13:21.298: atk-bridge: get_device_events_reply: unknown signature
Yup that's normal. I'm pretty sure the warning comes from using a graphical application (gedit) with sudo
If you're familiar with nano or vim, a command-line text editor won't give those warnings
I've finally got lspci -nnk to show using vfio-pci instead of pci-stub. However, I still can't get the vm to display out to the monitor (host on display port and vm's gpu on hdmi).
Now that it's being used by vfio-pci you're ready to add it to the VM and install the driver to the VM
Have you seen the part3 video?
@@BlandManStudios Yup. But it only covers Windows VM. I want it for Linux too. I know that it makes no sense, but that's what my goal is. I want to be able to boot into my main linux, shut it down, and boot into Windows without ever having to hit the reset button. I've already figured out how to add boot from the Linux storage device bare metal as it were.
@@BlandManStudios Out of curiosity, would I need to run the commands "sudo start libvirtd" and "sudo enable libvirtd"?
@@andromydous if virt-manager is started and "connected" I think its already running so you don't need to start it
You can run this to know for sure
sudo systemctl status libvirtd
@@BlandManStudios So after some searching, though not finding an exact answer, it seems that it's a video port issue. When I run lspci -nnk in my PopOS vm, it shows that the gpu was indeed passed through and is using the amdgpu drivers.
i get an error during boot, can anyone please help me how to fix it? I turned on virtualization in BIOS and disabled secure boot. Google search suggests disabling vt-d
[drm] VT-d active for gfx access
VT-d does need to be enabled
Have you been able to verify (by finding product spec or other people with the same hardware) that your MOBO and CPU support IOMMU/VFIO
@@BlandManStudios Thanks for the reply i really appreciate this! Yes, it does support virtualisation. My CPU is i7-13700K and motherboard is MSI Z690-A.
In fact, I event managed to get it working on Manjaro, but there was another issue and i decided to get through your complete Fedora tutorial.
But it doesn't seem to work on Fedora :( I did everything exactly like you showed in your guide. It does boot if i remove grub options related to virtualisation, but doesn't boot with them. I haven't tried reinstalling Fedora yet though, maybe it will help
I reinstalled fedora and now my boot is stuck at i915 0000:00:02.0: enabling device (0006 -> 0007)
so im following the tutorial word for word and when I got to the end my GPU audio driver was in pci-stub but the actually card was still in amd-gpu. is there something I missed?
There was a typo in the video that might be causing your problem
The command line argument is supposed to be "rd.driver.pre=vfio-pci" and not rd.driver.pre=vfio.pci" which I accidentally put on the screen in the beginning
Also, I haven't done this with an AMD GPU (I've only used Nvidia) so maybe you need to blacklist the "amd-gpu" driver
I had a problem in part3 of video and I come back here. I did some research but without success... I got " Please ensure all devices within the IOMMU_GROUP are bound to their vfio bus driver". Does I need to put every single item that I present in the IMMO GROUP to vFio?
In this video you doesnt show in the LSPCI as group, but my group has a lot of things that doesnt make any sense...
I read that I could ignore the PCI Bridge's but I stil receive the same error...
In my IOMMU Group I have 7 itens
1- PCI Bridge E3-1200 x16 Controller
1- PCI Bridge E3-1200 x8 Controller
1- PCI Bridge AMD Pcie Bridge
1- PCI Bridge AMD Pcie Bridge (just the id change)
1- VGA AMD VEGA 10 XL
1- Audio device AMD
1- Non volatile memory controller (x16 to m2)
Checking anothers computers just mine has a lot of stuff in the same group...
Tried to put all items from group and now my fedora doesn’t boot anymore
Great question!
Usually my recommendation is to pass everything in the GPU's IOMMU group into the VM (I assume cuz this error you're seeing)
I don't think you'll be able to do that because if you pass in both your m.2 drives you won't be able to run an OS on the host (if the m.2 drive has your host OS and is in the same group as the GPU)
Maybe you could try the GPU in a different PCI slot to see if that moves it to its own group? Or if it's within the return period maybe try another motherboard?
Other than that you could do some research on the vfio subreddit on the "ACS Patch". I haven't used it but I think it is a trick people use to get around bad IOMMU grouping
@@BlandManStudios thank you, I’ll try the ACS Patch, after I fix the boot again, I’m not a Linux user, I think I screwup my Fedora…
@@rodrigo_rodrigues I've been there many times
You're a Linux user now and it sounds like you're learning fast
Goodluck!
you could have used single gpu passthrough right? is there any reason to use 2 gpus?
I chose dual-GPU for this tutorial series because it's easier to setup and easier to use. The host OS stays on and has a monitor attached it, so you do have the ability to use them both at the same time
But you're right, single-GPU is possible. You'd just need a script that kills the Xserver on the host and unbinds the GPU when the VM starts
@@BlandManStudios why i asked this is itll be complicated to perform remote pc.
Also im waiting for hackintosh and other kvm series. 🔥🔥🔥
@@BlandManStudios im kind of confused how would that work
@@killertigergaming7395 it would be like you start up your computer in Linux and you have graphics and you can navigate windows
Then when you want to start your VM you would run a special script that turns off everything graphical, then gives the GPU to the VM, then starts the VM and you would see the VM graphics
That's is the way I think single GPU passthrough work work
Does that make more sense?
@@BlandManStudios yea i mean i kinda understood what you meant but what seems difficult is how would I actually do it idk how to make a script lol
Small problem: followed this guide and ended up with a black screen with an X cursor.
could it be a problem because im running with a IGPU and GPU?
Hey whatsup BlandMan! I followed the tutorial thru and thru but I only have 1 GPU and I want to do this project. So basically when I finished with this video (I did everything) and rebooted my PC. The resolution started looking terrible, I would assume that it is because my GPU is being used by the config that we did.
But is there a way to config it, in such a way that it only starts using my GPU when I start running the VM? Thank You in advance!
Yes it is possible. It's called "single GPU passthrough." Its a little annoying that you have to kill the graphics on the host to start using the guest, but it is possible
I followed this tutorial
github.com/joeknock90/Single-GPU-Passthrough
Then filmed this demo
th-cam.com/video/p3eJNp3OmrA/w-d-xo.html
But I'm still working on my video guide, so hopefully that's out soon
@@BlandManStudios Thank You very much! It helped me a lot to setup my VM
@@BlandManStudios One last thing though BlandMan, when I'm done using the VM and turn it off, my gpu doesn't re-attach to my Linux. So my monitors don't get any signal and I have to hard reset.
By any chance You have a discord server where I can seek out help on this?
is it possible do pass through the igpu and use the dgpu on my linux host
i know it makes no sense for most people but kinda curious
That's a really good question. Get some acceleration in the VM and save the good GPU for the superior OS haha
I haven't tried it, but you'd need two things for it to work
1. Host graphics need to show up as PCI device in "lspci"
2. Host graphics needs to be in it's own IOMMU group (or pass everything that is in the same group or find a way to hack around it)
That's a really good question. Get some acceleration in the VM and save the good GPU for the superior OS haha
I haven't tried it, but you'd need two things for it to work
Host graphics need to show up as PCI device in "lspci"
Host graphics needs to be in it's own IOMMU group (or pass everything that is in the same group or find a way to hack around it)
That's a really good question. Get some acceleration in the VM and save the good GPU for the superior OS haha
I haven't tried it, but you'd need two things for it to work
Host graphics need to show up as PCI device in "lspci"
Host graphics needs to be in it's own IOMMU group (or pass everything that is in the same group or find a way to hack around it)
not working for me, tested 20/04/22. after generating the new grub config and rebooting my fedora wont boot up anymore. do I have to blacklist the nautilus driver? can you make a tutorial how to run fedore on onboard graphics and use main gpu for passthrough only?
Hmmm, I'm not sure what the problem is without more info. Did you do this on a fresh install?
It's possible if you already had the Nvidia drivers installed and you disabled the GPU (using the arguments I gave in the video) now your computer has no GPU driver available
What onboard graphics (card and driver) is it supposed to be using when you have the GPU set aside for passthrough?
@@BlandManStudios thx for response. I fixed it by using my onboard graphics for fedora and nvidia for passthrough. Good job on the series. I've noticed you edited some mistakes away like when you added the virtio iso as sata storage instead of cd drive. This took me an hour to figure out cause i followed everything as you showed. It would have been great if mentioned that you changed it to cd drive instead of editing it away. Anyway great job and thank you 😀
somehow the iommu did not start and my video drivers stopt working. i am going to try with pop os with nvidia support
Is dracut a thing on arch? or is it only required on fedora? I remember doing vfio without it.
You're right. Arch uses mkinitcpio by default but you can install dracut
wiki.archlinux.org/title/Dracut
@@BlandManStudios Ah I see, thanks for the info mate :)
@@Battler624 Cheers! :)
does this works for iGPU?