I assume this requires the host motherboard and bios supports IOMMU and that it is enabled? Also, why the world is not just not an option in the UI? Same with Proxmox. Though, in Proxmox once you go through the steps to enable PCIe passthrough, you can just do everything else from the GUI. But still, the hypervisor should just detect if the hardware is capable of sharing PCIe devices and then let you set everything from the GUI.
The reason is that nobody cares enough (of the businness customers that actually pay the bills). Same for sr-iov. Afaik the only project that has full GUI for PCIe passthrough setup is Unraid, from version 6.9 onward.
ESXI *cough Broadcom has a really nice implementation. I tend to think the OP is more right than wrong tho. I do agree that this is more of a "home-lab" request... But we always want more. I absolutely love XCP-ng & Proxmox, but the full GUI implementation in ESXI was really nice. Just give it to us. It would FACTUALLY make their project better. Additionally, thank you Tom for putting this out. Your content is always great. Love the amount of videos you've been cranking out too. The community really appreciates you and all your hard work!
@@mt_kegan512 xpc-ng and proxmox (and Unraid for that matter) have a plugin API to let the community make GUI plugins for things like these that don't warrant the attention from the "main" dev team. That functionality in Unraid was originally a third party plugin as well.
@@mt_kegan512 ESXi does this much better indeed. Combined with SR-IOV it is very powerful and we do rely on this for our business. In telecom there are still plenty of reasons why you'd want a PCI card, or port, passthrough in a production environment.
I have also asked in the forums but for newer hardware to be supported they need update their kernel. Come on 4.19 back porting all everything because their storage solution depends on it
Anyone have success with this on Linux VMs? Very straightforward process, but the OS always hangs when trying to pass through to Linux VMs saying the device is busy (though it’s not)
Two things: 1) I noticed that in the VM, under the Advanced tab, that there is an option for GPUs. Is this the process that we would follow to ennumerate said GPUs so that it will be made available to the VMs? 2) I am guessing that by passing through the PCIe device (e.g. GPU) that you won't be able to SHARE that resource between VMs, correct? (Unlike Proxmox where you can share a GPU across multiple LXC containers.) Thank you.
This is voor vgpu only from what ive seen. If you run xcp-center you can pass through the gpu from there without having to go commandline for this. Although results may vary as this doesn’t work for the nuc 11 and 13 it seems. Devices are detected by the vm but you can’t use them. Did a post in the forum trying to get this resolved.
Hey Tom, watching this while trying to pass the SATA controllers and one of the M.2 drives through to a TrueNAS VM, and I have a feeling from something I've read that there may be issues in doing so for the AsRock Rack X570D4U-2L2T; is that a situation you've tested on your end? I know you use a couple of those (or at least similar), so figured I may as well ask. Will get a HBA if needed, I'd just love not to need to XD
Okay, never mind, got the drives passed through. Now it's just hanging on boot instead...but they're showing in the boot sequence, so maybe I can somehow work through that? May end up as a sticking point, but I'll keep at it for now to see if I can work my way through it.
@@LAWRENCESYSTEMS Didn't see this when I put up my other comment, whoops ':D Thanks for getting back to me though :) I seem to have got that sorted, but yeah, TrueNAS Scale's now hanging at "Begin: Running /scripts/local-premount ... done.", or occasionally working through at least some of mounting sda and hanging there instead. I did seem to get further with that than with Core, but I've yet to figure out just what's hanging. Having the issue both with 23.10.2 and 24.04-RC1, with Core 13.0-U6.1 not even getting through install. (Scale also didn't like rebooting after installation with drives connected, and went through install just fine before the SATA controller was passed through...might see how it goes with it passed through with no drives for testing purposes.)
And I got it sorted out, as far as I need to for now at least; half the SATA ports passed through and that's all I need at the moment. Issues passing the other controller through, but I can pick up a dedicated controller down the track to resolve that I guess. Either way, it's been a great thing to work through, and your video was a massive assist on it, both for this and many other things I've been working on for moving my setup over to XCP-ng.
Thanks for the video and explanation. Looking at moving away from VMWare, and relying on passthrough quite heavily, this is pretty much a no-go for XCP-NG. Doing this kind of config from the command line makes it basically unusable for anything other than a hobby environment. Thats a real pity. Also, an important aspect with this is SR-IOV support, allowing one PCI device to be shared over multiple VMs. For instance, a 4 port adapter shared over 4 VM's each getting their own port.
XCP-ng does have SR-IOV support and passthrough will probably be added to the web UI in the future but for now it's command line. We don't see too much passthrough in the enterprise work because it bind the VM to that host.
@@LAWRENCESYSTEMS It does not necessarily *have* to bind the VM to the host, though. If the same type of hardware device is available at multiple hosts in the cluster, nothing is stopping the software from allowing configuring devices on a cluster level (i.e. define a cluster-wide PCI device and assign a physical device to that resource from every host). The VM could then use the cluster-defined PCI device and run on any of the hosts that offer the PCI resource within the cluster. Of course things like live migration won't work, but HA for example could.
The inability to hot-plug USB, as well as the lack of snapshots while they're attached seems like a pretty severe limitation... I know QEMU can do both at least, but is this maybe just a UI limitation for XCP-ng (as it is for proxmox), or does the underlying tech stack have the same limitation as well?
Do you have to have the XO from sources in order to see the USB's from the gui? I have the same version of XCP-NG running and can't see an USB passthrough using XOA.
Make sure you are using the latest version as this is a brand new feature. You can check by going to XOA then Updates in the menu and choose "Release channels Latest" to be on the most current one.
Do pass through USB devices survive a host restart? Testing earlier in the year I found that the USB device would disconnect, workaround being pass through the USB pci controller instead of the USB device
2:20 Having to reboot a VM to enable a USB device doesn’t look good. Imagine trying to do that to a server OS. That would imply downtime unless you schedule it
In a past video you said that passing a pcie card connected to a bunch of HDD's might cause some troubles and it was better to connect the disks directly to the VM (it was a truenas example). Would you still recommend passing the disks individually or the PCI card?
Great video 👌🏼 Ive been moving stuff off vmware to xcp ng last week and all seem to work great except this part. Been passing through the igpu for years in vmware but for some reason this doesn’t work in xcp ng 8.3. The device gets detected in linux it even shows hardware transcoding but just hangs from there. This happens on a NUC11 and 13. You or anyone else have hint on what could be the problem? I could try 8.2 to see if it is a bug in 8.3…
Well... I just installed "patches" on my xcp-ng host with Home Assistant aaaaaaaaaand USB passthrough stopped working... All my Zigbee, BT, etc devices stopped working. Why? Because XCP-NG overwrites /etc/xensource/usb-policy.conf and who knows what else... Time to figure out what to do :./
I didn't experience this when installing the latest patches, I also have HA virtualized and use USB passthrough on a zigbee dongle. Everything started up and worked normally. There must be some other configuration difference that caused your usb-policy.conf to be overwritten, my usb-policy.conf hasn't changed in six months.
Not directly on video topic, but has Vates been moving features that used to be in the free version of XOA up into licensed levels? I spun up XOA about 3 years ago and I could do file level restore - now it looks like you have to have a Premium license - is that correct?
Thx for the video Lawrence! I tried GPU passthrough to a Windows VM and it works. However, when I tried GPU (RTX 3090) passthrough on an Ubuntu 22.04 VM, the OS would hang on startup. Is anyone else experiencing this? Does anyone know a solution to this?
Maybe I missed this, I understand that with USB passthrough you can't do snapshots and the VM has to be shutdown for backups, what about when passing a NIC or passing an HB, does one still have the restrictions on snapshots and backups, or is it only for USB devices?
Well, that's not a fun dance to attach PCI-devices. You'd think you could do that from XO, given how you can do the same from quite literally every other hypervisor's GUI utilities....!
Ugh. Wake me up when you don't need copy, paste into the command line! No snapshots possible when pass-through enabled can be a showstopper, but then, when, why would you need external USBs? Encryption? Passwords? Data? Can be done without the complexity (smb, rdp, even usb over ip). NIC/HBA pass-through to a VM is a real possibility and real scenario, but those solutions are usually "embedded", stateless or reproducible with 1 configuration file, so, not terrible, but they can surely improve or workaround the technical issues.
trying to follow along during your tutorials is difficult, you spend 3 minutes on reiterating what is needing and making sure that you have things this way, or that it's self compiled, and then 15 seconds on actually doing what the tutorial is for "go over here, and then click this, and then look at this". I want to learn XCP-NG because it seems like a less bloated hypervisor than most of the others i've delt with, but there is no explanation, it's just click here, do this, look at this.
@@LAWRENCESYSTEMS and i did. maybe before you post a tutorial, you should listen to it, without the video, like most people do when using it as a guide, and see if YOU can follow along before posting it. my point is you give us the reiteration of a warning, and that's needed, before spending like 15 seconds of go here, click, then go here click, make sure this is right, click. you are knowledgeable, and post a ton of these "guides" but they are amongst the worst tutorials i've ever seen. just because you can do something doesn't mean you are good at teaching someone to do it.
@@LAWRENCESYSTEMS Just because YOU can't teach something, doesn't mean others can't learn. this is why Linux isn't well adopted, "it's not me, it's you" well in this case it is you, you seriously need to listen to the audio only and try and follow along. it's impossible. you are a horrible advocate for xcp-ng, and i have been wanting a true hypervisor since i first debated between slackware and redhat in 1996.
Really enjoying these In-Depth XCP-NG video tutorials! Thanks for taking the time to make them. I am becoming such a fan of XCP-NG!
Thank you for making this video.
My pleasure!
Thank you so much for making this lovely video!!
I assume this requires the host motherboard and bios supports IOMMU and that it is enabled? Also, why the world is not just not an option in the UI? Same with Proxmox. Though, in Proxmox once you go through the steps to enable PCIe passthrough, you can just do everything else from the GUI. But still, the hypervisor should just detect if the hardware is capable of sharing PCIe devices and then let you set everything from the GUI.
The reason is that nobody cares enough (of the businness customers that actually pay the bills). Same for sr-iov.
Afaik the only project that has full GUI for PCIe passthrough setup is Unraid, from version 6.9 onward.
ESXI *cough Broadcom has a really nice implementation. I tend to think the OP is more right than wrong tho. I do agree that this is more of a "home-lab" request... But we always want more. I absolutely love XCP-ng & Proxmox, but the full GUI implementation in ESXI was really nice. Just give it to us. It would FACTUALLY make their project better.
Additionally, thank you Tom for putting this out. Your content is always great. Love the amount of videos you've been cranking out too. The community really appreciates you and all your hard work!
@@mt_kegan512 xpc-ng and proxmox (and Unraid for that matter) have a plugin API to let the community make GUI plugins for things like these that don't warrant the attention from the "main" dev team. That functionality in Unraid was originally a third party plugin as well.
Yup hardware support is required. Iommu support by intel is a mess. I hear AMD is much more sane.
@@mt_kegan512 ESXi does this much better indeed. Combined with SR-IOV it is very powerful and we do rely on this for our business. In telecom there are still plenty of reasons why you'd want a PCI card, or port, passthrough in a production environment.
Just did this two weeks ago from your older video lol passed an hba card. Super simple
One aspect that was missed on the PCI Device Passthrough, is how to check if the machine you are using even supports it. (xm dmesg)
Awesome video, Tom. One question: Is it not possible to run modprobe -r to remove the kernel module and avoid a reboot?
Not sure, also worth noting they have added pass through options to the web interface
I have also asked in the forums but for newer hardware to be supported they need update their kernel. Come on 4.19 back porting all everything because their storage solution depends on it
Anyone have success with this on Linux VMs? Very straightforward process, but the OS always hangs when trying to pass through to Linux VMs saying the device is busy (though it’s not)
Thank you, great video. Would the same steps also work for GPU-Passthrough?
It should
Two things:
1) I noticed that in the VM, under the Advanced tab, that there is an option for GPUs.
Is this the process that we would follow to ennumerate said GPUs so that it will be made available to the VMs?
2) I am guessing that by passing through the PCIe device (e.g. GPU) that you won't be able to SHARE that resource between VMs, correct?
(Unlike Proxmox where you can share a GPU across multiple LXC containers.)
Thank you.
This is voor vgpu only from what ive seen. If you run xcp-center you can pass through the gpu from there without having to go commandline for this. Although results may vary as this doesn’t work for the nuc 11 and 13 it seems. Devices are detected by the vm but you can’t use them. Did a post in the forum trying to get this resolved.
Hey Tom, watching this while trying to pass the SATA controllers and one of the M.2 drives through to a TrueNAS VM, and I have a feeling from something I've read that there may be issues in doing so for the AsRock Rack X570D4U-2L2T; is that a situation you've tested on your end? I know you use a couple of those (or at least similar), so figured I may as well ask. Will get a HBA if needed, I'd just love not to need to XD
Not something I have tested.
Okay, never mind, got the drives passed through. Now it's just hanging on boot instead...but they're showing in the boot sequence, so maybe I can somehow work through that? May end up as a sticking point, but I'll keep at it for now to see if I can work my way through it.
@@LAWRENCESYSTEMS Didn't see this when I put up my other comment, whoops ':D Thanks for getting back to me though :) I seem to have got that sorted, but yeah, TrueNAS Scale's now hanging at "Begin: Running /scripts/local-premount ... done.", or occasionally working through at least some of mounting sda and hanging there instead. I did seem to get further with that than with Core, but I've yet to figure out just what's hanging. Having the issue both with 23.10.2 and 24.04-RC1, with Core 13.0-U6.1 not even getting through install. (Scale also didn't like rebooting after installation with drives connected, and went through install just fine before the SATA controller was passed through...might see how it goes with it passed through with no drives for testing purposes.)
And I got it sorted out, as far as I need to for now at least; half the SATA ports passed through and that's all I need at the moment. Issues passing the other controller through, but I can pick up a dedicated controller down the track to resolve that I guess. Either way, it's been a great thing to work through, and your video was a massive assist on it, both for this and many other things I've been working on for moving my setup over to XCP-ng.
But the backups wouldd still work for my Home Assistant (I don't care about snapshots)?
I also just learned about the 2TB max disk size limitation :(
Stuffing large volumes of data into virtualized disks is not good storage design. th-cam.com/video/1u3aie4w32g/w-d-xo.html
@@LAWRENCESYSTEMS What would you recommend for 3-4TB MSSQL DB servers? Thank you so much for making these videos, we need to get rid of vmware...
@@AlexDiamantopulo if they are running in Windows than you can do an iSCSI attached drive
Thanks for the video and explanation. Looking at moving away from VMWare, and relying on passthrough quite heavily, this is pretty much a no-go for XCP-NG. Doing this kind of config from the command line makes it basically unusable for anything other than a hobby environment. Thats a real pity. Also, an important aspect with this is SR-IOV support, allowing one PCI device to be shared over multiple VMs. For instance, a 4 port adapter shared over 4 VM's each getting their own port.
XCP-ng does have SR-IOV support and passthrough will probably be added to the web UI in the future but for now it's command line. We don't see too much passthrough in the enterprise work because it bind the VM to that host.
@@LAWRENCESYSTEMS It does not necessarily *have* to bind the VM to the host, though. If the same type of hardware device is available at multiple hosts in the cluster, nothing is stopping the software from allowing configuring devices on a cluster level (i.e. define a cluster-wide PCI device and assign a physical device to that resource from every host). The VM could then use the cluster-defined PCI device and run on any of the hosts that offer the PCI resource within the cluster. Of course things like live migration won't work, but HA for example could.
That is use case for SR-IOV
The inability to hot-plug USB, as well as the lack of snapshots while they're attached seems like a pretty severe limitation... I know QEMU can do both at least, but is this maybe just a UI limitation for XCP-ng (as it is for proxmox), or does the underlying tech stack have the same limitation as well?
Do you have to have the XO from sources in order to see the USB's from the gui? I have the same version of XCP-NG running and can't see an USB passthrough using XOA.
Make sure you are using the latest version as this is a brand new feature. You can check by going to XOA then Updates in the menu and choose "Release channels Latest" to be on the most current one.
@@LAWRENCESYSTEMS Cheers Dude. Sorted 🤘🏼
Do pass through USB devices survive a host restart?
Testing earlier in the year I found that the USB device would disconnect, workaround being pass through the USB pci controller instead of the USB device
Rebooting the host does not remove the USB.
Thank you for the wonderful video. It shocks me that this simple thing cant be done in the gui
Maybe one day, but it takes time to write that into the UI. It's open source and if you are a dev feel free to help contribute the code.
2:20 Having to reboot a VM to enable a USB device doesn’t look good. Imagine trying to do that to a server OS. That would imply downtime unless you schedule it
USB passthough is not a very common server use case.
Damn, you literally did a video about the same thing I asked in the Q&A video!
In a past video you said that passing a pcie card connected to a bunch of HDD's might cause some troubles and it was better to connect the disks directly to the VM (it was a truenas example). Would you still recommend passing the disks individually or the PCI card?
When virtualizing something that uses ZFS I recommend passing the controller card through.
Great video 👌🏼 Ive been moving stuff off vmware to xcp ng last week and all seem to work great except this part. Been passing through the igpu for years in vmware but for some reason this doesn’t work in xcp ng 8.3. The device gets detected in linux it even shows hardware transcoding but just hangs from there. This happens on a NUC11 and 13. You or anyone else have hint on what could be the problem? I could try 8.2 to see if it is a bug in 8.3…
Well... I just installed "patches" on my xcp-ng host with Home Assistant aaaaaaaaaand USB passthrough stopped working... All my Zigbee, BT, etc devices stopped working.
Why? Because XCP-NG overwrites /etc/xensource/usb-policy.conf and who knows what else... Time to figure out what to do :./
I didn't experience this when installing the latest patches, I also have HA virtualized and use USB passthrough on a zigbee dongle. Everything started up and worked normally. There must be some other configuration difference that caused your usb-policy.conf to be overwritten, my usb-policy.conf hasn't changed in six months.
Is having to reboot when attaching a usb device a limitation in Xen? My laptop with kvm does handle usb redirection without rebooting the VM
Not directly on video topic, but has Vates been moving features that used to be in the free version of XOA up into licensed levels? I spun up XOA about 3 years ago and I could do file level restore - now it looks like you have to have a Premium license - is that correct?
You can have all the features in the self compiled version but to my knowledge the file level restore has always been a higher license.
Thx for the video Lawrence! I tried GPU passthrough to a Windows VM and it works. However, when I tried GPU (RTX 3090) passthrough on an Ubuntu 22.04 VM, the OS would hang on startup. Is anyone else experiencing this? Does anyone know a solution to this?
Maybe I missed this, I understand that with USB passthrough you can't do snapshots and the VM has to be shutdown for backups, what about when passing a NIC or passing an HB, does one still have the restrictions on snapshots and backups, or is it only for USB devices?
Only USB causes the issue
Well, that's not a fun dance to attach PCI-devices. You'd think you could do that from XO, given how you can do the same from quite literally every other hypervisor's GUI utilities....!
Ugh. Wake me up when you don't need copy, paste into the command line!
No snapshots possible when pass-through enabled can be a showstopper, but then, when, why would you need external USBs?
Encryption? Passwords? Data?
Can be done without the complexity (smb, rdp, even usb over ip).
NIC/HBA pass-through to a VM is a real possibility and real scenario, but those solutions are usually "embedded", stateless or reproducible with 1 configuration file, so, not terrible, but they can surely improve or workaround the technical issues.
A showcase of why people are just going to pay Broadcom.
trying to follow along during your tutorials is difficult, you spend 3 minutes on reiterating what is needing and making sure that you have things this way, or that it's self compiled, and then 15 seconds on actually doing what the tutorial is for "go over here, and then click this, and then look at this".
I want to learn XCP-NG because it seems like a less bloated hypervisor than most of the others i've delt with, but there is no explanation, it's just click here, do this, look at this.
You should be watching the "Getting Started" video before this one. th-cam.com/video/2wMmSm_ZeZ4/w-d-xo.htmlsi=H1xTGthQGGC8xcCL
@@LAWRENCESYSTEMS and i did. maybe before you post a tutorial, you should listen to it, without the video, like most people do when using it as a guide, and see if YOU can follow along before posting it. my point is you give us the reiteration of a warning, and that's needed, before spending like 15 seconds of go here, click, then go here click, make sure this is right, click. you are knowledgeable, and post a ton of these "guides" but they are amongst the worst tutorials i've ever seen.
just because you can do something doesn't mean you are good at teaching someone to do it.
Just because YOU can't understand my tutorials, does not mean they are bad.
@@LAWRENCESYSTEMS Just because YOU can't teach something, doesn't mean others can't learn. this is why Linux isn't well adopted, "it's not me, it's you" well in this case it is you, you seriously need to listen to the audio only and try and follow along. it's impossible. you are a horrible advocate for xcp-ng, and i have been wanting a true hypervisor since i first debated between slackware and redhat in 1996.
@@gwjones82009 It's not a podcast so I would suggest watching the video not just listening to the audio.