Proxmox GPU Virtualization Tutorial with Custom Profiles thanks to vGPU_Unlock-RS
ฝัง
- เผยแพร่เมื่อ 20 มี.ค. 2022
- Visit Keeps.com/Craft to get 50% off your order!
If you have a compatible nVidia graphics card, you can build your own Cloud Gaming Server in as little as 90 minutes! This video is a complete step-by-step tutorial for installing Proxmox, nVidia GRID, and defining custom GPU profiles for allocated memory and supported features.
So why am I posting this updated tutorial less than a year later? Mainly, this update fixes a number of problems with driver compatibility, as well as memory allocation in Maxwell GPUs. Previously, only 8GB was allowed to be used in total on Maxwell-based cards. Now GPUs like the Tesla M40s, Titan X, etc, can use their full 12GB/24GB to divide between vGPU profiles. The installation is easier, more stable, and customizable depending on your needs.
But first... What am I drinking???
From New Image Brewing comes the Can You Milk an Oat... an Oat Cream IPA.. This one caught me a bit by surprise. Very thick and creamy, but with a crisp finish. This is an IPA that doesn't know if it's supposed to be a Winter or Summer drink, but manages to pull both of them off.
Links to items below may be affiliate links for which I may be compensated
Build your own sub-$1,000 Cloud Gaming Server!
Intel Xeon E5-2690 v3 12-Core: ebay.us/ewszQG
Jingsha/Bewinner x99 ATX Motherboard: amzn.to/36lDlKm
OLOy DDR4 32GB 3600MHz: amzn.to/3tdfhlS
16GB DDR4 ECC-REG (Buy 4x): ebay.us/lBdn7x
Tesla M40 12GB: ebay.us/2OD6VL or ebay.us/40kdCU
Tesla M40 24GB: ebay.us/FW9GWQ
be quiet! Pure Base 600: amzn.to/34K6JJO
be quiet! Shadow Rock 3: amzn.to/3I99wde
be quiet! Silent Wings 3 140mm: amzn.to/3isPosl
Fractal Design Ion Gold 850W: amzn.to/3q514FG
Silicon Power 1TB NVMe: amzn.to/3IdVIOH
Delta BFB1012EH Blower Fan: amzn.to/3w4XRKa
nVidia Tesla Cooling Shroud: www.thingiverse.com/thing:474...
My previous Tesla M40 Benchmark video: • $220 for Titan X Perfo...
Instructions and Source links
Step by Step Instructions - drive.google.com/drive/folder...
vgpu_unlock Github: github.com/DualCoder/vgpu_unlock
vgpu_unlock-RS Github: github.com/mbilker/vgpu_unloc...
Register for a 90-Day Virtual Workstation Trial at nvid.nvidia.com
vgpu_unlock Discord server: / discord
Game Streaming Clients (Comparison video soon!)
Parsec
parsec.app/
Sunshine Streaming Server
github.com/loki-47-6F-64/suns...
Moonlight Streaming Client
moonlight-stream.org/
Head on over to craftcomputing.store and grab yourself an official Pint Glass
Follow me on Twitter @CraftComputing
Support me on Patreon or Floatplane and get access to my exclusive Discord server. Chat with myself and the other hosts on Talking Heads all week long.
/ craftcomputing
www.floatplane.com/channel/Cr... - วิทยาศาสตร์และเทคโนโลยี
Milk that cow, I love seeing it! I've been wanting to do something similar to this for a long time, but I've just been too busy \ lazy to do so.
Man, your content has just been getting better and better, once I got used to the brew aspects of your videos I really dig your interests. I have followed a few of your setups successfully for fun and then just sold the parts once I was able to do the project. Thanks for all the fun ideas!
Awesome tutorial Jeff! I don't have a practical use for this type of vGPU setup, but I like watching and learning what can be done. It's amazing what you can do with somewhat new hardware.
Very cool tutorial! This is one of the most accurate guides available to date. I always were interested in this technology and how this is going to evolve for using in Enterprise environments. Will certainly keep following you. Thanks Jeff :)
I greatly appreciate your clarification of what a gaming server is, the benefits of having one, and how to build it! Excellent video! Thanks Jeff!
Your kick off made me laugh like crazy. Great job. Further update. I was the guy who quickly said hello at CES 2020. Sadly I had to run to a meeting but I'm glad to see that you've updated your channel and are doing so well!
LOVE THIS JEFF! For those struggling with the Nvidia profiles, make sure you choose a profile with a Q beside the sizes. There are several letters, but I found this buried in the vgpu issues section and it was the solution!
OMG. Absolutely amazing. Thank you so much for taking the time to go through this. I am in the process of sorting out a setup like this for a plex server for encoding. This is exactly what I was looking for. Thanks so much. :)
Awesome, can't wait for next video for more streaming options.
Fabulous video. Probably the best Proxmox / virtualisation video I’ve ever seen. Impressed. Subscribed!
This video is GOLD! Thanks for taking the time to cover this so thoroughly.
Looking to render with an RTX3060, in a MacOS VM...I think you see where I'm going with this...
Definitely doing this sometime. Thanks Jeff!
Hey man, thank you so much for doing this. I discovered your channel while trying to do this on my server; for anyone wondering, I was able to follow these directions on Proxmox 7.4 Kernel 5.15.102 & dual Tesla P40s
hmmm im using 7.4-3 and getting errors with the dependencies
Great tutorial - Been using Proxmox for a while now as my daily rig is a linux box. But I still want to game so I use use steam play and make sure the windows VM is on the same subnet. works a treat
Never stop the virtual machine videos, next you should do bhyve on FreeBSD with gpu passthrough and hardware acceleration.
Caught that Archer reference. Nice touch.
does anyone else do this? say im installing something new or setting up software. i can know fully how to do the job, but ill still usually check a tutorial just as a sanity check that i do know how. surely this isnt that odd. jeffs vids reminded me that i did it. we do alot of the samethings lol. but always glad to see your new video, keep it up :)
You rock and so does proxmox
Finally this video is what I needed Since I have the Tesla M40 24GB model
Only for the fact that it's in a Dell PowerEdge R720
@@christopherspivey9117 I have got a R720 and does it all work out for you?
@@repairman2be250 Yes it does provided that you have the proper EPS12V to PCI-E
You will have to bridge both the 200W PCIE Power cards in to 1 GPU since the GPU is way more than 200W
@@christopherspivey9117 Thank you for your reply. Will keep an eye on for a M40 card.
KEEP GOING! - This is something I've wanted to do for so long, and I'm knee deep in my own build. ATM I would make a suggestion but it seems you've covered all the bases. This is not an easy thing todo, just the right combination of software and hardware must come together. Oh - How about a demo with Teradici PCoIP? - I'm not sure if it's free but I know it can do 3D graphics.
I am using moonlight works perfectly fine. No need for parsec with newer gpus.
I'm puzzled. I thought this was gonna walk down the Hyper-V route. That being said, as always - ace video, nice information.
Amazing tutorial Jeff, love it. You, sir, have inspired me to make my own gaming server. Just wondering when you were planning on making that video about different streaming programs. Was it going to be about different programs other than Parsec? Hopefully, the video drops soon. Would love to get started on my project as soon as I move and have my computer room all set up.
thanks bud that's a massive effort! hats off to you good sir :)
As a mostly retired techie I found that a great description of something I'll never need to do but like to know that it can be done. But I'd be very worried about a beer that is that strong but doesn't tastes like it isn't. I've been caught out too many times by these.
Lkg fwd to using this "latest and greatest" version 👍
Thank you Jeff.
Keep them coming Jeff! I love all these proxmox/tesla tutorials... Is there any chance you are planning on doing a guide on setting up a Linux guest to use the k80/M40 teslas with accelerated graphics? I had things partially working on Pop!OS VM over VNC with a RX570 passedthrough(video out), but i can't figure out how i did it now (much less with my K80).
I love this content ebaut gpu virtualization Thanks for doing it
Please don't stop making these videos
Can’t wait for this to become a thing in unraid. I’m thinking a Tesla M40 might become my go to for a steam link/moonlight box.
thanks Jeff, great vid
With regards to keeping the M40's cool @Craft Computing. I have heard people say that the Kraken G12 kit works to mount one of the Kraken AIO coolers to the GPU die
I'd like a step by step disaster recovery, of-course this would need to be "simulated" by Jeff, but this would really teach us some very important basics! Creating servers is fun and all, but disaster recovery won't be if you don't know what you're doing :)
Basic explanation of scrubbing and the different pool states would be really helpful.
Disaster recovery = backup proxmox disk before doing any changes.
Love the guides!
@4:17 you should look into using Ansible to setup desired state configuration. I'm just doing this for Proxmox as I'm installing my 'X'th (where X > too many) machine and it's like groundhog day will all the "tweaks" needed on a base install.
This guy is just genius ....
All of this video went over my head.
awsome tutorial! TNX!!!!
Pretty cool 😎. I have installed proxmox before as a test but forgot to partition the hard drive to give me space for my Mac VM. I was doing this on a older intel msi laptop with a ivy bridge cpu and 16GB of ram. 🤣😂 appreciate the video.
You should perhaps re-visit Proxmox with some more powerful hardware. VMs really need SSD storage it's worth noting.
@@wayland7150 appreciate the advice. I was considering that.
Having said that I opted for a z840 workstation with 2x xeons, 28, 2 x RTX A4000, 128 GB Ram and I am happy with the duct inside pushing a lot of air where it belongs for perfect cooling.
"or a hashtag....." i felt that haha
awesome thanks for the tutorial
Thanks, Jeff, for all your work on this! Do you know if any of this has become a little easier in Proxmox V7.3 or is this tutorial stil how this is done right?
This is pretty usefull for thoose who gaming who can find tesla cards for cheap and want to game. Thank you for the guides
This can be done with any Maxwell, Pascal, Volta or Turing based nVidia cards. They don't have to be Teslas.
@@CraftComputing yeah fair point but how i localy looked a maxwell based tesla was 1/3 the price of a GTX 970, yet had the perfromance of the GTX 980, honestly i would of gave it a crack with this guide now but the gpu was sold and i got a R9 380x 4gb.
And if my internet was actually good would try eitherway. But in speedtest i got 20pin, 30mb up and 30mb down. Pretty bad
@@dexsters5643 I've heard the game streaming only uses about 10Mbit/sec. It's worth trying if you want to play around.
@@eDoc2020 i dont have a nvidia videocard, i had a choice GTX 750ti 2gb vor 75€ or R9 380X 4gb for 100€ i took the R9.
@@dexsters5643 You can still do streaming with other cards, the difference is you can't split it across VMs.
Looking forward for the next video, the reason I don't like this setup is I don't want to have to connect through parsec to the same PC i'm using.
Could you also explore DDA (Discrete Device Assignment) in hyper-v? It's supposed to only be available on windows server for GPUs but maybe there's a way to trick it on regular windows
Awesome content!
Thank you for the detailed tutorial! I'm looking forward to building something similar (but probably lower-spec) for my two kids. In a follow-up video can you try out some popular games and tell us if there are issues with DRM systems that prevent the game from running on a virtual host? Or are there techniques to hide the fact that it's running on a virtual host? Also - what's the best way to pass through a microphone on the client to the VM so that you can talk while playing? Thanks again!
It's usually not the DRM but rather anti-cheat being hit or miss depending on title. For microphones you probably can just do a USB passthrough by the Proxmox Web GUI (or via LibVirt if you're using something else).
already weird! After you have explained it in great detail and detail, other youtube channels will make the same video almost in a timely manner...... 😂
Great job! ..please make a tutorial for linux and macos! ..It would be interesting to know if it works with those OS
very interested to see a raspberry pi running a game at 1080p streaming from this beast.
Gracias maestro!!!
I'd love to see an AMD GPU variant of this, especially since we're talking about older GPUs for something like this. AMD GPUs also wouldn't require dealing with enterprise licenses nor NVidia's terrible linux drivers.
Commenting for the algorithm. I agree with you! In his past videos, it makes it more difficult to follow when he doesn't show the AMD technique.
There is no AMD method for this that works with consumer cards. I'd love it if there was.
AMD hasn't released a vgpu capable gpu in many years, so the Linux solution just isn't there, but his videos on using windows hyperV to share a GPU (Google easy gpu-pv) does work for AMD cards.
@@CraftComputing That's disappointing, but good, to hear. I haven't done any digging myself, so it's good to know someone else has. Thanks for your content, and I appreciate it!
@@CraftComputing I've managed to passthrough a RX580 (2nd slot) in Hyper-V in windows 11 pro. The problem I encountered was that the pci-e DDA is not supported in my version of windows. I did some more digging and found that I need windows server OS in order to do what I wanted to do. As soon as I am feeling brave I will be attempting a 2 gamers 1 pc using hyper-v with my current system.
Thank you for the video, Jeff! How about Kepler based cards? I've got K80.
Amazing video tutorial.
I really want one of those Tesla cards but they much more expensive here in UK (around used 1080Ti prices). There are loads of cheaper Grid K2 8GB cards, i don't know if those would work with this system. Need to research that.
Thank you so much for "milking this cow", please don't stop! 😂
Also, would it be possible to do a similar thing on top of TrueNAS SCALE instead of Proxmox?
One thing I have found. If your install is EFI with zfs the grub change for IOMMU will not work. You need to instead create /etc/kernel/cmdline and put what cat /proc/cmdline returns with the iommu turn on command you need. Then run pve-efiboot-tool refresh and reboot. Being my system is a multi-function setup zfs was part of it which added a small twist.
Very cool! This is something I have been wanting to do, I'm wondering whether this will be possible with the latest 12gen Xe integrated GPU or even the coming Intel graphics card for a remote desktop for youtube and streaming. Would you be interested in digging into that?
The slight cringe cut and subtle tone of disgust when you said "hash tag" was amusing lol
I do believe we're in desperate need of a refresh. It's 2024 and Proxmox 8 (with the dropped support of Pascal and older cards on 17.0 and later) is its own beast in and of itself. Getting driver 16.5 to work on 8.2.2 with Kernel 6.8.4-3 is a total nightmare.
I like the virtualized gpu series and I can think of quite a few more passive entries. 1. Quieter fan, 2. Cooler swap 3. Truenas scale 4. Tesla m60 5. Affordable dual Tesla rack system. 6. Tesla p4, Tesla p40?
Oh yeah, also use for workstation applications instead of games.
Is there a water cooling kit that would work with those Tesla M40?
@@JoshuaBoyd I am into that as well. I don't play games.
If it works with this setup then, I would recommend Moonlight over Parsec. Enable nvidia gamestream on the server then run moonlight on the client side. I get lower latency and better picture quality with moonlight over parsec.
I've actually been using Sunshine, an open-source alternative to nVidia Gamestream. Works on Linux VMs too ;-)
@@CraftComputing Nice, never heard of Sunshine I will have to look into. (or you could make a video on it, wink wink)
Next week (wink wink)
What are your thoughts on virtualizing a CAD workstation? I don't desire remote access for it either but dedicated hardware (graphics) is a must.
"If your an IT admin worth it's weight in scotch" is the best quote I have heard in a while
should it be bullseye (Debian 11) instead of buster (Debian 10) at 4:12?
Would you mind doing a power draw/power usage update for us? I'm curious about comparing a setup like this to having 4 low-spec gaming pcs (comparatively) and weighing the power usage for gaming. For me, I'm concerned with the increasing cost of electricity, thanks to Idaho Power being a monopoly...
Great job! Super installation. The author is well done. I just don’t understand, I have a task to also assemble a machine for cloud gaming, but there are 1060 6GB and 1070 8GB cards available, could I run them according to this guide?
Awesome stuff, Jeff. Although there is one issue here...it's too late for me to prevent male pattern baldness with Keeps :/
There's always that Locks of Love sponsorship
great tutorial! I've been waiting for this, I only have on question is how does this impact network traffic per VM?
Streaming to a client requires ~10Mbps upload at 1080p.
@@CraftComputing That sounds like it would be OK over WiFi!
@@wayland7150 yeh, i've streamed VMs to crappy devices over wifi just fine. main problem with wifi is latency imo; you need decent 5ghz or better wifi.
I tried this with a Quadro K2000 I had lying around, didn't expect it to work with that card but strangely enough it did.
this is cool. i have one too. i'll give it a swing
I may be able to get a K2000 so this is interesting. It seems similar to the GRID K1 which is the oldest card supported by vGPU. I wonder why support for this generation isn't officially supported by the script. Is it just an oversight?
An idea for another video along these lines: The monster workstation: Simultaneously running windows, linux, macos and TrueNas Scale VM that all the other machines connect to develop code. And have all three build and run game code at the same time. You could do it two ways, one using 1 video card split like this and another each having a dedicated gpu.
jesus christ this tech is useful lol
Fairly easy to do with multiple GPUs(I have BigSur, Windows 11 and Ubuntu 20.04 on one workstation), but MacOS is very picky about video drivers. Newer versions are limited in their support of NVidia, it is much easier with AMD...
Morganaught shows this but she rushes through a bit fast so hard to follow. Easy enough to connect a screen keyboard and mouse to a VM in Proxmox, I got that far.
I like Morganaught, but she is a bit erratic and I think she wants to be purposely obscure sometimes to promote her pay for me to help you so this business. I can respect that, but I think Jeff/Craft Computing does a better more followable tutorial job.
@@colinbyrne7013 Huge compliment to Jeff. Margonot is always enthusiastic, but does not show anything useful in the end and calls people like "any idiot can do that"
1. How should I split a 11GB 1080ti? 2. Does this support non-windows clients? Like Androidx86 or ChromiumOS, or just straight up linux. 3. Can you configure mirroring on both display instead of disabling one?
"I think I can get a couple more." Me: YESSS!!!!!!!!! MORE!!! MORE!!!
Jeff I have a question. This method is for spliting a gpu for multiple VM's but If I want to get 1 physical gpu for every VM, I have to do the same thing like in this video or is easier to passthrough and no modifications needed?. I am a beginner in this world so I still have a ton of things to learn about VM's and stuff.
I second this
Passing through 1 GPU per VM is easier if you have the money. You have to do most of the stuff before the first reboot, but most of the stuff after the reboot is specific to this solution. There are guides on the Proxmox Wiki for GPU passthrough.
curious why the framerate limiter wasn't taking effect during your example running Heaven? was the profile not setup or applied correctly? I would have expected to see 60fps from heaven, and less than 100% GPU utilization.
Something I think needs to be noted EVERY time we talk about device passthrough...
Memory Ballooning is NOT SUPPORTED when passing through devices. It WILL cause craziness.
...
I learned this the hard way after several weeks of troubleshooting. Someone just happened to mention it like it was a 'well known' problem. Which it is clearly not.
It is listed somewhere in the fine print, but I haven't seen it mentioned pretty much anywhere else.
Thank you. I passed through an HBA to TrueNAS but fortunately was advised not to use ballooning with TrueNAS.
i may look at copying this system for a windows 11 build
@Craft Computing, 8 months ago you made a video on how to create 2 VM using Hyper-V including shared GPU. Now im confused, should I do it on Proxmox or on Hyper-V? I would like to access them remotely too. Thank you!
It's a "hash", not a "hashtag". Hashtags are tags prefixed by the hash symbol.
Now that there are custom profiles I really want to get back into this (merged drivers and Looking Glass)
But the newer Linux kernel is so goooood, with futex2 and lots of useful goodies for my hardware, and 5.17 just released with AMD-P state driver I need and 5.18 is going into feature freeze with better drivers for my mother. I don't think I can go back to 5.12 again to get the old Nvidia drivers :(
So, what do you do once your trial account with Nvidia ends? Keep making different accounts?
Friend, I have a question. I want to play around with an Nvidia P4 for graphics processing with OpenCV, without modifying the clock or using it as a video card. I won't be modifying the standard clock. Do I still need to add an additional cooler? Or can I use its passive dissipation without adding any cooler?
Still waiting on that video for steaming solutions!
Would you please repeat and test this on a fresh installed Proxmox 8 ?
Any luck?
I know this probably has been brought up but could the tesla m40's be water cooled? I know it would bring the price up a bit but it would be quieter.
hey Jeff, cool tutorial. i'm building one of those on dell t5610 with 2x p100 card. i have 126gb ram on it but i kind of curious can you use RAMDisk as a boot drive for virtual machine? if it's can i want to upgrade to 256gb RAM then partition it to RAMDisk to the VMs
I noticed you advised to split the CPU's resources evenly among the VMs, but I was thinking that for best results, Proxmox needs a core or two, doesn't it? Or would it just use whatever scraps of CPU were left over from the VMs?
from my experience it's not a problem, unless you plan to hit 100% from all the vms simultaneusly. it depends from the use case of course. I have a Zimaboard running much more than it's supposed to run with only 4 cores.
Thanks a lot for another great guide, worked like a charm with Parsec!!!...
But I am having a weird issue when I try to use Moonlight (I rather use that one to stream to the phone)... It errors out during RTSP handshake, giving me an error saying Ports need to be forwarded (I am in my LAN!!!).... This Happens only when using the VGPUS... removed the mdevs, passed the GPU directly to the same VMs, doing that, with the full GPU passed through, Moonlight works no prob... Added the mdevs again, trying with different newer and older drivers. With Parsec, not a single issue... Moonlight keeps giving me Port Forwarding errors when I am in the same network!
I could not make Moonlight work with Vgpus...Waiting for your video on the streaming clients... maybe you had the same issue, and hoping you give a solution. :)
Your videos taught me a lot, thanks a lot once again!
if you use a card with a hdmi out can you use that for the games X4 love your work
I am getting close to building a new rig, and I am wondering if I can set up my workstation with a vm for video editing I can run remotely. The second question is what would be the best way to process footage when I am editing out of state? I guess the best way is to set up a NAS that I need to upload to?
And I guess along side that I am wondering if perhaps it isn't just better to use something like parsec?
Tried using this awesome tutorial for Proxmox 7.2-5 and it doesn't work unfortunately. Trying to give my 1080ti a second life. Tonight i try a clean install of 7.1
I'm still not sure why patch argument for nvidia driver installer is not used to patch all the necessary lines with a patch file
please keep milking that cow. I am so SO keen to get this working so I can run steam games around my house with my girlfriend at the same time. basically the exact scenario you suggested.
Have you try to watercool the Teslas? Maybe Kraken G12 could work well to use an Asetek AIO
thanks for video, have a side question about passthrough gpu, mounted 3 gpu on my workstation, but can only use 2nd and 3rd, proxmox keep 1st in use and it gives error 43 when i try to use it.
when i remove 1st gpu the error appear in 2nd gpu ...
just little point, maybe i m wrong about vfio_iommu_type1, from version 6.4 or 6.5 it included by default in loaded models, no need to add it in modules file, except if there some other reasons.
when i check list of modules, i found it there
Have you thought about water cooling? Apparently an NZXT Kraken G12 bracket will adapt M40s to an AIO (according to one random forum poster)
Stay tuned.... 😉
@@CraftComputing Loved the video, the G12 bracket also works perfectly! One thing I noticed, it looks like the value you have listed for doing an override to create an 8GB vGPU seems to cause an assertion error that prevents the VM from starting. After changing the value from 7874106708 to 8053063680, it started up normally. I think it's because the first value is not divisible by 1024?
I picked up a Tesla P4. Which, if any, of the unlock elements can I skip when setting up?
whats the list of supported gpu`s? rtx 30 consumer series supported?
Hi!
I think many people dream of buying a "powerful server" instead of buying two or three computers. Less money spent, less noise, more convenient maintenance.
- Are there any examples of calculations what is more profitable to buy in 2024 ? 2 separate computers or 1 powerful one ? For example I want to play modern AAA games with my girlfriend in QHD resolution at 144 fps.
- Nvidia often releases driver updates. Sometimes they help to improve performance. How stable on long distance does the system behave with these c gpu hacks ?
- Have you tested different ways to connect peripherals remotely? Like DisplayPort/usb over lan ? Very curious how to connect real monitors to the server in such a configuration
- Is it possible to bypass multiple graphics cards? For example, if I want to connect more monitors (play on a powerful graphics card and watch content on another)
Can the vGPU unlock work on KVM natively, or just Proxmox? Cheers!
You could probably deshroud those GPUs and find a way to affix some 120mm fans.