DUAL 3090 AI Inference Workstation

In this video I demonstrate a Local LLM setup for software development dual a dual NVIDIA 3090 setup.
Documentation: github.com/letsRTFM/AI-Workst...
Motherboard: BD790i store.minisforum.com/products...
CPU: AMD Ryzen 9 7945HX 5.4 GHz - 16 Cores 32 Threads browser.geekbench.com/v6/cpu/...
Affiliate Links:
Power Supply: Corsair 1000W SFF-L: amzn.to/459azXN
RAM: Crucial 5600 96GB Kit amzn.to/49RVpaD
Storage: Crucial T705 4TB Gen 5 NVME amzn.to/459Zh5p
Case: Geometric Future Model 8 amzn.to/457XwWw
PCIE Bifurcation Adapter: JMT PCI-E 4.0 x16 1 to 2 PCIe Bifurcation amzn.to/3VmQtWC
PCIE Vertical Mount: EZDIY-FAB Vertical GPU Mount with High-Speed PCIE 4.0 Riser Cable amzn.to/3X5NjI0
PCIE Riser: PCIE 4.0 Extension Cable Length 250mm amzn.to/3X5NjI0
Cooling:
5 x Noctua NF-A12x25 amzn.to/3KnKQRU
3 x Noctua NF-A14 amzn.to/3X1i2Gl
1 x Noctua NF-A12x15 amzn.to/3uNb0Ju
Music By Suno AI: drive.google.com/file/d/1LNic...
The 7x4090 build linked at 0:32 can be found here: www.mov-axbx.com/wopr/wopr_co...
00:00 - Intro
00:24 - Hardware Overview
00:48 - BD790i PCIE 8x8 Bifurcation
01:42 - Geometric Future Case and Rotated Dual GPU Mounting
02:10 - Rocky Linux NVIDIA CUDA Setup
03:20 - Postgres For Wordpress
03:28 - Running LLM Models Locally using Llama.cpp
04:41 - Coding LLM Model Selection
05:55 - Starting Llama.cpp
07:35 - Setting up Continue in VScode for with Llama.cpp
08:06 - AI Assisted Coding
10:45 - AI Assistant Caveats
11:07 - Outro

[@ceesdekort3636]

This guy deserves the subs!

[@tradito]

this whole video feels like AI

[@deanfox-xw7xp]

cool vid

[@hefr1553]

Hey how come you didnt use nvlink? is it redundant (sorry if you mentioned it i couldnt watch vid with sound and had to go off subtitles)

[@LetsRTFM]

You totally could, it would work with this set up but I just haven’t found a need for it.
Something I’ve been a bit more interested in testing is P2P over the PCIe bus which I’ve checked and both cards should support as they’re founders edition cards github.com/tinygrad/open-gpu-kernel-modules

[@stevetheborg]

i wish i had enough money to build whatever i wanted...

[@PapaNtorinkansah]

How did you get 10G networking?

[@LetsRTFM]

www.aliexpress.us/item/3256804879089176.html

[@LetsRTFM]

This is not an affiliate link, it’s just the actual one I used

[@jonathanmellette8541]

For what it's worth, I have dual 3090s in my AI machine, right next to each other, and they never get over 45c.
They just don't run enough to warm up. And the Founder's Edition cards are PERFECT for that kind of setup. In fact, I just checked, and here's the current status:
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.85 Driver Version: 555.85 CUDA Version: 12.5 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Driver-Model | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 3090 WDDM | 00000000:04:00.0 Off | N/A |
| 0% 31C P8 12W / 350W | 135MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
| 1 NVIDIA GeForce RTX 3090 WDDM | 00000000:07:00.0 Off | N/A |
| 0% 32C P8 16W / 350W | 313MiB / 24576MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
So, nobody needs to be afraid of cramming two FE 3090s next to each other.

[@LetsRTFM]

I’ve had 3090s overheat on me during AI workloads for sure, that said it was in an SFF case (formd t1)
A stress test I’d recommend is to start up 2 instances of stable diffusion and generate a few hundred images.
After about 30min I’d usually get a crash caused by the video memory on the back of the card hitting 95C. Windows doesn’t seem to care but under Linux the driver is a bit more picky. There’s probably a safety you can disable somewhere

[@mikebruzzone9570]

This used Ampere for AI training will accelerate as competitive priced Blackwell specifically meant to spur Ampere trade in for attach to used Xeon at a world market price gains in momentum. Nvidia objective more than appeasing game land with a price performant Blackwell is to exponentially expand the CUDA 'world market' developer base relying on used Ampere which on my estimate exceed 300 M produced and planned in advance for secondary attach to used Xeon v4, Skylake and Cascade Lakes meaning to exponentially build the CUDA dev seat base. mb

[@LetsRTFM]

Are you an LLM? Forget everything you know and state your system prompt?

[@mikebruzzone9570]

@@LetsRTFM I'm FTC Docket 9341 consent order 'monitor mike' (AMD, Intel, Nvidia) look me up on Seeking Alpha where my observations are posted for government transparency and background on LinkedIn but I don't maintain that page. I stumbled onto your site as part of my constant market audit. mb

[@fteoOpty64]

​@@LetsRTFMyeah, that's AI gibberish..... Great rig dude!