วีดีโอ

1st 5:30 seconds of talking... I bet it was going to start any time but I just ran out of patients.. GR8T Job. You RoCk!

Thanks!

Hi, I am a new to all of this. I follow the video, but got an error when I run the docker common when all the docker images finished downloading. My Ubuntu system has one 512 nvme drive. Error response from daemon: error while creating mount source path '/var/localai/models': mkdir /var/localai: read-only file system

Absolutely excellent work! Just what I needed. Just in time. :)

Did bro just cover up the sound of his toilet flushing by going "wow did yall hear thise jets flying over the house?"😂😂

Can you have mix VRAM GPUs to run Llama? Ex 24gb and 16gb RTX 40 series?

Thanks for sharing all this invaluable information. Nicely explained from the start and understandably. 🎉

I was tinkering with llama2.cpp last year, and had a lot of fun with it in the home lab. Been away for a while, to say I'm more than impressed with both localai and this video series is an understatement. Excellent, excellent work. Whoever the hell you are, a big thank you!

Hey, I love the channel. I started making my own setup a few months ago and came across this when researching docker. New to programing environment and ubuntu. I come from the industrial automation side PLC's and networking. I am going to do a fresh install following your guide on a Node I am setting up for practice.

If i interpret your Test right, then you have enough memory with 3 cards ( 48gb ), but Not enough GPU Prozess Power.I have a 3090 and 2x 4060 ti 16gb. Will Test If i got more then 5t/s....

I have a r730xd with 1 m40 installed. I’m looking to install my second m40 but it looks like it physically doesn’t fit into the second pci slot. Do I have to cut something’s ? Hahah

Have you tested llama 70b on the cpu only?

Wow, I thougth that would be much much more traffic. So mi Thunderbolt 3 eGPU is perfectly fine as a complement to an internal card.

This video is really good. Very informative. I would like to know how different pcie size sockets affect tensor splitting, as the cards send their results to each other.

would be interesting to see some 13/20/70b models too

hi 2 rtx 4060 ti 16gb = to 32gb vram +-??

Thank you so much for sharing this; it’s been really useful. Could you please consider creating a written guide as well?

I am certainly learning stuff from your video's. Perfect pace. Thank you for your efforts.

This is amazing. Thanks for all the effort

Great video, the exact content that was necessary. I propose K8s setup for one of the next videos... I played locally with Ollama, vLLM and Triton and Triton with vLLM backend. But LocalAI was the next thing I wanted to check out...and just in time ;)

Hi, one question, if you mix cards, will they all run at the speed of the slower one? or each one would max? Or how much do the cards talk to each other, or they just split the work and go?

Excellent content as usual: Ash would be proud!

this is amazing! can't wait to see how the distributed inferencing works!

how are you using Mac OS with NVIDIA GPUs?

AI? Apple Intelligence? ;)

I am retired/disabled IT person myself. I have a unraid server I like a lot. I have been playing with AI stuff but find my Nvidia 3060 12gb vram has it's limits. I no longer have a budget to buy much anymore. My unraid server is so packed I cannot add anymore hardware to it and it has some key dockers running that I don't like playing on it to much. Plus my disability makes it hard to work on PCs in those big cases. So I have a old pc laying around so I ordered a test bench station, some PCIe extenders cables and some used Nvidia K40's. I plan to do what you are doing here so I will be following your channel. I'm just unsure if or how I can link all the NVidia K40's 24 gb Vram cards to get 48 gb vram. I plan to run Comfyui. maybe run dockers on it as well. Look forward to more of your videos.

Not a boring video at all! I plan to build my own AI server in the very near future! Thanks for the walk through.

The intent of using multiple GPUs is to run larger models. So this test is not very useful and the results might be not surprising. So what about 33b- or 70b-models with this setup?

I started building mine about 3 weeks ago. But just with an rx 7900 xtx 24gb. First time trying ubuntu with LLM. Very soon I was stuck with network card driver. Apparently some drivers are not available in Linux 😂😂 I am crying while looking for solutions.

When you run multi GPU I am assuming no SLI setup is required. Is that right?

What is LocalAI ? Is it like CGBT ?

An interesting part of this test that you didn't cover in your charts is the power draw (which was visible during the testing), beyond mentioning heating your house with M40s. The cost of power, plus the card should be considered for "bang for the buck". In this case, it appears that with the popularity gain pushing up prices for the Tesla cards and the a4500 ($1500 ish today on Ebay for used cards), the 4060 ti represents by far the best bang for the buck in a consumer machine of what you tested.

Thanks for doing the stable diffusion tests. I had written off cards this old because I thought they wouldn't do video. I'll have to re-think my server build now.

Have we discovered what AI can't do?

I'm sad that there are no $249 Jetson Orin Nano Super's available. I have a solution that needs 5 different small models running simultaneously and the 8GB cards would be have been great for that. I have two AI inference servers: One 7900 XTX and One 4060 Ti 16GB. Loved the hardware modification video, I definitely wouldn't call it carnage until you break out the giant tin snips and break a few PCBs and rewire a few traces :)

Have you tested P40s? Were you using llama.cpp directly? Can you dive into context length and context caching, tensor parallelism etc with llama.cpp on these older cards ❤

1st

Warning - if you are sensitive to hardware carnage don't watch first parts 🤣Cases are SO confining.... Happy holidays and what are you doing with your old junk hardware sitting around?

dunno, going from 10 tokens/sec to 30 tokens/sec, yes its 3x but still feels slow Excellent content, thank you

I am going back into your videos to see if you did one, but would LOVE to see what a p40 will do.

My Apple federal contact was telling me about TB -> fiber channel interfaces. Wondering about a cluster of Mac M4 minis with fiber channel.

I'm very curious how my brand new 64g M4 Pro mini will do compared to the M1 max. Do you know if any use of the inference cores is on the horizon or valuable? Maybe only for training? I'm pretty new to this. In fact so far I have only installed Ollama, which I assume is way slower, and not Metal?

Great video. Seems 4060Ti is still too slow for serious use.

Great video, thanks. I would suggest making the particular screens you're focusing on bigger and more readable.

is that a must to put same gpus there to run llm on multi gpu can i go with 3090 and 4090 together?

Matrix cores: Am I a joke to you?

So... What's your favorite pizza? And what's your AI's favorite pizza?

Happy Holidays Robots! 🤖🎄🎉

I'm an outdated high end 4GB gpu from 2014 (so no acceleration) and an outdated high end cpu from 2010 lmfao it takes several hours before the first letter of a response is typed back to me so I can only really do a single prompt a day hahaha. also the whole time it's thinking my system is sucking down 270W and only idles at 145W.... I guess if my gpu helped it would add another 250W or so...

great benchmark. 4060ti seems to be a good choice for running self-fine-tuned 8b models locally, decent speed and low cost.