Fun fact, Vulkan is *not* a Graphics API. It's a GPU programming API. "Presenting Graphics on a Screen" is a Vulkan extension the programmer will need to enable manually (VK_KHR_surface, if my memory does not let me down). Vulkan ultimately is meant to replace OpenGL *and* OpenCL
@@blisphul8084 I think the reason is flexibility. Vulkan supports a ton of different GPUs, so it is really hard to optimize Vulcan for any hardware. CUDA on the other hand runs on comparatively few GPUs, so Nvidia can pour tons of resources into optimizing it to be as fast as possible on their own hardware alone. It's similar to why Apple's ARM chips are so good, they are vertically integrated and by controlling the entire ecosystem they get a lot of opportunities that a more general vendor just does not have.
@@blisphul8084 as is OpenCL... CUDA is a proprietary Nvidia solution that is 100% vertically integrated. When you run CUDA, you run a nvidia programming language, processed by an Nvidia compiler (cudac) running on Nvidia hardware. I guess they are able to optimize things further. I have no idea, I am not a specialist in GPGPU programming…
I love videos on how to set up local AI, the companies always charge exorbitant amount for what you can do for free on a gaming GPU without being spied on
LOL.. Theres something of a huge difference between the LLMs that we mortals can run and the AIs such as Claude, chatgpt and the like... I could buy 4 4090s and have a world of fun building my local AI server.. but then what? Spend $10000 only for that hardware to be obsolete in 1-2 years. Not to mention the electricity cost of me running this 24/7 and the degradation of the hardware... Paying $20 a month is not that bad and not that i dont love the idea of hosting our own LLM, its just not even comparable.
It's also a good exercise for demystifying AI and realizing just how overblown the hype is. Especially with image generation-I'm no graphic designer, but I have consistently found it easier to composite an image in GIMP than get Stable Diffusion to follow a basic proompt.
I strongly disagree. We can sort of run a trained model to get an output. We cannot even come close to the computing power required to train these algorithms into a usable state.
The problem I see with this is model storage. I currently have a 1TB NVME drive connected to my Pi-5, which is pretty much a necessity for storing and playing with these models. This of course uses the only PCIe port on the Pi, so no way to connect a graphics card. I know, there are some adapters, but if you look into them you'll soon discover they have issues. If anyone has tested and tried one that'll allow an NVME disk and one of these graphics cards I'm all ears. Any thanks Jeff for finally testing and demonstrating this and all the work you've put into helping make the Pi better for everyone!
@@JeffGeerlingHere's hoping that the Raspberry Pi 6 will have more and faster pcie slots and lanes and maybe even an ITX version with a real PCIE slot.
If you keep the model loaded all the time, it's not an issue. The lag would get annoying if you swapped between different models like for a home assistant vs a coding assistant.
AI on a PI, rolls off the tongue. We oughtta get Weird Al to write a spoof song to lift the Pi from the obscurity of the general public not knowing ANYTHING about Raspberry Pi!
For me its Transcribing, Text Extraction/Reformatting (PDF's other weird formats to Json), and other general OCR are three area's I'm very happy to have local AI applications, llama3.2-vision:11b on the latest Ollama and hopefully eventually Pixtral:12b with "relatively open" licenses may have quite a lot of interesting small business use cases all around. Having a low power setup (even if its janky) like this is a rack versus an Ubuntu Server that guzzles 2-5x the power on idle seems like a big win for the home-labbers & early adopters.
I found that used thin clients offer way better performance per watt than any pi setup. Currently my server on a 8500T HP thin client runs at 3.4watts idle. Yes it pulls 58w on full tilt, but it outperforms the rpi by leaps and bounds that it'll take less power for the same task at a way shorter time. The rpi 4 on the other hand runs at 4.4w idle. So I still am not convinced that the pi is in any way a good option for a server application if you can get a thin client with more RAM, better performance and lower effective power consumption for cheaper than a 8gb rpi 5.
@@chocolateztif you can get a cheap used thin client. The Pi is more easily available. I'd get the Pi and make a custom NUC casing for it and the piggybacked GPU.
@fallinginthed33p they are abundant on eBay, at least here in Europe. Yes you have to check the options and know what is good and what is bad, check power efficiency etc, but if you wanna build a server I think that is something everyone can do. Also, a rpi won't ever get close the performance from a GPU you could get on a even just decent pcie 4x4 interface you get on thin clients compared to the pcie 2x1 you get on the pi, and then you lose all other fast up options. I think it's a cool project for tinkering, but if you could get better performance and energy efficiency for the same price, on a better supported x86/64 platform, why go with the overpriced pi? Maybe in the future if they decide to use a better soc, give us some actual PCI lanes and don't price it unreasonably high just because they can, then I'll might consider it an option. But if such a future comes it's more likely that the used thin clients at that point will still be better value...
Seriously. There is no "intelligence," in terms of reasoning. That's literally the point of the Transformer architecture-no recurrent layers, "Attention is All You Need."
Well that's only cause then you understand that true generative AI will never be accomplished on this platform but LLM will most likely be a small implementation of what that kind of model would look like. Or whatever.
@@Yuriel1981 generative AI I think is possible but it’s a long shot to have it before 2030. All the development gains in hardware and software simply are better methods of brute forcing data.
Can't wait to see how you managed the local voice control. When Home Assistant did there voice project it was exciting but the results where at best Painful to get to work. Hopefully newer versions have fixed a lot of that.
You don't have to worry about power usage if you have solar, wind, hydrothermal and/or hydroelectric power. I would suggest nuclear, but as you're a civilian that's not generally feasible.
@@JeffGeerlingI'd settle for an RTG. I have this fantasy that the DOE solves the Yucca Mountain bottleneck by paying citizens to bury spent fuel under their properties for a monthly stipend. I'd sign up in a heartbeat, especially if I could siphon some of the heat!
@@GSBarlevGood luck farming on it or occasionally suffering from radiation every once in a while. This needs to be done on a VERY small scale first, and IF it works, it should be implemented in a VERY safe manner.
@@fujinshu Oh naturally-it's pure fantasy. But what I will say is that you'd be surprised how much properly shielded material you'd need to bury-and how close it would need to be to the surface-in order for the dose to even come close to the background levels of radon already found in most basements.
AMD drivers for linux is the best decision ever. Nvidia drivers is windows exclusive. I won't be surprised if Nvidia buy out Microsoft for open AI or their cloud ideas.
Nvidia has their 'open drivers' now, which are a lot better than the status quo a few years ago, but they're miles behind AMD on the Linux front (IMO), but only from a community perspective. Unfortunately with how prevalent CUDA support is, they still lead based on hardware + dev tools.
Thanks for the shout out! Great idea with llama.cpp. Up until now I’ve been doing CPU inference on an old Mac Pro 5,1 (+ lots of cheap RAM 😂). Will compare it to the Pi + 6600 XT tonight!
oh man… this makes me really really really wish there was a PCI dock that was built on a CM# carrier board. It would be soooooo slick to pop a CM4+RX580 or a CM5+6700 together depending on your use case. Thanks for blazing this trail!
Neat! There's a nice Windows client on the MS store for ollama so I grabbed that to play around with local models on my laptop. Thanks for opening my eyes to that option.
I have to admit, I originally thought what you were doing (supporting graphics cards on a PI ) was just for bragging rights, but I have changed my mind. Combining something like this with a project like EXO would allow clusters of old (e-waste) GPUs to be turned into quite potent LLMs. I watch with interest!
Thanks Jeff. Too much we give up privacy for convenience, but the cloud is just some else's computer. A little effort and video watching can get the average computer guy up and running with alot of creature comforts without the loss of data control
When I watch this videos my first question which comes in my mind is: WHY ? I know it's useless to do things like this. BUT: It it's still very amusing to watch experiments like this. 😆
Question: Jeff, which donation platform gives the largest percentage directly to you? We support you and all benefit from your open source documentation. Happy to make another donation. - As someone who has set up a quad AMD 7900XT rig for LLM use, I don't envy the amount of debugging you did!
GitHub Sponsors is the best, with Patreon just slightly behind. TH-cam Memberships takes the biggest cut away, but it's more convenient for some people!
I just got inspired by this and installed my first local language model using an old GTX 1080 with a Ryzen 3080. With llama 3.2 I'm getting about 70 tokens per second with this venerable old GPU!
The main reason I subscribed this channel is to run llms on pi I am big fan of these sbc hope the next will come up on nvidia graphics card working with cuda
Yeah, I would be extremely interested in an AI home assistant that was mine, not belonging to large corporations! My home's own LLM that I could train, improve, and know it was looking out for my needs and not it's company's? Hell yeah!
I have a similar setup, but used a Latte Panda instead of a Pi, and was able to get a used Nvidia Tesla P41 running on it. I was able to pick the P41 up for ~$100 off of eBay. Going for the older server compute cards as a basis for a local LLM server is a much better IMO for a cost to performance ratio. They extra VRAM in the P41 compute card is hard to get close to in any consumer GPU.
Hi, Jeff. Nice video. But in my opinion, it's a bit incorrect comparison of "RPi vs RPi + GPU". You can take RockChip NPU or Jetson Orin NX for the models you are using. It will be cheaper, and it will solve the home assistant case (maybe not 40 t/s speed, but it is much better than RPi). I hope that Hailo-10 will also be released soon. A few less popular platforms are working with LLMs as well.
@@raulbustamante8337 i was a bit like that with the first risc v sbc i saw on a video all enthusiasm went away quickly days ago explaining compuers showed a recent new risc v chip implemented on a motherboard, it is better, yes, alot better than the first one i saw, but, still struggles loading a linux gui, something that even the oldest weakest single core celeron can do in a near future i expect it to handle it better, but, it is another racer in a race that began so long ago, and it will never win it interest, performance and support, i dont see it happening, that is how i see it
I'd love to see if you can put some setups like this into a cluster. Some other YTers have shown multi gpu clusters where cheaper gpus can share the model across all their VRAM and get good performance. But if you can cluster together enough cheaper GPUs to get 24GB VRAM on a few Pi's, that would be a great alternative.
I know the wizard trick here is using the Raspberry in unorthodox ways, but it would be so nice to see how it compares when pairing the graphics cards with an N100 board. That's in the same ballpark in terms of cost and power usage as the Pi5, but it would enable much more, since it's just kitchen standard x86.
LLMs are a small subset of AI tech. So are search algorithms like what power google maps, and so are the algorithms that curate your social media feeds. So LLMs are AI, but AI is much more than just LLMs.
This is an interesting topic-the ones available right now have the Hailo-8 and 8L, which aren't suitable for LLMs (unless extremely tiny), because they don't have much RAM. The Hailo-10 supposedly does, but I haven't seen one in the wild yet.
Thanks for mentioning the resources needed to run "AI". And thanks for using the right terms. LLMs are not AI. It makes me so angry these days because everything has AI these days. Another high quality video.
Given the cost of the setup, I'm curious how the base m4 mac mini might stack up against it. The storage on the base model is the obvious drawback. After seeing your storage hooked up through USB, perhaps a PCIE gen 3 or 4 m.2 through thunderbolt would be more than sufficient.
@@JeffGeerlingCan you see if you can get multiple Mac Minis in a cluster to run larger models too? The upgrade cost to add more ram is the same price as the whole system itself, so an m4 cluster is very appealing here.
I also wonder if you can scale to larger models with the number of mac minis too? So if you have 4 minis or 8 minis, can you run a 64 or 128GB model on the cluster with really good performance? If so, please test Mistral Large 123b on it.
I literally want a Raspberry Pi just to turn it into the ultimately small form factor PC. Probably not ready for my lifestyle but man it's enticing, even just for fun.
Caring about a couple of watts probably isn't worth the effort of using a different cpu. Running a gpu on the pi is a cool concept regardless. Im not sure about the 80+ specification on a regular pc psu but wouldn't it make more sense to measure at the dc bus instead of at the wall? Depending if you can get a quality psu you might be able to power the riser and gpu with some homemade adapters off a 12v only power supply, idk a meanwell for security cameras. That would cut out the other atx bits and voltages from a regular psu and possibly increase efficiency. They are trying to go that way with atx12vo for efficiency in pc's, but then your drives need power off the motherboard or another step down thing for 5v and 3.3. Or even go the other way and run the pi off the psu's 5v rail to eliminate the need for another charger. No idea if those chargers are efficient but they sure aren't free if you're considering them in the setup price and pc psus should have plenty of amps left on the 5v rail to run multiple PI's
So far it hasn't been an issue-I have both the GPU PSU and the Pi PSU plugged into the same power strip, and I also don't hold both at the same time that much, so can't say if it's going to be a perfect solution or just works for me. Ideally there'd be a riser that powers the Pi and the card at once.
@@GSBarlev I'm building my own cinema camera, will probaby resort to a off-the-shelf power bank yeah. It won't need to be powered by a typical power adapter anyway
For a few LLM workloads this might be ok, but a 6700 XT is still probably overkill. I'm using a much older GTX 1060, a GPU that's half the speed of the 6700 XT with a E5-1620, a CPU that's more than twice as fast as the Pi 5, and for a lot of workloads the CPU is the bottleneck. There are only a very few cases where it makes sense to pair a slow cpu with a high powered gpu... and even then a 6700 XT is still overkill and going to be leaving most of it's performance on the table. It's cool to see... but highly impractical.
the 4 kilobyte kernel page size issue were the same thing the asahi linux devs ran into with the m-series chips trying to run x86 games, i wonder how well their work on it with the micro-vms might translate over to the raspberry pi stuff
Interesting. There's a lot of information available around Cuda/Tensor performance for "AI" (for me that goal is LLM for Home Assistant and inference for Frigate) but I'm unable to understand how AMD fits in this space. Notionally, there's better value to be had from AMD GPUs but how does that stack up in the tokens/sec comparison against NVidia? I'm starting out with an old intel platform so not limited by the ARM compatibility/drivers or concerned about power (it'll be an OMV server too).
Interesting video, thank you for sharing this! Out of curiosity: how long does it take to compile llama.cpp? You are compiling using the Raspberry Pi’s SOC, right?
Jeff, do you have an affiliate link to the smart power outlet things that you use and connect to your home assistant? Gonna add to my holiday gift wishlist..
I love this idea, but how practical is it from daily use perspective? Assuming I want to run a home lab with NAS, collab apps, home assist, video streaming, this personal assistant. I would think that a setup like this would require multiple rPIs over running on one desktop PC? Would we see real power savings in this case?
How about dual GPU's? And I'd automate it to make it more power efficiënt. With two smart sockets and automate it with entering zones in home assistant. Like if you're entering the zone home, then switch on power plug named GPU, wait 10 seconds and then turn on the power plug of the pi. This makes it even more affordable, since it doesn't have to run 24/7 since you're not at home. With my calculations you'll get two hours on full power when in use. You can even turn off the plugs at midnight if you want, by making an automation.
Three pies for ASR under the cloud, Seven pies for TTS in their halls of sound, Nine pies for LLMs doomed to be computed, One pie for the Master on its dns throne, In the land of Nvidia where the GPUs lie. One pie to map them all, one pie to find them, One pie to reduce them all, and in the code bind them, In the land of closets where the old GPUs lie
There is an open PR to add Vulkan support to Ollama which would support Intel and AMD GPUs better and make it easier to integrate in home assistant and vscode, but ollama seems uninterested for some reason.
thanks for another great video! Maybe you've been asked this before, but where did you get the stickerboard that's in the background of your videos? i can't find one in that size!!
you got your units wrong 0.15 t/s per w, 0.29t/s per w or you could say 0.15 tokens per joule, and 0.29 tokens per joule or, 540,000 tokens per kWh vs 1044,000 tokens per kWh
Love the setup with PI 5. But I think a used Nuc 9 extreme + 3060 12 Gb VRam (in total around $600) could be more cost effective. I would trade the different in price for the idle power consumption to the difficulty in the setup process.
Or just buy an older second hand PC and a second hand rtx 3090 with 24gb of memory 3090s are now really cheap because for gaming they are slower the a 4070 super. But do still have twice the vram
@@Merlinvn82 got mine for €600. Also bought a CPU, RAM and motherboard for €120. Did need to buy a new PSU though. So yeah it will be more expensive, but will have 2x the VRAM and more CUDA cores
I had good luck and performance with an Intel N100 CPU running Ollama and llama3.2:3b. It would put out good T/s for shorter answers while boosting, and you were not hitting it constantly with question after question. Let it cool down a bit (~30 secs) and ask another question and it would do well. It isn't as good as my 3060 12GB, but usable.
It's really hard to find a good chart showing global pricing that isn't behind a paywall. I think the relative comparison (mostly that European countries are all more expensive than US) is the thing I was trying to get across.
thanks for the amazing videos, i'd like to sugest that you try the "useless grapgics cards for gaming" such as Nvidia GT430, GT1030, to see if it has a usable perpose it would be great to see the tests results Thanks again for the Lab just amazing
I'm making a wearable computer based on the OpenEyeTap instructable. Just trying to figure out a CM4 version now! Need me some of that pcbway money! :D
Always has been. AI is a research field and researchers usually don't see a point in moving away from Nvidia if their cards work fine (and even the same cards can be used in production). Many projects also use custom CUDA kernels code, which complicates things more
I wonder if Stable Diffusion/Onnxstream will recognize the GPU card through Vulkan? Local AI Images, Videos and LLM's. I priced a PC box needed to do this and it is outside my budget. Plus I love pushing my Pi's to doing useful things.
I'd love a Pi5 in Micro-ATX form factor with one or more PCIE sltos so you would not need all those cables and wires. Also, it would look smug if put inside a nice case together with the GPU. Is there a Pi5-Compute-Module to Micro-ATX solution out yet?
Fun fact, Vulkan is *not* a Graphics API. It's a GPU programming API. "Presenting Graphics on a Screen" is a Vulkan extension the programmer will need to enable manually (VK_KHR_surface, if my memory does not let me down).
Vulkan ultimately is meant to replace OpenGL *and* OpenCL
So why is it slower than cuda?
@@blisphul8084 I think the reason is flexibility. Vulkan supports a ton of different GPUs, so it is really hard to optimize Vulcan for any hardware. CUDA on the other hand runs on comparatively few GPUs, so Nvidia can pour tons of resources into optimizing it to be as fast as possible on their own hardware alone. It's similar to why Apple's ARM chips are so good, they are vertically integrated and by controlling the entire ecosystem they get a lot of opportunities that a more general vendor just does not have.
😮
@@blisphul8084 as is OpenCL... CUDA is a proprietary Nvidia solution that is 100% vertically integrated. When you run CUDA, you run a nvidia programming language, processed by an Nvidia compiler (cudac) running on Nvidia hardware.
I guess they are able to optimize things further. I have no idea, I am not a specialist in GPGPU programming…
@@lbgstzockt8493 So..cute! Closing the environment using elemental math on a closed system! More useful than competition.
I love videos on how to set up local AI, the companies always charge exorbitant amount for what you can do for free on a gaming GPU without being spied on
LOL.. Theres something of a huge difference between the LLMs that we mortals can run and the AIs such as Claude, chatgpt and the like...
I could buy 4 4090s and have a world of fun building my local AI server.. but then what? Spend $10000 only for that hardware to be obsolete in 1-2 years.
Not to mention the electricity cost of me running this 24/7 and the degradation of the hardware...
Paying $20 a month is not that bad and not that i dont love the idea of hosting our own LLM, its just not even comparable.
@@RocketLR newer Llama is surprisingly close
If you use their api instead of the public consumer product it’s much cheaper and they can’t train on your API calls
It's also a good exercise for demystifying AI and realizing just how overblown the hype is. Especially with image generation-I'm no graphic designer, but I have consistently found it easier to composite an image in GIMP than get Stable Diffusion to follow a basic proompt.
I strongly disagree. We can sort of run a trained model to get an output. We cannot even come close to the computing power required to train these algorithms into a usable state.
Alexa, the gift that keeps on taking. You are absolutely right that subscriptions are not needed to do this
The problem I see with this is model storage. I currently have a 1TB NVME drive connected to my Pi-5, which is pretty much a necessity for storing and playing with these models. This of course uses the only PCIe port on the Pi, so no way to connect a graphics card. I know, there are some adapters, but if you look into them you'll soon discover they have issues. If anyone has tested and tried one that'll allow an NVME disk and one of these graphics cards I'm all ears. Any thanks Jeff for finally testing and demonstrating this and all the work you've put into helping make the Pi better for everyone!
For my testing, I'm using a 1TB USB SSD. A little slower than NVMe, but only slightly-plenty fast for loading models in and out of RAM though!
@@JeffGeerlingHere's hoping that the Raspberry Pi 6 will have more and faster pcie slots and lanes and maybe even an ITX version with a real PCIE slot.
If you keep the model loaded all the time, it's not an issue. The lag would get annoying if you swapped between different models like for a home assistant vs a coding assistant.
AI. the 'cloud' buzzword of 2024.
Need a new version of the Cloud-to-Butt plugin!
AI on a PI, rolls off the tongue. We oughtta get Weird Al to write a spoof song to lift the Pi from the obscurity of the general public not knowing ANYTHING about Raspberry Pi!
Heh, sans-serif does NOT like making it easy to distinguish between an I and an l!
Lol we need an update, it's not All About The Pentiums anymore
Weird Al is heroic
@@JeffGeerling blame Bob Marley, as he famously _shot the serif_ ... 🥁
For me its Transcribing, Text Extraction/Reformatting (PDF's other weird formats to Json), and other general OCR are three area's I'm very happy to have local AI applications, llama3.2-vision:11b on the latest Ollama and hopefully eventually Pixtral:12b with "relatively open" licenses may have quite a lot of interesting small business use cases all around.
Having a low power setup (even if its janky) like this is a rack versus an Ubuntu Server that guzzles 2-5x the power on idle seems like a big win for the home-labbers & early adopters.
I found that used thin clients offer way better performance per watt than any pi setup. Currently my server on a 8500T HP thin client runs at 3.4watts idle. Yes it pulls 58w on full tilt, but it outperforms the rpi by leaps and bounds that it'll take less power for the same task at a way shorter time.
The rpi 4 on the other hand runs at 4.4w idle.
So I still am not convinced that the pi is in any way a good option for a server application if you can get a thin client with more RAM, better performance and lower effective power consumption for cheaper than a 8gb rpi 5.
@@chocolateztif you can get a cheap used thin client. The Pi is more easily available. I'd get the Pi and make a custom NUC casing for it and the piggybacked GPU.
@fallinginthed33p they are abundant on eBay, at least here in Europe. Yes you have to check the options and know what is good and what is bad, check power efficiency etc, but if you wanna build a server I think that is something everyone can do. Also, a rpi won't ever get close the performance from a GPU you could get on a even just decent pcie 4x4 interface you get on thin clients compared to the pcie 2x1 you get on the pi, and then you lose all other fast up options.
I think it's a cool project for tinkering, but if you could get better performance and energy efficiency for the same price, on a better supported x86/64 platform, why go with the overpriced pi?
Maybe in the future if they decide to use a better soc, give us some actual PCI lanes and don't price it unreasonably high just because they can, then I'll might consider it an option. But if such a future comes it's more likely that the used thin clients at that point will still be better value...
Finally, someone else that isnt brain damaged with the "AI" hype. If you understand how an LLM works, you will never see it as "AI" again.
Seriously. There is no "intelligence," in terms of reasoning. That's literally the point of the Transformer architecture-no recurrent layers, "Attention is All You Need."
a better name for the hype is artificial stupidity.
Well that's only cause then you understand that true generative AI will never be accomplished on this platform but LLM will most likely be a small implementation of what that kind of model would look like. Or whatever.
@@Yuriel1981 generative AI I think is possible but it’s a long shot to have it before 2030.
All the development gains in hardware and software simply are better methods of brute forcing data.
Can't wait to see how you managed the local voice control. When Home Assistant did there voice project it was exciting but the results where at best Painful to get to work. Hopefully newer versions have fixed a lot of that.
2:19 "This is my rifle. There are many like it, but this one is mine."
Would you be honoured or offended if we call our voice assistants "Jeff"?:D
have the AI say “as always, i’m jeff” 🤣
You don't have to worry about power usage if you have solar, wind, hydrothermal and/or hydroelectric power. I would suggest nuclear, but as you're a civilian that's not generally feasible.
If I have any shot at nuclear power for personal use, I'm taking it! :D
@@JeffGeerlingI'd settle for an RTG. I have this fantasy that the DOE solves the Yucca Mountain bottleneck by paying citizens to bury spent fuel under their properties for a monthly stipend.
I'd sign up in a heartbeat, especially if I could siphon some of the heat!
@@GSBarlevGood luck farming on it or occasionally suffering from radiation every once in a while.
This needs to be done on a VERY small scale first, and IF it works, it should be implemented in a VERY safe manner.
@@fujinshu Oh naturally-it's pure fantasy. But what I will say is that you'd be surprised how much properly shielded material you'd need to bury-and how close it would need to be to the surface-in order for the dose to even come close to the background levels of radon already found in most basements.
AMD drivers for linux is the best decision ever. Nvidia drivers is windows exclusive. I won't be surprised if Nvidia buy out Microsoft for open AI or their cloud ideas.
Nvidia has their 'open drivers' now, which are a lot better than the status quo a few years ago, but they're miles behind AMD on the Linux front (IMO), but only from a community perspective. Unfortunately with how prevalent CUDA support is, they still lead based on hardware + dev tools.
YES! I was waiting for this video!
Thank you again *THIS* was what I was looking for to complete a specialist project idea I have been working on for years now
I just love your videos! Can't wait to see a local voice interface for Homeassistant !
That's very cool. Having your very own ChatGPT at home.
One that doesn't suck up all your private data, too!
Thanks for the shout out!
Great idea with llama.cpp. Up until now I’ve been doing CPU inference on an old Mac Pro 5,1 (+ lots of cheap RAM 😂).
Will compare it to the Pi + 6600 XT tonight!
Super sweet Jeff! Wow, its been a journey but here we are, thanks for all the hard work!
What a fantastic initiative. I’m going to try this in the holidays
Keep it up Norm, this is GOLDEN!
2:20 missed opportunity! there are other boards like it, but this one is mine!
oh man… this makes me really really really wish there was a PCI dock that was built on a CM# carrier board. It would be soooooo slick to pop a CM4+RX580 or a CM5+6700 together depending on your use case. Thanks for blazing this trail!
Neat! There's a nice Windows client on the MS store for ollama so I grabbed that to play around with local models on my laptop. Thanks for opening my eyes to that option.
These are super fun videos. Not practical for most of us, but cool to see.
WOW, Jeff - keep up the great work, just got a RP4, hope this works.
Unfortunately Pi 4 is limited to slower and smaller Ollama use.
I have to admit, I originally thought what you were doing (supporting graphics cards on a PI ) was just for bragging rights, but I have changed my mind. Combining something like this with a project like EXO would allow clusters of old (e-waste) GPUs to be turned into quite potent LLMs. I watch with interest!
Thanks. Didn't knew about running on vulkan option, now my 5700 XT works for some models
Smooth move on the BlueSky reference by asking about... the sky.
Thanks Jeff. Too much we give up privacy for convenience, but the cloud is just some else's computer. A little effort and video watching can get the average computer guy up and running with alot of creature comforts without the loss of data control
Of all the youtubers I watch, I am confident that Jeff is the one who spends the most time watching youtube videos
When I watch this videos my first question which comes in my mind is: WHY ? I know it's useless to do things like this. BUT: It it's still very amusing to watch experiments like this. 😆
Question: Jeff, which donation platform gives the largest percentage directly to you?
We support you and all benefit from your open source documentation. Happy to make another donation.
- As someone who has set up a quad AMD 7900XT rig for LLM use, I don't envy the amount of debugging you did!
GitHub Sponsors is the best, with Patreon just slightly behind. TH-cam Memberships takes the biggest cut away, but it's more convenient for some people!
I just got inspired by this and installed my first local language model using an old GTX 1080 with a Ryzen 3080. With llama 3.2 I'm getting about 70 tokens per second with this venerable old GPU!
The main reason I subscribed this channel is to run llms on pi I am big fan of these sbc hope the next will come up on nvidia graphics card working with cuda
Oh man, you make Raspberry Pi so interesting.
Yeah, I would be extremely interested in an AI home assistant that was mine, not belonging to large corporations! My home's own LLM that I could train, improve, and know it was looking out for my needs and not it's company's? Hell yeah!
I have a similar setup, but used a Latte Panda instead of a Pi, and was able to get a used Nvidia Tesla P41 running on it. I was able to pick the P41 up for ~$100 off of eBay. Going for the older server compute cards as a basis for a local LLM server is a much better IMO for a cost to performance ratio. They extra VRAM in the P41 compute card is hard to get close to in any consumer GPU.
You finally did it 🎉🎉🎉
Just amazing Jeff. And I think this will encurise manufactorers to create somethink like this to compute with Microsoft and Sony.
Waiting for your local voice assistants video. Great stuff!
Hi, Jeff. Nice video.
But in my opinion, it's a bit incorrect comparison of "RPi vs RPi + GPU". You can take RockChip NPU or Jetson Orin NX for the models you are using. It will be cheaper, and it will solve the home assistant case (maybe not 40 t/s speed, but it is much better than RPi).
I hope that Hailo-10 will also be released soon. A few less popular platforms are working with LLMs as well.
Hailo-10 will be an interesting bit of hardware to test. Hope they are more available at some point.
i was thinking the exact same thing when you did the other video with the gpu for gaming
Very interesting contraption. I'd like to see content about risc-v sbc's
Explaining Computers just did video on Mini AX Risc-v board, it has PCIe slot.
so you can feel sad because it is slow and no one cares because it is used on tvs only?
@@FlintStone-c3s I did watch it!
@@arch1107 just let people be enthusiastic about stuff
@@raulbustamante8337 i was a bit like that with the first risc v sbc i saw on a video
all enthusiasm went away quickly
days ago explaining compuers showed a recent new risc v chip implemented on a motherboard, it is better, yes, alot better than the first one i saw, but, still struggles loading a linux gui, something that even the oldest weakest single core celeron can do
in a near future i expect it to handle it better, but, it is another racer in a race that began so long ago, and it will never win it
interest, performance and support, i dont see it happening, that is how i see it
I'd love to see if you can put some setups like this into a cluster. Some other YTers have shown multi gpu clusters where cheaper gpus can share the model across all their VRAM and get good performance. But if you can cluster together enough cheaper GPUs to get 24GB VRAM on a few Pi's, that would be a great alternative.
We're finally here! GPU's on pi's
I know the wizard trick here is using the Raspberry in unorthodox ways, but it would be so nice to see how it compares when pairing the graphics cards with an N100 board. That's in the same ballpark in terms of cost and power usage as the Pi5, but it would enable much more, since it's just kitchen standard x86.
Finally someone calling the right turn, LLM is not an AI, all of marketing junks need to be cleared out of the tech space
LLMs are a small subset of AI tech. So are search algorithms like what power google maps, and so are the algorithms that curate your social media feeds. So LLMs are AI, but AI is much more than just LLMs.
Lmao LLM is an AI model
WAIT!
Didn't I just watch this about 3 hours ago ?
Great work
James McKenzie's channel is very good
Hey ... a comparison of the GPU and PI AI Hat will be interesting 😊 ... how is fastest on tokens ... and so on 😮
This is an interesting topic-the ones available right now have the Hailo-8 and 8L, which aren't suitable for LLMs (unless extremely tiny), because they don't have much RAM. The Hailo-10 supposedly does, but I haven't seen one in the wild yet.
Someone needs to turn this setup into a super low profile rack mount setup..
Jeff just killing it with the coding!
We really need a Pi Pro mini-ITX, with m.2 and a full PCI slot, and about 10x the performance of the 5.
Teşekkürler Allah razı olsun
Thanks for mentioning the resources needed to run "AI". And thanks for using the right terms. LLMs are not AI. It makes me so angry these days because everything has AI these days. Another high quality video.
And it's so size efficient and stream lined 😅
Given the cost of the setup, I'm curious how the base m4 mac mini might stack up against it. The storage on the base model is the obvious drawback. After seeing your storage hooked up through USB, perhaps a PCIE gen 3 or 4 m.2 through thunderbolt would be more than sufficient.
I've been testing, will maybe post something on Level2Jeff soon!
Definitely addling a gpu soon after I buy more ram. Playing with a ryzen 7 5825u and 16gb ddr4, 21 tokens at 8b.
Ok but the base model Mac Mini is smaller and cheaper than this setup, even more power efficient. Could you do a direct comparison?
A worthy idea! I haven't had a chance to get all my testing done on my M4 mini yet.
@@JeffGeerlingCan you see if you can get multiple Mac Minis in a cluster to run larger models too? The upgrade cost to add more ram is the same price as the whole system itself, so an m4 cluster is very appealing here.
I also wonder if you can scale to larger models with the number of mac minis too? So if you have 4 minis or 8 minis, can you run a 64 or 128GB model on the cluster with really good performance? If so, please test Mistral Large 123b on it.
I literally want a Raspberry Pi just to turn it into the ultimately small form factor PC. Probably not ready for my lifestyle but man it's enticing, even just for fun.
Caring about a couple of watts probably isn't worth the effort of using a different cpu. Running a gpu on the pi is a cool concept regardless. Im not sure about the 80+ specification on a regular pc psu but wouldn't it make more sense to measure at the dc bus instead of at the wall? Depending if you can get a quality psu you might be able to power the riser and gpu with some homemade adapters off a 12v only power supply, idk a meanwell for security cameras. That would cut out the other atx bits and voltages from a regular psu and possibly increase efficiency. They are trying to go that way with atx12vo for efficiency in pc's, but then your drives need power off the motherboard or another step down thing for 5v and 3.3. Or even go the other way and run the pi off the psu's 5v rail to eliminate the need for another charger. No idea if those chargers are efficient but they sure aren't free if you're considering them in the setup price and pc psus should have plenty of amps left on the 5v rail to run multiple PI's
Thank you, really😊
Brilliant!
The thumbnail is throwing major Johnny Five vibes. 😆
Johnny five is alive!
Great job! Awesome video. Did you have to do anything to share the ground between graphics card and raspberry pi?
So far it hasn't been an issue-I have both the GPU PSU and the Pi PSU plugged into the same power strip, and I also don't hold both at the same time that much, so can't say if it's going to be a perfect solution or just works for me. Ideally there'd be a riser that powers the Pi and the card at once.
This looks like a cool prototype for a eGPU closure-esque chassis but instead doesn't rely on a pc
Id love to learn about RPI's with batteries. Battery packs? Powerbanks? UPSs?
I looked into this a while back. There are a few UPS HATs. With that, given how little power Pis require, you can use any off-the-shelf power bank.
@@GSBarlev I'm building my own cinema camera, will probaby resort to a off-the-shelf power bank yeah. It won't need to be powered by a typical power adapter anyway
Could you go into more detail about the Raspberry's PCI-E lanes and which cards can cope with fewer lanes and how this actually works?
Thank you 😊
So when's the first full-sized GPU HAT+ coming I wonder?
CM5 with CM5I/O board with PCIe slot?
For a few LLM workloads this might be ok, but a 6700 XT is still probably overkill. I'm using a much older GTX 1060, a GPU that's half the speed of the 6700 XT with a E5-1620, a CPU that's more than twice as fast as the Pi 5, and for a lot of workloads the CPU is the bottleneck. There are only a very few cases where it makes sense to pair a slow cpu with a high powered gpu... and even then a 6700 XT is still overkill and going to be leaving most of it's performance on the table. It's cool to see... but highly impractical.
the 4 kilobyte kernel page size issue were the same thing the asahi linux devs ran into with the m-series chips trying to run x86 games, i wonder how well their work on it with the micro-vms might translate over to the raspberry pi stuff
Interesting. There's a lot of information available around Cuda/Tensor performance for "AI" (for me that goal is LLM for Home Assistant and inference for Frigate) but I'm unable to understand how AMD fits in this space.
Notionally, there's better value to be had from AMD GPUs but how does that stack up in the tokens/sec comparison against NVidia?
I'm starting out with an old intel platform so not limited by the ARM compatibility/drivers or concerned about power (it'll be an OMV server too).
Interesting video, thank you for sharing this! Out of curiosity: how long does it take to compile llama.cpp? You are compiling using the Raspberry Pi’s SOC, right?
yeah. it was maybe 10 minutes or so? I cant recall exactly.
Been waiting for a low cost/ low power way to get rid of alexa.
Hi Jeff id like to see you do all kinds of emulation on this setup
Jeff, do you have an affiliate link to the smart power outlet things that you use and connect to your home assistant? Gonna add to my holiday gift wishlist..
I love this idea, but how practical is it from daily use perspective?
Assuming I want to run a home lab with NAS, collab apps, home assist, video streaming, this personal assistant.
I would think that a setup like this would require multiple rPIs over running on one desktop PC? Would we see real power savings in this case?
How about dual GPU's? And I'd automate it to make it more power efficiënt. With two smart sockets and automate it with entering zones in home assistant. Like if you're entering the zone home, then switch on power plug named GPU, wait 10 seconds and then turn on the power plug of the pi. This makes it even more affordable, since it doesn't have to run 24/7 since you're not at home. With my calculations you'll get two hours on full power when in use. You can even turn off the plugs at midnight if you want, by making an automation.
Three pies for ASR under the cloud,
Seven pies for TTS in their halls of sound,
Nine pies for LLMs doomed to be computed,
One pie for the Master on its dns throne,
In the land of Nvidia where the GPUs lie.
One pie to map them all, one pie to find them,
One pie to reduce them all, and in the code bind them,
In the land of closets where the old GPUs lie
There is an open PR to add Vulkan support to Ollama which would support Intel and AMD GPUs better and make it easier to integrate in home assistant and vscode, but ollama seems uninterested for some reason.
We'll see... I'm going to try running that on my Pi setup!
I'm big on Pi's and solar panels or windmills. I am going to have a crack at this project!
Also Jeff has been a dog with a bone on the GPUs and I for one am totally grateful
thanks for another great video! Maybe you've been asked this before, but where did you get the stickerboard that's in the background of your videos? i can't find one in that size!!
It's custom from a 2x4 sheet of marker board from Lowe's!
@ thank you!! (:
Your doing the AI overlords proud Jeff, Skynet and Hal will be proud....... lol.
you got your units wrong
0.15 t/s per w, 0.29t/s per w
or you could say 0.15 tokens per joule, and 0.29 tokens per joule
or, 540,000 tokens per kWh vs 1044,000 tokens per kWh
Love the setup with PI 5. But I think a used Nuc 9 extreme + 3060 12 Gb VRam (in total around $600) could be more cost effective. I would trade the different in price for the idle power consumption to the difficulty in the setup process.
Definitely a viable option.
Or just buy an older second hand PC and a second hand rtx 3090 with 24gb of memory
3090s are now really cheap because for gaming they are slower the a 4070 super. But do still have twice the vram
@@FuZZbaLLbee a used RTX 3090 alone could be around $700 already....
@@Merlinvn82 got mine for €600. Also bought a CPU, RAM and motherboard for €120.
Did need to buy a new PSU though. So yeah it will be more expensive, but will have 2x the VRAM and more CUDA cores
I can imagine slot for CM5 on back side of big gfx cards.
You could 3D print an enclosure for this.
but, why
I had good luck and performance with an Intel N100 CPU running Ollama and llama3.2:3b. It would put out good T/s for shorter answers while boosting, and you were not hitting it constantly with question after question. Let it cool down a bit (~30 secs) and ask another question and it would do well. It isn't as good as my 3060 12GB, but usable.
French electricity price is 30% less expensive as it is shown on your charte. 🤔
It's really hard to find a good chart showing global pricing that isn't behind a paywall. I think the relative comparison (mostly that European countries are all more expensive than US) is the thing I was trying to get across.
Combined with lower wages too
thanks for the amazing videos,
i'd like to sugest that you try the "useless grapgics cards for gaming" such as Nvidia GT430, GT1030, to see if it has a usable perpose
it would be great to see the tests results
Thanks again for the Lab just amazing
I'm making a wearable computer based on the OpenEyeTap instructable. Just trying to figure out a CM4 version now!
Need me some of that pcbway money! :D
CM5 boards seen in the wild, they are probably in mass production, getting ready for launch, new CM5I/O board too?
Sadly, with the power money savings, you can buy a bag of chips to celebrate.
Are a lot of AI/ML projects still tied to cuda or is this no longer the case?
Always has been. AI is a research field and researchers usually don't see a point in moving away from Nvidia if their cards work fine (and even the same cards can be used in production). Many projects also use custom CUDA kernels code, which complicates things more
good attempt for AMD gpus on the pi; but ultra low power x86 mini pc board works well with Nvidia gpus as well.
but will never be as cool as this.
Looking forward to plug in my rx 6400 into the pi, to make a tiny form factor arm pc
So why not submit a patch to the Pi foundation?
We're considering this soon...
I wonder if Stable Diffusion/Onnxstream will recognize the GPU card through Vulkan? Local AI Images, Videos and LLM's. I priced a PC box needed to do this and it is outside my budget. Plus I love pushing my Pi's to doing useful things.
I'd love a Pi5 in Micro-ATX form factor with one or more PCIE sltos so you would not need all those cables and wires. Also, it would look smug if put inside a nice case together with the GPU.
Is there a Pi5-Compute-Module to Micro-ATX solution out yet?