วีดีโอ

Thanks very much!

Thank you very much!

Thanks for watching!

I have a video on the 8gb pi 5 vs Opi here: th-cam.com/video/OXSsrWpIm8o/w-d-xo.html

Thanks very much!

No problem, hope it is helpful!

Not well if possible haha

@OminousIndustries 😆

For science!

I started with an Ender 3 a couple years later. These will be as close to plug-n play as you can get.

Unfortunately the current Hailo AI hat does not work for llms (based on what they themselves have said) I have not personally tried it.

@@OminousIndustries thanks for taking the time to comment. Appreciate you and your efforts, and this video.

@@mooninthewater3705 No problem at all, thanks for the kind words!

Thank you! This is the 13 TOPS version.

Yes, it was quite hot! I am going to get an active cooler for it and will test again to see if there is a difference. I was also wondering about throttling.

Thank you, here is the link: www.microcenter.com/product/661858/inland-tn320-256gb-ssd-nvme-pcie-gen-30x4-m2-2280-3d-nand-tlc-internal-solid-state-drive

@@OminousIndustries thanks

Thanks very much. Yes, that was quite frustrating to me as well until I tried the different container.

Exactly, I still have a bunch of 2gb 4B models that I am using for random things. I was very excited when I saw these were in stock and ran out to grab one haha

I personally prefer Ubuntu, but if someone is used to windows and does not want to have to trouble shoot a lot it might not be a bad idea to stick with windows haha

Yes there are small models like llama 1b and 3b that would be much much quicker. I wanted to test the larger models for this video as the new 16gb ram variant was able to run them, something that the previous 8gb max RAM pi could not!

Yes it is, I was impressed it got that, I would love to have the HP to run the 90b one at a decent quant but I need more than 48gb vram.

This monitor is not curved so that would probably factor into your decision.

@@OminousIndustries It doesn't matter whether the monitor is flat or curved, image quality and a large screen are more important to me. Thanks for reply

You're absolutely right. I will make sure to better show temps/etc in the future!

I can't definitively answer this, but I have had good luck with some of the Qwen models and function calling abilities.

Good to know I wasn't going nuts haha

Thanks very much!

I do actually have the components to do an e-gpu on the pi, at some point I would like to try it as his videos showcasing that were very very cool!

@ yeah with the 16Gb that would be a first and with power consumption. I have the equipment also just busy

@@ESGamingCentral I may try it sooner than later but only have a 3060 12gb to do it with.

@ as far I’m aware you need an AMD card; I have a 6600XT. I don’t believe there are drivers for Nvidia cards in arm RPI

@@ESGamingCentral That's very interesting and something I wasn't aware of. I don't actually have any modern AMD cards so damn haha

If you were going to use a small Llama 1b or 3b I don't believe there would be a speed difference between the 8gb and 16gb pi.

That's the tough part about comparing to the Pi, the long term ecosystem has had the support of a lot of talented people over the years which makes them far better supported.

I'm okay with having more options! haha. Use cases are a different story

Thanks very much! Yes, I would like to see the introduction of a more AI focused Pi.

Glad to hear, they will definitely fit well into some of your projects!

From what I have seen about Veo2 it is very very impressive.

That sounds like an awesome project and something right up my alley. Yes, being able to run things locally is very important to a lot of us!

LOL

Yes, but if I am not mistaken, if the model was a smaller parameter model like a 1b or 3b the total system ram wouldn't make a difference in the speed as it would only be allocated across what was needed and the extra ram wouldn't come into play.

Yes, these smaller models are much better suited for lower-powered hardware.

I did touch the cpu and it was extremely hot haha. Active cooling is on the agenda ASAP.

I've been meaning to try phi4 at some point.

@@OminousIndustries step by step tutorial without ollama. Only Python, only hardcore

@@MrKim-pt2vm LOL hackathon vibes

Thanks very much! Soon the hardware will be more and more accessible :)

Thanks so much, glad you found it useful!

no, after the initial load it's stored in the ram

I don't believe it would make a difference in this scenario, no.

Was scared to without any form of cooler haha

Absolutely!

I have 2 Orins ordered in UK. Do you recommend any particular m2 SSD? Is it just set up as swap? What command did you use? Thanks in advance

@@ianfoster99 Any fast SSD should be fine. I did a 256Gb but only set a ram swap of 20Gb. You don't want to over allocate. Plus wear rates will be higher so I have lots of unallocated space as regions go bad. You can set it up with the Disks utility in Ubuntu, or with python code. ChatGPT can help with that. Good luck.

Thats a very cool setup, with the increased swap. I am interested to know your results when you test the larger llms. Definitely exciting time for local llm and SBC's

Interesting! Could you or anyone on this thread share a link to a video explaining how to add this additional virtual ram with a second m.2 on the Jetson Orin Nano Super? Thanks!

That's a great idea, I will try to squeeze that in!

Thanks very much!

Cheers!

Cheers!

Thanks! I think this needs a 3090 minimum, though for a minimal setup a Zotac 3060 12gb is a very good price/performance option and I have two of those in a machine

Thanks! It's been a fun build!

Unfortunately I don't see myself getting the AI hat + or new Pi 5b but that would be a good comparison indeed. The original pi AI kit was not able to do any llm work because of pcie bandwidth limitations (according to the manufacturer) so it was really only for vision tasks.