It's Clusterin' Time!
ฝัง
- เผยแพร่เมื่อ 27 พ.ค. 2024
- I bought a bunch of Turing Pi RK1 modules... then Turing Pi sent me some more!
I'm testing the RK1 32GB modules on my Turing Pi 2 SBC cluster board. We'll see if we can get them working, then have some fun once we do! Could be a while, so sit back, relax, and start your clusters!
Turing Pi RK1: turingpi.com/product/turing-r...
My SBC Reviews page for RK1 with test data: github.com/geerlingguy/sbc-re...
Some things I mentioned in the stream:
- MyElectronics dual ITX 2U rackmount enclosure: www.myelectronics.nl/us/19-in...
- 250W PicoPSU: amzn.to/3uT1aWt
- 12V 8A Power supply: amzn.to/42NLyQG
- 1TB TeamGroup SSDs: amzn.to/48rxRIj
- Home Assistant Yellow: www.home-assistant.io/yellow/
- Aqara Wireless Mini Switch (for studio lights): amzn.to/49lTzir
- Gator frameworks patch cable holder: amzn.to/3IcEUtT
Support me on Patreon: / geerlingguy
Sponsor me on GitHub: github.com/sponsors/geerlingguy
Merch: redshirtjeff.com
2nd Channel: / geerlingengineering
Contents:
00:00:00 - Are we live?
00:02:00 - Turing Pi 2
00:04:52 - The first livestream setup, featuring NDI
00:06:32 - Setting up the clusterboard, featuring PicoPSU
00:09:30 - NVMe SSDs on the bottom
00:12:20 - RK3588 SoC and the RK1 SoM
00:14:20 - Heatsinks and fans
00:19:52 - RTFM, Jeff
00:20:50 - Heatsinks and fans, take 2
00:35:00 - Getting plugged in, featuring Diet Dr. Pepper
00:37:44 - No magic smoke, board LEDs light up!
00:40:42 - SBC Review repo - a gauntlet of tests
00:45:16 - BMC Web UI overview, featuring risky TP2 handling
00:47:40 - Testing the first RK1 - neofetch and Geekbench
00:53:58 - RK1 overview, what is an SoM?
01:00:00 - Geekbench 6 results (and Pi 5 comparison)
01:03:17 - eMMC performance testing
01:08:05 - Ethernet performance testing
01:12:00 - What is a cluster? Featuring live chat
01:13:04 - Memory bandwidth
01:15:04 - 1 Gbps Ethernet is a major bottleneck
01:17:36 - Impromptu office and studio tour (with some technical difficulties)
01:27:04 - Impromptu server/rack room tour
01:31:37 - Impromptu desk/studio tour
01:36:07 - Impromptu front office and retro corner tour
01:39:55 - Flashing finally finished, didn't fail
01:42:11 - Getting connected to the nodes (SSH)
01:45:15 - Pi Cluster project setup
01:48:31 - Timestamp guy!
01:49:10 - Plans for this cluster - วิทยาศาสตร์และเทคโนโลยี
I loved the part where jeff said "It's clusterin' time" and clustered all over the place
truly one of the clusters of all time. (I really enjoy clustered computing though so these videos are always fun to see)
I just realized why I enjoy this channel so much. Apart from his knowledge and intelligence, I find his channel soothing. It's nice to relax and enjoy learning. I'm trying to obsurb as much knowledge from Jeff as I can. Maybe one day if I'm blessed to win the lottery, I can hire him to setup my home network. One thing about it, I know my network would be setup the right way. God bless you sir. Go Navy.
I love this cluster now it’s 10 days on my desk… all it’s working.. great product. Thanks for your work
awww missed the live ... you always get to play with the cooles gadgets Jeff. in most of your videos my mind just goes "i want i want i want i want"
Hopefully I can help distinguish between the wants that are worth pursuing and what's too expensive or not good enough :D
I'm so glad to see this as a recording. I have my Turning Pi 2 and four RK1s waiting to be set up. I've successfully flashed the BCM to V2 but can seem to get a RK1 to work.
Can't wait to be able to watch this so I might be able to follow along.
Most people didn't get used to the "voice auto cut-off" for saving bandwith, it's OK for me, if the threshold can be higher, that means allow some background noise go through may help not total silent when not speaking.
I had a Turing Pi 2, partly inspired by your early videos. I managed to get through the older Phoenixsuite updates and get connected to BMC. Got it upgraded to the later and improved software. But there were so many hardware caveats, things to solder or boards to add for basic functionality. I had two Orin Nanos and was never able to get those recognized. Finally realized that the point for me was to learn cluster software, not trouble-shoot hardware so I sold everything off. But I don't have anywhere near a millionth of your level of expertise. Took the case and power supply and built an AMD PC running Ubuntu. That sort of hardware I was able to build and get running in no time.
Yeah it's definitely not as beginner-friendly as I'd hoped, especially the earliest revisions of the board. It seems like the new 2.5 version has some of the hardware bugs ironed out, and hopefully they can make another Turing Pi (3?) that fixes some of my last concerns, most especially networking.
yeah for 99.99% of people that want a cluster it's better to just make a Proxmox server on a PC and run muliple VMs as a cluster, since the software is going to be the same anyway. These things are more for embedded enthusiasts to do weird and wondrous things
@@JeffGeerling Exactly! I've seen a lot of people struggle and then sell it off like I did. I had a RK1 on order but cancelled that. The Discord group is very knowledgeable and generally helpful but it was apparent that I was out of my league. The PC build helped restore some of my confidence that I can make something work! I still have one of my Orin Nanos so I'll keep an eye out for future updates. It was good to see this stream and get an idea of the general workflow. I find your channel and general enthusiasm super helpful!
Heh yeah, sometimes you need a simpler project to keep your sanity. I am working on some old Macs right now, and it's nice to have hardware I can hold in my hands instead of tiny single board computers that either work or are trash :D@@dollhausenx
nice, my RK1's arrived yesterday, so look forward to watching this!
If you had scrolled down the instructions a few more pictures, you would certainly have noticed that the installation was correct and that all you had to do was run the cable "under" the cooler.
love your work, so easy to listen too, i'm in Australia, your accent is so easy
Always learn something from Jeff. Please let us know when the live events will occur! Would loved to have seen this live!
When put the heatsink on the module, it resembled the Slot I Pentium II.
Haha so it does (though a little smaller!)
I’m excited for the new 2.5 version of the Turing pi. Will you be doing a review of that one? Pre orders are open now I think.
Every time you bump something onto the desk we get a hell of a thud in the audio.
I'll have to figure that out-the shockmount on the Azden must need a little work :P
@@JeffGeerling - Thank you. My headphones are a little bass sensitive!
@@JeffGeerlingyou need a high-pass filter, if not on your microphone or camera or audio interface in the analog domain, then in the digital domain via your interface’s DSP, or in OBS (assuming you’re using OBS for the stream)
Shockmount and an OC White Ultima undersling arm is way to go.
Glad to see a diet Dr Pepper there. Wouldn’t be a project worth doing without that!
The RK1 looks like an interesting processor option. The 16 cpus you saw says the 8 core processor has 2 threads per core. Linux shows a thread as a single cpu.
The single 1Gb ether to me would be the greatest negative for this Turin board.
Thanks for this video. For being a live event, you did great. Best made plans sometimes just don't work out in execution. You were really resilient. No need for apologies!
Soooo jealous! I am still waiting for my order since November. It should be fulfilled next month...
Unbelievable you have subtitles on a live stream, the production quality is through the roof.
Gotta tighten up the audio a little but nowadays, I think everyone should try to subtitle everything!
For a few years i used a PCIe low profile bracket as a finger saver, could cause palm issies for cage nuts that were too tight though.
The parcel service also handed me my 4 RK1s today. Another K3S cluster will be built this weekend 😅
Jeff, don't worry, you can always adopt a nerd if your own kids "disappoint" in the computing interest department 😎
I bought a n100 mini pc with 16gb ram and a 512gb nvme for €200. That includes 21% tax. Great deals are still available 😁
Amazing stream !
awesome idea
Sorry that I missed your live cast, Jeff.
Great job on your new studio. Looks great and I didn't hear any background noises, while listening on a 5.2 AV system. Perhaps your viewers are picking them up on headphones.
Amazed at how well you're able to multitask between video production, the build, chatting with viewers, and while dealing with being hacked into (which would have seriously pissed me off in the moment), yet you brushed it off as if it were no big deal.
Looks like you're gaining a little weight after your health issues, which must be a good thing.
Great one man show. 👏👏👏
😎
Yeah on headphones the little bumps are more audible. I've ordered a better shockmount so hopefully next time it will be cleared up a little better! Thanks!
Definitely got a boomy desk there. Need to filter low frequencies or keep the mic from contacting the desk.
2:26 Pro tip: Set the set-up up. Position the camera as well. Aim the camera. Make sure the camera is focused.
*THEN turn off the autofocus and lock the focus*
That way the subject area always stays, more or less, sharp.
Get an antistatic, heat-resistant silicone soldering/magnetic work mat so you don't lose all your screws. Amazon has all kinds. I went with the Kaisi S-180, but you do you!
And stop touching the contacts with your fingers! The oil on our skin is corrosive. 😧
Once I have a proper electronics bench I plan on doing that; I don't have a great spot for it on my desk though, at least not with my current setup. Maybe I'll get a smaller pad.
Great video Jeff, and just in time 😊
What are your thoughts on turning this into a ultimate kubernetes & nas combo homelab?
Either using node 3 as a storage server leveraging sata ports and pcie to generate a raid-z1 and the rest as bare metal kubernetes,
Or try and flash proxmox on these nodes and cluster them together
Considering my options with the TuringPi_2.4 and 4 RK1 16GB modules plus NVMe drives. I am thinking to setup Docker Swarm across all 4 RK1's vs Microk8s kubernetes with Ceph on the NVMe drives. I might split the NVMe partitions and only use part of it for Ceph persistent storage. Alternatively, I could NFS mount and use that for persistent storage for kubernetes.
There is a good bit of audio from bumps on the table coming through the mic. Might need more isolation away from the desk.
Looks a nice ha dashboard. Any chance you can do a video on your instance of home assistant?
You might want to apply some/more low-pass filtering on your audio, I'm hearing a lot of low-free thumps every time you touch the desk, and that's definitely a new phenomenon.
Yep, gotta listen to the audio on this stream and fix it up a bit more (probably a better shockmount too).
You are live
Glad I'm not dead!
Well, he was live. Now he's just alive.
It would be nice if someone, (hint!) would show how to setup for live stream. What plugs into what, and software used.
---
I'm 76, and I've never used a grounding strap in my life. And have NEVER blown anything! It's over rated!!
Yeah, grounding straps are mostly bogus except in more specialized environments these days. But it is good to take at least basic precautions like touching a grounded object while working on something!
Dear Jeff, I saw the Turing Pi RK1 has 6 TOPS NPU. Can it be used to run a 2B Large Language Model?
Appreciated for all the great work you have done.
would be really cool to run whisper on the NPU and have it send messages to and from ollama llms
I love this system. A pity for me i don't have the budget for it.
is the presence sensor not like being near the metal rack bar, and/or is the bigger metal rack blocking or otherwise messing with the reflective pattern ?
I wonder if that could be causing issues. It is a bit sporadic, so I'll have to mount it on the wall away from the rack.
I do like to save the box in case of any warranty claims. Then at times I have the box years past the warranty end date, lol.
Is there any chance that we can have any video on esp32 cluster ?
Hi Jeff, great video. Can you Tell me what NDI-Software you are using ob your iPhone?
"OBS Camera"
An Apple-Chromecast is the most evil thing that has never been! 🤣
Wow, I was literally just talking to a colleague of mine about how I wanted to get this (but the Turing Pi 2.5 is out of stock till May 😫)
Yeah, I'm still on the 2.4, which has a few little quirks (luckily none that make my experience bad!).
Another thought... why not mount your tabletop camera from the ceiling and point it top down? You'll be able to get the tripod out of your way.
Perfect timing. My two 32GBs are on their way to join two CM4s in my cluster. Have you tested the NPU already for AI acceleration workloads?
Not yet, want to see if I can get it working with Frigate.
sounds like this would make a heck of a NVR...
is there any reason you use github issues instead of markdown or something in a repo
Good question! Easy to pop stuff in comments. Probably should have a GitHub Pages site for it too, though. Discussion in comments, but main pages on the site, something like that.
Hey Jeff! I'm just starting to get into some clustering and 'portable computing' for a car project. I was curious if you've had a chance to look at the Radxa NX5 SoM? It looks like it might be compatible with the turing pi 2
No but it does look nice-the RK3588s could be even better for this form factor since it takes off a few interfaces in favor of a smaller package and a tiny bit less power consumption. But it would probably need some support from the Turing Pi board for hardware support.
11:25 It looks like the thread for the shorter SSD is in the way and bending the SSD module when you tighten it 🤔
Nice video , but this looks very bulky , would switch to min pc with 12 core cpu on it?, what the benefit for using something like this?, what implementation would it serve?
Are there no cluster backplanes that have 10Gb+ network bandwidth? Makes them a bit of a joke being limited so much with network connectivity.
Thanks! Seems like all of the stuff you review is almost instantly sold out or backordered. Wonder why? ;D
That's I guess a good reason to get on the notification bell-be the first to see it sold out! :D
Are you going to make a follow up video setting up the cluster?
I plan on it! Hopefully I'll get this board up, plus a Pi board, and compare the two side by side :)
Can't promise a timeline for that but "soon" :D
Nice idea they had , I followed this years ago during the crypto mining craze when it could of had a good use network staking ETH or trade, BOT farming ECT, now days its just an expensive Toy as when you cost it up you can just buy a powerful ATX , ITX or embedded board that can do more for less hassle and that has all the software prepackaged, they missed the bus but by all means have fun if you have the cash to spend and want the challenge.
@Jeff Geerling have you tried the DeskPi, I am thinking about it, it's $199 and says it can work with 6 CM4
Yep! Check out my video on the 'Ceph Supercluster' :)
Just got mine too. While installing the heatsinks did you observe the pads not touching the back of the heatsink properly?
They seemed to make contact on mine.
I wonder what the best way to solve this is. Fatter thermal pads? Can I double up on pads or maybe find some fatter ones and cut them down. Hmm.
Best "job" ever :)
Any chance on a shorter form cut down video. I sadly don't have this kind of time anymore.😢
I added in timestamps; I'd recommend just browsing through for the bits you want to watch. I'll have another video on the cluster later though!
can Proxmox be installed on the Turing RK1?
Dr. Pepper better sponsor you soon! :)
What's clustering good for if you don't get redundancy? 🤔
When you're on the same mainboard a hardware/power -fail will probably take all 4 CPU's out. 😕
This setup can of cause be used for training/experimenting, but I would not use something with a possible ( single point of failure ) in an production environment. 😉
Yeah, what I would like to see is a better/faster network connection, so you could run multiple boards-alternatively, the Compute Blade will *hopefully* start shipping soon, and that has a little more built-in redundancy!
have you done any of hte nvidia jetson orins with this? i'm having a hell of a time finding a usb cable that works to flash the card. if anyone has had any luck and has a part number for a usb cable that worked i'd appreciate it, tired of getting another one form amazon and them not working (usb a- a data transfer)
Not I-but I've heard USB flashing is not working that well; I'd check out the Discord.
Which button is that Home Assistant for the lights?
It's an Aqara Mini Switch, Zigbee: amzn.to/49lTzir
@@JeffGeerling thanks
instead of neofetch you can do 'uname -a' shows all the specifications
Edit : i talked to fast
Hehe
The Battery adapter made no sense to me. Why would a small battery fix a bug?
You are not using the ifixit ;P EDIT: ow I think it was also mentioned in the chat ys.
Please enable the display of subtitles.
I'm working on that, it'll be another hour or two.
Shame turing don’t have an adapter for the Toradex imx8mp 😂
Can you run multiple BMC tabs and do the flashing in parallel?
That's... a good question.
Nope :( It cancels the first one if you try a 2nd :(
Bummer. Hopefully the one it cancelled wasn't too far in.
Heh, it was 70% but that's okay, I have it going again, will finish up by morning when I get back! @@mysticknight487
The first gen iPad Air I got no longer gets TH-cam updates so the app isn’t useable for me anymore. Maybe it could be possible to make it more useful with a jailbreak. It is annoying Apple doesn’t allow to root the device so I could try and install a different OS now that they aren’t supporting it anymore.
I cannot watch live chat on the Safari browser, so if yours is also unsupported it probably cannot either.
Link to your patch panel organizer?
Oh shoot, will add! Gator Frameworks wall mount cable holder: amzn.to/3IcEUtT
hehe Windows progress bars are truly nonsensical! hehe you have microsoft minutes and Microsoft percentages and both are very unique to normal minutes or percentages!
Waiting on my 32GB Rk1, still shame they didn't add pwm fan to the turing pi, been contemplating adding it to my board just can't find anyone to do it. Hopefully CM5's will be compatible also !
The fan control is PWM, it's just there's a weird bug with the 2.0.0 firmware, it seems. Three of the nodes are coming up with PWM, but node 2's fan still spins full blast for right now. Will upgrade soon.
Is the mic built inside that desk? Listening with the headphones is especially annoying with all those thumping coming through.
I'll have to figure out a better mic shockmount solution.
Jeffgerling what happened to the tower was stolen, did you recover it
Ha! Not sure yet, still waiting on some details from that crazy event.
Low speed switching fabric, fat TCP stack sucking up processor time, kills the dream of parallel computing and efficient clustering.
Any discount codes?
What up dude
Lost live chat 😅
It should come back soon, fingers crossed!
Jeff can you do a proper shakedown on the "Super 6C" ive been thinking of doing a project with this calculating Pi with my eldest, dunno why but im Pi obsessed and autistic and thought it would be fun, though a fun dent in my wallet!
Check out my 'Ceph supercluster' video from last year, I think-I covered the Super6C there :)
How does someone go about acquiring all these old boards you have.... say the 8GB RK1's
Audio issues - That's why you should go android! lol
Ask the Ubuntu guy... why it takes 4 separate clicks to power off? Sincerely - Wallwart Yanker
how'd you get the 32GB ?
They sent a couple early production units to me for testing-they just started shipping them this week I think!
I was going to build the cluster with the 8 GB modules I had pre-ordered (they came a few weeks ago), but I decided to test with 32GB so I can see if tests like HPL get a nicer number :)
@@JeffGeerling fantastic. I've had the Tp2 for a few months now and it's great. I've got my 16G RK1 now and it hopefully prove to be a powerful workhorse to add to the cluster.
thanks for responding and thanks for all your content and everything.
feb 32nd lol
AMD Ryzen™ 7 7800X3D Processor - purchase this and youre done, not sure why even do this entire project
Test
Works!
interesting, but a 20 min summary would be greatly appreciated (please not a youtube short)
Maybe I'm just old, but the idea of flashing something over Ethernet gives me the screaming willies.
Also, as for mispronunciations of "Ubuntu": I watch a set of training videos (not naming names), where the instructor consistently pronounces it "oom-BOON-too". Drives me nuts to hear that non-existent "m" in there.
That's a computer.... lolz. You are gunna need more power supply than that....
Ok, good vid, good brick... however.... would you plugin a 750 Full ATX power supply then re-run the benchmark? In this design...
I'd forget the fans, lay it in fiberglass pan with mineral oil, sat in a freezer with wiring harnesses cut out and filled insulated, build that, then run it again.
Freezer on lowest, some stuff shrinks..... some stuff doesn't as much...
Such a nice computer... just try it with a Raspberry Pi 4B, good board, totally designed to be run supper cooled exactly like that, in mineral oil, run from in a freezer. You've never used it correctly until you build them like that.
Then... when you want it to go faster.... you turn the freezer up, and make it colder... until something cracks. Cold clocking.
Cray cray over clocking. I mean super computer, and I mean crazy.
heh
Your mic(s?) pick up the bass wait to much.Or the compressor add way too much boost on the deep deep lower end.
I believed it was a neighbor doing some work, but no, it's the video.
listen to it on a hifi system and you will hear it. (during the assembly of the heatsink for example)
I am surprised how annoyed I am with your hand coming in and out of focus, I'd like to see through to the end, but I just can't watch. Static focus (set focal point) would be fine if set right in the middle of the motherboard, maybe increase the f-stop for better DOF, anyway, anyway just a thought.
I set it to f/8 but unfortunately that camera is full frame and even at like f/11 at closer ranges the DoF is very shallow :(
I may consider a different setup for future streams, maybe my other camera with APS-C which would give a little more depth.
They are cool but still cost too much. Heck you've got 4 there so you're talking $600 just for the boards. For that price you could get a much more capable small form factor computer with better grafix.
For many use cases, that is a perfectly reasonable option!
Saya berhenti nonton hanya gara-gara dia minum pakai tangan kiri 😂
49:53 你说中文吗??
Heh, no, but I should! I work a lot with various companies and individuals in Asia... it would be helpful.
What
I always said Ubuntu the right way, but hearing you use the right pronounciation all of a sudden is weird
Too bad SBC have all become and still are overpriced like nvidia :( An RK3588 dev board should be like 30$ and not around 180$.
sound was terrible, use body microphone - sound much better
noise cancelling awful when something works - but when no noises - noise cancelling not needed too...