Building a Ceph cluster on a bunch of Raspberry Pis was a fun project, and I think perfect for this board! If you're looking for some Pis, they pop up from time to time on rpilocator.com (no affiliation, I just use that site to check up on Pi stock around the world). Thanks especially to KIOXIA for sending the NVMe SSDs to fill in the rest of this board - check out their XG6 and their new XG8 if you need a great, reliable NVMe drive for your computer. I'd been working on this video since a few weeks ago (before the hospitalization), so I was able to get it out for this normal weekly release. I'll probably be taking a break and won't be posting a video next Wednesday-just check back the week after, and in the mean time, go check out Geerling Engineering, my 2nd channel! th-cam.com/users/GeerlingEngineering
I really like your videos because you answer all the relevant questions directly and always get to the point. So thank you, Jeff, for always covering fascinating topics and products. And best greetings to your wife and dad! I really hope you feel better soon!
Can you use something like this as a desktop replacement? I am not clear on if you can configure Linux to use all these Pis as 'one' system transparent to the user and just do normal desktop stuff with it.
AWESOME video! Superb content. But right now, TAKE A BREAK, take care of your health, please. That's the priority. We all are cheering for you, Jeff. God bless you and be better soon. 🙏🏻
Truly impressive video especially given that most of the time your being ill in bed But I have no idea how your keeping Red shirt Jeff under control handcuffs? Video games? A really long leash? So back to my bed to watch you till I fall asleep
Heh... those are literally all the Lite CM4s I have. I have a couple eMMC models too but I typically stick to the Lite modules since they're easier to re-flash (don't have to wait for slow eMMC writes over USB).
@@JeffGeerling I was diagnosed with Collitis the day before you posted your video about your colostomy. Gotta say knowing someone I look up to has the same/similar issue to me healthwise has made it easier to cope.
@@JeffGeerling real clone wars is rhinos like bush=nute gunray invade naboo bush could of prevent europe migrant crisis since obama target Libya Sudan Somalia Syria was originally bush admin rumsfield idea pnac! we can give all medicare by reduce warfronts, no more allowing new daca abusers, force senators to cut their own wealth in gov by half if they want to keep above 70% of current daca population so inflation/living cost stay reasonable wesley clark foreshadow reveal 2000 to 2012 all rig for kill iraq to syria th-cam.com/video/_mrJRHwbVG8/w-d-xo.html establishment kill 50 in vegas/portland, thugs attack with stand down cops san jose/charlotte, burn loot several months, sabotage afgan withdraw using russia bounty smear to give taliban equip, crash car in to wisconsin parade thanks to nbc follow jury bus smearing ritten house too th-cam.com/video/UxoL8tHSa7g/w-d-xo.html bush 14y ago said add ukraine to nato with nuland f eu coup 2014 support = th-cam.com/video/nTQ3D1a-j20/w-d-xo.html current ukraine gov is proxy since cia drew red line just like did in syria earlier arming rebels telling russia not to interfere while zelensky ethnic cleanse donbass region 7y= th-cam.com/video/ta9dWRcDUPA/w-d-xo.html th-cam.com/video/IBeRB7rWk_8/w-d-xo.html
@@JeffGeerling It is a fun game but losing it sucks and even when winning you then get to play the second round of the game which is named 'Desperately search the floor on your hands and knees because you know that if you dont find the pin you will feel it in three days when you step on it barefoot...' So many times.....
It's way quicker to cut than to desolder. Desolder on 100 boards and I'm pretty sure you will be quite tired. Even lots of massproduced motherboards has cut pins.
@@perwestermark8920 I've never seen a mass-produced motherboard with snipped pins, do you have an example? Desoldering is usually quicker though, touch the pin with the soldering iron and pull it out just like that, no need to line up cutters and no risk of leaving a pin loose on the motherboard ready to short circuit everything when an unsuspecting customer connects power...
The quality of your TH-cam videos is outstanding. You have such a great mix of technical and communication skills. The things you show (and link to in the descriptions) end up being so approachable. Thank you for putting all this time into the videos for us!
An interesting alternative, but if I were to try this I think I'd go with the Turing 2 for the reasons you mentioned. I am glad to see that you are feeling better. I know from experience that gastric diseases are no fun. Thank you for all you do!
Thank you for marking this as a paid promotion. I've seen too many creators lately that don't count product(that they get to keep, not stuff they have to send back) as payment. Cash didn't change hands, but they were still paid. Also, I love that trick with the sticky note. I wouldn't have thought of that, but it's brilliant for dealing with so many small microsd cards.
Haha yeah I started doing that about the fifth time I lost track of microSD cards on a cluster build... it's perfect since I always have a pile of Post-Its nearby!
You don't need 5 power supplies when clustering normal Pi4s, just one feisty 5V supply, which you can also then feed to the Ethernet switch. Meanwell comes highly recommended, and they have a 90W/18A model. You might want to 3D print a MicroSD card puller that you can clip onto the cards to retrieve them. I've seen a bunch of designs of those floating around.
Great to have you well enough to post Jeff. I do feel for companies that have developed such cool stuff for the rpi and especially the cm4. I don’t think I’ve actually seen a cm4 available for sale through an Australian based website at all. Someday!
Man it is good to see you up and around and cranking out these great videos again. I hope you are feeling better. You must be feeling some better since you’re up and around at least. That’s a great thing. Keep cranking man
Good news: no individual power adapters and cables needed. Bad news: the PSU that powers the board is now a single point of failure. Thanks for the Ceph tutorial, Jeff. I think I now have the intestinal fortitude to try a Ceph deployment myself. And BONUS - what a great hands-on opportunity to really learn and understand Ansible.
Happy to see you're back and hope you recovered full. Were waiting on your video and it's quite interesting to see how your RPI cluster have evolved. I will dive more in ceph/ansible using VMs instead...
Great to see you back to making videos. Pace yourself . Thanks to your family for helping you and weathering this episode. You are my favorite tech youtuber. The amount you give back to the community is awesome. Keep up the great work
1. I am SUPER happy to see you producing content again. It implies you are feeling better! 2. This board appears to be really useful for learning ceph, or other technologies. 3. This board is almost useless because it is the foundation for rpi boards that are mostly unavailable. 4. I have been having some really good luck with the "Le Potato" to fill most of the needs for raspberry pi's. I don't always need the power of an rpi4 and the "potato" is just a bit more power than a 3b, and at a slightly lower power consumption so almost any phone charger will now work very well. 5. Maybe with the whole world going into a recession, maybe these boards will be available soon?
I just randomly stumbled upon this video and just realized you are "The" geerlingguy. Holy frick mate, your tools have made my life so much easier. Thanks for all the hard work. Great video.
It is definitely the number one issue with these projects; you either have to spend a bunch of money or have a huge amount of patience to get Pis anymore :(
Thanks Jeff. Great video. I came across your post about two months ago and really enjoy the content and your presentation style! Loved your Turning Pi videos and now this one too. I am waiting on my Turing Pi 2 to ship and I'm going to order this board too! Thanks again!
Am glad to see an upload. I know this is from before but I hope you are taking care of yourself and that you are getting better. Get well soon dude! We miss you! :) 🤗🤗🤗👍👍👍❤️❤️❤️
This is super neat! As a testbed, this makes so much sense. Or just running a small homeserver - its really neat! However, I am still waiting for just one CM4 on a miniITX board - I have an old thin MiniITX board lying around here and I would love to put a CM4 into that to use it to run HomeAssistant and friends. The board you showed off might actually fit, which is kinda insane. xD But the case's built-in PSU just won't do 100W...
It's not fair! I've been trying to buy just one CM4 for nearly a year now and it's out of stock everywhere. Now here come's Jeff with his SIX CM4s on a single board! 😭
Ah but I ordered four of these CM4s in October 2020, and the other two in June 2021, heh... they took months to come in! I have been seeing 2 and 4 GB CM4 modules coming in stock every week or so, if you need one make sure to get notifications from rpilocator!
@@JeffGeerling Yeah, I'm keeping an eye on that site. Hopefully there'll be some UK stock soon. It's super frustrating seeing all these amazing carrier boards on your channel and not being able to make use of any of them. 🙂
I like your clustering vids (touring pie, deskpie, conventional with router poe, etc). The parts described in the des just expensive when shipping to Indonesia (vat, shipping cost). Thats really put me into tears
One of the features of a cluster is a reduction in the single-point-of-failure elements. For home use, you can't do anything about AC power, but your original cluster at least had separate power adapters and ethernet ports, although just a single switch. This board appears to have more single-point-of-failure components.
I'm pretty sure I can get my normal heatsinks to fit, I would just need to use a different set of spacers and screws so I can still tighten them directly to the board (the inserts on the board are threaded, I believe M2.5).
You could use multipath TCP for getting more speed, and as it's CEPH and storage you could use SMB3 multichannel feature which works with at least Windows 10 and modern Samba. Neither of these solutions require a managed switch.
But both of these require multipath connections to the nodes. Here, each Pi has only 1 Ethernet connection and the switch has two ports exposed. Connect those two ports to another switch and you have a loop, causing a smarter switch to disable one of the ports or when using a not too smart switch will just lock up the network... So SMB multipath won't work here unfortunately, that is why it is hard to understand why they chose an unmanaged switch...
@@someguy4915 Duh that was a Homer Simpson moment. However with a suitable managed switch you could put down MAC based ACL's that would mask half the PI's down one port and half down the second port, to get more upstream speed. That should shut down the loop, but yeah it's slightly wacho.
@@jonathanbuzzard1376 True but even then you'd need a bunch of routing etc. to make sure that traffic sent on Port 1 also returns through Port 1 etc. which while certainly possible is just a bit of a headache when a simple managed switch would fix all of that and provide so much more functionality. Besides LACP port bonding you'd also get VLAN support, now you could just have the Pi's spit through VLAN tags and hope that the buffer of the switch is big enough, usually this does work, but it is unsupported and a great way to get very complex network issues (don't ask how I learned that lesson... :P)
Finally getting to what I wanted with Ceph. Really cool I knew you would come around. HA!!! I have some reading to do with Ceph; I did not know it existed.
Hopefully I'll get some more time to dive into Ceph; it impressed me with how simple it was to get going, and the concept is similar to Gluster (which I'm more familiar with), but I need a bit more time with it before I'd be comfortable going deeper in a video!
I do like SFP+, though just having an SFP+ cage wouldn't give faster speeds, the switch chip would also need an upgrade. 2.5G could be realistic but going to 5 or 10G on this board design might have issues with heat or signal routing. The other difficulty with this board in particular is there's not enough physical room for SFP+; you'd have to take out one of the Pis to do it.
@@JeffGeerling Oh, yes I see now, you are right. The advantage with a SFP+ slot is that it can run on less power than a 10GE copper connection. But perhaps one or two 2,5GE slots would be good? The Intel I225 should perhaps fit well? It had issues in the first two revisions. But yes the whole switch thing is an issue as well.
@@JohnAngelmo But that Intel is a NIC, not a switch, you'd need to replace the entire Realtek switch chip for something more useful (managed and 2.5-10GbE) before adding SFP+ has any benefit.
You weren't kidding when you said it had it's trade offs. Honestly I'd prefer the Turing Pi, as compatibility with Jetson platform, and ATX power compatibility would make it a great fit in home-labs; my biggest gripe with the Turing is how IO is handled by specific slots rather than one master.
Much better! Though still taking it easy right now, as part of that is due to some of the medications I'm on... always hard to figure out how much improvement comes from *actual* improvement in underlying health!
4:06 You don't get much air trough those hole when you try to suck trough them. You will only get a fan that spins at a high rpm without any air going trough. You can easily test it by holding a fan and with your hand you starts to covering the side the fan suck from. You will now hear the fan spinning up because less air going trough. Do the same thing on the blow side. You will not be able to block the outgoing air that easy.
I can see this working really well in distributed TV editing storage, particularly for outside broadcast or mobile facilities. Low power requirements, easily configured to access 6TB at time. The whole thing, including monitor, keyboard and mouse could be put in a flight case and carried to the next job. The only downside is the gigabit connecting. 150 minutes minimum per terabyte is slow but that's where the price and portability come in. Two or three cases per reporting team would keep everything flowing nicely.
This is yet another step forward, I love it. Maybe in the future we'll see more home super computers. I'm glad you recorded this earlier, hopefully you're still taking it easy. And I'll never stop using the SI prefixes to refer to 1024 increments of bytes. I don't care if hard drive manufacturers cheat, computers are special and it makes no sense to make up new terms this late in the game, aside from the fact that they sound dumb.
@@tomservo5007 Not really, considering how they were built in the past. It's just that processors have gotten so much better now that we, most of us anyway, have as much computing power in our pockets as supercomputers did in the 70's.
@@anon_y_mousse And as a note, this board's linpack scores would put it somewhere in the Top500 list from around the year 2000... so it just depends on your definition of 'Supercomputer' ;)
I seem to recall that a netgear (dumb) switch that I owned (may still be in a box here some where) actually supported link agregation, even if it didn't have management or vlan support. So it's possible that getting link aggregation on the board may just be a firmware update, but I'm kind of fond of the vlan support as it allows me to move management of devices off of the vlan that the user data is running over. Hopefully the manufacturer will get that all worked out.
Can the Pis still be overclocked in such a setup? If so, can they be individually OCed? 2:58 A minor mod with transistors can take care of the power LED and fan always being on. Throw an NPN transistor on the ground of each, and connect the transistors' bases to an unused, power-switched voltage out (with an appropriate resistor in between). The USB header would work fine if you can't find any other switched power rail.
I noticed with my Pi 4b that it was thermal throttling with the regular plastic cases most kits provide, at the stock 1.8 GHz but that when I switched to a passive cooling metallic case (Flirc) I was able to overclock it to 2.0 GHz and it didn't thermal throttle at all at that speed. I'd like to see a sleek passive cooling case for this thing. It would probably do a decent job distributing the heat, being so flat, having so much surface area for the 6 compute modules.
@@JeffGeerling Sorry, I was unclear. I meant can they still be overclocked on this board, and if so can you set different clocks for the different Pis? For my standalone Pis, I overclock almost all of them.
Fascinated by the idea of something like this or the Turing Pi for a little home cluster to do some Docker experiments. This beats the Turing Pi by virtue of being available but on the other hand I'm not sure if it is worth going all-in on six CM4s, six SD cards and six m.2 drives vs just going to state surplus and picking up an old pc or two.
Protip on the SD cards. Just put little scotch tape tails on em. Makes em easy to pull out of tight spots. Just don't do that in an application where they may get snagged and pulled accidentally.
whenever I see pis clustered like this, I definitely find myself wondering about storage clustering like you explored here. If this board had SATA instead of nvme, I'd be very tempted to try and build something with larger HDDs for some NAS testing
Interesting. I wonder what use cases there are for this, for people who don't need the performance of the $12k setup but who could benefit off of having so many cores with efficient networking.
Building a Ceph cluster on a bunch of Raspberry Pis was a fun project, and I think perfect for this board! If you're looking for some Pis, they pop up from time to time on rpilocator.com (no affiliation, I just use that site to check up on Pi stock around the world).
Thanks especially to KIOXIA for sending the NVMe SSDs to fill in the rest of this board - check out their XG6 and their new XG8 if you need a great, reliable NVMe drive for your computer.
I'd been working on this video since a few weeks ago (before the hospitalization), so I was able to get it out for this normal weekly release. I'll probably be taking a break and won't be posting a video next Wednesday-just check back the week after, and in the mean time, go check out Geerling Engineering, my 2nd channel! th-cam.com/users/GeerlingEngineering
I really like your videos because you answer all the relevant questions directly and always get to the point. So thank you, Jeff, for always covering fascinating topics and products. And best greetings to your wife and dad! I really hope you feel better soon!
Can you use something like this as a desktop replacement? I am not clear on if you can configure Linux to use all these Pis as 'one' system transparent to the user and just do normal desktop stuff with it.
AWESOME video! Superb content. But right now, TAKE A BREAK, take care of your health, please. That's the priority. We all are cheering for you, Jeff. God bless you and be better soon. 🙏🏻
I'd love to see some of the points per day it could generate on folding@home if that is possible.
Truly impressive video especially given that most of the time your being ill in bed
But I have no idea how your keeping Red shirt Jeff under control handcuffs? Video games? A really long leash?
So back to my bed to watch you till I fall asleep
If anyone is wondering why there are no rpis in stock its because Jeff has them ALL!
Heh... those are literally all the Lite CM4s I have. I have a couple eMMC models too but I typically stick to the Lite modules since they're easier to re-flash (don't have to wait for slow eMMC writes over USB).
No doubt!!!!
@@JeffGeerling totally sus how he has so many LoL he works for RPi maybe 🤔
@@JeffGeerling That's what someone who bought *all* the Pis would say! :D
dont forget red "shirt" jeff
It is good to see you back healthy. Hope you are fully recovered Jeff!
Like always awesome content +1.
Not quite fully recovered, but we'll get there!
@@JeffGeerling 👍🏼Hang in there! Hope it's (relatively) easy for you and family! 😎✌🏼
@@JeffGeerling I was diagnosed with Collitis the day before you posted your video about your colostomy. Gotta say knowing someone I look up to has the same/similar issue to me healthwise has made it easier to cope.
@@JeffGeerling real clone wars is rhinos like bush=nute gunray invade naboo
bush could of prevent europe migrant crisis since obama target Libya Sudan Somalia Syria was originally bush admin rumsfield idea pnac!
we can give all medicare by reduce warfronts, no more allowing new daca abusers, force senators to cut their own wealth in gov by half if they want to keep above 70% of current daca population so inflation/living cost stay reasonable
wesley clark foreshadow reveal 2000 to 2012 all rig for kill iraq to syria
th-cam.com/video/_mrJRHwbVG8/w-d-xo.html
establishment kill 50 in vegas/portland, thugs attack with stand down cops san jose/charlotte, burn loot several months, sabotage afgan withdraw using russia bounty smear to give taliban equip, crash car in to wisconsin parade thanks to nbc follow jury bus smearing ritten house too
th-cam.com/video/UxoL8tHSa7g/w-d-xo.html
bush 14y ago said add ukraine to nato with nuland f eu coup 2014 support =
th-cam.com/video/nTQ3D1a-j20/w-d-xo.html
current ukraine gov is proxy since cia drew red line just like did in syria earlier arming rebels telling russia not to interfere while zelensky ethnic cleanse donbass region 7y=
th-cam.com/video/ta9dWRcDUPA/w-d-xo.html
th-cam.com/video/IBeRB7rWk_8/w-d-xo.html
5:19 I don't know why they're snipping off the pin. it's a really easy task to desolder the single pin and make it look good.
Probably they enjoy the challenge of the "don't let the pin fly into your eyeball when you snip it" game!
Red shirt jeff has to have his moments too
@@JeffGeerling It is a fun game but losing it sucks and even when winning you then get to play the second round of the game which is named 'Desperately search the floor on your hands and knees because you know that if you dont find the pin you will feel it in three days when you step on it barefoot...'
So many times.....
It's way quicker to cut than to desolder. Desolder on 100 boards and I'm pretty sure you will be quite tired. Even lots of massproduced motherboards has cut pins.
@@perwestermark8920 I've never seen a mass-produced motherboard with snipped pins, do you have an example?
Desoldering is usually quicker though, touch the pin with the soldering iron and pull it out just like that, no need to line up cutters and no risk of leaving a pin loose on the motherboard ready to short circuit everything when an unsuspecting customer connects power...
Hope you fully recover soon!
The quality of your TH-cam videos is outstanding. You have such a great mix of technical and communication skills. The things you show (and link to in the descriptions) end up being so approachable.
Thank you for putting all this time into the videos for us!
An interesting alternative, but if I were to try this I think I'd go with the Turing 2 for the reasons you mentioned.
I am glad to see that you are feeling better. I know from experience that gastric diseases are no fun. Thank you for all you do!
Hey! I just found out that I need to learn about Ansible for work. I'm absolutely going to check out your videos, on the company dime. Thanks so much.
They're a great set of videos, and is how I learned Ansible. 👍🐧
Great to see you are doing good Jeff
I appreciate all your content, best of luck.
Thank you for marking this as a paid promotion. I've seen too many creators lately that don't count product(that they get to keep, not stuff they have to send back) as payment. Cash didn't change hands, but they were still paid.
Also, I love that trick with the sticky note. I wouldn't have thought of that, but it's brilliant for dealing with so many small microsd cards.
Haha yeah I started doing that about the fifth time I lost track of microSD cards on a cluster build... it's perfect since I always have a pile of Post-Its nearby!
You don't need 5 power supplies when clustering normal Pi4s, just one feisty 5V supply, which you can also then feed to the Ethernet switch. Meanwell comes highly recommended, and they have a 90W/18A model.
You might want to 3D print a MicroSD card puller that you can clip onto the cards to retrieve them. I've seen a bunch of designs of those floating around.
Great to have you well enough to post Jeff. I do feel for companies that have developed such cool stuff for the rpi and especially the cm4. I don’t think I’ve actually seen a cm4 available for sale through an Australian based website at all. Someday!
spoiler it can't well I would hope not it's not any where near the 12k price tag
Man it is good to see you up and around and cranking out these great videos again. I hope you are feeling better. You must be feeling some better since you’re up and around at least. That’s a great thing. Keep cranking man
Good news: no individual power adapters and cables needed. Bad news: the PSU that powers the board is now a single point of failure.
Thanks for the Ceph tutorial, Jeff. I think I now have the intestinal fortitude to try a Ceph deployment myself. And BONUS - what a great hands-on opportunity to really learn and understand Ansible.
The switch and ethernet cable are also single points of failure.
This board is cool and I'm sure it's good for something, I just don't know what...
Happy to see you're back and hope you recovered full. Were waiting on your video and it's quite interesting to see how your RPI cluster have evolved. I will dive more in ceph/ansible using VMs instead...
Great to see you back to making videos. Pace yourself . Thanks to your family for helping you and weathering this episode. You are my favorite tech youtuber. The amount you give back to the community is awesome. Keep up the great work
Happy to see your up and aboard again. I'm waiting for my Turing Pi2 but no CM4s.
1. I am SUPER happy to see you producing content again. It implies you are feeling better!
2. This board appears to be really useful for learning ceph, or other technologies.
3. This board is almost useless because it is the foundation for rpi boards that are mostly unavailable.
4. I have been having some really good luck with the "Le Potato" to fill most of the needs for raspberry pi's. I don't always need the power of an rpi4 and the "potato" is just a bit more power than a 3b, and at a slightly lower power consumption so almost any phone charger will now work very well.
5. Maybe with the whole world going into a recession, maybe these boards will be available soon?
Good to see you back and hope you are doing well. Also; leave some RPIs for us regular folks. Great videos as always!
I just randomly stumbled upon this video and just realized you are "The" geerlingguy. Holy frick mate, your tools have made my life so much easier. Thanks for all the hard work. Great video.
Haha, thanks, glad you found the channel :)
Hey Jeff. Glad to see you finally got this one out. 👍The shortage of Pi's however is sadly becoming a real problem for innovative projects like this.
It is definitely the number one issue with these projects; you either have to spend a bunch of money or have a huge amount of patience to get Pis anymore :(
Tell me about it, I'm waiting since January for my RBPi4 8GB and I actually wanted to buy 7 to replace my x86 server...
Glad to see you are feeling well enough to post again!
Hey Jeff.
Glad to see you uploaded again. Hopefully you are feeling better. I'm sorry things have been rough for you. Take care.
Thank you!
Glad you managed to get to this, hope you're feeling better!
Much better! Going to keep on with my break though, so next video will likely come in two weeks instead of one!
Take your time, health is always more important and the videos are worth the wait
Thank you for showing us the Ansible parts! I found that very helpful.
Welcome back Jeff, wonderful video!
Nice to see you over your sickness, you always make great videos!
4:40 Love that SD card management thingy.
Heh, post it notes have built in sticky pads to hold the cards in place!
I hope you are feeling better man you are my go to for anything Pi related !
Great build and review, Jeff. I hope you're taking it easy though. We can wait longer between videos if it means you getting healthier!
Thanks Jeff. Great video. I came across your post about two months ago and really enjoy the content and your presentation style! Loved your Turning Pi videos and now this one too. I am waiting on my Turing Pi 2 to ship and I'm going to order this board too! Thanks again!
Still waiting on my Turing Pi 2 board, but this board arrives in two weeks. Can't wait! Just watched your video again to help keep me calm. LOL
"DNS is the root of all problems" I would ask for his autograph because of his shirt alone :)
but why., dns is good
So glad to hear you're doing better! Amazing video as always man
Am glad to see an upload. I know this is from before but I hope you are taking care of yourself and that you are getting better.
Get well soon dude! We miss you! :) 🤗🤗🤗👍👍👍❤️❤️❤️
The last pi home server you will ever need. Gigabit lan, 24 Gb of ram, and 6 ssd’s!!!
Just wish it exposed more GPIO-it'd be cool to be able to integrate a HAT into the platform too!
@@JeffGeerling we just may need a bigger board and more pcie lanes for that.
But it's not. It's 6 servers on a single board. Resources cannot be be summed up like that.
I like the part where he says "It's clusterin' time!", and then he clustered all over those Pis
morbius gone F4
This shit is violently unfunny
This is super neat! As a testbed, this makes so much sense. Or just running a small homeserver - its really neat!
However, I am still waiting for just one CM4 on a miniITX board - I have an old thin MiniITX board lying around here and I would love to put a CM4 into that to use it to run HomeAssistant and friends. The board you showed off might actually fit, which is kinda insane. xD But the case's built-in PSU just won't do 100W...
This board only really needs 25W... maybe 30-40W total for overhead. That is, unless you use some really high-powered NVMe drives!
Every video is a commercial from this guy. He comes off really ingenuous.
It's not fair! I've been trying to buy just one CM4 for nearly a year now and it's out of stock everywhere. Now here come's Jeff with his SIX CM4s on a single board! 😭
Ah but I ordered four of these CM4s in October 2020, and the other two in June 2021, heh... they took months to come in!
I have been seeing 2 and 4 GB CM4 modules coming in stock every week or so, if you need one make sure to get notifications from rpilocator!
@@JeffGeerling Yeah, I'm keeping an eye on that site. Hopefully there'll be some UK stock soon. It's super frustrating seeing all these amazing carrier boards on your channel and not being able to make use of any of them. 🙂
BerryBase DE has a load of CM4s in stock, maybe give that a look, although I suppose shipping would be a pain
@@alexbissessur5013 Thanks. Do you know if they would deliver to the UK?
@@hadron2 Sadly you have to be a member of their club and I've not found out how you join it, so it won't let you add to basket ...
I like your clustering vids (touring pie, deskpie, conventional with router poe, etc). The parts described in the des just expensive when shipping to Indonesia (vat, shipping cost). Thats really put me into tears
Wow! Awesome video! Thank you very much for this.
Glad to see you "full of energy", and the cutting edge news.
One of the features of a cluster is a reduction in the single-point-of-failure elements. For home use, you can't do anything about AC power, but your original cluster at least had separate power adapters and ethernet ports, although just a single switch. This board appears to have more single-point-of-failure components.
Now that is a supercomputer, would be even better if the SOCs and other on-Pi chips had heatsinks.
I'm pretty sure I can get my normal heatsinks to fit, I would just need to use a different set of spacers and screws so I can still tighten them directly to the board (the inserts on the board are threaded, I believe M2.5).
If they wanted fast and cheaper they wouldn't be using RPis to begin with.
@@trinidad17 Nvidia jetsons? Pine64 whatevers?
Or perhaps use a submerged liquid cooler like the one just shown on LTT channel
it's still not a supercomputer. even with a lot more boards, the overall limitations are too many for it to reach such levels of performance.
Another good 'un, Geerling!
Certainly hope you are doing much , much better!
Look forward to the next installment of JG!
Thanks!
You could use multipath TCP for getting more speed, and as it's CEPH and storage you could use SMB3 multichannel feature which works with at least Windows 10 and modern Samba. Neither of these solutions require a managed switch.
But both of these require multipath connections to the nodes.
Here, each Pi has only 1 Ethernet connection and the switch has two ports exposed.
Connect those two ports to another switch and you have a loop, causing a smarter switch to disable one of the ports or when using a not too smart switch will just lock up the network...
So SMB multipath won't work here unfortunately, that is why it is hard to understand why they chose an unmanaged switch...
@@someguy4915 Duh that was a Homer Simpson moment. However with a suitable managed switch you could put down MAC based ACL's that would mask half the PI's down one port and half down the second port, to get more upstream speed. That should shut down the loop, but yeah it's slightly wacho.
@@jonathanbuzzard1376 True but even then you'd need a bunch of routing etc. to make sure that traffic sent on Port 1 also returns through Port 1 etc. which while certainly possible is just a bit of a headache when a simple managed switch would fix all of that and provide so much more functionality.
Besides LACP port bonding you'd also get VLAN support, now you could just have the Pi's spit through VLAN tags and hope that the buffer of the switch is big enough, usually this does work, but it is unsupported and a great way to get very complex network issues (don't ask how I learned that lesson... :P)
Thank you so much Jeff! In just this post you let me discover 3 solutions I ignored. Great post.
Finally getting to what I wanted with Ceph. Really cool I knew you would come around. HA!!! I have some reading to do with Ceph; I did not know it existed.
Hopefully I'll get some more time to dive into Ceph; it impressed me with how simple it was to get going, and the concept is similar to Gluster (which I'm more familiar with), but I need a bit more time with it before I'd be comfortable going deeper in a video!
@@JeffGeerling I'd have to see if it supports a fail over head node.
Hmm, I guess I would prefer a SFP+ slot for networking so you can use 1GE, 2,5GE, 5GE or 10GE depending on need if you can get 1GE from each pie.
I do like SFP+, though just having an SFP+ cage wouldn't give faster speeds, the switch chip would also need an upgrade. 2.5G could be realistic but going to 5 or 10G on this board design might have issues with heat or signal routing.
The other difficulty with this board in particular is there's not enough physical room for SFP+; you'd have to take out one of the Pis to do it.
@@JeffGeerling Oh, yes I see now, you are right. The advantage with a SFP+ slot is that it can run on less power than a 10GE copper connection. But perhaps one or two 2,5GE slots would be good? The Intel I225 should perhaps fit well? It had issues in the first two revisions. But yes the whole switch thing is an issue as well.
@@JohnAngelmo But that Intel is a NIC, not a switch, you'd need to replace the entire Realtek switch chip for something more useful (managed and 2.5-10GbE) before adding SFP+ has any benefit.
This video forces you to want to purchase more Pi's. I love this board soooo much!
Glad to see you up and running.😁
Glad to see you back. 💯
Looks like Jeff is feeling better! Thank goodness.
I hadn't heard of Ceph before, super cool combo of tech!
Jeff: Im feeling a bit sick and I need to take some time off.
Also Jeff: *dropping one video after another*
👏🏼♥️💩💨
Heh this video was close to ready a couple weeks ago when I wound up in the hospital. It'll be a bit before I'm back on a normal schedule :(
I'm happy that your back.
Love your channel. Keep up the great work.
You weren't kidding when you said it had it's trade offs. Honestly I'd prefer the Turing Pi, as compatibility with Jetson platform, and ATX power compatibility would make it a great fit in home-labs; my biggest gripe with the Turing is how IO is handled by specific slots rather than one master.
Jeff, single-handedly keeping the Raspberry-Pi industry afloat
HYPE HYPE HYPE!
Good to see you back at it Jeff
Thanks Jeff! I am going to get on of these. Thanks for leading the way and this is so cool!. Oh, and I hope you are feeling better!
Much better! Though still taking it easy right now, as part of that is due to some of the medications I'm on... always hard to figure out how much improvement comes from *actual* improvement in underlying health!
Awesome, I loved the W.O.P.R. reference, I just watched that movie for the umpteenth time not too long ago.
Shall we play a game?
just glad your feeling better
2:50 The board has it's own little PMU -> The board has ITS own little PMU.... for the rest great videoarticle!
4:06 You don't get much air trough those hole when you try to suck trough them. You will only get a fan that spins at a high rpm without any air going trough. You can easily test it by holding a fan and with your hand you starts to covering the side the fan suck from. You will now hear the fan spinning up because less air going trough. Do the same thing on the blow side. You will not be able to block the outgoing air that easy.
Oh, Christmas can’t come soon enough🥰
Finally we in Germany get to have cm4s! They once hat the "rich kid" cm4 like a month ago with everything maxxed!
I see your feeling better Jeff. I hope you stay that way for bit.
I can see this working really well in distributed TV editing storage, particularly for outside broadcast or mobile facilities. Low power requirements, easily configured to access 6TB at time. The whole thing, including monitor, keyboard and mouse could be put in a flight case and carried to the next job. The only downside is the gigabit connecting. 150 minutes minimum per terabyte is slow but that's where the price and portability come in. Two or three cases per reporting team would keep everything flowing nicely.
This is yet another step forward, I love it. Maybe in the future we'll see more home super computers. I'm glad you recorded this earlier, hopefully you're still taking it easy. And I'll never stop using the SI prefixes to refer to 1024 increments of bytes. I don't care if hard drive manufacturers cheat, computers are special and it makes no sense to make up new terms this late in the game, aside from the fact that they sound dumb.
Whenever I say mebibyte, I think of a little toddler trying to say 'megabyte' :D
'supercomputer' , that's stretching the term a wee bit
@@tomservo5007 Not really, considering how they were built in the past. It's just that processors have gotten so much better now that we, most of us anyway, have as much computing power in our pockets as supercomputers did in the 70's.
@@JeffGeerling I wasn't going to go there, but you said it, not me.
@@anon_y_mousse And as a note, this board's linpack scores would put it somewhere in the Top500 list from around the year 2000... so it just depends on your definition of 'Supercomputer' ;)
It's a different board, you are looking good. I hope everything has become more manageable and your health is back.
Keep baking new Pies.
Nice to see you up and about.
great content as usual. It's also good to see that you're feeling better.
so encouraging watching all these people with their bunch of pis... I can't even get one
I seem to recall that a netgear (dumb) switch that I owned (may still be in a box here some where) actually supported link agregation, even if it didn't have management or vlan support. So it's possible that getting link aggregation on the board may just be a firmware update, but I'm kind of fond of the vlan support as it allows me to move management of devices off of the vlan that the user data is running over. Hopefully the manufacturer will get that all worked out.
God bless you jeff, praying for you buddy,we appreciate everything you do , your health comes first, your pretty awesome to share with others🤓
Yet another great video. Hope you're feeling better, mate.
For a fan controller you could fit a PWM 4-Wire Fan Temperature Controller board internally.
Thx!
Nice to see you back on business 😉
Can the Pis still be overclocked in such a setup? If so, can they be individually OCed?
2:58 A minor mod with transistors can take care of the power LED and fan always being on. Throw an NPN transistor on the ground of each, and connect the transistors' bases to an unused, power-switched voltage out (with an appropriate resistor in between). The USB header would work fine if you can't find any other switched power rail.
They can be OCed, and I've gotten stable overclocks to 2.0 GHz, sometimes 2.2 GHz with good cooling.
I noticed with my Pi 4b that it was thermal throttling with the regular plastic cases most kits provide, at the stock 1.8 GHz but that when I switched to a passive cooling metallic case (Flirc) I was able to overclock it to 2.0 GHz and it didn't thermal throttle at all at that speed.
I'd like to see a sleek passive cooling case for this thing. It would probably do a decent job distributing the heat, being so flat, having so much surface area for the 6 compute modules.
@@JeffGeerling Sorry, I was unclear. I meant can they still be overclocked on this board, and if so can you set different clocks for the different Pis?
For my standalone Pis, I overclock almost all of them.
@@JeffGeerling Glad that you've gotten some of your wind back, btw.
@@davidg5898 Yes, I tested a 1.8 and 2.0 GHz overclock on this board and it worked fine.
Now i understand what monitoring system i do for. Interesting work have my older colleagues.
That Ansible wow made my day.
Great video as usual Jeff. Hope you are feeling better. I look forward to more big brained stuff from you in the future :-)
6:58 "How am I going to manage six Raspberry Pi's?"
We all knew exactly where you were going with this, Mr. Ansible master.
Great video as always, thanks Jeff !
Nice to see you back on your feet. :)))
Good to see you again!
Hope the poohtubes feeling better!
Somewhat!
If this isn't art, if this isn't beautiful, I don't know what is.
Good luck getting enough CM4's !! Even if you find some available..there are often order limits of 1 per customer.
Wow Jeff, this projects looks soo cool, I think I will try to reproduce it! :) Thank you, fantastic work as always.
Fascinated by the idea of something like this or the Turing Pi for a little home cluster to do some Docker experiments. This beats the Turing Pi by virtue of being available but on the other hand I'm not sure if it is worth going all-in on six CM4s, six SD cards and six m.2 drives vs just going to state surplus and picking up an old pc or two.
Glad to see youre doing better!
I like this guy, he doesn't hide that 50% of his appeal for this thing are the beepy boopy lights
Why do raspberry pi videos bang more than most other computer videos... It's insane
Protip on the SD cards. Just put little scotch tape tails on em. Makes em easy to pull out of tight spots. Just don't do that in an application where they may get snagged and pulled accidentally.
whenever I see pis clustered like this, I definitely find myself wondering about storage clustering like you explored here. If this board had SATA instead of nvme, I'd be very tempted to try and build something with larger HDDs for some NAS testing
Interesting.
I wonder what use cases there are for this, for people who don't need the performance of the $12k setup but who could benefit off of having so many cores with efficient networking.