6-in-1: Build a 6-node Ceph cluster on this Mini ITX Motherboard
ฝัง
- เผยแพร่เมื่อ 5 พ.ค. 2024
- It's time to experiment with the new 6-node Raspberry Pi Mini ITX motherboard, the DeskPi Super6c! This video will explore Ceph, for storage clustering, since the board has six NVMe SSDs on the bottom, thanks to the video's sponsor, KIOXIA!
Products used in this build (some links are affiliate links):
- DeskPi Super6c: amzn.to/3psoOmE
- Raspberry Pi Compute Module 4: rpilocator.com
- Goodisory MX01 Mini ITX case: amzn.to/3AsCtQW
- Arctic F12 Silent 120mm fan: amzn.to/3AmTOul
- SanDisk 32GB Extreme microSD Cards: amzn.to/3CaJyXh
- Kioxia XG6 NVMe SSD: amzn.to/3zXxeHr
Open source code to set up the cluster is in this GitHub repository:
github.com/geerlingguy/deskpi...
And the IO shield design: www.thingiverse.com/thing:546...
My book and TH-cam series on Ansible!
- Book: www.ansiblefordevops.com
- TH-cam series: • Ansible 101
Support me on Patreon: / geerlingguy
Sponsor me on GitHub: github.com/sponsors/geerlingguy
Merch: redshirtjeff.com
2nd Channel: / geerlingengineering
#RaspberryPi #Ceph #homelab
Contents:
00:00 - It's CLUSTERIN' Time!
00:50 - DeskPi Super6c
03:48 - The build
05:33 - It boots!
06:30 - Ansible orchestration
07:43 - Distributed storage
09:08 - Ceph setup and benchmarks
11:16 - Can it beat a $12k appliance?
11:48 - vs Turing Pi 2
12:35 - What it's good for - วิทยาศาสตร์และเทคโนโลยี
Building a Ceph cluster on a bunch of Raspberry Pis was a fun project, and I think perfect for this board! If you're looking for some Pis, they pop up from time to time on rpilocator.com (no affiliation, I just use that site to check up on Pi stock around the world).
Thanks especially to KIOXIA for sending the NVMe SSDs to fill in the rest of this board - check out their XG6 and their new XG8 if you need a great, reliable NVMe drive for your computer.
I'd been working on this video since a few weeks ago (before the hospitalization), so I was able to get it out for this normal weekly release. I'll probably be taking a break and won't be posting a video next Wednesday-just check back the week after, and in the mean time, go check out Geerling Engineering, my 2nd channel! th-cam.com/users/GeerlingEngineering
I really like your videos because you answer all the relevant questions directly and always get to the point. So thank you, Jeff, for always covering fascinating topics and products. And best greetings to your wife and dad! I really hope you feel better soon!
Can you use something like this as a desktop replacement? I am not clear on if you can configure Linux to use all these Pis as 'one' system transparent to the user and just do normal desktop stuff with it.
AWESOME video! Superb content. But right now, TAKE A BREAK, take care of your health, please. That's the priority. We all are cheering for you, Jeff. God bless you and be better soon. 🙏🏻
I'd love to see some of the points per day it could generate on folding@home if that is possible.
Truly impressive video especially given that most of the time your being ill in bed
But I have no idea how your keeping Red shirt Jeff under control handcuffs? Video games? A really long leash?
So back to my bed to watch you till I fall asleep
If anyone is wondering why there are no rpis in stock its because Jeff has them ALL!
Heh... those are literally all the Lite CM4s I have. I have a couple eMMC models too but I typically stick to the Lite modules since they're easier to re-flash (don't have to wait for slow eMMC writes over USB).
No doubt!!!!
@@JeffGeerling totally sus how he has so many LoL he works for RPi maybe 🤔
@@JeffGeerling That's what someone who bought *all* the Pis would say! :D
dont forget red "shirt" jeff
It is good to see you back healthy. Hope you are fully recovered Jeff!
Like always awesome content +1.
Not quite fully recovered, but we'll get there!
@@JeffGeerling 👍🏼Hang in there! Hope it's (relatively) easy for you and family! 😎✌🏼
@@JeffGeerling I was diagnosed with Collitis the day before you posted your video about your colostomy. Gotta say knowing someone I look up to has the same/similar issue to me healthwise has made it easier to cope.
@@JeffGeerling real clone wars is rhinos like bush=nute gunray invade naboo
bush could of prevent europe migrant crisis since obama target Libya Sudan Somalia Syria was originally bush admin rumsfield idea pnac!
we can give all medicare by reduce warfronts, no more allowing new daca abusers, force senators to cut their own wealth in gov by half if they want to keep above 70% of current daca population so inflation/living cost stay reasonable
wesley clark foreshadow reveal 2000 to 2012 all rig for kill iraq to syria
th-cam.com/video/_mrJRHwbVG8/w-d-xo.html
establishment kill 50 in vegas/portland, thugs attack with stand down cops san jose/charlotte, burn loot several months, sabotage afgan withdraw using russia bounty smear to give taliban equip, crash car in to wisconsin parade thanks to nbc follow jury bus smearing ritten house too
th-cam.com/video/UxoL8tHSa7g/w-d-xo.html
bush 14y ago said add ukraine to nato with nuland f eu coup 2014 support =
th-cam.com/video/nTQ3D1a-j20/w-d-xo.html
current ukraine gov is proxy since cia drew red line just like did in syria earlier arming rebels telling russia not to interfere while zelensky ethnic cleanse donbass region 7y=
th-cam.com/video/ta9dWRcDUPA/w-d-xo.html
th-cam.com/video/IBeRB7rWk_8/w-d-xo.html
The quality of your TH-cam videos is outstanding. You have such a great mix of technical and communication skills. The things you show (and link to in the descriptions) end up being so approachable.
Thank you for putting all this time into the videos for us!
One of the features of a cluster is a reduction in the single-point-of-failure elements. For home use, you can't do anything about AC power, but your original cluster at least had separate power adapters and ethernet ports, although just a single switch. This board appears to have more single-point-of-failure components.
An interesting alternative, but if I were to try this I think I'd go with the Turing 2 for the reasons you mentioned.
I am glad to see that you are feeling better. I know from experience that gastric diseases are no fun. Thank you for all you do!
Happy to see you're back and hope you recovered full. Were waiting on your video and it's quite interesting to see how your RPI cluster have evolved. I will dive more in ceph/ansible using VMs instead...
Great to see you back to making videos. Pace yourself . Thanks to your family for helping you and weathering this episode. You are my favorite tech youtuber. The amount you give back to the community is awesome. Keep up the great work
Thank you for showing us the Ansible parts! I found that very helpful.
Love this! Going to build a public level 8 plan9 grid with it. Mostly for research for IoT/scada and Mobile Edge networking via plan9 design principles but also for individuals to do computational time / research.
Happy to see your up and aboard again. I'm waiting for my Turing Pi2 but no CM4s.
Great to have you well enough to post Jeff. I do feel for companies that have developed such cool stuff for the rpi and especially the cm4. I don’t think I’ve actually seen a cm4 available for sale through an Australian based website at all. Someday!
spoiler it can't well I would hope not it's not any where near the 12k price tag
Hey! I just found out that I need to learn about Ansible for work. I'm absolutely going to check out your videos, on the company dime. Thanks so much.
They're a great set of videos, and is how I learned Ansible. 👍🐧
Great to see you are doing good Jeff
5:19 I don't know why they're snipping off the pin. it's a really easy task to desolder the single pin and make it look good.
Probably they enjoy the challenge of the "don't let the pin fly into your eyeball when you snip it" game!
Red shirt jeff has to have his moments too
@@JeffGeerling It is a fun game but losing it sucks and even when winning you then get to play the second round of the game which is named 'Desperately search the floor on your hands and knees because you know that if you dont find the pin you will feel it in three days when you step on it barefoot...'
So many times.....
It's way quicker to cut than to desolder. Desolder on 100 boards and I'm pretty sure you will be quite tired. Even lots of massproduced motherboards has cut pins.
@@perwestermark8920 I've never seen a mass-produced motherboard with snipped pins, do you have an example?
Desoldering is usually quicker though, touch the pin with the soldering iron and pull it out just like that, no need to line up cutters and no risk of leaving a pin loose on the motherboard ready to short circuit everything when an unsuspecting customer connects power...
Man it is good to see you up and around and cranking out these great videos again. I hope you are feeling better. You must be feeling some better since you’re up and around at least. That’s a great thing. Keep cranking man
You don't need 5 power supplies when clustering normal Pi4s, just one feisty 5V supply, which you can also then feed to the Ethernet switch. Meanwell comes highly recommended, and they have a 90W/18A model.
You might want to 3D print a MicroSD card puller that you can clip onto the cards to retrieve them. I've seen a bunch of designs of those floating around.
Thanks Jeff. Great video. I came across your post about two months ago and really enjoy the content and your presentation style! Loved your Turning Pi videos and now this one too. I am waiting on my Turing Pi 2 to ship and I'm going to order this board too! Thanks again!
Still waiting on my Turing Pi 2 board, but this board arrives in two weeks. Can't wait! Just watched your video again to help keep me calm. LOL
Another good 'un, Geerling!
Certainly hope you are doing much , much better!
Look forward to the next installment of JG!
Good to see you back and hope you are doing well. Also; leave some RPIs for us regular folks. Great videos as always!
Great video as always, thanks Jeff !
Good news: no individual power adapters and cables needed. Bad news: the PSU that powers the board is now a single point of failure.
Thanks for the Ceph tutorial, Jeff. I think I now have the intestinal fortitude to try a Ceph deployment myself. And BONUS - what a great hands-on opportunity to really learn and understand Ansible.
The switch and ethernet cable are also single points of failure.
This board is cool and I'm sure it's good for something, I just don't know what...
So glad to hear you're doing better! Amazing video as always man
Thanks for sharing your I/O shield design.
Nice to see you over your sickness, you always make great videos!
1. I am SUPER happy to see you producing content again. It implies you are feeling better!
2. This board appears to be really useful for learning ceph, or other technologies.
3. This board is almost useless because it is the foundation for rpi boards that are mostly unavailable.
4. I have been having some really good luck with the "Le Potato" to fill most of the needs for raspberry pi's. I don't always need the power of an rpi4 and the "potato" is just a bit more power than a 3b, and at a slightly lower power consumption so almost any phone charger will now work very well.
5. Maybe with the whole world going into a recession, maybe these boards will be available soon?
Good video, it's amazing you managed to build all those, very fascinating.
Great build and review, Jeff. I hope you're taking it easy though. We can wait longer between videos if it means you getting healthier!
I hadn't heard of Ceph before, super cool combo of tech!
Glad to see you "full of energy", and the cutting edge news.
Great to see you looking better Jeff. Great content and presentation on an interesting board - I am looking at what and where I can put something like this to work but I may favour the Turing board.
Thank you so much Jeff! In just this post you let me discover 3 solutions I ignored. Great post.
just glad your feeling better
Thanks Jeff! I am going to get on of these. Thanks for leading the way and this is so cool!. Oh, and I hope you are feeling better!
Much better! Though still taking it easy right now, as part of that is due to some of the medications I'm on... always hard to figure out how much improvement comes from *actual* improvement in underlying health!
Nice to see you up and about.
I hope you are feeling better man you are my go to for anything Pi related !
Wow! Awesome video! Thank you very much for this.
great content as usual. It's also good to see that you're feeling better.
Welcome back Jeff, wonderful video!
Glad you managed to get to this, hope you're feeling better!
Much better! Going to keep on with my break though, so next video will likely come in two weeks instead of one!
Take your time, health is always more important and the videos are worth the wait
Wow Jeff, this projects looks soo cool, I think I will try to reproduce it! :) Thank you, fantastic work as always.
Hey Jeff. Glad to see you finally got this one out. 👍The shortage of Pi's however is sadly becoming a real problem for innovative projects like this.
It is definitely the number one issue with these projects; you either have to spend a bunch of money or have a huge amount of patience to get Pis anymore :(
Tell me about it, I'm waiting since January for my RBPi4 8GB and I actually wanted to buy 7 to replace my x86 server...
Glad to see you up and running.😁
This was brilliant, love your content!
I can see this working really well in distributed TV editing storage, particularly for outside broadcast or mobile facilities. Low power requirements, easily configured to access 6TB at time. The whole thing, including monitor, keyboard and mouse could be put in a flight case and carried to the next job. The only downside is the gigabit connecting. 150 minutes minimum per terabyte is slow but that's where the price and portability come in. Two or three cases per reporting team would keep everything flowing nicely.
Glad to see you are feeling well enough to post again!
Am glad to see an upload. I know this is from before but I hope you are taking care of yourself and that you are getting better.
Get well soon dude! We miss you! :) 🤗🤗🤗👍👍👍❤️❤️❤️
Man, great job on this video, very detailed and I found it pretty interesting. I'd like to see more stuff on Kubernetes (or similar) and what you can actually do with a cluster.
Love your channel. Keep up the great work.
I like your clustering vids (touring pie, deskpie, conventional with router poe, etc). The parts described in the des just expensive when shipping to Indonesia (vat, shipping cost). Thats really put me into tears
Hey Jeff.
Glad to see you uploaded again. Hopefully you are feeling better. I'm sorry things have been rough for you. Take care.
Thank you!
I appreciate all your content, best of luck.
I see your feeling better Jeff. I hope you stay that way for bit.
Now i understand what monitoring system i do for. Interesting work have my older colleagues.
It's not fair! I've been trying to buy just one CM4 for nearly a year now and it's out of stock everywhere. Now here come's Jeff with his SIX CM4s on a single board! 😭
Ah but I ordered four of these CM4s in October 2020, and the other two in June 2021, heh... they took months to come in!
I have been seeing 2 and 4 GB CM4 modules coming in stock every week or so, if you need one make sure to get notifications from rpilocator!
@@JeffGeerling Yeah, I'm keeping an eye on that site. Hopefully there'll be some UK stock soon. It's super frustrating seeing all these amazing carrier boards on your channel and not being able to make use of any of them. 🙂
BerryBase DE has a load of CM4s in stock, maybe give that a look, although I suppose shipping would be a pain
@@alexbissessur5013 Thanks. Do you know if they would deliver to the UK?
@@hadron2 Sadly you have to be a member of their club and I've not found out how you join it, so it won't let you add to basket ...
Thank you very much, Jeff. I have been waiting for this video for a long time. BTW, I hope you’re doing well now and got better. Also, can you make a video about the Ansible config you used in this video and explain more about it in depth? I would really appreciate it. Great job as always.
This board looks awesome!
4:40 Love that SD card management thingy.
Heh, post it notes have built in sticky pads to hold the cards in place!
Yet another great video. Hope you're feeling better, mate.
Love that WOPR ref.
Fascinated by the idea of something like this or the Turing Pi for a little home cluster to do some Docker experiments. This beats the Turing Pi by virtue of being available but on the other hand I'm not sure if it is worth going all-in on six CM4s, six SD cards and six m.2 drives vs just going to state surplus and picking up an old pc or two.
Glad to see you back. 💯
That Ansible wow made my day.
This is super neat! As a testbed, this makes so much sense. Or just running a small homeserver - its really neat!
However, I am still waiting for just one CM4 on a miniITX board - I have an old thin MiniITX board lying around here and I would love to put a CM4 into that to use it to run HomeAssistant and friends. The board you showed off might actually fit, which is kinda insane. xD But the case's built-in PSU just won't do 100W...
This board only really needs 25W... maybe 30-40W total for overhead. That is, unless you use some really high-powered NVMe drives!
I just randomly stumbled upon this video and just realized you are "The" geerlingguy. Holy frick mate, your tools have made my life so much easier. Thanks for all the hard work. Great video.
Haha, thanks, glad you found the channel :)
God bless you jeff, praying for you buddy,we appreciate everything you do , your health comes first, your pretty awesome to share with others🤓
Great video as usual Jeff. Hope you are feeling better. I look forward to more big brained stuff from you in the future :-)
I think it is a nice concept. If I would be designing one, I would make Pis plus vertically (but with more slots than Turing Pi), but I do see benefits of this design too.
Glad to see youre doing better!
I'm happy that your back.
The last pi home server you will ever need. Gigabit lan, 24 Gb of ram, and 6 ssd’s!!!
Just wish it exposed more GPIO-it'd be cool to be able to integrate a HAT into the platform too!
@@JeffGeerling we just may need a bigger board and more pcie lanes for that.
But it's not. It's 6 servers on a single board. Resources cannot be be summed up like that.
Finally getting to what I wanted with Ceph. Really cool I knew you would come around. HA!!! I have some reading to do with Ceph; I did not know it existed.
Hopefully I'll get some more time to dive into Ceph; it impressed me with how simple it was to get going, and the concept is similar to Gluster (which I'm more familiar with), but I need a bit more time with it before I'd be comfortable going deeper in a video!
@@JeffGeerling I'd have to see if it supports a fail over head node.
Excellent presentation.
whenever I see pis clustered like this, I definitely find myself wondering about storage clustering like you explored here. If this board had SATA instead of nvme, I'd be very tempted to try and build something with larger HDDs for some NAS testing
HYPE HYPE HYPE!
Good to see you back at it Jeff
Now that is a supercomputer, would be even better if the SOCs and other on-Pi chips had heatsinks.
I'm pretty sure I can get my normal heatsinks to fit, I would just need to use a different set of spacers and screws so I can still tighten them directly to the board (the inserts on the board are threaded, I believe M2.5).
If they wanted fast and cheaper they wouldn't be using RPis to begin with.
@@trinidad17 Nvidia jetsons? Pine64 whatevers?
Or perhaps use a submerged liquid cooler like the one just shown on LTT channel
it's still not a supercomputer. even with a lot more boards, the overall limitations are too many for it to reach such levels of performance.
Amazing product you brought to us today 😉
Thank you for marking this as a paid promotion. I've seen too many creators lately that don't count product(that they get to keep, not stuff they have to send back) as payment. Cash didn't change hands, but they were still paid.
Also, I love that trick with the sticky note. I wouldn't have thought of that, but it's brilliant for dealing with so many small microsd cards.
Haha yeah I started doing that about the fifth time I lost track of microSD cards on a cluster build... it's perfect since I always have a pile of Post-Its nearby!
great review
Every video is a commercial from this guy. He comes off really ingenuous.
This video forces you to want to purchase more Pi's. I love this board soooo much!
I seem to recall that a netgear (dumb) switch that I owned (may still be in a box here some where) actually supported link agregation, even if it didn't have management or vlan support. So it's possible that getting link aggregation on the board may just be a firmware update, but I'm kind of fond of the vlan support as it allows me to move management of devices off of the vlan that the user data is running over. Hopefully the manufacturer will get that all worked out.
Always love your content dude
You could use multipath TCP for getting more speed, and as it's CEPH and storage you could use SMB3 multichannel feature which works with at least Windows 10 and modern Samba. Neither of these solutions require a managed switch.
But both of these require multipath connections to the nodes.
Here, each Pi has only 1 Ethernet connection and the switch has two ports exposed.
Connect those two ports to another switch and you have a loop, causing a smarter switch to disable one of the ports or when using a not too smart switch will just lock up the network...
So SMB multipath won't work here unfortunately, that is why it is hard to understand why they chose an unmanaged switch...
@@someguy4915 Duh that was a Homer Simpson moment. However with a suitable managed switch you could put down MAC based ACL's that would mask half the PI's down one port and half down the second port, to get more upstream speed. That should shut down the loop, but yeah it's slightly wacho.
@@jonathanbuzzard1376 True but even then you'd need a bunch of routing etc. to make sure that traffic sent on Port 1 also returns through Port 1 etc. which while certainly possible is just a bit of a headache when a simple managed switch would fix all of that and provide so much more functionality.
Besides LACP port bonding you'd also get VLAN support, now you could just have the Pi's spit through VLAN tags and hope that the buffer of the switch is big enough, usually this does work, but it is unsupported and a great way to get very complex network issues (don't ask how I learned that lesson... :P)
You weren't kidding when you said it had it's trade offs. Honestly I'd prefer the Turing Pi, as compatibility with Jetson platform, and ATX power compatibility would make it a great fit in home-labs; my biggest gripe with the Turing is how IO is handled by specific slots rather than one master.
Thanks, Jeff. I can see this being used for a micro computational cluster for use at the edge. I like to take low cost clusters and then benchmark them to see if/where they would have shown up on the Top 500 Supercomputers list. That is, what year. For example, your configuration might result in a “score” of 1998 - meaning that 6-node cluster would have been a top 500 supercomputer back in 1998.
Haha amazing video! Glad to see your feeling better
I like this techy RPI stuff
so encouraging watching all these people with their bunch of pis... I can't even get one
Looks like Jeff is feeling better! Thank goodness.
Oh, Christmas can’t come soon enough🥰
For a fan controller you could fit a PWM 4-Wire Fan Temperature Controller board internally.
That unmanaged network switch is a real killer for the utility of this unit outside of personal users/home labs. It would have been trivial to add a smartswitch with basic management, or a managed chip from realtek with a serial connection to the first pi or a console port even.
Thanks!
The main reason to have 2 ports at the back is the chip supports 8 ports, and 6 are used by Pis, so 2 at the back are natural way to have it. It allows daisy chaining multiple clusters for example. Useful if you do not have much network traffic.
Ooh, I'm really tempted by this. But I'll wait until I can get the Pis. Feels like the version two of this board or the cm5 might be out before I can buy them
Awesome, I loved the W.O.P.R. reference, I just watched that movie for the umpteenth time not too long ago.
Shall we play a game?
You rock, brother.
Even without clustering I can see a lot of fun server/headless use scenarios for having 6 computers attached to the network while taking very little power and physical space. I can even see uses for having the two ethernet ports such as having them connected to two separate switches so two different networks can each talk to those servers but nor each other directly.
Think about things like having one pi dedicated to just doing git pulls and nightly builds of software, one as a media server for the home, the one with HDMI out being for emulation or a media player or a low end desktop, one dedicated to testing software builds before deploying to other machines separate from the one that just auto build nightlies for you, and two for a low end server with a redundant backup in case of a software config screwup or data corruption.
Those port act as a dumb switch, so connecting them to 2 switch will end like you used a wire between the switch NOT 2 separates networks
Basictly, the board is a 8 port dumb switch, where 6 are connected to the pi.