Jeff, Thank you for your honesty in who sent you what and how things were paid for. That’s very refreshing in this space and shows you value your audience
It's not even legal to _not_ disclose things. Sadly, many creators aren't up-front about where they get things. Honesty's the best policy... and the best way to ensure the FTC doesn't have to get involved haha.
@ave14401 im not sure who you see doing that on here, but that would be an easy way to lose your channel/sponsor/pay a fine. But if its someone who kept on repeatedly not disclosing advertisers, I wouldn't be surprised if they got locked up...
@@ave14401 Sadly, this is the case-honest sponsors actually state the requirements in a contract they'll have a creator sign. But there are a lot of dishonest sponsors too, who don't tell creators to do that, and hope any fallout doesn't touch them.
I remember when data rates went from 110 baud to 300 baud. That speed increase was amazing for the time. And years later the first 14,400 baud modem came to market at $14,400 each. These young folks think 1Gbps is slow. They do not know what slow is. It did teach us patience.
My wallet wishes I had never discovered your channel ... keep up the great coverage of your SBC adventures, you play in the same space I'm interested in.
I've done a fair bit of cluster computing, on big clusters like Stampede at the University of Texas, so I'm familiar with the desire of cluster creators to have fast shared filespace and high speed interconnect fabrics etc - *but* - at least for all the real projects (as opposed to demos or benchmarks) that I've used clusters for in real life, I've always been able to structure my code so that high speed interconnects and fast shared filespace are just are not needed. Even for applications where there was big data rather than just a lot of CPU being used. My observation is that these extreme hardware facilities are mostly used by people who don't make the effort to optimise their application for parallelism and who try to shoehorn code that was really designed with uniprocessing in mind, into a multiprocessing environment without taking on the necessary restructuring. Now I'm retired, when I need to throw CPU at a problem I just farm it out to a few dozen regular Pi's and a couple of spare x86 portables I have around the house, running over Wifi, with one regular NFS mount to supply shared files across the lightweight cluster. In fact - although I do have proper cluster MPI software installed, I generally can get by with just kicking off tasks using "ssh". So I mention all this to suggest that perhaps the features you describe and the cost of the system you're using is perhaps a little overkill, that could be avoided by a little more coding effort. An interesting approach for a subsequent video might be in terms of computation done per dollar rather than just what is the fastest shinyest new hardware available?
Sadly when you talk to anyone about making code better versus beefing up the hardware, 99 times out of 100 you'll be told the hardware's easier :D One of the reasons I love Pi clusters is I've changed my own site architecture many times to make it faster (avoiding disk IO where I can, caching in RAM on individual nodes, that sort of thing), and made big sites scale (think millions of dynamic page views per hour) on relatively modest AWS resources... all because I would run the same application on my Pi cluster that we were running on $20,000/month servers :D But not every project allowed me the time to optimize. Sometimes the plan was to build up hardware for a month, then tear it all down, and they were happier doing that spending $100k+, than to spend an extra two weeks optimizing the site (for a lot less).
Read "In Search of Clusters", a fine book. Amdahls Law; show me where the data is and I'll show you where the computing has to take place. At some point all parallel computations reduce to a set of serial instructions. It's the law. Clusters live and die by the speed of their switching fabric, and that will always be the case.
The RK1 boards are crazy fast! I have two of these TuringPi 2 boards, running 2 RK1s and 2 CM4s each. A sort of P-module, E-module configuration. 92GB of RAM and 82 GHz of CPU. I use Hashicorp Nomad to orchestrate tasks using a combination of the Docker and Exec drivers. Ceph to manage the persistent volumes. Soon I'll be able to retire my power-hungry my Dell R710s in favor of a system that draws < 200W. Couldn't be happier with the results!
Thank you for showing the debugging steps with the cluster, including how to see which pod had stalled and finding error logs using "describe". You can watch 100 happy path tutorials without ever seeing these things.
It won't. But I do recommend considering getting it either used or older gen like Raspberry Pi 3 for example which is lot cheaper than Pi 4 or 5, especially if you don't need the performance of those.
Pi 3 or Pi 4 (or even Zero 2W) are probably the sweet spot for value in the Pi lineup right now. If they release a 2GB Pi 5 for $40, that would be a great value.
@@JeffGeerling That's be like putting a v4 in an old muscle car. We all know that we don't build these things because we need them. We build them because we NEED them
Haha if you know, you know. Usually you go down every other avenue and then the 100th google search you get a path that leads to a solution. I got lucky in this case and skipped the first 99 dead ends!
Break out the soldering iron and swap the direction those power leads exit the PSU board. No more fan-blade jeopardy, no more stressing of the board connector.
When my son was in cub scouts, he sold popcorn like all the scouts did. We borrowed a wagon to put the popcorn in, and found a cardboard box that fit perfectly! As I was walking behind him I realized that we'd have to ditch the cardboard box. It was one that we got at the local library book sale (They were selling unpopular books to make room for more books.) Written across the back of the box someone wrote "ADULT BOOKS" in large letters. That phrase has a completely different meaning outside of libraries. Lol!
Really seems like the Turing Pi board could benefit from having a switch chip. Even if you still only had 1 Gb or 2.5 Gb connectivity to the external network, having a 40/80Gb switching fabric between nodes would make something like Ceph internal replication or any node to node comms in the cluster much better. Of course, having a 10 Gb SFP+ would also be ideal, since so much 10 Gb fiber gear is plummeting in price these days, and might obsolete the need for finding another big chip and traces on the board, so that might the smarter play.
I do hope they do a Turing Pi 3, where they maybe bump the price a little, but include some features that will let the board really go all-out on performance. The problem is as the price of the board sneaks past $200, $300, or more, the audience gets even more limited :(
@@JeffGeerling I hear you, but I think this space could do with some better comms around what makes sense and what you _can_ do with these platforms. Like, a 4 node cluster on the Turing Pi 2 at $1500 does not make sense for most home lab operators and especially not the folks running single purpose GPIO heavy configs, but $1500-2000 for a 4 node Ceph backed lightweight k8s platform with a slough of groupware appliance containers is a stellar HA homelab / business server in a single box. For the load of businesses I know trying to de-SaaS their OpEx while still looking for a reliable solution, this is barking up the right tree for sure.
In my m.2 slots I now have adapters that go to a 10Gb single network card. I have proxmox on each node and the 10Gb is being used for iscsi for the vms/containers. Its working great.
I've been using the turingpi 2 with 3x RK1's for a few weeks, and really enjoying it. Ive been managing it using the Ansible for DevOps and Ansible for Kubernetes books you put out, so many thanks for that!
Love these tiny videos. As someone who does Van Life traveling full time in my van, I’m always looking for ways to run NAS and cluster solutions in a compact “portable” way. Especially love it when you can run on 12v, 24v, or 48v DC.
this stuff does not make sense when for this price you can buy 3-5 mini pc with 8 core ryzen cpus. you don't have to deal with arm specific software and the performance are 10 times faster.
I agree with you. My co-workers are all about SBCs but it doesn't make any sense to use them most of the time. When it does make sense are mobile applications like car and boat PCs, and remote stations that are powered by solar and nighttime battery banks. I don't know what you'd use a cluster for in a remote application. In a fixed site it would probably be better to point to point wireless to a place with more cost-effective high power output and use regular processors. An 11 year-old Ivy Bridge Xeon 2S can crush the compute performance of this entire cluster. But presumably someone needs them for something, or they would go out of business quickly.
This thing is sooo cool. I stopped running a homelab or server for my house because of power consumption. This setup is making me rethink that. I could do crazy stuff with one of these boards.
Practically, you don't need an SBC cluster. Single CPU x86 machine will be faster, and more important, easier to support. So performance comparison between different SBC cluster does not have a practical meaning. But it is still fun!
its good for 24/7 use at home without using much electricity. I too have x86 based ones and they break the bank. btw did you know the orange pi 5 rk3588 is so much cheaper than the best x86 SBC for the same performance? it matches the intel N200 CPU in performance, thats how fast the rk3588 is. It beats intel in some things but loses in anything avx.
That little mini rack thingy is rather neat. I wonder if one could make a small easy to move server with a rack for compute, a rack for storage, a switch, and if there is such thing, a UPS.
Fever dreaming out loud for engagement: I want to stick one of these cluster boards in a 2x 3.5" bay NAS case for a 3 node apps cluster and the 1 node that has the sata slots being the bulk storage. Alternately, would love to see a laptop form factor that could take one of those RK1s (like the MNT reform, or similar).
MNT reform + RK1 would rock! It's a niche use case and a niche laptop, but if you're in that niche, the RK3588 would probably be the closest thing to a 'good' average x86 laptop experience in Arm-land right now.
I'm in that boat! I have quite a few Libre Computers and Pis, but also small PCs sitting around waiting for projects, while also having a large NAS which sits on top of a Dell Poweredge... The NAS gets use very very regularly, but not all the SBCs or the Dell.
But it will run.... whatever. As in, nothing you want to do. I'll stick to my 7950x. I would be more interested in a lower powered NAS board with just the 4 nvme drives on it or better, loaded with SATA ports.
Heh, I need to make a Mini ITX board with just tons of little LEDs around random circuits and chips, that blink in random patterns. The Blinkenlights ITX Board.
@@JeffGeerling Jeff, I'm not gonna lie, but I even bought the IKEA "OBEGRÄNSAD" just to have more blinkenlights in my room. I'm a huge fan of the Connection Machine's by Thinking Machines corporation; even went to NYC just to see the CM-5 at the MoMA. I love the thought process behind it; to give the machines a visual representation of the computation going on as if the black box is thoughtful. I have a few of books and saved YT videos about it too. There's some other pure blinkenlight projects but I couldn't afford them. But I've basically built my own homelab around maximizing the lighting. I was sad to even leave my X79 DDR3 era boards when I upgraded because Crucial's Ballistix Tracer DDR3 were the only RAM kits with true electrically driven LEDS instead of pure software. Made quite a show. Wish you could see my setup, lol.
The mini UPS sounds like a cool idea. I'm thinking, make it be a shared power supply as well, so integrate a switch mode power supply in there that keeps the batteries charged, and have a bunch of 12v outputs with individual reset-able fuses, plus maybe some 5v outputs. Include a micro-controller in it with some temperature and voltage sensors to keep the charge levels and temperature in check, with an output to a character LCD and a USB port for UPS management.
I saw a talk a few years ago at kubecon where the speaker was the guy who first named kubectl and he said it was "kube control". And that fits my narrative, so I'm sticking with it
Thanks for the mini-rack link. I've been looking for something like this for a while, thinking I was just not looking in the right place. Great for a mini home lab.
I wonder how it compares to a Milkv pioneer, double the cores, same memory, not a cluster but a "desktop", it does have 10G eth (x2) and NVME. It's less than double the price, for double the cores and the same amount of ram but probably faster as it is all in one place. No NPU on the pioneer, but it does have pci-express ports so you can add a large video card if the drivers support it, and if you upgrade the power supply. I absolutely loved the 10 inch rack idea, this is new to me. I have a (official status pending) RISC-V lab in Costa Rica, with a bunch of licheepi4a, vision five I&II, beagleV, mangopi, etc, etc. I think the milkv pioneer is mini-itx, so it would be cool to fit in all of the lab in this rack. It's a bit pricey, but would look and function way nice than what I have right now.
@JeffGeerling In reply to your comment on 17:46 about the availability of 10" racks in Europe: I got lucky getting a 30cm deep closed 12U rack from a now defunct Polish company (COVID stock/supply issues I guess) I only could get the MyElectronics 2U enclosure in there by putting in some custom spacers to get the ITX case closer to the glass front door. Otherwise the cables in the back would just not fit. Any deeper rack would not have fitted in my utility closet. I once revved out the diesel my driving instructor used when I was still learning for my driverslicense, some 300 meters from where MyElectronics is currently based in Alkmaar. I now have one more reason to chuckle about that event, 'some' American guy made realise that they were based there.. :)
I ordered new Turing Pi 2.5 with those RK1s with 32GB RAM each. I can't wait! But yeah... elephant is big, and even dual 2,5G would be great. 1Gigs for this is too small bandwith and just because of that I was postponing my ordering since like a year... I finally decided that it might be ok, but I still wish it has faster network.
@@concinnusI thought about doing a DIY ups from lithium ion for my rack but was thinking gee i hope it never sets on fire lol. I think the UPS batteries use lead acid because they are more reliable long term or just in general for a few reasons.
@@clutchboi4038 Lead-acid is just cheap, that's it (at least if you exclude lead core charge). LFP is the right option for a DIY UPS right now, it's very safe. Probably safer than lead-acid, honestly, which can create H2 gas. Then in 5-10 years sodium-ion will likely beat out both LFP and any last argument for lead-acid.
At about 18:30 he talks about a UPS and moving a live rack, I did that with a server at home about 1999-2000, unplugged everything except the two power cables from the UPS to the server and a serial cable to the UPS, loaded the server and UPS on a cart, unplugged the ups, loaded it into the truck, moved to the new location and plugged it back in, zero downtime lol good times
These RK3588 SoCs are quite amazing to behold anyway, it seems. From a regular user's perspective, I'd still love to see an ITX board for desktop use but the board you used here is interesting to watch too.
This rack is a master piece of engineering for home datacenters. 🙂 A little UPS would be indeed amazing. Whoever creates it, shut up and take my money! 😀
I am eagerly awaiting my Turing RK1 modules. My plan is to build an ARM based CI/CD cluster and I am curious to see if the gigabit connection ends up being a bottleneck.
Thanks Jeff I love that you're willing to review these niche devices. From a power standpoint this is great from a compute standpoint not so much. 24 TOPS is nothing, an RTX 4090 can crush over 1200 TOPS
It's such a minor thing, but "sudo su" always bugs me. It feels like when the switch from using su to using sudo went mainstream people said "but how can I use su now?" and found a hacky way of doing it without looking at the documentation. "sudo -i" gets you an interactive shell using the target user's default shell and runs the login files for that shell. "sudo -s" uses the invoking user's shell. Either feels cleaner imo. There's many ways to cut a cake, and there's rarely a right and a wrong, but I wanted to put "sudo -i" out there in case others maybe think it sounds neater too.
@@JeffGeerlingI have the same issue: muscle memory! I know about the “proper” way, but “sudo su -“ is so embedded in my brain I can’t seem to switch completely, even though I do some times… 😅🫣
Well, I think "sudo su" is just easier to remember and more in line with all the other "sudo ". Id rather do "sudo su" knowing that its well tested and works rather than using possibly untested parameters that maybe 100 people in the world knows about.
@@hubertnnnIt's well tested, so the limitations are well known. If you want to start an interactive root shell from sudo, it's just not the correct way.
Idea for a cluster: take Framework Laptop mainboards (the ones with USB 4 for PCIe tunneling and 10Gbps+ networking) and bolt them into these enclosures. I think you might even be able to fit a full 4 of them in there. the keystone slots at the back up top could be USB-C Keystones for power and for the big opening you could 3d print an IO shield so you could put in other keystones for IO. Fully x86, BLAZING fast IO for that small of a cluster, and possibly pretty energy efficient.
I'm intrigued by the idea ... as a hobby / educational endeavor ... being a grad student in computer science ... it would be fun to play around with to try clusters for machine learning in my own home... but on the realistic side ... it's a very expensive toy ... you get a lot of this stuff for free from vendors ...and ... a dedicated machine learning platform with an NVIDIA GPU built around the same price point will definitely blow all of those out of the water in performance ... Fun factor 10+ ... practical use 1. The price of the R-Pi hardware is getting to the point where the average computer hobbyist is better off spending their cash elsewhere. That 6 Pi board alone if you're using the 8GB versions & 6 NVMe cards will cost over $1000. Unfortunately all it's really good for is experimenting.
@@JeffGeerling Perhaps someone (not suggesting you take this on) could design some brackets that could either be 3D printed, or sent to a service like SendCutSend or OshCut to be laser cut and bent as specified. Yes, additional ones available from them would be nice. They list additional rack shelves for sale, but don't seem to have considered additional support brackets as an item people might want.
If you are going to standardize on 12 VDC devices, you could put a battery in the bottom of the rack. Then you can feed the rest of the rack from there. Could even get a DC power bank and use that to connect to the AC or DC source of your choice and keep the rack powered even in transport.
I have the TP 2.4 with 3 CM4's and an RK1, would be great to see a series based on using ansible and k3s to add services like plex, owncloud, etc. I have it up and running k3s and want to add mulitple server applications using symfony and maybe how to use argoCD etc.
The mini rack needs a 10" PDU that has a 110v input power. And as Red Shirt Jeff pointed out, a 10" UPS would be a nice option. EDIT: A Ubiquiti EdgeSwitch 8 PoE (ES-8-150W) would fit nicely in the 10" rack.
I think that is too niche for it to be ever made, but maybe a 12V UPS would be a good idea for those small racks. It would be much easier and cheaper to make (since 12V batteries is a standard) and skipping the conversion from 12V (battery) to 220V (AC) back to 12V (PSU) would be nice.
if you want the led activity indicators from your cluster board, I would try to make a passive light pipe solution with some pieces of clear petg filament, perhaps with a little bit of gaff tape or electrical tape or something to keep the light in the pipes seperate. diy fiber optic.
i am not an experienced PC Builder, but there are two reasons to push air out instead of in. 1: Dust, pushing air in also pushes dust in. Arguably dust can be pulled in by the vacum created. 2: Reducing airpressure in an environment has an great cooling effect. I would be interested to hear your other peoples opinions on this?
6:24 is the important moment. If you want to use two of this boards in a cluster, 10 Gbit/s Ethernet is what you need. It was mentioned twice in the beginning (a little redundant) and it is what stops me from buying such a board: Ethernet is too slow. The connections between the Compute Modules must be very fast and the connection with the external world should be as well. Everything else is great. Storage, compute power, all great.
I have to say that the 10 Inch Rack just makes sense for small builds where space is very limited. hell I've debated about going that path for somethings after you've mentioned 'myelectronics' a couple of times on top of you deploying their cases for some of the builds
For the UPS - it is 12v, so 3s LiPo or 6s SLA would fit the bill nicely - no need to do conversions into and out of AC and DC unnecessarily. For more runtime, just add parallel sets of whichever type you choose (but don't mix them - it is either SLA or LiPo, not both in parallel unless you want a fairly spectacular fireworks display - DAMHIKT). If you go for SLA put them at the bottom of the rack for stability - not a bad plan for either, really.
I got plenty 10" racks at work and we are using the TP-Link TL-SG1210MPE switch for it (: PoE, SFP, managed.... downside is the external power supply... fits perfectly in a 10" shelf
Not a bad option! I do wish it had all ports + lights on front though, instead of the desktop configuration with the ports on back. Though, depending on what kind of gear you have, it may be cleaner that way!
We need to resurrect open Mosix from the dead. It got shut down because the industry was going more virtual machines on single nodes.. now there are clusterboards in small sizes.
OK, have to say this jokingly. Damn you Jeff, costing me money. I use bought the T1 and the 6 Pi ITX board. I am going to use that case for a talk I am going to give at the Southeast Linux Fest in June. I love this because I was using mini milk crates for my SBC lab that I am trying to replace big bare metal with.
with the mini UPS thing, could use some of those USB-PD triggerboards and a compatible battery bank. course might want to collab with one of the eletrical engineer youtubers to work out how to make it do passthrough+charge cause some batterybanks turn off when charged up
Over $249 for that little rack... wow... I'll stick to my big server racks... was hoping to get a small one for my pi's, but not paying that price for that rack. yes thats thru your link here in description... maybe they are running low and prices going up from your video, LOL. I'll watch and see if it comes down.
Wait really? It was $119 from their site when I posted the video, and $199 on Amazon US. Edit: still showing as $119 + $60 shipping from their site right now. Are you outside US?
I love to see how tech gets smaller and more usefull, and how open source software reigns among certain communities, but boy do I feel 'lesser' nowadays, in the late 2000's I grew up daily driving and tinkering with linux (mainly mandriva at the time), got into programming a bit early still do some, but I'm mainly a digital artist to pay the bills, and I feel so out of the loop, when I hear about tech like kubernetes, having no clue what it is about ahaha
I found with k8s that setting up calico or flannel (can’t remember which one) messed with mDNS on the cluster, it would change the source port of the response and avahi would drop it. It would add some iptable entries and would always put its own rules first even if you manually inserted your own rules before. No idea if you use those in your script but noticed you’re using mDNS figured it was worth shouting about
Another question - do you think that the tower cabinet could handle two of the Mini-ITX cases? [I have two Turing Pi 2 boards from the campaign, along with eight RK1s, ready to go... and have been instigating Mini-ITX cases for a while now...]
"Worst case, we'll just have a little smoke come out here." and then a little later "all's well that ends in not a fire". Sounds like a good day at the office.
I'm still trying to wrap my head around the use case for such an esoteric machine; what can be accomplished on this that can't be done on a similarly priced SMP server?
As an electronic engineer, sometimes I just wish I had more time so I could also be a TH-camr, allowing me to work on silly projects for weeks, like building a little 10" UPS for this cute little rack ^^'
That’s pretty cool - I want For price comparison- there are plenty of used epyc cpu + mobo + ecc ram combos on offer for less money.. more grunt, more power bills
@@JeffGeerling then again.. there is no substitute for having an arm cluster in your home lab, when production deployments are likely to be arm based in future as well. Been using arm AWS instances a little lately, and they do hum along quite nicely
Server on the go? I think this would make an awesome little travel server. Your main mini itx pc, your cluster server, and a travel router would probably fit really well and it looks like it could fit in a little carrying case.
Interesting that you’ve been using Gflops per Watt as a benchmark - seems useful for efficiency testing. Have you thought about showing Gflops per dollar as well? Would help show the differences between the Pis (both Raspberry and Turing) based on price!
Lovely labels on the fan box ;) How does this compare to the Ampere 64 core ASRock Rack? Withouth memory that is also $1500, so slightly more expensive.
It's a lot easier to build with the Ampere, it has a LOT more PCIe expansion available, and if you just want a high performance Arm setup, I'd go with the ASRock Rack setup. I like the cluster so I can use it in my lab for little K3s/Kubernetes testing on real hardware, but it is still not a match for a single high performance workstation/CPU.
@@JeffGeerlingdoes one server motherboard will be faster than Turing pi with 4 boards ? I’d like to use kubernetes for my apps as well, is it worth it to set up a single node cluster on a board you mentioned?
Thanks Jeff, so x86 is still a cheaper and more practical option as of today? I can buy a stock G9 HPE server with the same number of Cores/Threads and RAM for about 500$.
@JeffGeerling In reply to your comment on 17:46 about the availability of 10" racks in Europe: I got lucky getting a 30cm deep closed 12U rack from a now defunct Polish company (COVID stock/supply issues I guess) I only could get the MyElectronics 2U enclosure in there by putting in some custom spacers to get the ITX case closer to the glass front door. Otherwise the cables in the back would just not fit. Any deeper rack would not have fitted in my utility closet. I once revved out the diesel my driving instructor used when I was still learning for my driverslicense, some 300 meters from where MyElectronics is currently based in Alkmaar. I now have one more reason to chuckle about that event, 'some' American guy made realise that they were based there.. :)
After my trauma with the preceding board, I find myself unable to even consider this. You really need to be a Geerling to get those things to work. The discord rocks though, but out of my league. No Turing, no cry.
The Turing Pi 1 was a lot more of a beta-level experience. This new board with the latest firmware is like a late beta, heh. But it has gotten a lot better!
Have been working with the TuringPi2 and RK1 Boards for a few months and recently also received a few different mPCIe devices, unfortunately with very bad results. Only 1 in 5 boards would show up. I do hope you look into testing mPCIe devices with the TuringPI2 similar to what you have done with the CM4 as it seems to be either a bit of a jungle compatibility wise or alternatively the faulty connection on the TuringPi2 side. only board I got working was a IOCREST 4 Port SATA with a 88SE9215 chip, Tried a few realtek ethernet adapters and a different SATA adapter.
PCIe is a weird beast with the RK3588, not to mention through the Turing Pi 2 board. I have had weird issues as well, to the point I don't try many anymore.
20:46 Is that case lid a USB-A socket? lol flipping it three times to get it to fit? :P Also, do you have a video already that goes over how you set up your infrastructure at home/office? Do you use monorepos for all your infrastructure? :)
@@xanderplayz3446 I missed a word in my question actually. I meant to refine it to how he sets up his ANSIBLE infrastructure. :) But yeah, I love that series. I don't think he's done an overview (or I haven't caught/found it) of how he organizes his complete infrastructure for ansible.
As a current student, I'd kill to have that mini rack with a Turing Pi board, a 2.5GiB switch, and a UPS, for learning and to bring it in to share with my peers. Sadly, as a current student, I'm also suffering the curse of not being able to splash out >$2k for much of anything at all... it's a shame that homelabs are so costly, especially when my roommates don't want me to drive up their power bill and I have to go ARM or go home.
Some universities have grant money available for that kind of project though-I know in St. Louis, my alma mater has an open source program that grants funds to student groups if they propose a good enough idea for it! It'd be cool to have little 'clusters in a box' like this to learn on.
I was initially dismissive of the Turing Pi 2 because of the so-dimm connector and having to deal with external PSUs and so on, but I must say I wasn't aware it had BCM so I might be regretting that a bit now... So I guess the only good news for me is that the CM5 seems to be closer 😆
Hopefully. And yeah, the SO-DIMM means you have to buy extra adapters for CMs, unfortunately. But it's cool they made a BCM for the board, and it actually works pretty reliably in my testing. Would love to see more standardization in that area too, for others building clusterboards.
I feel we should move to something similar to this for regular computers. The mother board itself would be responsible for power delivery and some basic IO but after that you would simply have a bunch of PCIE slots that could be used to communicate to other devices. The ATX standard really has been around for too long and I wish instead of trying to do incremental improvements like bottom power delivery that we could actually update the entire standard. Drop ATX PSUs for ATX12VO and use SFX while we are at it. Drop DDR for CAMM on the CPU blade now that it is vertical. Drop SATA and SAS for an all NVME solution. Our hardware really hasn't changed much in the past 25 years.
This may be taking it too far but instead of a SFX PSU I would even consider Flex ATX. Currently they go up to 600W but if we adopted ATX12VO they would drop 3.3v and 5v which would mean they would gain space internally and could probably go up to like 750W-800W. Another modification we could do it to treat them more like server PSUs and separate the actual power cables from the power conversion. If done like this you could combine 2 Flex ATX PSUs together to get either high availability or support up to 1500W-1600W. Again it is about the possibilities if we actually make some breaking changes.
Jeff, Thank you for your honesty in who sent you what and how things were paid for. That’s very refreshing in this space and shows you value your audience
It's not even legal to _not_ disclose things. Sadly, many creators aren't up-front about where they get things. Honesty's the best policy... and the best way to ensure the FTC doesn't have to get involved haha.
@ave14401 im not sure who you see doing that on here, but that would be an easy way to lose your channel/sponsor/pay a fine. But if its someone who kept on repeatedly not disclosing advertisers, I wouldn't be surprised if they got locked up...
@@bretb8403 i see it all the time. with small-medium sized channels its hard to spot or enforce so a lot of people get away with it.
@@ave14401 Sadly, this is the case-honest sponsors actually state the requirements in a contract they'll have a creator sign. But there are a lot of dishonest sponsors too, who don't tell creators to do that, and hope any fallout doesn't touch them.
I remember when data rates went from 110 baud to 300 baud. That speed increase was amazing for the time. And years later the first 14,400 baud modem came to market at $14,400 each. These young folks think 1Gbps is slow. They do not know what slow is. It did teach us patience.
My wallet wishes I had never discovered your channel ... keep up the great coverage of your SBC adventures, you play in the same space I'm interested in.
"All's well that ends in "Not A Fire"" - a motto to live by.
This is my new favorite quote
I've done a fair bit of cluster computing, on big clusters like Stampede at the University of Texas, so I'm familiar with the desire of cluster creators to have fast shared filespace and high speed interconnect fabrics etc - *but* - at least for all the real projects (as opposed to demos or benchmarks) that I've used clusters for in real life, I've always been able to structure my code so that high speed interconnects and fast shared filespace are just are not needed. Even for applications where there was big data rather than just a lot of CPU being used. My observation is that these extreme hardware facilities are mostly used by people who don't make the effort to optimise their application for parallelism and who try to shoehorn code that was really designed with uniprocessing in mind, into a multiprocessing environment without taking on the necessary restructuring. Now I'm retired, when I need to throw CPU at a problem I just farm it out to a few dozen regular Pi's and a couple of spare x86 portables I have around the house, running over Wifi, with one regular NFS mount to supply shared files across the lightweight cluster. In fact - although I do have proper cluster MPI software installed, I generally can get by with just kicking off tasks using "ssh". So I mention all this to suggest that perhaps the features you describe and the cost of the system you're using is perhaps a little overkill, that could be avoided by a little more coding effort. An interesting approach for a subsequent video might be in terms of computation done per dollar rather than just what is the fastest shinyest new hardware available?
Sadly when you talk to anyone about making code better versus beefing up the hardware, 99 times out of 100 you'll be told the hardware's easier :D
One of the reasons I love Pi clusters is I've changed my own site architecture many times to make it faster (avoiding disk IO where I can, caching in RAM on individual nodes, that sort of thing), and made big sites scale (think millions of dynamic page views per hour) on relatively modest AWS resources... all because I would run the same application on my Pi cluster that we were running on $20,000/month servers :D
But not every project allowed me the time to optimize. Sometimes the plan was to build up hardware for a month, then tear it all down, and they were happier doing that spending $100k+, than to spend an extra two weeks optimizing the site (for a lot less).
Read "In Search of Clusters", a fine book. Amdahls Law; show me where the data is and I'll show you where the computing has to take place. At some point all parallel computations reduce to a set of serial instructions. It's the law. Clusters live and die by the speed of their switching fabric, and that will always be the case.
The RK1 boards are crazy fast! I have two of these TuringPi 2 boards, running 2 RK1s and 2 CM4s each. A sort of P-module, E-module configuration. 92GB of RAM and 82 GHz of CPU. I use Hashicorp Nomad to orchestrate tasks using a combination of the Docker and Exec drivers. Ceph to manage the persistent volumes. Soon I'll be able to retire my power-hungry my Dell R710s in favor of a system that draws < 200W. Couldn't be happier with the results!
Thank you for showing the debugging steps with the cluster, including how to see which pod had stalled and finding error logs using "describe". You can watch 100 happy path tutorials without ever seeing these things.
Yeah that is the cool part.
Man i really hope that Raspberry Pi prices comes down to a reasonable level, so that us enthusiast can do stuff like these
It won't. But I do recommend considering getting it either used or older gen like Raspberry Pi 3 for example which is lot cheaper than Pi 4 or 5, especially if you don't need the performance of those.
Pi 3 or Pi 4 (or even Zero 2W) are probably the sweet spot for value in the Pi lineup right now.
If they release a 2GB Pi 5 for $40, that would be a great value.
appears to be heading in the right direction. :)
@@JeffGeerling That's be like putting a v4 in an old muscle car. We all know that we don't build these things because we need them. We build them because we NEED them
> corpo enshitification enters the chat
As soon as you said pod was pending and you were using nfs I said to my self "nfs-common". Great video!
Haha if you know, you know. Usually you go down every other avenue and then the 100th google search you get a path that leads to a solution.
I got lucky in this case and skipped the first 99 dead ends!
@@JeffGeerling I've integrated that into my playbooks because Longhorn Storage Class uses it. Took me a while to figure that one out!
I have one of these, and it is spectacular! I couldn't afford to get 4x of the 32gb nodes, but I have two of em. They are heckin nice.
It is nice to have x86-level performance in the same power footprint as a Pi!
That rockship kicks butt, I can tell, I have a NanoPC-T6 powered by that baby, it ROCKS!
Break out the soldering iron and swap the direction those power leads exit the PSU board. No more fan-blade jeopardy, no more stressing of the board connector.
True. May do that before I move the mini rack into its final location.
Yo, Only Fans reveal at 11:12!
That would be one scary channel ;-)
I always snicker when I see the "Only Fans" container.
When my son was in cub scouts, he sold popcorn like all the scouts did. We borrowed a wagon to put the popcorn in, and found a cardboard box that fit perfectly! As I was walking behind him I realized that we'd have to ditch the cardboard box. It was one that we got at the local library book sale (They were selling unpopular books to make room for more books.) Written across the back of the box someone wrote "ADULT BOOKS" in large letters. That phrase has a completely different meaning outside of libraries. Lol!
Really seems like the Turing Pi board could benefit from having a switch chip. Even if you still only had 1 Gb or 2.5 Gb connectivity to the external network, having a 40/80Gb switching fabric between nodes would make something like Ceph internal replication or any node to node comms in the cluster much better. Of course, having a 10 Gb SFP+ would also be ideal, since so much 10 Gb fiber gear is plummeting in price these days, and might obsolete the need for finding another big chip and traces on the board, so that might the smarter play.
I do hope they do a Turing Pi 3, where they maybe bump the price a little, but include some features that will let the board really go all-out on performance.
The problem is as the price of the board sneaks past $200, $300, or more, the audience gets even more limited :(
@@JeffGeerling I hear you, but I think this space could do with some better comms around what makes sense and what you _can_ do with these platforms. Like, a 4 node cluster on the Turing Pi 2 at $1500 does not make sense for most home lab operators and especially not the folks running single purpose GPIO heavy configs, but $1500-2000 for a 4 node Ceph backed lightweight k8s platform with a slough of groupware appliance containers is a stellar HA homelab / business server in a single box. For the load of businesses I know trying to de-SaaS their OpEx while still looking for a reliable solution, this is barking up the right tree for sure.
In my m.2 slots I now have adapters that go to a 10Gb single network card. I have proxmox on each node and the 10Gb is being used for iscsi for the vms/containers. Its working great.
These cluster boards keep getting cooler and cooler. Not sure what i would do with one but it would be cool to play with.
I've been using the turingpi 2 with 3x RK1's for a few weeks, and really enjoying it.
Ive been managing it using the Ansible for DevOps and Ansible for Kubernetes books you put out, so many thanks for that!
So glad I could help! Reminds me I need to get around to updating those books again...
Love these tiny videos. As someone who does Van Life traveling full time in my van, I’m always looking for ways to run NAS and cluster solutions in a compact “portable” way. Especially love it when you can run on 12v, 24v, or 48v DC.
this stuff does not make sense when for this price you can buy 3-5 mini pc with 8 core ryzen cpus. you don't have to deal with arm specific software and the performance are 10 times faster.
Rule of cool means this is better
And probably consumes 10 times the power.
I agree with you. My co-workers are all about SBCs but it doesn't make any sense to use them most of the time. When it does make sense are mobile applications like car and boat PCs, and remote stations that are powered by solar and nighttime battery banks. I don't know what you'd use a cluster for in a remote application. In a fixed site it would probably be better to point to point wireless to a place with more cost-effective high power output and use regular processors. An 11 year-old Ivy Bridge Xeon 2S can crush the compute performance of this entire cluster. But presumably someone needs them for something, or they would go out of business quickly.
...But this is more fun...
@@josuesantos871😂agree
Excellent! Thanks for sharing. Looking forward to the new base board with 10GbE.
This thing is sooo cool. I stopped running a homelab or server for my house because of power consumption. This setup is making me rethink that. I could do crazy stuff with one of these boards.
Like what ?
Practically, you don't need an SBC cluster. Single CPU x86 machine will be faster, and more important, easier to support.
So performance comparison between different SBC cluster does not have a practical meaning.
But it is still fun!
I think you forgot that the main point of this was how power-efficient it was for the performance.
@@xanderplayz3446 Modern x86 will be more power efficient than a cluster of SBCs.
its good for 24/7 use at home without using much electricity. I too have x86 based ones and they break the bank. btw did you know the orange pi 5 rk3588 is so much cheaper than the best x86 SBC for the same performance? it matches the intel N200 CPU in performance, thats how fast the rk3588 is. It beats intel in some things but loses in anything avx.
@@Lincos321what would you use with x86 as a replacement ?
That little mini rack thingy is rather neat. I wonder if one could make a small easy to move server with a rack for compute, a rack for storage, a switch, and if there is such thing, a UPS.
Fever dreaming out loud for engagement: I want to stick one of these cluster boards in a 2x 3.5" bay NAS case for a 3 node apps cluster and the 1 node that has the sata slots being the bulk storage. Alternately, would love to see a laptop form factor that could take one of those RK1s (like the MNT reform, or similar).
MNT reform + RK1 would rock! It's a niche use case and a niche laptop, but if you're in that niche, the RK3588 would probably be the closest thing to a 'good' average x86 laptop experience in Arm-land right now.
“I don’t need a cluster”
“What would I use it for”
“It’s just gonna be running some bullshit”
I so really want one
This is the way.
@@JeffGeerling Why not walk both ways?
First, I completely agree with you. I think this is amazing, just to learn how to build an HPC System, and to see how it works.
I'm in that boat! I have quite a few Libre Computers and Pis, but also small PCs sitting around waiting for projects, while also having a large NAS which sits on top of a Dell Poweredge... The NAS gets use very very regularly, but not all the SBCs or the Dell.
But it will run.... whatever.
As in, nothing you want to do.
I'll stick to my 7950x.
I would be more interested in a lower powered NAS board with just the 4 nvme drives on it or better, loaded with SATA ports.
I just like how it looks, a compact blinkenlights thingy. So cool.
Heh, I need to make a Mini ITX board with just tons of little LEDs around random circuits and chips, that blink in random patterns. The Blinkenlights ITX Board.
@@JeffGeerling Jeff, I'm not gonna lie, but I even bought the IKEA "OBEGRÄNSAD" just to have more blinkenlights in my room. I'm a huge fan of the Connection Machine's by Thinking Machines corporation; even went to NYC just to see the CM-5 at the MoMA. I love the thought process behind it; to give the machines a visual representation of the computation going on as if the black box is thoughtful. I have a few of books and saved YT videos about it too. There's some other pure blinkenlight projects but I couldn't afford them. But I've basically built my own homelab around maximizing the lighting. I was sad to even leave my X79 DDR3 era boards when I upgraded because Crucial's Ballistix Tracer DDR3 were the only RAM kits with true electrically driven LEDS instead of pure software. Made quite a show. Wish you could see my setup, lol.
The mini UPS sounds like a cool idea. I'm thinking, make it be a shared power supply as well, so integrate a switch mode power supply in there that keeps the batteries charged, and have a bunch of 12v outputs with individual reset-able fuses, plus maybe some 5v outputs. Include a micro-controller in it with some temperature and voltage sensors to keep the charge levels and temperature in check, with an output to a character LCD and a USB port for UPS management.
I don't have a use for a cluster system, but they're entertaining.
Make it work, do some brutal testing with it. Really brutal testing.
I saw a talk a few years ago at kubecon where the speaker was the guy who first named kubectl and he said it was "kube control". And that fits my narrative, so I'm sticking with it
So is Kube pronounced "Cube" or "Koob" or "Coob" or "CooBee" ?
Great question, "cube control" if I remember correctly... It was the "control" bit I was worried about
Red shirt seems calm in this video.
Thanks for the mini-rack link. I've been looking for something like this for a while, thinking I was just not looking in the right place. Great for a mini home lab.
So cool to see the channel has grown so much! You’ve earned it Jeff! (But not that red shirt guy) :)
I wonder how it compares to a Milkv pioneer, double the cores, same memory, not a cluster but a "desktop", it does have 10G eth (x2) and NVME.
It's less than double the price, for double the cores and the same amount of ram but probably faster as it is all in one place. No NPU on the pioneer, but it does have pci-express ports so you can add a large video card if the drivers support it, and if you upgrade the power supply.
I absolutely loved the 10 inch rack idea, this is new to me. I have a (official status pending) RISC-V lab in Costa Rica, with a bunch of licheepi4a, vision five I&II, beagleV, mangopi, etc, etc. I think the milkv pioneer is mini-itx, so it would be cool to fit in all of the lab in this rack. It's a bit pricey, but would look and function way nice than what I have right now.
I did not know about 10" rack systems. Very cool.
@JeffGeerling In reply to your comment on 17:46 about the availability of 10" racks in Europe: I got lucky getting a 30cm deep closed 12U rack from a now defunct Polish company (COVID stock/supply issues I guess) I only could get the MyElectronics 2U enclosure in there by putting in some custom spacers to get the ITX case closer to the glass front door. Otherwise the cables in the back would just not fit. Any deeper rack would not have fitted in my utility closet.
I once revved out the diesel my driving instructor used when I was still learning for my driverslicense, some 300 meters from where MyElectronics is currently based in Alkmaar. I now have one more reason to chuckle about that event, 'some' American guy made realise that they were based there.. :)
I ordered new Turing Pi 2.5 with those RK1s with 32GB RAM each. I can't wait! But yeah... elephant is big, and even dual 2,5G would be great. 1Gigs for this is too small bandwith and just because of that I was postponing my ordering since like a year... I finally decided that it might be ok, but I still wish it has faster network.
I’m always highly impressed when someone gets Drupal to do anything sensible at all, let alone a usable deployment of it.
Really love the screen time during installation and configuration so didactic!!
You read my mind when you talk about a UPS for a mini rack. That would be siiiick
APC's most popular models are like 7 years old and use lead-acid batteries. They don't seem to feel much competitive pressure.
@@concinnusI thought about doing a DIY ups from lithium ion for my rack but was thinking gee i hope it never sets on fire lol. I think the UPS batteries use lead acid because they are more reliable long term or just in general for a few reasons.
@@clutchboi4038 Lead-acid is just cheap, that's it (at least if you exclude lead core charge). LFP is the right option for a DIY UPS right now, it's very safe. Probably safer than lead-acid, honestly, which can create H2 gas. Then in 5-10 years sodium-ion will likely beat out both LFP and any last argument for lead-acid.
Letting red shirt Jeff in to the studio seems reckless.
PI's are getting insane dude! 😲
That Quad board look like such a nice cluster training tool!
The 32GB Turing is still less than the price to upgrade to 32GB Ram in a Mac.
Ouch.
So true it hurts!
At about 18:30 he talks about a UPS and moving a live rack, I did that with a server at home about 1999-2000, unplugged everything except the two power cables from the UPS to the server and a serial cable to the UPS, loaded the server and UPS on a cart, unplugged the ups, loaded it into the truck, moved to the new location and plugged it back in, zero downtime lol good times
14:00 you could even solder in the power leads on the other side of the pcb
These RK3588 SoCs are quite amazing to behold anyway, it seems.
From a regular user's perspective, I'd still love to see an ITX board for desktop use but the board you used here is interesting to watch too.
Radxa Rock 5 ITX is perfect for desktop use case.
This rack is a master piece of engineering for home datacenters. 🙂 A little UPS would be indeed amazing. Whoever creates it, shut up and take my money! 😀
The Rackmate T1 in this video is awesome!
Awesome detailed look. The 1Gbps is definitely showing its age. The price is kinda crazy also. Hope they can increase production to make it cheaper
I am eagerly awaiting my Turing RK1 modules. My plan is to build an ARM based CI/CD cluster and I am curious to see if the gigabit connection ends up being a bottleneck.
Brought to you by the Pi Cluster King!
Thanks Jeff I love that you're willing to review these niche devices. From a power standpoint this is great from a compute standpoint not so much. 24 TOPS is nothing, an RTX 4090 can crush over 1200 TOPS
It's such a minor thing, but "sudo su" always bugs me. It feels like when the switch from using su to using sudo went mainstream people said "but how can I use su now?" and found a hacky way of doing it without looking at the documentation.
"sudo -i" gets you an interactive shell using the target user's default shell and runs the login files for that shell. "sudo -s" uses the invoking user's shell. Either feels cleaner imo.
There's many ways to cut a cake, and there's rarely a right and a wrong, but I wanted to put "sudo -i" out there in case others maybe think it sounds neater too.
True; for me I just have muscle memory from whenever ago to use 'sudo su' :D
@@JeffGeerlingI have the same issue: muscle memory! I know about the “proper” way, but “sudo su -“ is so embedded in my brain I can’t seem to switch completely, even though I do some times… 😅🫣
Well, I think "sudo su" is just easier to remember and more in line with all the other "sudo ".
Id rather do "sudo su" knowing that its well tested and works rather than using possibly untested parameters that maybe 100 people in the world knows about.
How about 'sudo sh'̈...
@@hubertnnnIt's well tested, so the limitations are well known. If you want to start an interactive root shell from sudo, it's just not the correct way.
Idea for a cluster: take Framework Laptop mainboards (the ones with USB 4 for PCIe tunneling and 10Gbps+ networking) and bolt them into these enclosures. I think you might even be able to fit a full 4 of them in there. the keystone slots at the back up top could be USB-C Keystones for power and for the big opening you could 3d print an IO shield so you could put in other keystones for IO. Fully x86, BLAZING fast IO for that small of a cluster, and possibly pretty energy efficient.
I'm intrigued by the idea ... as a hobby / educational endeavor ... being a grad student in computer science ... it would be fun to play around with to try clusters for machine learning in my own home... but on the realistic side ... it's a very expensive toy ... you get a lot of this stuff for free from vendors ...and ... a dedicated machine learning platform with an NVIDIA GPU built around the same price point will definitely blow all of those out of the water in performance ... Fun factor 10+ ... practical use 1. The price of the R-Pi hardware is getting to the point where the average computer hobbyist is better off spending their cash elsewhere. That 6 Pi board alone if you're using the 8GB versions & 6 NVMe cards will cost over $1000.
Unfortunately all it's really good for is experimenting.
Felt like I unlocked a core memory when Jeff slid that Pi Blade out. Forgot I'm still waiting on mine. Soon™
for deep stuff like the RK1 in that rack, it'd be nice if they included rear supports like what the shelf had.
That seems like an easy enough accessory to make-put a few bends in some metal.
@@JeffGeerling Perhaps someone (not suggesting you take this on) could design some brackets that could either be 3D printed, or sent to a service like SendCutSend or OshCut to be laser cut and bent as specified. Yes, additional ones available from them would be nice. They list additional rack shelves for sale, but don't seem to have considered additional support brackets as an item people might want.
If you are going to standardize on 12 VDC devices, you could put a battery in the bottom of the rack. Then you can feed the rest of the rack from there. Could even get a DC power bank and use that to connect to the AC or DC source of your choice and keep the rack powered even in transport.
I have the TP 2.4 with 3 CM4's and an RK1, would be great to see a series based on using ansible and k3s to add services like plex, owncloud, etc. I have it up and running k3s and want to add mulitple server applications using symfony and maybe how to use argoCD etc.
The mini rack needs a 10" PDU that has a 110v input power. And as Red Shirt Jeff pointed out, a 10" UPS would be a nice option.
EDIT: A Ubiquiti EdgeSwitch 8 PoE (ES-8-150W) would fit nicely in the 10" rack.
I think that is too niche for it to be ever made, but maybe a 12V UPS would be a good idea for those small racks.
It would be much easier and cheaper to make (since 12V batteries is a standard) and skipping the conversion from 12V (battery) to 220V (AC) back to 12V (PSU) would be nice.
There are 10" C13 PDUs.
if you want the led activity indicators from your cluster board, I would try to make a passive light pipe solution with some pieces of clear petg filament, perhaps with a little bit of gaff tape or electrical tape or something to keep the light in the pipes seperate. diy fiber optic.
I wonder when failsafe systems and ECC memory will be popular🤔
i am not an experienced PC Builder, but there are two reasons to push air out instead of in. 1: Dust, pushing air in also pushes dust in. Arguably dust can be pulled in by the vacum created. 2: Reducing airpressure in an environment has an great cooling effect. I would be interested to hear your other peoples opinions on this?
BTW I literally saw the person who wrote kubectl at a talk in person and he said that it is pronounced "cube control".
So a bit less cuddly!
@@JeffGeerling to be fair, virtually everyone else seems to call it "cube cuddle" no matter what the creator says.
At the moment I am a little fed up with clusters. But this one is nice
Heh, that's the secret. Anyone who deals with clusters is always fed up with them! Otherwise you're inhuman :D
6:24 is the important moment.
If you want to use two of this boards in a cluster, 10 Gbit/s Ethernet is what you need.
It was mentioned twice in the beginning (a little redundant) and it is what stops me from buying such a board: Ethernet is too slow. The connections between the Compute Modules must be very fast and the connection with the external world should be as well.
Everything else is great. Storage, compute power, all great.
I have to say that the 10 Inch Rack just makes sense for small builds where space is very limited. hell I've debated about going that path for somethings after you've mentioned 'myelectronics' a couple of times on top of you deploying their cases for some of the builds
For the UPS - it is 12v, so 3s LiPo or 6s SLA would fit the bill nicely - no need to do conversions into and out of AC and DC unnecessarily. For more runtime, just add parallel sets of whichever type you choose (but don't mix them - it is either SLA or LiPo, not both in parallel unless you want a fairly spectacular fireworks display - DAMHIKT). If you go for SLA put them at the bottom of the rack for stability - not a bad plan for either, really.
Making them line interactive or at least work with an AC input is a little harder, though there are some good BMS boards out there these days.
Thanks for all your hard work, Jeff.
Jeff, you should design your own desk mat!! :-D
Red shirts and rip saws and blow torches, oh my!
I got plenty 10" racks at work and we are using the TP-Link TL-SG1210MPE switch for it (: PoE, SFP, managed.... downside is the external power supply... fits perfectly in a 10" shelf
Not a bad option! I do wish it had all ports + lights on front though, instead of the desktop configuration with the ports on back. Though, depending on what kind of gear you have, it may be cleaner that way!
We need to resurrect open Mosix from the dead. It got shut down because the industry was going more virtual machines on single nodes.. now there are clusterboards in small sizes.
OK, have to say this jokingly. Damn you Jeff, costing me money. I use bought the T1 and the 6 Pi ITX board. I am going to use that case for a talk I am going to give at the Southeast Linux Fest in June. I love this because I was using mini milk crates for my SBC lab that I am trying to replace big bare metal with.
with the mini UPS thing, could use some of those USB-PD triggerboards and a compatible battery bank. course might want to collab with one of the eletrical engineer youtubers to work out how to make it do passthrough+charge cause some batterybanks turn off when charged up
Over $249 for that little rack... wow... I'll stick to my big server racks... was hoping to get a small one for my pi's, but not paying that price for that rack. yes thats thru your link here in description... maybe they are running low and prices going up from your video, LOL. I'll watch and see if it comes down.
Wait really? It was $119 from their site when I posted the video, and $199 on Amazon US.
Edit: still showing as $119 + $60 shipping from their site right now. Are you outside US?
I love to see how tech gets smaller and more usefull, and how open source software reigns among certain communities, but boy do I feel 'lesser' nowadays, in the late 2000's I grew up daily driving and tinkering with linux (mainly mandriva at the time), got into programming a bit early still do some, but I'm mainly a digital artist to pay the bills, and I feel so out of the loop, when I hear about tech like kubernetes, having no clue what it is about ahaha
I found with k8s that setting up calico or flannel (can’t remember which one) messed with mDNS on the cluster, it would change the source port of the response and avahi would drop it. It would add some iptable entries and would always put its own rules first even if you manually inserted your own rules before.
No idea if you use those in your script but noticed you’re using mDNS figured it was worth shouting about
Another question - do you think that the tower cabinet could handle two of the Mini-ITX cases?
[I have two Turing Pi 2 boards from the campaign, along with eight RK1s, ready to go... and have been instigating Mini-ITX cases for a while now...]
"Worst case, we'll just have a little smoke come out here." and then a little later "all's well that ends in not a fire". Sounds like a good day at the office.
I'm still trying to wrap my head around the use case for such an esoteric machine; what can be accomplished on this that can't be done on a similarly priced SMP server?
Sick video, Jeff as always. Love to see some RGB on that rack. 😊
Thanks Jeff you're such a nice person someone I'd love to meet next time you're in England. David
As an electronic engineer, sometimes I just wish I had more time so I could also be a TH-camr, allowing me to work on silly projects for weeks, like building a little 10" UPS for this cute little rack ^^'
That’s pretty cool - I want
For price comparison- there are plenty of used epyc cpu + mobo + ecc ram combos on offer for less money.. more grunt, more power bills
If power's cheap, then some used servers or mini PCs are definitely a better option, cost-wise. Power's luckily pretty cheap most places in the US!
@@JeffGeerling then again.. there is no substitute for having an arm cluster in your home lab, when production deployments are likely to be arm based in future as well.
Been using arm AWS instances a little lately, and they do hum along quite nicely
I, (like so many others) are going back to Mint! We've had it with Snap Packages and Ubumtu slowing down. So we'd like to see you using Debian more.
Server on the go? I think this would make an awesome little travel server. Your main mini itx pc, your cluster server, and a travel router would probably fit really well and it looks like it could fit in a little carrying case.
Interesting that you’ve been using Gflops per Watt as a benchmark - seems useful for efficiency testing. Have you thought about showing Gflops per dollar as well? Would help show the differences between the Pis (both Raspberry and Turing) based on price!
Actually not a bad idea, I should consider adding that metric to my sbc-reviews perhaps!
Lovely labels on the fan box ;) How does this compare to the Ampere 64 core ASRock Rack? Withouth memory that is also $1500, so slightly more expensive.
It's a lot easier to build with the Ampere, it has a LOT more PCIe expansion available, and if you just want a high performance Arm setup, I'd go with the ASRock Rack setup.
I like the cluster so I can use it in my lab for little K3s/Kubernetes testing on real hardware, but it is still not a match for a single high performance workstation/CPU.
@@JeffGeerlingdoes one server motherboard will be faster than Turing pi with 4 boards ? I’d like to use kubernetes for my apps as well, is it worth it to set up a single node cluster on a board you mentioned?
Thanks Jeff, so x86 is still a cheaper and more practical option as of today? I can buy a stock G9 HPE server with the same number of Cores/Threads and RAM for about 500$.
@JeffGeerling In reply to your comment on 17:46 about the availability of 10" racks in Europe: I got lucky getting a 30cm deep closed 12U rack from a now defunct Polish company (COVID stock/supply issues I guess) I only could get the MyElectronics 2U enclosure in there by putting in some custom spacers to get the ITX case closer to the glass front door. Otherwise the cables in the back would just not fit. Any deeper rack would not have fitted in my utility closet.
I once revved out the diesel my driving instructor used when I was still learning for my driverslicense, some 300 meters from where MyElectronics is currently based in Alkmaar. I now have one more reason to chuckle about that event, 'some' American guy made realise that they were based there.. :)
Ha! Yeah, they could even shave a little more depth from their designs but if you have a deep enough rack or open back, no problem at all.
After my trauma with the preceding board, I find myself unable to even consider this. You really need to be a Geerling to get those things to work. The discord rocks though, but out of my league. No Turing, no cry.
The Turing Pi 1 was a lot more of a beta-level experience. This new board with the latest firmware is like a late beta, heh. But it has gotten a lot better!
Have been working with the TuringPi2 and RK1 Boards for a few months and recently also received a few different mPCIe devices, unfortunately with very bad results. Only 1 in 5 boards would show up.
I do hope you look into testing mPCIe devices with the TuringPI2 similar to what you have done with the CM4 as it seems to be either a bit of a jungle compatibility wise or alternatively the faulty connection on the TuringPi2 side.
only board I got working was a IOCREST 4 Port SATA with a 88SE9215 chip, Tried a few realtek ethernet adapters and a different SATA adapter.
PCIe is a weird beast with the RK3588, not to mention through the Turing Pi 2 board. I have had weird issues as well, to the point I don't try many anymore.
for $1500, could easily get a 16 core 32 thread and with 128GB dd5 and a great motherboard, case and power for less than the cost of the max version.
20:46 Is that case lid a USB-A socket? lol flipping it three times to get it to fit? :P
Also, do you have a video already that goes over how you set up your infrastructure at home/office? Do you use monorepos for all your infrastructure? :)
The studio moving videos would answer the office part, but I dont know for the home part.
@@xanderplayz3446 I missed a word in my question actually. I meant to refine it to how he sets up his ANSIBLE infrastructure. :)
But yeah, I love that series. I don't think he's done an overview (or I haven't caught/found it) of how he organizes his complete infrastructure for ansible.
As a current student, I'd kill to have that mini rack with a Turing Pi board, a 2.5GiB switch, and a UPS, for learning and to bring it in to share with my peers. Sadly, as a current student, I'm also suffering the curse of not being able to splash out >$2k for much of anything at all... it's a shame that homelabs are so costly, especially when my roommates don't want me to drive up their power bill and I have to go ARM or go home.
Some universities have grant money available for that kind of project though-I know in St. Louis, my alma mater has an open source program that grants funds to student groups if they propose a good enough idea for it!
It'd be cool to have little 'clusters in a box' like this to learn on.
I was initially dismissive of the Turing Pi 2 because of the so-dimm connector and having to deal with external PSUs and so on, but I must say I wasn't aware it had BCM so I might be regretting that a bit now...
So I guess the only good news for me is that the CM5 seems to be closer 😆
Hopefully. And yeah, the SO-DIMM means you have to buy extra adapters for CMs, unfortunately. But it's cool they made a BCM for the board, and it actually works pretty reliably in my testing. Would love to see more standardization in that area too, for others building clusterboards.
With two cases in one, it would even be possible, to use two Turing Pi 2 boards with a combined 8 RK1 SOMs. For the ultimate insanity.
This is good for those who need several virtual machines with Php and MySQL base data Maria
Great vid - the sound is very different, and it looks like you green screened your eclipse photo/workspace - is this in your new space ?
Any interest in testing the DeskPi Super6C with 6x Orange Pi CM5 modules?
I feel we should move to something similar to this for regular computers. The mother board itself would be responsible for power delivery and some basic IO but after that you would simply have a bunch of PCIE slots that could be used to communicate to other devices. The ATX standard really has been around for too long and I wish instead of trying to do incremental improvements like bottom power delivery that we could actually update the entire standard. Drop ATX PSUs for ATX12VO and use SFX while we are at it. Drop DDR for CAMM on the CPU blade now that it is vertical. Drop SATA and SAS for an all NVME solution. Our hardware really hasn't changed much in the past 25 years.
This may be taking it too far but instead of a SFX PSU I would even consider Flex ATX. Currently they go up to 600W but if we adopted ATX12VO they would drop 3.3v and 5v which would mean they would gain space internally and could probably go up to like 750W-800W. Another modification we could do it to treat them more like server PSUs and separate the actual power cables from the power conversion. If done like this you could combine 2 Flex ATX PSUs together to get either high availability or support up to 1500W-1600W. Again it is about the possibilities if we actually make some breaking changes.
With the NPUs, you could maybe run some GPT models? Those generally do not require speedy networks, just a lot of FLOPS and RAM
you can add 4pin fan header to v 2.4 boards but you have to solder a coonecot and a chip onto the board.