Best part, is when you say there are links below. They are there just as you said. You're probably the only TH-camr that actually puts the link they promise into the description. So many well known big TH-camrs promise links, then never post them, and never respond to people asking for them. Thank for your well planned projects and thoroughness. How clean your website is, and how just because computers are so much more powerful today you don't put in inefficient or lazy code like everyone else does. Really says a lot about the kind of person you are. You're a true asset to the community.
- 7 RPi B+ Power - A pcb that save the mess of cables and power - A standar case - Software tailored for clusters ready for Raspberry Pi This project have every thing that Raspberry Pi user will dream, nice work.
I think you shouldn't instruct people to create unencrypted SSH private keys. "Press the enter key three times" after ssh-keygen is resulting in unencrypted private key. Form a security standpoint those keys are worthless!
How'd that work out? Did it happen? There's another thing you can do with this, if you are into coding and already know some C. MPI_lib. It gives you a parallelization API such that you can divide a task across 28 devices. With careful coding, you can have an enormous speedup with certain categories of tasks. (multidimensional modelling, sorting, {en | de}-cryption, file compression, DSP)
They say any piece of hardware while in operation is obsolete. I was recently given five Minus Forum N-40 minicomputers to do something with during lockdown, and I want to cluster them. I got a Pi 4B to manage things, and would like to eventually use it for video processing. Thanks for your series so far, I think it will come in handy.
Here to learn more about clusters in general. Since I'm not really familiar with Pi clusters, I was expecting some type of PXE boot for each node to pull from a central image file. Interesting that each needs to be flashed manually. The more you know!
Currently there's no IO shield for it, but I asked Turing Machines about it and they said they may make one for it. I could fashion one up myself, but it's not a huge deal for right now-this is a prototype that I have to return. For a more permanent build, I would be integrating a large, quiet fan, and probably an internal PSU.
@@wmonk5642 Then buy them and make a video about it and upload it to TH-cam. I'd watch it. I take your point though. The Turing Pi motherboard is pretty expensive for what it is, though I think the Compute Modules themselves are good value and you are comparing new with very old.
Fantastic job. Great presentation, scripting, content. I'm now a subscriber and sponsor. Looking for future content on Pi clusters, kubernetes, containers, AI software and hardware on Pi.
Now what I really want to know is if there's something out there like a PCIe card with a connection for the compute module that would allow the compute module to effectively function as an ARM Coprocessor on an x86 or x86-64 PC.
Unless you write your code to specifically load binaries onto those arm cores, this will not help you one bit. And the x86 cpu in your desktop/laptop/whatever will very likely be much much faster than the rpi's arm cores anyway. Not very useful.
Think you're missing the point of the video. Nobody "needs" this, he's just doing it cause he can and he is curious. Plus if you need a serious server you're not going after an ARM based server anyway.
@@admiraldirtbag4336 any X86 CPU made recently, particularly if you're talking a used 1700x or something along those lines and 16GB of RAM. The point of this, though, is to play around with clustering without having to fill up a room with ATX towers.
I would go with AWS. I've setup some T3 Nano instances, with spot prices you can get each one down to less than £0.02 a day. You can also stop them at any point and not be charged more.
As always, thanks for a great series of videos. Patiently waiting for Ep3. I have 4 Pi 4's which I bought to try and get my head around Kubernetes/Docker (my experience is with more traditional server/VMs/hypervisors). The goal is to run some containers to manage media at home and share around the network (Airsonic, plex). I get stuck on how to set-up the networking, especially how this would integrate or be accessed from existing network: - Can I put all 4 on my home LAN, so I can access them directly? - Should they get their own network using the physical ports and I manage them over the WiFI? - Do I need a separate device/another Pi to act as the router between the 2 networks (IP masquerading/forwarding etc.)? - Can the Master node also act as this router? Hopefully you will cover the network setup in detail in a future episode.
I built a cluster out of 5 4GB Pi 4's. I've got them all on my home network, using an 8-port switch. I have a sixth Pi 4 running as a file server (using gluster.) Each of these boards has a 16gb SD card for booting and a 16gb USB flash for the root file system. The file server also has a 256gb flash drive. Each of the cluster nodes mounts a shared volume which is housed on that drive. I have a seventh 2gb Pi 4 running pi-hole as a DHCP and DNS server, and use static DHCP leases for the cluster nodes. Each of the systems runs Ubuntu 20.04 minimal. For the actual clustering, I'm using k3s from Rancher, which I highly recommend over other approaches if you decide to run Kubernetes. At the moment, my cluster runs a web server, a git repository, a docker registry, and a Drone CI server. I can launch Drone agents within the cluster, but my current projects are building multi-architecture docker images, so I have a static Drone agent instance on an Intel system (it's an Atomic Pi, fwiw.) I'm housing it all in the Cloudlet Custer case from c4labs, with some modifications for better airflow and cable management. I'm using the aluminum heatsink cases (just the top parts) that cover the entire top of the Pi 4 board, which keeps things nice and cool. I'm powering it all with a 10A USB power supply, which is adequate but not stable enough to support overclocking. I plan on building a better power system for this at some point. I did most of the OS and software setup of the cluster boards using ansible (I write ansible infrastructure code for a living.) At some point, I'll make these roles and playbooks public (there's a few secrets in the files I need to obfuscate first.) You can visit the cluster at project.kube.thejimnicholson.com.
Always wanted to learn and apply. Although RPI4 KNOWN for heat. A special designed ice cluster mount over blades with thin line per blade and super cooler on top would do the trick. Have it rest above blades mounted to corner post of base board. Mini turbine blade type most efficient in moving air.
Looking forward to your exploration. I am hoping I can run my machine-learning stack on this :D Its mainly build around doing a lot of small calculations. I am curious to IO latency and how well it runs Dask.
I would be interested in doing the turing cluster, if only to run something like boinc or folding@home for distributed computing research like seti or covid-19 study. I can also recommend looking into etcher pro when it launches, a multi-device flashing pod, if you want to save time and constantly mirror image files.
Neat stuff, but (as you say) preparing media for every node is a PITA. Is there something in the Pi's PXE network boot function that stops you from using it for clustering?
5:15 "using PV to monitor the progress of DD command" - /me looking at the man page: dd - option: "status" - value: "progress" so - using "dd if=image of=disk status=progress" does exactly the same without using additional PV ... today you learned
It depends on the version of dd you have installed - I'm guessing some platforms have newer versions by default that support it, but I know at least on RHEL 6 (which is an OS I have to work with from time to time, even though it's not even supported now!), dd doesn't have a progress parameter, so I still use pv there, and old habits die hard :) - pv is useful in many other contexts too, so not a bad tool to have handy.
@@JeffGeerling well - I honestly have to apologize to you on that one I started using linux on my root around mid-2014 - but at least as far as I recall it always had this option according to different sources I was able to look up it seems that this function just was added recently - and just to the version of GNU coreutils v8.24 back in 2016 (at least according to one source - while others dates to 2019) unfortunately as there's no "one official" version of DD but it's rather a common tool in way too many different versions it's hard to tell since when and on which platforms the status=progress option was introduced first as I'm just so used to that option I wasn't even aware of the existence of PV ... so, well, SORRY ... I guess?
Be so nice if there was a retro Pi build that let you use this Turing cluster. That way you could just add as many as you need, and if you need more just add them and be playing stuff like GameCube, and 3DS
Logan : Wouldn’t it be neat to run two or three Pi3’s for emulation gaming at normal temps, instead of one overclocked Pi4 with active cooling? Maybe it’s just me...
@@totophi Well couldn't one be solely dedicated for GPU task, and the other for CPU? I honestly dont know much about cluster computation, but i would imagine that would speed it drastically if possible
This isn't available in the dd version installed on mac by default. You'd need to upgrade your dd version for this to work. The pv + dd solution works on mac by default.
Thank you for this video. I have a few questions... 1) can you expand the turing pi to use two board or more? 2) can I use a PC as the head node to run computations along with the pi’s?
1) 6:22 I believe you could connect the two Turing Pi's via ethernet / a router to scale this setup 2) Depends upon software / configuration - if the software runs on both Windows (assuming this is what you meant by "PC") and Linux and it communicates over the network, then it should work
Thanks for the video, but I didn't get the math on the cost totals you gave until I realized you just summed together the cost of each line item, but didn't multiply each line item through by the quantity needed.
@@JeffGeerling No problem! After re-reading my comment I realize I may have rushed the wording and made it sound like a complaint, which wasn't really what I intended, sorry! More of just a friendly "hey this was confusing to me, maybe it was to others too". Overall, the content of the video obviously doesn't have anything to do with the specific pricing of any of the components really, so I don't even know why I keyed in on the specifics of that. All I can say is I must be getting bored in lock-down, and I hope you can forgive my pedantry. I've been really enjoying the recent videos you've been putting out, keep it up!
0:40 - Why 7 compute modules? I'm guessing it's the upper limit of the power supply chip/circuitry or something else on the Turing Pi board? Or is it just a common cluster size? My head says it should be an even number! lol 2:16 - Regarding the PoE HAT (www.raspberrypi.org/products/poe-hat/), istarter guides/videos mention is the fact that they don't have the necessary 'power management' chip/circuitry? Something like that lol
We use kubernetes where I work, and I'd rather NOT use it in my personal projects. Some other videos have shown that there are alternatives, and I'd like to see some content on those. Since I'm using elixir, that may give me some extra alternatives that aren't applicable with other technologies.
Thanks Jeff for all you do, I know I have the turning pi 1, and what you show does not seem to work for me. For some reason it's not honoring the user-data file on the Hyperiot. When I look at my network all I see is black pearl, so that tell me it's not reading the file or my public key for the ssh_athorized_keys: at the bottom. Anyone else finding this issue?
@Jeff Geerling - I have a Raspberry Pi 3 B+ that I’m using for a Pi Hole, but I’ve been thinking about building a homemade router. I usually have 4 to 8 devices connected to the network, usually streaming or sending high-speed data to a data center. Do you think a Bramble would be able to fill this function without overheating or becoming a bottleneck?
Is there a recent alternative to the turing pi? I am not able to find anything (at least not in the EU) as of now... But even without the turing pi: loads of good information in here. Thanks and keep it up!
Yes; note that the netboot setup is different depending on model of CM, it also requires one separate server (or the master Pi would suffice) to share filesystems via NFS. It's a little more complicated and I might cover that later, but didn't want to get too much into the weeds in this episode, especially since the way you set up netboot is different per Pi and the Pi 4 doesn't work with it yet (as of this writing?).
just for completeness the pi4 does already support netboot, I tried it and it does work, but you do need the latest firmware and to set it up you do need to boot each pi with an SD first in order to write the proper configuration in the eeprom.
One point Jeff. In episode 1 you spoke of the benefits of learning all the things that are needed no matter what the speed of the nodes. And the learning process for managing them can be as important as getting the required performance. But the Turing Pi by-passes many of those steps that the aspiring cluster student really should be learning. I agree that much of this is a combination of rote effort and boring repetitive work; but that is exactly what a student needs to be exposed to. It is not easy to build a major cluster, and the cabling will inevitably get to be a bit of a nightmare. This does not mean that I do not appreciate what the board provides, I just question it for the teaching role.
True; in that sense the model B is a little better of a teaching tool as it has all the basics that you'd find in a typical multiple-server installation. But even the Turing Pi requires flashing each Pi (which is a bit of a manual and boring process), and especially if not using HypriotOS, you still need to do the networking configuration/DNS for each Pi. The main thing taken away is plugging in power cables and network cables, which, to be fair, is something that you'd bypass on modern blade-based servers too. But your point is well taken-for the purest educational experience hardware-wise, multiple model Bs are a little better.
@@JeffGeerling As a systems engineer, I always found the routine of building servers into the racks was the hard part. Setting the servers up was easy. Of course that likely has more to do with how many servers I set up, vs how few racks I had to work on.
@@JeffGeerling Sorry Jeff, but I am retired now. It would also be a problem sending them from Australia. Also I used to work for Compaq, and now that is a name reduced just another trade mark.
The Turing Pi certainly seems simpler and more elegant than a “bramble” build of regular pi. However, am I missing something, or isn’t the end cost cheaper for buying the full pi units? You have to get SD cards and what not, but the Turing board itself is so expensive it appears to push it over the edge. Maybe it’s addressed in a later video, but I’m also not clear on which would perform better, or if they perform about the same?
Equivalent spec Pis will perform identically; the main things the Turing Pi board brings to the table are a more compact footprint with less cabling mess required. The total cost (all-in) is similar, but you can mount up the Turing Pi in a standard PC case or server case, which may push somone towards that solution if they're not just running it on a desk or shelf somewhere.
I've never really fully understood pi clusters. Can you combine the processing power of the clusters to run a single application faster or is this just for the purpose of getting faster computational power with combining cpu's for calculations?
It depends on the application. It's not like plugging ten CPUs together gets you 10x faster processing. Applications have to be not only aware of multiple CPU cores (many aren't), they also have to work in a way that can be parallelized / split up on different computers. Many web applications and some scientific applications are made in a way that can be scaled in a cluster like this, but many applications aren't (e.g. games). So a cluster could let you run more things but maybe not the same things faster, depending on what you're trying to run.
I can't get the ssh_authorized_keys: part to work in the user-data file. Which Jeff shows at 11:55 in the video. I have this in my file: ssh_authorized_keys: - ssh-rsa (this is where my SSH key would go like in the video) But I get this error in my YAML Linter (): found character that cannot start any token while scanning for the next token at line 21 column 1 Line 21 is the - ssh-rsa line. I have looked at: cloudinit.readthedocs.io/en/0.7.9/topics/modules.html#users-and-groups But it doesnt give any examples for me to follow. Any help would be appreciated.
I figured out how to get around the issue I was having. I didnt use the ssh_authroized_keys: this time, but did it another way. In the user-data file down near the WiFi section I added this, enter in your own RSA key at Your-Key-Here. And make folder name changes if you are not using pirate as user name. This worked for me and I was able to login with pirate and my private key on first attempt on Rasp-Pi bootup: runcmd: - [ mkdir, /home/pirate/.ssh ] write_files: - content: | ssh-rsa Your-Key-Here owner: root:root path: /home/pirate/.ssh/authorized_keys permissions: '0644'
The main use of this sort of cluster is doing things like distributed computing or kubernetes on the cheap. Each node is a discrete computer, with it own CPU, mass storage and RAM and therefore needs its own OS, even if the IO is offloaded to the Turing Pi. The threads are balanced separarely by the OSes running on each module, but you can run k8s or use MPI to coordinate the cluster.
How’s the coil whine on your PoE hats? I’ve used one in the past but it had a terrible noise and since these are in the room where I sleep, I don’t use it
Can you build something like this to execute trades? Stock exchanges, crypto currency and other financial instruments. For example, I want to get a 6% gain on X instrument. would it be possible to train the machine to search for stocks or any traded instrument that are very low and possibly increase and have it execute the trades? In addition can it learn by itself like AI? Thanks!
I'll probably be exploring performance differences in two episodes, after Kubernetes is installed. But yes, in general the Pi 4 will beat the 3 in most benchmarks.
Looking forward to getting the released turing pi... (IIRC I'm in for a bit over $500 with a full rack of 32GB Pi3+'s). I know learning Kuberntes probably makes more sense in an AWS sandbox... but I liked the idea of physical hardware.
$500 would buy you a few commercial usff boxes with i7's and 16gb ram each, retired from offices, something you could actually do a lot of computing on instead of a handful of pi's, but whatever. I don't get what people are using pi clusters for when you can do more real processing on a single 4 core desktop than a whole ton of pi's
Did you try to use Erlang/Elixir languages stack on this cluster? Its VM has native support for transparent clustering, and languages has some special features because Erlang was especially designed for such like installations from the 1998. PS: maybe I already asked it...
Very interesting video! Subscribed :) Doesn’t the Pi support PXE boot? “Real” cluster nodes don’t have dedicated storage as far as I know, this could save the hassle of dealing with SD cards entirely.
Could something like this be used as a dedicated server? Maybe for game servers such as Minecraft? Or maybe some more resource intensive stuff like rust?
Ok? What if I want to take this and stack 10 full cases of Pie cluster as shown in the video? Hi do I do this? Does the main board on raspberry pi cluster have POE or POE+ Integrated? I want to use this for my kube development lab for telecommunications use cases in seconds world countries, and my lap is Powered by green energy.
Yes; each Pi has it's GPIO pins available next to it, meaning you can use GPIO just like with a normal Pi. I believe the final version of the board will have headers installed too (so you wouldn't have to solder your own on, like with the prototype I have in these videos).
3:58 "into a card reader" .. I had to go back and replay that 2x cause I heard "into a carburetor" .. thought it was an unexpected joke from a totally dry video :D
Serious question: In part 1 you mention cores and reference AMD at $2k, but aside from 'fun' of the cluster, why not go with something like Intel E5 v2 series used for
For most applications, almost any current or even slightly-older gen Intel or AMD chip is going to be a better value than a Pi cluster, if you're looking for performance (or even efficiency). But for light but bursty workloads (e.g. web requests) an ARM processor like the one in the Pi does just fine, and with more CPU cores the cluster can distribute requests more efficiently as long as the task is not CPU-bound.
a few years ago i saw a small Chinese bitcoin mining company using a ton of Galaxy S8 as miners, just wanna ask is the pi cluster applies the same and can be used as miner too? im just curious, i dont know pi until it was suggested by youtube
For input / output options will something like this be slower than many pi 3 or pi 4 clusters without that board? Since I’m assuming less input output options unless built upon. Just curious. Fairly new to this
The Pi 4 has much faster I/O with its USB 3.0 ports and full Gigabit ethernet. The Compute Module has equivalent functionality to the 3 B+ (or whatever generation it matches), meaning it has USB 2.0 and ~100 Mbps ethernet. The Pi 4 gives the best I/O capabilities, if that matters to you. You could also use the compute modules as nodes and have a separate Pi 4 acting as a master, with attached SSD storage on NFS for more speed. It really depends on your needs.
what is the usecase for a raspberry pi cluster? is it used for educational pourposes? a blade cluster is not that expensive used on ebay and probably a lot faster so..
The one I'm using is the latest prototype, which I have to return soon. They're currently working on the final production run, hopefully they'll start shipping soon!
Okay...Tell me where you're going with all this. What is this able to do? Work (What can this do) I just ordered a Raspberry Pi 4. For example.. Can this do anything with X-Y-Z moves in a laser engraver? 3D printing? Just asking...
great I have a rpi4 with docker and on it influxDB, nodered, portainer.io, pihole, mosquitto, grafana, rtl_433. I would like to create a cluster so to add also tensorflow and move the containers... so I am very interested in what you are doing, but I think I am doing my cluster with RPI4 and not with the compute modules which have only 1 GB of RAM. I am very interested in your project, go on my friend...
Best part, is when you say there are links below. They are there just as you said. You're probably the only TH-camr that actually puts the link they promise into the description. So many well known big TH-camrs promise links, then never post them, and never respond to people asking for them. Thank for your well planned projects and thoroughness. How clean your website is, and how just because computers are so much more powerful today you don't put in inefficient or lazy code like everyone else does. Really says a lot about the kind of person you are. You're a true asset to the community.
"you don't have to spend an hour wiring everything..."
Bold of you to assume that this isn't my favorite part
that sweet feeling of an ethernet cable latching on......
@@pakidood I fully agree
just as bold to assume some people wont be trying to add wires that don't even belong
- 7 RPi B+ Power
- A pcb that save the mess of cables and power
- A standar case
- Software tailored for clusters ready for Raspberry Pi
This project have every thing that Raspberry Pi user will dream, nice work.
I think you shouldn't instruct people to create unencrypted SSH private keys.
"Press the enter key three times" after ssh-keygen is resulting in unencrypted private key.
Form a security standpoint those keys are worthless!
The Touring pi is $280 with power, $300 with all the addons. Cheaper to do it otherways. Ridiculously over-priced board. Absurd.
I have no idea why I am watching this series. I have no idea why I can't stop watching this.
Because it’s informative and you have a curious mind?
I'm now actually thinking to build a few of these for my School's Innovision Center
How'd that work out? Did it happen? There's another thing you can do with this, if you are into coding and already know some C. MPI_lib. It gives you a parallelization API such that you can divide a task across 28 devices. With careful coding, you can have an enormous speedup with certain categories of tasks. (multidimensional modelling, sorting, {en | de}-cryption, file compression, DSP)
or you can take Elixir/Erlang language stack which include OS-like facilities with clustering in its VM
15:08 favorite part of this video - I love working on projects because they push my boundaries even if they're not pushing the industry's.
They say any piece of hardware while in operation is obsolete. I was recently given five Minus Forum N-40 minicomputers to do something with during lockdown, and I want to cluster them. I got a Pi 4B to manage things, and would like to eventually use it for video processing. Thanks for your series so far, I think it will come in handy.
Really looking forward to following this project!
Can't wait for the next episode.
Here to learn more about clusters in general. Since I'm not really familiar with Pi clusters, I was expecting some type of PXE boot for each node to pull from a central image file. Interesting that each needs to be flashed manually. The more you know!
Thanks for making this series, they're very clear and easy to understand 👍
Great video, looking forward to your next part!
This is cool and definitely more efficient, but for me, the challenges of using standard Pis were the most fun part of the project
Whats the consumption like on your rig? Does it chug much?
respect for using drag from finder into terminal, and piping to pbcopy.
14:00 Turing Pi should have an IO shield.
Oh yeah,Even if they had one 3D printed :)
Hot glue does it too
Currently there's no IO shield for it, but I asked Turing Machines about it and they said they may make one for it. I could fashion one up myself, but it's not a huge deal for right now-this is a prototype that I have to return. For a more permanent build, I would be integrating a large, quiet fan, and probably an internal PSU.
@@JeffGeerling For 420$ you can get 5 Intel Xeons on 775 Socket from Ali
@@wmonk5642 Then buy them and make a video about it and upload it to TH-cam. I'd watch it. I take your point though. The Turing Pi motherboard is pretty expensive for what it is, though I think the Compute Modules themselves are good value and you are comparing new with very old.
Fantastic job. Great presentation, scripting, content. I'm now a subscriber and sponsor. Looking for future content on Pi clusters, kubernetes, containers, AI software and hardware on Pi.
I don't have a Ra pi, will never own one for some time. But, still watching it like I understand each and every word of it.
How can you not? They are incredibly cheap.
Now what I really want to know is if there's something out there like a PCIe card with a connection for the compute module that would allow the compute module to effectively function as an ARM Coprocessor on an x86 or x86-64 PC.
There is/was Intel Xeon Phi, which placed up to 72 cores essentially stripped out Atom cores, on a PCIe card.
Unless you write your code to specifically load binaries onto those arm cores, this will not help you one bit. And the x86 cpu in your desktop/laptop/whatever will very likely be much much faster than the rpi's arm cores anyway. Not very useful.
@@Messugga Those were x86 cores though.
@@Chuckiele Yip, as I said, stripped down Atom cores. I didn't mean to imply they were ARM in any way, shape or form.
@@Messugga I know, you didnt mention ARM, but thats what the original comment asked for.
Wow, I need one of these but I can't exactly spare £500+ for something that realistically isn't any faster than my server I got on eBay for £60.
Think you're missing the point of the video. Nobody "needs" this, he's just doing it cause he can and he is curious. Plus if you need a serious server you're not going after an ARM based server anyway.
Which £60 server outperforms this?
@@admiraldirtbag4336 anything, even an old HP DL360 G7
@@admiraldirtbag4336 any X86 CPU made recently, particularly if you're talking a used 1700x or something along those lines and 16GB of RAM.
The point of this, though, is to play around with clustering without having to fill up a room with ATX towers.
I would go with AWS. I've setup some T3 Nano instances, with spot prices you can get each one down to less than £0.02 a day. You can also stop them at any point and not be charged more.
As always, thanks for a great series of videos. Patiently waiting for Ep3.
I have 4 Pi 4's which I bought to try and get my head around Kubernetes/Docker (my experience is with more traditional server/VMs/hypervisors). The goal is to run some containers to manage media at home and share around the network (Airsonic, plex). I get stuck on how to set-up the networking, especially how this would integrate or be accessed from existing network:
- Can I put all 4 on my home LAN, so I can access them directly?
- Should they get their own network using the physical ports and I manage them over the WiFI?
- Do I need a separate device/another Pi to act as the router between the 2 networks (IP masquerading/forwarding etc.)?
- Can the Master node also act as this router?
Hopefully you will cover the network setup in detail in a future episode.
I built a cluster out of 5 4GB Pi 4's. I've got them all on my home network, using an 8-port switch. I have a sixth Pi 4 running as a file server (using gluster.) Each of these boards has a 16gb SD card for booting and a 16gb USB flash for the root file system. The file server also has a 256gb flash drive. Each of the cluster nodes mounts a shared volume which is housed on that drive. I have a seventh 2gb Pi 4 running pi-hole as a DHCP and DNS server, and use static DHCP leases for the cluster nodes. Each of the systems runs Ubuntu 20.04 minimal. For the actual clustering, I'm using k3s from Rancher, which I highly recommend over other approaches if you decide to run Kubernetes. At the moment, my cluster runs a web server, a git repository, a docker registry, and a Drone CI server. I can launch Drone agents within the cluster, but my current projects are building multi-architecture docker images, so I have a static Drone agent instance on an Intel system (it's an Atomic Pi, fwiw.)
I'm housing it all in the Cloudlet Custer case from c4labs, with some modifications for better airflow and cable management. I'm using the aluminum heatsink cases (just the top parts) that cover the entire top of the Pi 4 board, which keeps things nice and cool. I'm powering it all with a 10A USB power supply, which is adequate but not stable enough to support overclocking. I plan on building a better power system for this at some point.
I did most of the OS and software setup of the cluster boards using ansible (I write ansible infrastructure code for a living.) At some point, I'll make these roles and playbooks public (there's a few secrets in the files I need to obfuscate first.)
You can visit the cluster at project.kube.thejimnicholson.com.
@@TheJimNicholson Thank you for the detail, most helpful (and interesting).
Always wanted to learn and apply. Although RPI4 KNOWN for heat. A special designed ice cluster mount over blades with thin line per blade and super cooler on top would do the trick. Have it rest above blades mounted to corner post of base board. Mini turbine blade type most efficient in moving air.
Looking forward to your exploration. I am hoping I can run my machine-learning stack on this :D Its mainly build around doing a lot of small calculations. I am curious to IO latency and how well it runs Dask.
if you want to use mashine learning, i would use a Jetson nano
Hi Jeff. Thank you for doing this thorough tutorial!!
Awesome part, good to know such thing exists, it's looks wayyy more convenient than setting up all of these cables. Thanks for sharing!
Im super new to servers and clustering and i love all your vids keep giving me KNOWLEDGE lol
Great video! I'll be waiting for the next one. Love the project.
I would be interested in doing the turing cluster, if only to run something like boinc or folding@home for distributed computing research like seti or covid-19 study. I can also recommend looking into etcher pro when it launches, a multi-device flashing pod, if you want to save time and constantly mirror image files.
Neat stuff, but (as you say) preparing media for every node is a PITA. Is there something in the Pi's PXE network boot function that stops you from using it for clustering?
Awsome! Can't wait for the next episode.
So glad I found this channel! Definitely subscribing. 👍
5:15 "using PV to monitor the progress of DD command" - /me looking at the man page: dd - option: "status" - value: "progress"
so - using "dd if=image of=disk status=progress" does exactly the same without using additional PV ... today you learned
It depends on the version of dd you have installed - I'm guessing some platforms have newer versions by default that support it, but I know at least on RHEL 6 (which is an OS I have to work with from time to time, even though it's not even supported now!), dd doesn't have a progress parameter, so I still use pv there, and old habits die hard :) - pv is useful in many other contexts too, so not a bad tool to have handy.
@@JeffGeerling well - I honestly have to apologize to you on that one
I started using linux on my root around mid-2014 - but at least as far as I recall it always had this option
according to different sources I was able to look up it seems that this function just was added recently - and just to the version of GNU coreutils v8.24 back in 2016 (at least according to one source - while others dates to 2019)
unfortunately as there's no "one official" version of DD but it's rather a common tool in way too many different versions it's hard to tell since when and on which platforms the status=progress option was introduced first
as I'm just so used to that option I wasn't even aware of the existence of PV ...
so, well, SORRY ... I guess?
Great video, can't wait for the next one :)
Be so nice if there was a retro Pi build that let you use this Turing cluster. That way you could just add as many as you need, and if you need more just add them and be playing stuff like GameCube, and 3DS
Logan : Wouldn’t it be neat to run two or three Pi3’s for emulation gaming at normal temps, instead of one overclocked Pi4 with active cooling? Maybe it’s just me...
@@totophi Well couldn't one be solely dedicated for GPU task, and the other for CPU? I honestly dont know much about cluster computation, but i would imagine that would speed it drastically if possible
For those of us using Linux with gnome, just right click on img
File, open with image writer, then just select your SD card
You can use the status=progress parameter to dd instead of piping to pv
This isn't available in the dd version installed on mac by default. You'd need to upgrade your dd version for this to work. The pv + dd solution works on mac by default.
Thank you for this video. I have a few questions... 1) can you expand the turing pi to use two board or more? 2) can I use a PC as the head node to run computations along with the pi’s?
1) 6:22 I believe you could connect the two Turing Pi's via ethernet / a router to scale this setup
2) Depends upon software / configuration - if the software runs on both Windows (assuming this is what you meant by "PC") and Linux and it communicates over the network, then it should work
Sooo good broooo
Awesome! Cannot wait for your update!
Super interesting! Thank you for this video
Thanks for the video, but I didn't get the math on the cost totals you gave until I realized you just summed together the cost of each line item, but didn't multiply each line item through by the quantity needed.
Larry May ah, sorry that wasn’t so clear!
@@JeffGeerling No problem! After re-reading my comment I realize I may have rushed the wording and made it sound like a complaint, which wasn't really what I intended, sorry! More of just a friendly "hey this was confusing to me, maybe it was to others too". Overall, the content of the video obviously doesn't have anything to do with the specific pricing of any of the components really, so I don't even know why I keyed in on the specifics of that. All I can say is I must be getting bored in lock-down, and I hope you can forgive my pedantry. I've been really enjoying the recent videos you've been putting out, keep it up!
That's a really awesome picture on the wall behind you at 0:30 - Do you know if it's possible to buy that anywhere?
Loving this. Thanks dude!
0:40 - Why 7 compute modules? I'm guessing it's the upper limit of the power supply chip/circuitry or something else on the Turing Pi board? Or is it just a common cluster size? My head says it should be an even number! lol
2:16 - Regarding the PoE HAT (www.raspberrypi.org/products/poe-hat/), istarter guides/videos mention is the fact that they don't have the necessary 'power management' chip/circuitry? Something like that lol
We use kubernetes where I work, and I'd rather NOT use it in my personal projects. Some other videos have shown that there are alternatives, and I'd like to see some content on those. Since I'm using elixir, that may give me some extra alternatives that aren't applicable with other technologies.
Thanks Jeff for all you do, I know I have the turning pi 1, and what you show does not seem to work for me. For some reason it's not honoring the user-data file on the Hyperiot. When I look at my network all I see is black pearl, so that tell me it's not reading the file or my public key for the ssh_athorized_keys: at the bottom. Anyone else finding this issue?
Best quote of the day "A bramble of pies"
Looks cool man ! Great work
@Jeff Geerling - I have a Raspberry Pi 3 B+ that I’m using for a Pi Hole, but I’ve been thinking about building a homemade router. I usually have 4 to 8 devices connected to the network, usually streaming or sending high-speed data to a data center. Do you think a Bramble would be able to fill this function without overheating or becoming a bottleneck?
Excellent video very informative!!! Sounds like a project I could try
Great! Can't wait to see the next videos. One thing, ethernet spec minimum cable length of 1 meter.
The v2 of this board will only support 4 compute module v4. So for that can you connect two boards together and do testing if it works.
Is there a recent alternative to the turing pi? I am not able to find anything (at least not in the EU) as of now... But even without the turing pi: loads of good information in here. Thanks and keep it up!
Could you just netboot them, either to initially flash the eMMC's, or to enable the use of non-eMMC Compute Modules?
PXE boot would be awesome, so it would be possible to provisioning a cluster with something like Ubuntu MAAS or similar.
Yes; note that the netboot setup is different depending on model of CM, it also requires one separate server (or the master Pi would suffice) to share filesystems via NFS. It's a little more complicated and I might cover that later, but didn't want to get too much into the weeds in this episode, especially since the way you set up netboot is different per Pi and the Pi 4 doesn't work with it yet (as of this writing?).
Seems like that at least usb boot for the RPi 4 is in the beta firmware since May 15th :)
just for completeness the pi4 does already support netboot, I tried it and it does work, but you do need the latest firmware and to set it up you do need to boot each pi with an SD first in order to write the proper configuration in the eeprom.
One point Jeff.
In episode 1 you spoke of the benefits of learning all the things that are needed no matter what the speed of the nodes. And the learning process for managing them can be as important as getting the required performance.
But the Turing Pi by-passes many of those steps that the aspiring cluster student really should be learning.
I agree that much of this is a combination of rote effort and boring repetitive work; but that is exactly what a student needs to be exposed to. It is not easy to build a major cluster, and the cabling will inevitably get to be a bit of a nightmare.
This does not mean that I do not appreciate what the board provides, I just question it for the teaching role.
True; in that sense the model B is a little better of a teaching tool as it has all the basics that you'd find in a typical multiple-server installation. But even the Turing Pi requires flashing each Pi (which is a bit of a manual and boring process), and especially if not using HypriotOS, you still need to do the networking configuration/DNS for each Pi. The main thing taken away is plugging in power cables and network cables, which, to be fair, is something that you'd bypass on modern blade-based servers too.
But your point is well taken-for the purest educational experience hardware-wise, multiple model Bs are a little better.
@@JeffGeerling As a systems engineer, I always found the routine of building servers into the racks was the hard part. Setting the servers up was easy.
Of course that likely has more to do with how many servers I set up, vs how few racks I had to work on.
@@JeffGeerling I forgot... Thank you for the videos. They are very well done.
@@John.0z I would not be opposed to you sending me a few dozens servers and a rack for my homelab, I'll even make a video of the installation ;)
@@JeffGeerling Sorry Jeff, but I am retired now. It would also be a problem sending them from Australia.
Also I used to work for Compaq, and now that is a name reduced just another trade mark.
05:18 for this you can use dd with status=progress so it will you same information xD
The Turing Pi certainly seems simpler and more elegant than a “bramble” build of regular pi.
However, am I missing something, or isn’t the end cost cheaper for buying the full pi units? You have to get SD cards and what not, but the Turing board itself is so expensive it appears to push it over the edge.
Maybe it’s addressed in a later video, but I’m also not clear on which would perform better, or if they perform about the same?
Equivalent spec Pis will perform identically; the main things the Turing Pi board brings to the table are a more compact footprint with less cabling mess required. The total cost (all-in) is similar, but you can mount up the Turing Pi in a standard PC case or server case, which may push somone towards that solution if they're not just running it on a desk or shelf somewhere.
@13:58 why no I/O shield for it :(
How can I use it for rendering videos? (H264 & H265) Say using Sony Vegas or Filmora.
I've never really fully understood pi clusters. Can you combine the processing power of the clusters to run a single application faster or is this just for the purpose of getting faster computational power with combining cpu's for calculations?
It depends on the application. It's not like plugging ten CPUs together gets you 10x faster processing. Applications have to be not only aware of multiple CPU cores (many aren't), they also have to work in a way that can be parallelized / split up on different computers.
Many web applications and some scientific applications are made in a way that can be scaled in a cluster like this, but many applications aren't (e.g. games). So a cluster could let you run more things but maybe not the same things faster, depending on what you're trying to run.
If someone wanted to, could they link multiple Turing Pis together?
Yes
I guess you could network multiple clusters together. The board has a gigabit ethernet port (max 100MB/s per module).
Thats illegal, CERN will come for you!
Why don't you use network boot (PXE) instead of putting an SD card into each RasPi? You can save a lot of work and some money.
Exactly what I was thinking! Perhaps use iPXE to boot them over http and have an iSCSI boot volume?
I can't get the ssh_authorized_keys: part to work in the user-data file. Which Jeff shows at 11:55 in the video.
I have this in my file:
ssh_authorized_keys:
- ssh-rsa (this is where my SSH key would go like in the video)
But I get this error in my YAML Linter
(): found character that cannot start any token while scanning for the next token at line 21 column 1
Line 21 is the - ssh-rsa line.
I have looked at: cloudinit.readthedocs.io/en/0.7.9/topics/modules.html#users-and-groups
But it doesnt give any examples for me to follow.
Any help would be appreciated.
I figured out how to get around the issue I was having.
I didnt use the ssh_authroized_keys: this time, but did it another way.
In the user-data file down near the WiFi section I added this, enter in your own RSA key at Your-Key-Here. And make folder name changes if you are not using pirate as user name. This worked for me and I was able to login with pirate and my private key on first attempt on Rasp-Pi bootup:
runcmd:
- [ mkdir, /home/pirate/.ssh ]
write_files:
- content: |
ssh-rsa Your-Key-Here
owner: root:root
path: /home/pirate/.ssh/authorized_keys
permissions: '0644'
dumb questions, what is this for and do you need to have operating systems for each node?
how does this do thread ballancing? is that kubernetes?
The main use of this sort of cluster is doing things like distributed computing or kubernetes on the cheap. Each node is a discrete computer, with it own CPU, mass storage and RAM and therefore needs its own OS, even if the IO is offloaded to the Turing Pi. The threads are balanced separarely by the OSes running on each module, but you can run k8s or use MPI to coordinate the cluster.
Jeff, Turing pi is not hot pluggable , all node reboots when i add any node . Did you check uptime after the node addition /removal ?
Hmm is there a possiblity to use the tuning pi to make a cpu miner for XMR ? Since its 1/6 the price of a treadripper ??
How’s the coil whine on your PoE hats? I’ve used one in the past but it had a terrible noise and since these are in the room where I sleep, I don’t use it
Can't seem to find the turing pi cluster board ... where did you get it? I searched under that name and it doesn't show up anywhere
Can you build something like this to execute trades? Stock exchanges, crypto currency and other financial instruments. For example, I want to get a 6% gain on X instrument. would it be possible to train the machine to search for stocks or any traded instrument that are very low and possibly increase and have it execute the trades? In addition can it learn by itself like AI? Thanks!
so a cluster of raspberry pi 4's would be faster than this right? Since a compute module is < pi4. What would be the difference in computing power?
I'll probably be exploring performance differences in two episodes, after Kubernetes is installed. But yes, in general the Pi 4 will beat the 3 in most benchmarks.
@@JeffGeerling ok cant wait this is good stuff.
Thanks for this Video!
Is it possible to compare the performance between this Pi Cluster and a MacBook Pro ?
how well does the turing pi work as a web server?
Looking forward to getting the released turing pi... (IIRC I'm in for a bit over $500 with a full rack of 32GB Pi3+'s). I know learning Kuberntes probably makes more sense in an AWS sandbox... but I liked the idea of physical hardware.
It's definitely a lot more fun to do it with physical hardware!
$500 would buy you a few commercial usff boxes with i7's and 16gb ram each, retired from offices, something you could actually do a lot of computing on instead of a handful of pi's, but whatever. I don't get what people are using pi clusters for when you can do more real processing on a single 4 core desktop than a whole ton of pi's
Looking forward for this !
Did you try to use Erlang/Elixir languages stack on this cluster? Its VM has native support for transparent clustering, and languages has some special features because Erlang was especially designed for such like installations from the 1998.
PS: maybe I already asked it...
Very interesting video! Subscribed :)
Doesn’t the Pi support PXE boot? “Real” cluster nodes don’t have dedicated storage as far as I know, this could save the hassle of dealing with SD cards entirely.
I've been meaning to do a video on netboot at some point...
Sounds great, as long as you want to buy such a turing pi, you have to predorder ...
Could something like this be used as a dedicated server? Maybe for game servers such as Minecraft? Or maybe some more resource intensive stuff like rust?
Ok? What if I want to take this and stack 10 full cases of Pie cluster as shown in the video? Hi do I do this?
Does the main board on raspberry pi cluster have POE or POE+ Integrated?
I want to use this for my kube development lab for telecommunications use cases in seconds world countries, and my lap is Powered by green energy.
Check this guy's blog out.... www.kaibader.de/3d-printed-raspberry-pi-rack-mount-with-heat-sink-passive-cooling/
What about using this as a sensor hub for agricultural sensors. Love to talk about the farm and applications.
does the on-board memory bus allow faster speeds?
There's now a 64bit OS from the Raspberry Pi Foundation and an EEPROM with USB-boot support.
Once you connect them, you can add many i/o options for machinery like servos, sensors, etc? I’m new to this
Yes; each Pi has it's GPIO pins available next to it, meaning you can use GPIO just like with a normal Pi. I believe the final version of the board will have headers installed too (so you wouldn't have to solder your own on, like with the prototype I have in these videos).
I just ordered one of those boards. Did you have any issue getting SD cards in an out of the board when it was in a case?
3:58 "into a card reader" .. I had to go back and replay that 2x cause I heard "into a carburetor" .. thought it was an unexpected joke from a totally dry video :D
My new hybrid "hacking a raspberry Pi cluster into a small block 350" series, coming soon :D
多謝！
Serious question: In part 1 you mention cores and reference AMD at $2k, but aside from 'fun' of the cluster, why not go with something like Intel E5 v2 series used for
For most applications, almost any current or even slightly-older gen Intel or AMD chip is going to be a better value than a Pi cluster, if you're looking for performance (or even efficiency). But for light but bursty workloads (e.g. web requests) an ARM processor like the one in the Pi does just fine, and with more CPU cores the cluster can distribute requests more efficiently as long as the task is not CPU-bound.
@@JeffGeerling Thanks for the reply! Great video and I'll give it a shot with a few spare pis I have around :)
It looks like hypriot is no longer supported, do you have a replacement that you like?
a few years ago i saw a small Chinese bitcoin mining company using a ton of Galaxy S8 as miners, just wanna ask is the pi cluster applies the same and can be used as miner too? im just curious, i dont know pi until it was suggested by youtube
For input / output options will something like this be slower than many pi 3 or pi 4 clusters without that board? Since I’m assuming less input output options unless built upon. Just curious. Fairly new to this
The Pi 4 has much faster I/O with its USB 3.0 ports and full Gigabit ethernet. The Compute Module has equivalent functionality to the 3 B+ (or whatever generation it matches), meaning it has USB 2.0 and ~100 Mbps ethernet. The Pi 4 gives the best I/O capabilities, if that matters to you. You could also use the compute modules as nodes and have a separate Pi 4 acting as a master, with attached SSD storage on NFS for more speed. It really depends on your needs.
what is the usecase for a raspberry pi cluster? is it used for educational pourposes? a blade cluster is not that expensive used on ebay and probably a lot faster so..
Can I use the lower price Orangepi board with the same method to make it
Where did you buy the Turing pi board? On their page they say it is still a beta and is not available before June/July
The one I'm using is the latest prototype, which I have to return soon. They're currently working on the final production run, hopefully they'll start shipping soon!
Waiting for your next video.
Okay...Tell me where you're going with all this. What is this able to do? Work (What can this do) I just ordered a Raspberry Pi 4. For example.. Can this do anything with X-Y-Z moves in a laser engraver? 3D printing?
Just asking...
Skip _dd_ and save the overhead:
• Enter a root shell with _sudo su_
• _pv /path/to/image.img >/dev/disk_
great I have a rpi4 with docker and on it influxDB, nodered, portainer.io, pihole, mosquitto, grafana, rtl_433. I would like to create a cluster so to add also tensorflow and move the containers... so I am very interested in what you are doing, but I think I am doing my cluster with RPI4 and not with the compute modules which have only 1 GB of RAM. I am very interested in your project, go on my friend...
I want to build a 30-40 sbc cluster what do u recomend?
Is RPi able to boot via network without an SD card? I did it with the first model, but card was required.
Can you show configure Pi 4 cluster or drop any resource, please?
I am in need.
I have no clue what I would use a raspberry pi cluster for, but it would be a unique build