The RPiCluster
ฝัง
- เผยแพร่เมื่อ 20 ก.ค. 2024
- Documentation, Source code, and EagleCAD designs: bitbucket.org/jkiepert/rpiclu...
Summary:
The RPiCluster is a 33 node Beowulf cluster built using Raspberry Pis (RPis). During my dissertation work at Boise State University I had need of a cluster to run a distributed simulation I've been developing. The RPiCluster is the result. Each of the 33 RPi is overclocked to 1GHz and is running Arch Linux. This demo shows the RPiCluster running a parallel program I developed using MPI to control all of the RGB LEDs installed on each of the nodes.
The Whole Story:
The RPiCluster project was started in Spring 2013 in response to a need during my PhD dissertation research. My research is currently focused on developing a novel data sharing system for wireless sensor networks to facilitate in-network collaborative processing of sensor data. In the process of developing this system it became clear that perhaps the most expedient way to test many of the ideas was to create a distributed simulation rather than developing directly on the final target embedded hardware. Thus, I began developing a distributed simulation in which each simulation node would behave like a wireless sensor node (along with inherent communications limitations), and as such, interact with all other simulation nodes within a LAN. This approach provided true asynchronous behavior and actual network communication between nodes which enabled better emulation of real wireless sensor network behavior.
So, why I would want to build a Beowulf cluster using Raspberry Pis? The Raspberry Pi has a relatively slow CPU by modern standards. It has limited RAM, slow USB-based 10/100 Ethernet, and its operating system runs directly on a SD card. None of these "features" are ideal for a cluster computer! Well, there are several reasons. First, when your dissertation work requires the use of a cluster it is nice to ensure that there is one available all the time. Second, RPis provide a unique feature in that they have external low-level hardware interfaces for embedded systems use, such as I2 C, SPI, UART, and GPIO. This is very useful to electrical engineers (like myself) requiring testing of embedded hardware on a large scale. Third, having user-only access to a cluster (which is the case for most student-accessible systems) is fine if the cluster has all the necessary tools installed. If not however, you must then work with the cluster administrator to get things working. Thus, by building my own cluster I could directly outfit it with anything I might need. Finally, RPis are cheap! The RPi platform has to be one of the cheapest ways to create a cluster of 32 nodes. The cost for an RPi with an 8GB SD card is ~$45. For comparison, each node in one of the clusters available to students here at BSU, was about $1,250. So, for not much more than the price of one PC-based node, I could create a 32 node Raspberry Pi cluster!
Update: While the BeagleBone Black was not available when I started this project, I would have chosen it rather than the Raspberry Pi had it been available. It is the same cost once you include an SD card, but it has onboard 2GB of flash storage for the operating system. It also uses a Cortex-A8 ARM processor running at 1GHz.
Cluster Performance:
I measured basic computing performance in a number of ways (see the paper). MPI performance was measured using HPL (www.netlib.org/benchmark/hpl/) with ATLAS (www.netlib.org/atlas/). The RPiCluster achieved 10+ GFLOPS peak, with 32-nodes running HPL. The single 3.1GHz Xeon E3-1225 (quad-core) system, I used for comparison, showed about 40 GFLOPS peak (when the HPL problem was optimized for Xeon system).
When I run the HPL problem that achieves 10 GFLOPS on the RPiCluster, the Xeon system achieves about 2 GFLOPS. This is because the HPL problem size is so large that it causes paging on the Xeon system. The Xeon system has 8GB of RAM (~6GB usable after OS, etc) whereas the RPiCluster has about 16GB of RAM (~15GB usable after OS, etc).
More information: coen.boisestate.edu/ece/raspbe...
Update:
I finished my PhD Spring 2014. For those interested in further details on what I was doing, an electronic copy of my dissertation is available here: scholarworks.boisestate.edu/td...
Now I know why I was having a hard time finding a Raspberry Pi in stock.
lol
Haha
lmao
Ha ha
Amazon LOL
Watching this exactly 10 years after the video published. Still a badass setup to look at.
I accidentally clicked 2 rpi2 purchases; maybe this guy made a similar mistake but on a bigger scale
ROFL!
walter0bz He wanted 3 and bought 33. :3
Lol
What in the World am I going to do with 33 Raspberries !?!?!
Step. 1 Complete a dissertation to justify to Significant Other
Step. 2 ???
I heard Michael J. Fox built a 400 RPI cluster! Amazing!
Why are there so many people who just have negative things to say. Every hobby does not have to make money or even sense. There is the experience, the learning, and just fun. I mean it's like people who put extra lights on their cars. It's not my thing, and I would not use my money there, but it they are having fun who's place is it to tell them it's wrong. Well as long as it's not hurting anyone. So come people. Be kind. Or at least constructive.
There are a lot of posers out there who think they know everything because they have blinked an led on the arduino.
Very nice job on your project, wish I had the know how to do something like this.
hobbyhands yes here is someone who agrees with me!
human beings are toxic. that's why.
@Блядь Россия It's not a waste. There are a huge number of applications that benefit from being run in parallel, such as physics simulations and raytracing. For those specific use cases, that cluster can perform dozens of times better than a thousand dollar desktop PC. And in fact, if you pay attention at all to what he says in the video, that's exactly what he's doing: he's using it to parallelize a physics simulation for his dissertation, and I'm sure the difference in performance saves him days of time.
Turn this into a RuneScape bot farm and make the Venezuelan economy crumble even more
Hayden Keast underrated comment
sseth type comment
oh yes
So there isn't already enough human suffering there?
But would it be cost efficient? Assuming you pay 35$ each, the total will be around 1k. Assuming you can run 2 bots on each of them, thats around 60 bots. If you built a PC with the same value, wouldn't it be able to run over a 100?
The LEDs are what got me. Fuckin' dank.
RPi now stands for Razer pi :)
For anyone out there confused about what this cluster of computers are doing, he's basically simulating a real world problem where you have several computers/devices wirelessly communicating data to the same database at the same time, but in different instances.
This explanation doesn't do it justice, but this is it in Layamon's terms. Read the description for the real one ;)
MNCpoops
MNCpoops thank you. I just started with electronics and Raspberry pi and stumbled on to this video. It's cool looking but I was waiting for an explanation.
WTF is Layamon's terms? Can you maybe explain that to me in layman's terms? I don't know most of these technical terms.
This was a real world problem 10 years ago.
Well, to be honest, this video was posted on 17 mei 2013, so like 6 years ago. So I guess it was still a bit relevant back then?
Now i don't need to imagine a beowulf cluster made of raspberry pi anymore. Thanks.
I realy like the 5v feed you use.
saves alot of extra wires and generaly made your setup look realy clean.
The controller for the lighting is more powerful than the cluster itself.
As an Electrical/Computer Engineer... I can say that this is one impressive feat! Well done sir! I'd love to see some benchmarks on this little bugger.
I like that you do both send/receive as well as broadcast messages, clever idea!
Me: Mom, can we have a 32 core system?
Mom: We already have one 32 core system.
*The 32 core system at home:*
@Badr Ahmed These are Raspberry Pi 1's so they do have only 1 core per cpu.
The synchronization of the blinking lights is well done. You must have good programming skills.
Beautiful, this is why I love TH-cam creators.
That was so cool. I'd love to read through your programming
I love the power solution! I was trying to imagine all those micro USB power connectors, but applying power to the header and adding your own safety fuse (and programmable LED indicator) was completely clever. What became of the cluster after your research? Does anyone still try experimental loads on it? Are there practical, economic workloads for it?
Mining, hacking, tons of applications = economics
Very cool, hope you got top marks on your dissertation
I'd love to see a build like this with ODROID U3. The U3 is a new $59 board with a 1.7GHz Exynos 441x Cortex A9 and 2GB
Wish I had the money to spare, would be a fun project :)
A really nice project. A very worthwhile investment. Hats off to you and keep up making more inspiring projects like this.
I dig it. It's simple looking and clean and I'm sure highly functional. Good stuff.
It looks really cool but I'm wondering what you are simulating on it.
At last, we can properly use the classic Slashdot meme: "Imagine a Beowulf cluster of these things!"
Nice work. I stumbled upon your video as I was looking for alternative ways to power a RPi. (Now I know I need to fuse the line if I use the GPIO pins to power it). Back in the early 90s I built a 64 node Inmos Transputer (look it up) cluster for my Masters thesis. Seeing your build reminded me of that one.
Amazing job! I wish my dissertation 10 years ago involved something as neat as this!
Upgrade those to Pi 3's!!! Then you'd be killin it!
Omfg yes
Upgrade those to Pi 3 B+!!! Then you'd be PoEin it!
That would be soooooooo expensive
@@SomeNot it already was
Scottie D369 now 4 b’s
Give this man all of your attention!
Vedant Mathur Should I do this?
Do it!
This is very impressive, I wish I had the know-how to do this. It's exciting to think of the possibilities of all this.
This project is done very professionally. This Josh Kiepert dude is a shoe in for almost any HPC job out there - he probably already has one
parallel programming: 100
I finished this project too called the 'Sleeper Pi'.
Very impressive, even w/ 2013 tech, & great video presentation, thanks!
A thing of beauty...Well done, mate!
i have no idea whats going on in this video and you might as well be speaking chinese but i am impressed regardless. way cool
He's programmed them to be one computer.
This is a cluster, the computers behave as if they are one PC.This is basically the equivalent of 32ghz and 16gb of RAM.The computer on the top is controlling the cluster, you don't use it's resources in the cluster itself..You can definitely produce a computer of similar power for much cheaper, and MUCH easier to design, but the purpose of this project. I heard some people call it a super computer, but a computer with 1 or 2 (at most) high end processors should top this, and you can put 16gb on practically any new motherboard out there. A supercomputer would be dozens of times more powerful than that. Anyways, looks great man!
Diego Garcia do GHz add up?
Wonder how much better this cluster could be with the new Pi3B+
This is the most beautiful cluster f#*k I've ever seen. Nice work. I recently found out about the RPi less than two hours ago and I am in love.
thats some savage work with the pcb and leds
Does this make smoothies?
I'm assuming virtual machines weren't going to cut it? a number of projects in my degree required network environments and vmware worked beautifully for me :)
And this is far more fun.
Well done sir, MPI and all. Now, render some fractals on this badboy!
I am proud of your work! I just read your dissertation
When you only wanted to scroll down two times using the keypad "Pg Dn", but forgot to disable NumLock and hit "ENTER" to purchase.
awesome...
This is pretty spectacular. Really impressed!
Impressive design - nice job
The problem is that so many facebook pages and stupid portals present this as a "cheap supercomputer" project. It is not. Ok, you can experiment with parallel processing with 30 or 64 nodes, but on the whole video I couldn't find a single piece of information about (i) total power consumption and (ii) total processing power of project (except the ARM processor frequency). So in conclusion, the whole project gets more attention that it should from people who look after something different. I assume that an OpenCL software on a single R9 280X will be x10 times faster than the whole 33 node project - and around 7 times cheaper too!
Well, as you said he did say he was going to overclock them to 1ghz each. But to cut the guy some slack, There are some valid scenario's for such as setup. 1) he could be building this as a prototype for an application he wants to run on a real big ass super computer. Which would cost money to rent time on. So having the behavior of your application across multiple nodes well tested will definitely help here. 2) He seeks to replace an existing supermachine program. But it's too risky to just "swap the program" and see if the modified version works. 3) He just wants to educate himself on the behavior of programs across multiple nodes. 4) everything else! Either way there could be many valid reasons for building this stack. And i personally think this stack was well executed.
So you're saying he can get more raw flops with a graphics card. Duh? If you had paid attention to the first 35 seconds of the video you would have noticed that he said he needed a cluster. While a graphics card would definitely provide more raw flops, that might not matter at all: remember that the ARM chips in the RasPi's are optimized and designed for different types of computing. If this guy's application is designed specifically to run on 32 individual ARM processors, then he's going to get MUCH higher performance running it on what it was designed for, instead of running it on some shitty consumer-grade graphics card.
dizzious
i think you mised the part of when he said he is writing a paper on how this works ... never went onto which part of it he is writing about though. In that how would you be able to emulate a 32 node project on a OpenCL software program? it is a project on and about how 32 nodes work together in sync thus if you only have 1 node even acting as 32 nodes you will not be able to get the same result with all the same errors, bugs and gliches that needed to be fixed in it.
Why the paper while probably because he is going for his master's and/or Ph.D thus actually haveing a 32-node cluster is needed even though he did go a little overkill with the power supplys and case fans. The power supplys could have been replaced with a custom power adapter made for powering mutliple devices on a 5V power line but not knowing how many amps are actually needed a PC power supply might have been the best solution not only the quickest solution to it.
dizzious
Which part of what I wrote was hard to understand?
Teodore Hatzikostas i think you missed the point of the project as it is a proof of concept device not an actual computing device there is a major difference thus yeah it has 32 nodes that can work together and all are overclocked to a speed that they could be synced at without noticeable latancy time but it is for a paper not a business nor a consumer application thus after he is done with the paper he will most likely either 1 keep it for future explaining about the how he got the data and/or 2 sell it to the university who sponsered it most likely as $1,000 for a device to just write 1 paper is a pretty steep cost to which is probably only going to be a foot note and/or a small paragraph in the paper while the rest is about how a 32 node system works together and how well.
Again he mentioned that using the raspberry Pis was the cheapest way but still $1,000 USD is not that cheap of a price ... probably was that high because of the PC PSUs though to which he could have gotten away with only 1 and used a diffenret PSU with more adapters on to the wires but that is assuming any PSU on the market will have 4 independent 5V outputs with enough amps to power that thing.
Is there a way to show it in action somehow. It looks so cool I would like to see it run something.
I was thinking the same thing, I would like to see some kind processor-intensive application run on this, but with the work distributed over all of the nodes, let's see if it speeds things up vs a single, high powered device.
It's kind of pointless otherwise (although I'm not saying it isn't an awesome idea, I just need to see some application and some context to really get the point)
Josh, my friend! Thanks a lot ...keep the cool stuff going.
Nice work one of the biggest clusters I've seen
can it run minecraft at 300 fps?
just go to Amazon and search there for raspberry pi.... very funny question because it will still run mc with 3~4fps 😜 cpu isn't important for mc... the ram is important
MichalPlays well, given that there are like 30 pi's in a cluster, with about 1 GB or ram each, 30 GB of ram is wayy more than enough for any type of Minecraft playing.
Go ahead and try to run minecraft with 32gb of ram and a really crappy cpu, you'll get FPS problems.
Jett Plays true, but since there are like 30 raspis connected together, I infer that several pi's will have much superior processing power, since 1 raspi can run Minecraft PC at about 10 fps, 30 pis should be able to pull 300 fps, that is, if the processing power is perfectly added up and used.
you are stupid it doesn't work that way you will know if you had a sli config. in your pc or atleast studied this for 2 minutes..
Yes, but can it run crysis?
+iTheStopSigni It was a rhetorical question, but thanks ;)
+iTheStopSigni No...
But, Crysis is Windows only...
+davidagat521 Actually, It's available for linux, which many raspberry pi OS' are based on.
+Mohammed Hamza its ARM architecture not x86 so it cannot even run windows only specialized linux distro's. Also the 33 pi's performance would not scale and its on board graphics are very, very bad, its pretty much playing crysis on an iphone.
+Mohammed Hamza Now it's "can it run Witcher 3?"
That Looks Pretty Cool =) Nice Job And Thanks For The Upload!
Amazing work thanks!!!! for sharing, look forward to more!!!!!
can you do some kind of benchmark of yhe cluster computing perreformance
Check out the video description additional details :)
yeah i like to see that also
What are the practical applications of a cluster like this?
+Benjamin Norton he needed to test a cluster program and needed a cluster to do that so this was the cheapest option
I guess what I'm curious about is if these could speed up the rendering process with something like video editing or making 3d fractals.
I have no idea how this is practical at all
+Benjamin Norton what are the practical application of the Christmas tree lights.
+Benjamin Norton Can't you see in the video? It flashes LEDs in fancy patterns! :P
looks very cool with the leds
That is really freaking cool. Thank you for doing this!
I haven't seen a response on any of these super computing raspi builds. What is the overall computational capability of your cluster?
I guess it would be 33 times the power of a raspberry pi b
+Nathan Rogers Please see the video description for lots of details regarding the performance of this cluster ;)
+Phoenix Wright
I think it would have 2^31 times the amount of power. That's like 2 billion!
Richard Vasquez really ?
lol. You're good.
That's a really smart RGB led matrix xD
The leds are only indicators, each one of those PIs is a node in a bigger cluster of computers, together they can do certains types of calculations far better than a desktop pc.
Ho boy ...
@@AishaDracoGryph It was a joke lol.
Nice work man, loving it!
1,000 Subscriber
Congrats!
One big mistake tho.. When one of the "pies" die.. you got alot of work to change it. Why not make small slides to them you'd get a quick "hot swap" or rather "cold swap" system to replace dead nodes. Othewrwise quite cool dude
What's the possibility of it breaking and it wouldn't even be a big job to change.
How much u want for it
Nice job getting your PhD, man. Impressive stuff.
This is very memorizing to look at.
So this is why Raspberry Pi's are always out of stock..
Just curious, it's now been 3 years since you posted this video. Are you still utilizing this project? Have you been able to repurpose this for anything else?
After completing the build, I continued to use the RPiCluster to finish simulations for my PhD research up until I graduated in May 2014. The electrical and computer engineering department at Boise State University provided the funding to build the cluster, so it currently resides at the university and I no longer have access to it. I haven't heard if anyone else is using it since I graduated.
+Josh Kiepert maybe they use it to mine bitcoin xD (not effective but you know)
It's their money :/
Chris Grabo I'd be excited about just having my hands on 30+ pis that's nuts.
Just curious, but how many SD cards did you chew through using this cluster?
downloaded your dissertation and although I barely understand exactly what is you are trying to accomplish I respect your work and will try to better understand not only your build but your research/work. Thanks!
+Juan A. Tena Thanks!
Yo dude congrats on your PhD!!
Seems fun!
But Can It Play Crisis?
So what are you using them for? Code compiling? Encoding/transcoding media? Running a videogame server? Torrenting? Downloading porn? DDOS'ing?
See video description
***** totally geeeey, eh?
He couldn't use it for ddos' ing as the traffic would still only come from a single source. But he could for dos attacks but not really useful
Jake Guard anon has spoken
Faster
Nice one dude! Congrats on your PhD!
This is Incredible! Very impressive!
But what do you DO with a supercomputer?
Besides, like, cracking passwords extremely fast?
He said in the video he wanted to use it for physics simulations for his dissertation
GET *_ALL_* THE DOGECOINS
Very nice project! Keep going!
You are a genius and this is a work of art!
And I can barely write a "Hello world"-program..
Youll get there! Keep going man!
I wouldn't do that...
I think is better and your should be in your body tags.
HTML isn't exactly programming. Here's an example:
C/C++: cout
***** Well thanks for repeating my comment. Lol
C would be printf("Hello World");
C++ would be cout
i like pie.
Very nice build!
my god you condensed a whole server room into the size of a home PC BAD ASS
Here is a really nice 40-node RPi cluster build! hackaday.com/2014/02/17/40-node-raspi-cluster/
Thanks! I think your cluster is one of the nicest out there. I'm interested to see what kind of work you've done on it once your dissertation is published.
Now combine the 33 node with 40 node for a 73 node cluster.. Wow that'd be awesome.
***** Wireless sucks and is slow.
I live in the other side of the house, far from the router. We would have to drill into the attic and we have a racoon ;P
Seth Tipping there is wireless usb wifi dongles you can get to wirelessly connect to the internet
now replace all of the pi's with raspberry pi gen2's and we are talking some serious performance
***** waste of money if he doesn't need more performance
waste of money anyway. He could just have used virtual machines to test his simulation software.
Great one, a 40-node banana pi cluster build is a great new idea.
Very intersting project, clever and cool.
rough calculation, this costs at least 1200 dollars. I am just wondering if this will be faster or more powerful than a 1200 dollars build i7 computer.
Marta Kurtovic thanks for replying. i just want to compare this 33-node rpi cluster with one i7 computer. which one is faster (more powerful)? assume they have the same cost. any one knows?
Heya.. the answers ur looking for are near the bottom of description. Always good to read that; just like FAQs b4 posting in Forums. Anyhow, a single proper PC would cook the pants off this cluster. But it was made to emulate a wireless-- --- and a good idea too! Imagine all the processing power just sitting idle in academic or business networks!! Which could otherwise be used for protein folding or something...!
glytchd thanks for your kindly reply
***** Yeah. The answer is not that simple. It depends on the program you are runing. This is not built to run a game. It's built for parallel computation. Where you need to do lots of calculation in a short period of time or even realtime in some cases. It's like you want to compare apples with grapes, each one has pros and cons.
666Pulsar666 like your answer, thanks
What do you use it for or why bulid it, to what purpose ?
He clearly says its for his dissertation for university
i think this for different hacks something like brutforce wi-fi networks
I think you should learn English.
This gets my vote simply for its uniformity! Well played.
even after watching the entire video a few times, i still have no idea what something like this is used for :p
looks amazingly cool tough!
What are you using it for?
Just read the *ABOUT* section of this video.
David Essex Thanks. Didn't help much.
Andreas Suojanen There's all kinds of info around. Not sure if this guy is reading here on g+ to help figure it out.
David Essex Simple? I have no idea what a beowulf cluster is for. :P
God damn it Andreas! You're not understanding something that's so simple if you took computing as a masters degree! Gah!
To me this is a mini supercomputer
It is that to you because that is what it actually is, and why it was built
+Robogames You took the thought right out of my head lol
Dude that thing is friggin awesome
Dude!! This is sweeet! Great job
But...
Can it run minecraft?
But can it run crysis?
BreeZy HD Yes.
Probably. I mean it's essentially 33 N64s more or less.
falt And can it simulate x86?
Amazing build!
I saw the thumbnail for this and thought it was a sweet ass home server build. But I continued watching coz this is pretty interesting.
yea you could finally play minesweeper over 60 fps. congrats
lol
It’s worth do it to mine Bitcoin??
No where near industry level performance for Bitcoin . The guy states its for poc for his dissertation Bravo i say.
Dave Jb Excuse me!
Flavio Borges Ammm ok , I was not giving out just saying .
nice setup mate
Nice presentation, A special "like" for the serious docs, Eagle as well, As the question of benchmarks always comes up, the comparison with "standard" nodes did it. As a owner of the both boards, I agree with your opinion about the BeagleBone Black with some slight remarks : the cost of the both is not really equal, almost when talking about 32 boards it's non negligible. On the other side, the performances are really different too : while the PI use USB for the ETH the BBb have a true controller + onboard flash +2 PRU to go Real time, without talking about headers and capes which could ease clustering.
Thank you for this