Can My Water Cooled Raspberry Pi Cluster Beat My MacBook?
ฝัง
- เผยแพร่เมื่อ 4 มิ.ย. 2024
- I recently built this water-cooled Raspberry Pi cluster and now it's time to put it to the test. I ran it against a Windows PC, a MacBook Air and a single Raspberry Pi to see how it compares. On each platform, we searched for all of the primes numbers up to a certain limit and compared the times it took to reach the target limit. You'll be surprised by the results!
People have also asked to compare the cluster to one of AMD's Ryzen CPUs, so if any of you have one then please download the script below and try running it on your computer. I'd also be interested in seeing how a Pi 400 compares.
Download the Python script - www.the-diy-life.com/can-my-w...
Download the multi-process Python script - www.the-diy-life.com/can-my-w...
Read the full write-up, including details on the cluster setup - www.the-diy-life.com/can-my-w...
Basic Parts List & Purchase Links
Display Used For Testing - amzn.to/2NHJlUn
8 x Raspberry Pi 4B (2GB Model Used) - amzn.to/3i0eYni
TP-Link 16 Port Ethernet Switch - amzn.to/2Ldn9km
Rav Power USB Charging Hub - amzn.to/2MNCSH3
HD Touchscreen Monitor - amzn.to/38ug1Zq
Water Cooling Kit - bit.ly/3sfTAiO
8 x 30mm Cooling Blocks - bit.ly/38vZa8I
8 x Ethernet Patch Leads - amzn.to/2XPjfAS
8 x USB C Cables - amzn.to/35rOwOp
3m RGB LED Strip - amzn.to/3nyaPbt
M3 Standoff Mount Kit - amzn.to/38u6Qs8
M3 Red Aluminium Standoffs - www.banggood.com/custlink/KKv...
M3 Screw Kit - amzn.to/38wl63J
3mm MDF Board Approx. 600 x 600mm
Laser Cutter Used - K40 40W CO2
Option 1 - amzn.to/3sB9xAb
Option 2 - amzn.to/3nSGrZz
Note: Some of the above parts are affiliate links. By purchasing products through the above links, you’ll be supporting this channel, with no additional cost to you. - แนวปฏิบัติและการใช้ชีวิต
Adi Sieker has put together a multi-process version of the script - www.the-diy-life.com/can-my-water-cooled-raspberry-pi-cluster-beat-my-macbook/#multi_test_script
This makes use of multiple cores and threads on the CPU it's being run on, so is more representative of the processing power of the whole CPU.
I'll post updates as I run it on each system:
HP Laptop:
10,000: 0.9 s
100,000: 18.27 s
200,000: 66.99 s
500,000: 374.3 s (6 min 15 s)
MacBook:
10,000: 1.42 s
100,000: 35.64 s
200,000: 142.12 s
500,000: 827.53 s (13 min 45 s)
Pi 2.0Ghz:
10,000: 0.9 s
100,000: 109.02 s
200,000: 448.02 s
Pi Cluster 2.0Ghz
10,000: 0.9 s
100,000: 15.88 s
200,000: 57.67 s
500,000: 312.45 s (5 min 10 s)
CPU: i7-10750H 6 cores / 12 threads at 2.6GHz
10K: 0.26 s
100K: 17.59 s
200K: 64.13s
500K: 375.43s
The code on the blog is also buggy, need to change chunks to 100 instead of 1 in last parameter. Ryzen 5900x does 200k in 4.1 seconds with stock amd cooler.
@@esteban- just changing chunks to 100 brought the 500K down to 97.94s on mine.
@@acidspark You can probably halve that by a small fix on the finding primes code too; it's ilogical to divide from 2 to candidate number, if the number is not divisible by 2 then it's not divisible by any number bigger than N/2 so range(2, candidate_number/2) will cut in half the tries. That anyway would be cheating to compare with the video numbers, but using all processes correctly no.
With max_number on 200000, mine run 121 seconds on the single threaded version vs 78 seconds on the multi-thread version.
Intel i5-8300H (8) @ 4.000GHz 4-core/8-threads
Please tell me you called it Octopi.
Octopi already exists
It’s used for 3d printing
Octopis
This comment is underrated.
Nice name🙂
Tested the script on my Ryzen 7 5800x running linux 5.10 and python 3.9.1 with the schedutil governor:
To 10000: 0.27 seconds
To 100000: 21 seconds
To 200000: 73 seconds
So faster than the cluster but costs about the same (I think I paid about 520€) just for the CPU. That said, the script only used a single core, if it was multicore it would have been 8-16 times faster.
Do basically one Ryzen core can do the work of a pi cluster?
@@alouisschafer7212 Essentially
88 seconds for 200000 on a Ryzen 7 3700X, for comparison's sake.
That's really impressive for a single core!
Yeah, if the script were multi-threaded then the Pi's would use their additional 3 cores each as well, so they should also be 3-4 times faster, but still no match for your Ryzen 7 5800x
@@MichaelKlements it can be easily just import multiprocessing library in python, I tried that can running with full 40 cores in my server.
THIS MAN SETUP A CLUSTER TO RUN SINGLE THREADED CODE WTF AM I WATCHING
I was thinking that
I'm pretty sure mpi just takes the loop and divides it evenly between the nodes.
Each loop is independent so this works fine but once you get into loops with dependencies (shared info / mutexs) the cluster is going to slow significantly .
It's a shame this wasn't highlighted in the video.
Yeah, the cluster's performance is better when the task can be parallel. But single core still suck.
this comment needs to be pinned.
also why didnt he test the pc on both single core and multi core performance. just to make it complete
It's kind of pointless to compare single core performance of a laptop CPU to performance of 8 Raspberry Pis.
You could get much more meaningful results by loading every core on laptops (with and w/o hyperthreading) while monitoring for thermal throttling, and also loading each core on your Pi cluster, not just one per node.
Yes, it would definitely have been a better comparison to utilise all cores on each system. I haven't been able to get a multi-threaded Python script to run stably on multiple platforms yet, but I'll keep working on it.
This still gives you a pretty good idea of how they would compare if you know the core count of each device being tested.
@@MichaelKlements Yeah, multithreading in python is virtually non-existent (due to GIL), but it should be not too hard to run one python process per core, the same way you did with MPI.
@@MichaelKlements Using Parallel from joblib is pretty trivial to run in multiple cores: drive.google.com/drive/folders/1_VUNGTMIvpuy_7pAXjTvD0MCGfQaf2NM?usp=sharing
With Python GIL multi-thread is out of the equation (so on some systems one can gain more performance by leveraging other languages)
Thanks José, I'll look into joblib and try that out on the cluster as well.
I would have really liked to see this as well. I tested the script on my i9-9900k @ 4.8 ghz and got 0.33, 24, and 84. If the script were multi-threaded then the pi's performance would likely falter to some other laptops/PC's
I don't even care if it's faster, that cluster looks so awesome!
I ran it on my pentium 4.
Stay Tuned.👌
*5 hours ago*
10 hours ago xD
16 hours ago
1 day ago
2 days ago ...
Tested the script on my Ryzen 7 3800x running Windows 10 and python 3.8.7 (More of a mid range cpu 8 cores 16 threads, max all core was 4.47ghz)
10000: 0.33 s
100000: 26.95 s
200000: 102.24 s
I have a Ryzen 7 2700 I'll check it when I get home
aw man you got destroyed on the 200k test by a raspberry pi lmao a r7 3800x getting rekt by 8 little tiny computers x(
it only runs on one thread so core count is irrelevant. the important thing is clock speed.
if someone makes an efficient multi thread script then core count would be important along clock speed.
i dont know python well enough but i can do C++ or C#.
@@darkshadowsx5949 but the RPI 4B also has 4 cores
man that's a weak 3800, my 3600 has 78 seconds on 200k. Are your clocks/temps okay?
WOW, Was not expecting that performance from a PI. Really powerful little machines....😁
Not very powerful by themselves, but when strung together yes they are very powerful
@@coderdude9417 Apes together strong
Not 'a' pi
'8' pi ;)
@@Chrisknot94 🤣🤣🤣🤣
@@Chrisknot94 or pi³
Ryzen 5 3600XT results:
single process script:
10k: 0.59 seconds
100k: 28.49 seconds
200k: 104.89 seconds
multi-process script:
10k:0.5 seconds
100k:6.82 seconds
200k:23.62 seconds
As someone pointed out in the comment section, the performance improved greatly only when I changed line 41 in the multi process script to :
parts = chunks (range(2, max_number, 1), mp.cpu_count())
To get the Multicore Working you have to change the line 41:
#Orginal:
parts = chunks(range(2, max_number, 1), 1)
#now:
parts = chunks(range(2, max_number, 1), mp.cpu_count())
with 24 cores:
10000 = 0.08 seconds
100000 = 1.67 seconds
200000 = 6.16 seconds
500000 = 37.01 seconds
Thanks much better.
8 cores 16 threads
10000 = 0.47 seconds
100000 = 5.16 seconds
200000 = 21.03 seconds
500000 = 119.29 seconds
stock clock and undervolted 16 core 3950x:
10k: 0.22 seconds
100k: 1.5 seconds
200k: 4.78 seconds
500k: 26.5 seconds
Awesome build and great explanations, thank you and well done!
Here are the results on an i9-9900K at stock speeds:
Single Core:
10,000 - 0.35s
100,000 - 25.85s
200,000 - 100.74s
Multi Core:
10,000 - 0.29s
100,000 - 3.23s
200,000 - 10.94s
2,000,000 - 2,220.28s
I tried it out on my M1 Mac mini, 5 runs of each script and averaged the times:
10000: 0.45 s
100000: 39.77 s
200000: 155.84 s
and because I had the time...
500000: 985.56 s
Thanks for sharing your results. I've heard good feedback from people using M1 Mac's so far, I'll definitely be looking at getting one when they've expanded the range.
Thats weird. I forced my gf to try it and she got 24 seconds on her MacBook Pro on the 100.000 test.
Try with the MP code fixed (chunks last parameter 100+) to get some real numbers for the test. M1 is probably as fast or faster than the cluster
My iPhone XS got nearly identical results, within the margin of error using a-Shell
Hi. Do you use python running on rosseta or python native for M1?
I love all the people here just participating and sending their test result. apes together strong
We're going to the moon with these results!
Hi @Michael, sorry if you already stated it in the video in which case I might have missed it.
Were you using the exact same algorithm on either systems ? were they all concurrent computations ?
also, how have you configured your cluster setup to distribute the jobs ?
I guess, what I am struggling to understand is, lets say if the scripts were same, how did your cluster setup broke down the job to delegate it to follower nodes?
awsome video man i put a comment on the cluster asking what the benefits were quiet clear now aha :)) welldone
Oh, dang... now I kind of want to put a cluster like that together and finally get around to get an MPI implementation of my raymarched mandelbox thingie. Ihis cluster is excessively cute
Tested the Multi-process script on my Pi400, 2.2 Ghz (overclock) running Raspbian 10 (buster) and Python 3.7.3:
10.000: 0.99 sec
100.000: 108.01 sec
200.000: 448.25 sec
Very cool experiment and demonstration of cluster computing. Thanks for showing us one of many applications of cluster computing. I will stay tuned for thermals. Great work Maker Michael!
Thank you Deech!
could over clock more with the water cooling? If you rewrote your script to use threading and workers does the PI cluster still hold up?
You only ran the test on a single thread on each laptop, so the test isn't very fair when comparing to multiple pi's. With multithreading the laptops would be about twice as fast if they're dual cores, or more with hyperthreading.
For full performance it doesn't show the performance of the laptops but it does represent why a pi cluster is useful over a laptop for this purpose in both price and performance. It also shows that if you are doing the kind of work that would benefit from a cluster even a small performance boost can be worth it while still allowing your main computer to be fine while with a laptop to use multithreading would make the machine slow while the processes are running basically wasting the use of the laptop.
Yes it is only running on a single thread on the laptops, but it's also only running on a single thread on the Pis (they're each running quad core processors). So with a multithreaded script, the laptops would have been around twice as fast, but the Pis would then have been 3-4 times as fast as well.
@@MichaelKlements still makes for an unrepresentative comparison. especially since you told people to try it themselves, especially with ryzen processors which can have a lot more than just 4 cores not to mention hyper threading.
@@MichaelKlements yeah, but with a cluster of 8 or 7 pi it's like running the same script on a 7 thread cpu, you should make it multithreaded for both the pc and the rpi, so it can be comparable, if you run a single threaded app on a ryzen and on an intel the intel will win for higher ghz, and it's the same for the rpi's, if you manage to make it multithreaded on a 8 threads 4ghz cpu it will be much faster than 8 rpi's at 2ghz
@@aerispalm6523 intel has hyperthreading, amd has smt (simultaneous multi threading)
Impressive and there are still a bunch of optimizations that can be done with in the cluster
What are the practical applications running the Pi's in such a way for someone such as me (not a computer science major or programmer, etc)? Would you be able to run an OS capable of running Windows or Linux and associated apps or will this only run specific programs written for the Pi? Thanks and sorry for my lack of knowledge in this subject!
Pi's themselves are perfectly capable of running a range of Linux based apps and many people use them as standalone desktop computers, typically the Pi 4Bs. Clustering them is purely to improve processing power, mainly for simulations, rendering or bulk computing tasks.
Awesome build! Does this cluster have failover redundancy; not stopping processing if one node fails?
Only testing single core performance seems unreasonable when testing something that is essentially designed for parallel workloads.
I've been working on a some render farm software. Basically a cluster interface. If you would like to test it out hit me up. Its pretty early stages but it's pretty fun to use. It's originally designed for render farms but has been more generalized over the last few months to run most tasks you can think of.
Where i can find it?
@@techbrosita9698 Its in active development but if you want access to some early test builds feel free to join our discord
discord.gg/V5jXaBgGtp
This is so cool! Would have been very curious to see some analysis of the power consumption of the different computers, and what your cluster could do without the cooling running. Great Video!
Thanks for the suggestions, I'll have a look at that.
Very interesting stuff! Thank you!
As others have already posted the 5800X, I thought I'd add the 10700K (at stock) running ubuntu 20.10.
Find all primes up to: 10000
Time elasped: 0.3 seconds
Find all primes up to: 100000
Time elasped: 21.63 seconds
Find all primes up to: 200000
Time elasped: 86.98 seconds
And as others have said, it would be great to have it run multithreaded (and clustered).
Shows how much and has improved 😂. Never thought I'd see the day when AMD beats Intel ate single core.
Thanks for this Andy. That's quite impressive for a single core!
I haven't been able to get a multithreaded Python script to run stably on different platforms, but I'll keep trying.
@Andy Monks,@@MichaelKlements 10700K @5.2 Ghz, 3466 Cl14 32gb ram, Win 10 pro, python-3.9.1
Find all primes up to: 10000
: 0.4 seconds
Find all primes up to: 100000: 14.16 seconds
Find all primes up to: 200000: 52.64 seconds
Impresive! I can only imagine what a cluster of 100 Pi can do!!!!!
How about 1 million arm cores:
en.m.wikipedia.org/wiki/SpiNNaker
The SpiNNaker is not a home DIY project...
But yeah is impresive
Does anyone know any detailed video or an article about the proper setting up of a cluster software-wise :) like dividing the tasks between each pi's etc.
did you make use of multithreading on the computers?
U SOUND HAPPY THATS THE MAIN THING
he sounds monotone and emotionless. So yeah, agreed, /s
5900 X here! Python 3.9.1 Win10, 5 runs each setting
Normal Run:
10000 - 0.23 - 0.26 s
100000 - 18.28 - 20.87 s
200000 - 69.83 - 77.14 s
Multi-Threaded:
10000 - 0.46 - 0.49 s
100000 - 11.68 - 11.89 s
200000 - 43.05 - 48.01 s
Are comments being removed?
how did you enable multi threading????
@@MichaelMantion there is another script for multithreading
Have the same CPU, but am on Linux and every time is extremely similar. Some faster, some slower, most within just a second or two.
What's interesting though is the 10000 multi-thread. Your fastest is 2.3x slower than my slowest, plus your threaded is slower than your unthreaded (which our times are identical for). Not sure what would cause that besides maybe a process already using some of the core it picked. Interesting result.
Are you sure you ran the multithreaded properly? Your single threaded scores are what I would expect compared to my Ryzen 9 4900HS, but your multithreaded scores are 4-10x slower than mine. My laptops multithreaded scores were
10,000: 0.05s
100,000: 2.98s
200,000: 11.57s
There’s no way my little laptop is faster in multithreaded than a 5900X.
@@-Burb I've got the same processor as the person above, turns out the multi-thread script was only using 1 core. With that fixed my times were
10000 - 00.06
100000 - 01.20
200000 - 04.36
500000 - 25.18
Would there be a way to treat a PC with multiple cores similarly to the cluster where the task is divvied up to the different cores resulting in potentially faster computation by more fully utilising the power available? It seems great care was taken to fully utilise each Pi, but were the PC or Mac using more than a portion of their power?
Also the weight of the OS seems like it could be a hindrance. I know that's probably outside of the scope.
I see this has already been brought up and well discussed in another thread here. Don't mind me.
In this case, you benchmark using integer arithmetic, not floating point. I saw another video that benchmarked a Turing CM 3 cluster. But there too the applications were running not floating point processes but tasks in Drupal and Wordpress. How do your 3 systems compare performing floating point math benchmarks?
RaspberryPie's website is hosted on a cluster of 16 pies.
With cloudflare set up. Less impressive
@@Sam-cp6so Why do you say that?
@@jaredschmidt4614 cache
How is the cluster able to parallelise the script? I mean it does not look like the script can easily be shared between cores. What am I missing?
Man, you do good work! Great video!!!
Thanks Timothy!
It makes me anxious how you are manually setting up every pi via ssh instead of using ansible
Hey, what other tools are available like ansible for this task?
Please share more details about doing it this way.
@@stalinsampras terraform, chef, and puppet are the main ones i know of
@@gunstorm05 Hey, thanks for replying.
@@stalinsampras absolutely
6:49 Wait a second, you're running a script that contains no logic for splitting up work and makes no MPI calls. What are you actually benchmarking here?
About to ask.. how is the work load being shared between the nodes?
Are you running them in an SDE? This might have slowed down the script because it could add some debugging code
Cool Video really interesting nur with what Software did you Made the Diagramms ?
They’re just made in Excel
So, I ran this at 200k on a few different pieces of hardware that I have access to:
Westmere (2.27GHz) - 1 core: 340.88s (python 3.5)
Westmere (2.27GHz) - 60 cores (10 CPUs): 14.77s (python 3.5)
Ivy bridge (4.0GHz) - 1 core - python 3.6: 148.49s
Ivy bridge (3.6GHz) - 16 cores (2 CPUs) - python 3.6: 10.36s
Zen2 (4.35GHz) - 1 core - python 3.9: 108.53s
Zen2 (4.25GHz) - 24 cores (1 CPU) - python 3.8: 8.19s
CPU TDPs for these systems?
zen2 - 1x280W
ivy bridge - 2x120W
westmere cluster - 10x60W + networking
I imagine your raspberry pi cluster is quite a lot more power efficient than any of my systems!
P.S. end_number = int(sys.argv[1])?
You've imported the sys module, might as well use it!
Thanks for trying this out and sharing your results!
Yeah this cluster runs at about 80W on full load, so it's quite a bit more power efficient than traditional computers.
Could you make a video of the OLED display which you then connect to the raspberry Pie 4
And with instructions
Sure, I'll put together a video for connecting an OLED display to the Pi and running a basic script. The issue is often with the software, it's updated so often that tutorial videos are out of date just a few weeks/months after posting them.
@@MichaelKlements ok thank you
when do you make the video for this OLED display?
Can they be connected infinity? And can they run games or do you need a gpu also?
I don't know why, but the multiprocessing python script seems to be pinned on a single core. I've been using Python for years and I might check out the code later as to find out what's the problem, but htop shows barely one core at 85/90% on my Ryzen 5 3600X rig
“Running mathematical computations” 😂 dude be crypto mining.
Or machine learning.
I was also thinking can is do mining
I've seen mining on a commodore 64.
This cant mine lol since they dont have any powerful gpu's
@@David-wg5ce u can mine with cpu too
i'd be interested in seeing the power draw of each device as well, or at least the cluster's
I know that the two sharing USB ports started struggling at 2.0Ghz full load, so I didn’t some measurements and add an extra supply to these. They use 1.7A at full CPU load, so the full cluster uses about 70W when running at full load.
@@MichaelKlements thank you !
wont recording and testing on same machine at same time affect the results?
Ive never owned a cluster of any sort. does ffmpeg have a cluster render option? I know programs like mandelbulber2 have cpu clustering features for fractal rendering... Host needs some RAM. We need a motherboard that will take 8 pi4 compute modules.
Well, that settles it. I'm building a cluster computer to handle computations. That's freaking sick!
NVIDIA makes a GPU microprocessor (Jetson Nano) with 120+ cores. I wonder how viable that would be in terms of GPU super cluster, lol.
This was cool. If I understood it right, they communicate via Ethernet? So some better hardware connection should speed up even more.
Yes correct
There is an actual picluster board and compute modules that look a bit like PCIe cards.
Awesome hardware build. I’m on the hardware side of things and I think it pretty cool what you have made. I can’t speak for software side but still awesome to see. Thank you for sharing this program with us.
Thanks for the great feedback!
I hope by now you have sorted out that power connector on the top left one
This shows how group projects can be done way faster with the whole group working. I'm talking about you Dave, shut up and do your work.
Tested on a Xeon e5 2678 v3 @ 3.3 ghz on all cores
Edit (tested again on singlecore and multicore version):
Singlecore -> Time elapsed: 0.52s, 41.91s, 157.99s
Multicore -> Time elapsed: 0.88s, 26.36s, 96.32s
How mpiexec helps here when there is no parameter to split data processing across all 8 nodes? I would expect different range for each IP
So is there a way to use all the pi’s cpus to make a virtual cpu just as a test of concept
On my galaxy tablet A, I got 18.3 seconds for 10k. When I tried 100k, the app would just say, "experiencing server issues, try again later" after a couple of minutes 😐
You should try transcoding videos on that stack.
Awesome video!! Love the production quality and thought that finding primes was a clever way to assess performance
I wondered this about the TuringPi, I also wonder how these would compare in blender, as blender supports clustering
Cool, so you tested a single core application on multithreaded systems, then made it run parallel on 7 cores (basically). Yeah no, this is flawed.
First half of the video he did single core for all the systems, nothing flawed. All the results are clearly tagged.
Them he show it running on multicore for the pi's. Have a blessed day good sir.
The whole world is behind pi pico now. when we are going to have videos on that.
I've got one on order, so hopefully soon!
I am not sure, but I am not sure the code you are using is multithread, so basically you are comparing single thread v.s. multicore?
could you build a simple interface for that script to just run tests like this?
And so pretty!
When you were testing the Mac I thought it was going to be one of the new arm laptops
This is pretty remarkable. I don't have any projects at home that would require this much computing power... but my goodness, for the cost, that's ridiculous.
Awesome video! I would 100% suggest something like ansible to manage your cluster though :D
Thanks Kyle. Yeah this is one of the most basic packages to get running on a cluster. I'm going to try Ansible and Kubernetes next.
*I ran it on my ryzen 7 (laptop):*
10000: *0.58s*
100000: *47.41s*
200000: *172.33s*
I have a Ryzen 9 3900XT, 10000: 0,32 sec 100000: 25,57 sec 200000: 98,33 sec
thanks i was to lazy to test it my self
Excellent experiment, very simple. It really shows just how much of a difference it makes to utilize cluster computing.
Thanks for the great feedback!
@@MichaelKlements Absolutely! I'm a student learning about databases, cluster computing, and MapReduce right now. So it's very cool to see that just how effective this strategy is, even when equipped with Pi's rather than a full sized PC. I feel like my knowledge is rudimentary at the moment.
Are you aware there's a cluster hat available for the Pi4 that controls 4 Pi zeros at once? Might be a cool prospect!
That sounds awesome! I’m also still learning a lot about clusters as I go along (as you can probably tell), it’s been really interesting and fun.
Yes I have seen them. It would be quite interesting to try clustering a cluster of a Pi 4 and zeros, I should have a look at that. Thanks for the tip
Ryzen 5 2600 (on a pretty bloated Windows install) using the multi core edit:
Python 3.7.5:
10,000: 0.6s
100,000: 33.62s
200,000: 120.19s
Python 3.9:
10,000: 0.93s
100,000: 27.87
200,000: 102.48s
Pypy (for fun):
10,000: 4.55s
100,000: 6.27s
200,000: 11.34s
Hope this helps
ryzen 5 5600x with PBO set to 5ghz, win10 python 3.7.8
10000 0.28sec
100000 22.75sec
200000 87.59sec
only 1 threat whas in use
for the second run
10000 0.28sec
100000 22.73
200000 85.21sec
Thanks for sharing your results
An IPad Air 2020 does it in:
To 10000: 0.28 seconds
To 100000: 22 seconds
To 200000: 82 seconds
Quite impressive for an Ipad.
How did you display each node on one screen?
It's just 7 different SSH connections in their own terminals
Awsom Video thanks for making this great content. ;-)
The frustrating thing here is how inefficient the alg for finding primes is!
> Tests perfromances
> Uses python
Why not write a C/C++/Rust program?
Here is a version in C:
pastebin.com/fAjwP61f
On my box it runs in 1.79 seconds (54.03 seconds for Python3 version).
Or... use an algorithm that doesn't suck. Implementing a sieve of Eratosthenes on my machine for 200k to .03 seconds.
@@InfiniteLemurs I'm sorry but it's not the algorithm's fault, for example a for loop in python is going to take you ages compared to one in Any real programming language or even scripting languages like php/js
Go try it yourself.
@@JustKatoh There is significant overhead in python, (though JS is arguably just as bad) but my argument is that inefficiency of the algorithm matters much more than the inefficiency of the language. A good prime sieve is magnitudes of scale faster than the one shown in this video-- a factor of utmost importance here. Writing bogosort in cpp is still going to be worse than quicksort in python; obviously things would be more performant in cpp/rust, but this isn't even beginning to approach optimization yet. The C code above gets absolutely stomped simply by using a better alg. Python isn't the best choice here, but the algorithm is the truly poor one.
@@InfiniteLemurs It does play a major role, lookup nodejs performance compared to python, is just as fast as JAVA, a fully fledge programming language. People trying to defend python's performance is just funny, python is literally a prototyping language to write hacky scripts quick and dirty, it's the only thing good for.
I can't find 1 thing made in python, that works well, performant and is preferred over other language alternatives.
I think you're misunderstanding what an algorithm is, algorithms usually are written in pseudocode and adapted to languages, be it scripts or programming, Take for example Huffman's compression algorithm, sure there is prewritten assembly and C code but 99% of the time it's going to be presented under a pseudocode format, you can look at it however you want, it's not Huffman's fault python is incapable of it's implementation in a real world scenario when even one of the most inefficient scripting languages ( PHP, nodejs is so much faster ) can still easily run it in a production environment reliably.
I really don't get what you're trying to bang at with the "algorithm" argument when algorithms are simple stencils to use independent of the language at hand,
Madlad :) this is great :)
6:42 You say the cluster needs to manage communication between the nodes. However, the MPI script splits the task in independent, almost identically sized parts which means there is minimal overhead in parallelizing on a cluster. That said, your cluster does look really neat and you clearly put some effort in making this video, nice job!
Yeah this script and software setup in general is not a good example of that because the script already splits the tasks up. There are much better software packages for clusters which better manage the split up and delegation of tasks.
This is a very poor comparison. It would have been great maybe in the 90's, but it's single threaded. Even the PI's have more than one thread each. I actually have the Ryzen 5 2600x, so 11 cores are sitting at absolute idle while one luckless bastard of a core gets pegged while it's working on the larger datasets.
Double + for the use of 'luckless bastard' - Made my coffee experience much better today.
It would definitely have been a more thorough comparison to get the script running across all cores of each device, but I haven't been able to get a multithreaded Python script to run stably on multiple platforms. I'll keep working on it.
The Pi cluster is also sitting with 24 unused cores during this test.
@@MichaelKlements that's a video i would like to see, as i have 3 older server systems needing something to test out. Should you get it working i might even go out and buy more pi4's just to build and play with. thanks michael
AMD Ryzen 9 5950X - Win10
RTX 3080
128GB 3600MHz RAM
2TB NVMe
10000 = 0.22
100000 = 18.03
200000 = 66.82
Thanks David, that's really impressive for a single core!
Should I splurge for higher ram capacity if I want to do this? I noticed you got 2gb models so I was wondering if there were tangible benefits with more ram.
It depends what you're going to be doing with them, more ram for this application doesn't really make much difference as it's mainly processing power that is required for simulations etc.. Ram makes quite a big difference on the Pi when multitasking, particularly with multiple browser tabs open or multiple applications running.
I would like to see the code write the found numbers into a txt file can you modify it for that purpose?
If you uncomment the last line then it will display all of the numbers in the terminal window afterwards. You could add a couple of extra lines to write this to a text file pretty easily.
this script runs on single core, so this isnt the way to compare these...
Would you say that it is possible to use multiple raspberry pi clusters to run a mini cloud service for a small to mid-sized company?
I think it depends on what you're expecting the cloud service to do. If it's just serving documents and files then yes, Raspberry Pi's would be perfectly capable. You would probably have to do quite a bit of your own programming and tweaking of software to get it working though.
Awesome video!
Well, what will be impressive too is, what will happen, when using 20 PIs and chunking encodes, ffmpeg splice a file in chunks of e.g. 10 seconds.
After that each node renders x of chunks and compare it energywise with an Ryzen 9 or so...
So for video encoding (software gets better visual quality than gpu based) this might be an good option too.
cluster setup is awesome
Awesome Work .. please more ... and thank you
Thank you! Will do!
I’ve done a lot of benchmarks for distributed computing with the Pi’s and although they’re great little boards, very often a used desktop will perform more for less money. I’ll only add Pi’s to the cluster when replacing them with faster versions. To keep up in cluster workloads pi’s need to get cheaper or faster at the same price.
Thanks for your thoughts on clustering. I agree, you can definitely get better performance out of some old desktops, but the Pi’s are still quite competitive in terms of power consumption and size.
I don't really know about clustering but if I'm understanding what's shown in this right, isn't it technically possible to nest clusters? so you'd have 1 main board with 7 slaved to it all running as main boards for their own clusters, essentially making it that each nested tier multiplies the computing of the previous tier by 8? So if there's 4 tiers you'd have 4096 Pi's all set to a single task funneling down from a single master board.
Very interesting, nice job
THANK YOU! I've been trying to understand what clusters are used for, everyone builds them but no one explains why, I know "because we can and it's cool" but I just wanted to know the usablity is all.
Liked from me :D
Thanks Rungeon!
Clusters are used to scale out tasks that can be done in parallel, so you won't be limited by the number of cores in a processor.
On my win 10 with Xeon E-2276 2.8GHz:
0,82 for 10k
31,5 100k
and 112,74 for 200k
But even that it's saying it was run on 48 processes in fact is was using only only 5% of CPU power on just single process.
It's not shown on the video, but can You shate your CPU usage? or am I missing someting?
One thing id like to know is could you put such a cluster to work verifying blockchain or mining other coins which might not have asic miners dedicated. I.e. a cost effective way of running the cluster 24/7 and would it pay for itself and the electricity it uses.
It all depends on the value of the currency being mined. This is definitely capable or doing blockchain verification and mining, whether it could mine enough value to be profitable depends on the difficulty of the currency being mined and its value.