CLUSTER MULTI THREADED NODEJS -- Run threads in parallel to speed up long processes (CORRECTED)
ฝัง
- เผยแพร่เมื่อ 22 ส.ค. 2024
- My JavaScript that creates thumbnails for hundreds of files is taking too long. So I have decided to speed things up with cluster.fork() that is built into Node.js. This will allow the workload to be spread out across my multi-core processor system.
⚠ 8:53 should read: index % forks === cluster.worker.id - 1
To follow along, you can fork this repository. I have added a tag you can check out that will bring you right to the point this video was created.
```
git checkout github.com/joe...
cd mojo-gallery
git checkout youtube-video-cluster
```
Your content is straightforward without much bloatware. I respect that. Big up!
Thanks! I appreciate the feedback. I'm trying to create the content I would like to view. So I'm glad others are enjoying it also. Cheers 🍻
I respected that you came back to fix the video
Cheers 🍻
You made it super easy!!
Bet no one can teach clustering that much easy.
Subscribed :)
Great video man best one I found
Thanks. Much appreciate the feedback. Cheers 🍻
Love from India, Amazing
Hello friends from India! Cheers 🍻
@@JoelCodes Your student 🙏🙏🙏
Really helpful, thanks
I appreciate the feedback! Thanks!
Anyone know how to get real user sys execute time in terminal? Somekind of plugin or extension?
Awesome work!! Thank you!!
Very nicely done, thanks
I appreciate the feedback. Glad you found the video to be useful. Makes me happy. Cheers 🍻
Great video!
I am curious if you have a solution to pass a predetermined list to each thread?
Instead of each thread having to process the entire list.
Hopefully that makes sense... thoughts?
Great lesson, thx
Appreciate the feedback! Hope to see you around. Cheers 🍻
Hi am confused between worker_threads and workerpools package
could you please explain?
you didn't give the source media file so i used some other photos as a source, but it's not working. though application is running but photo resizing mechanism is not working
subscribed !!!
Thanks. Much appreciated 🙏
Great content my man. Can you give some other way in which we can segregate the work to each worker other than using the filter method
I'm not sure I fully understand what you're asking. I have been thinking about making a part 2 for this, but I haven't got it all planned out yet. Still thinking about it.
@@JoelCodes There is a huge array and on every item there is some heavy computational task being performed. So in the video you are allocating different parts of the array to different process using the filter method of array. This is one way of distributing load. Can you give some other example other than using the filter method
Very nice, do you know how can i debbug worker threads or clusters using typescript files on vscode?
I'm still learning TypeScript, so I don't have any videos that cover TypeScript. But the process shouldn't be much different, other than setup. I'll put this one in my queue of videos to make. Great suggestion. Cheers 🍻
Thanks to its Very helpful. How implement CRON side using Cluster.
Glad to hear you found this useful. Can you expand on what you mean about CRON?
@@JoelCodes When cron was running and a huge amount of data was being added, the site went down.
I am master, I am cluster. I am master of clusters 😉Like & sub, thanks for sharing.
Hello Joel, can you explain why your code didn't get stuck when the worker id is 1? It means your filter will always pass since the calculation will always be 0? Thanks and love the video!
not stuck per se, but that worker did not get all the load?
What is the time stamp in question? I'll go through it and try to explain it better for you. Cheers 🍻
@@JoelCodes 9:47 line 45 -- let's say the first forks value is 1 so the result of modulo calculation will always be 0 and all of images will be included on that clusterFiles since it will pass the filter everytime. right?
@@calvinrogercanas6737 Yes you have got it correct. The cluster (index 1) would have processed the full load and cluster (index 2) would have processed all the evens, index 3 would have done every 3rd. Not the result I was looking for. 😂
Any possibility you can share code for this video? I've tried to follow the video and with your github project, but alas not working. It goes back and forth to quickly for me.
No problem! I put a tag in on the section of the code in the repository so you can go right to it. Here's the download link github.com/joelnet/mojo-gallery/releases/tag/youtube-video-cluster
@@JoelCodes thanks alot! Your vid helped me see why my workers didn't want to cooperate with me
@@JoelCodes Im not 100% on it, can you help me with the code? It looks like this right now, I removed resize from your file, but it aint firing away. hastebin.com/yifiqihayi.js
@@valdemarjohannesson2720 One thing I see in this code is you are awaiting `hugeData`, but `hugeData` is an Array. So you don't have to await that. You are also creating the function `makeGetRequest` inside of the for loop. Move this to outside of the for. Inside of `makeGetRequest` you are also referencing `i`, but it is not declared.
@@JoelCodes Thanks alot. Do you have any video on worker-threads?
any update on how much speed it has saved if you use the corrected code?
The speed improvement will vary based on the application. So while I could have run benchmarks on this code, they would only apply to this specific code and not represent your app.
It will have more effect with apps that use a lot of cpu.
But its still single thread ,multiple processes.
Of course it increases performance as it uses multi core cpu , but you can't share memory within processes i.e you must use databases or external storage at all times , you can't use or share dynamic shared state with clustering module.
Clustering is OK for http web based apps, but not necessarily for IOT , online multi user games or Big Data.
Messages can be passed between threads. I would disagree with staying it's use is only limited to web apps. There are many successful multi process js applications out there that prove this.
@@JoelCodes Yes I agree ,
I think what you are saying is about worker threads
That is a important difference... I got a bit confused because of that
can I use this to scale apps and/or games to multiple cores with multiple cpu's? Or am I just dreaming? lol
Ya you can definitely use this. I believe most game engines will automatically do this (multi-processor) for you though. So you don't have to manually offload work.
@@JoelCodes Well in the case that I'm using older hardware to run a modern games... For example star citizen. I have to use sde.exe ( Intel's software developer executable) to emulate the AVX instruction set because I use old hardware. And for that reason, I like to see what software improvements I can do to bring them up to speed. I think you see where I'm going with this, hopefully.
I think your cpu has multithreading and generally logical cores aren't as efficient as real cores
You may be right about that and I didn't think about that at the time. It would be nice to separate out the real cores and run a test on it. Next time I guess. Cheers 🍻
guess what , i subed for learning english rn
Awesome! Hope to see you around then. Cheers 🍻
what is the name of the theme vscode you use
Right now I am using a theme called Moonlight and the font FiraCode.
@@JoelCodes thank bro
360p ?
Dude, I'm sorry. But this is BAD. You really, and I mean really, need to delete or recreate this video.
Hi Dzenis, could you be more specific on what part of the video you would consider bad? Or an approach that you would recommend instead?