That's an amazing resource! It's great to see you sharing such detailed information on a complex topic. Your effort to explain everything clearly will really help others understand and apply these concepts. Keep up the great work!
Great video, thanks for creating this. I have use DDP quite a lot but seeing the visualizations for communication overlap helped me build a very good mental model. Would love to see more content around distributed training - Deepspeed ZeRO, Megatron DP + TP + PP
Starting to watch my 3rd video on this channel, after transformer from scratch and quantization. Thank you for the great content and also for the code and notes to look back again. Thank you.
This is second video Ive watched from this channel after "quantization". And frankly wanted to express my gratitude towards your work as it is very easy to follow and the level of abstractions is tenable to understand concepts holistically.
The video was very interesting and useful. Please make a similar video on DeepSpeed functionality. And in general, how to train large models (for example LLaMa SFT) on distributed systems (Multi-Server) when GPUs are located on different PCs.
well explained each and every detail, Great work Great Explanation👍 can you make this type of detailed video on distributed training through tensor parallelism? it would be very helpful. Thank you!
In broadcast , if we are sending the copy of file from rank 0 and rank 4 node to other node. How is the total time still 10 second. Because still I am having same internet speed of 1MB/s. Could anyone explain? I am bit confused. Also what happens if I am having odd numbers of nodes
If time permits for you, Please make an video for entire GPU and TPU and how to them effectively and most of us donno . please create a playlist for pytorch for beginners and intermediates. Thanks for reading.
@@umarjamilai not sure if it’s in your plans, but if you are open to suggestions, I would love to watch a video on multimodal models. Again, awesome work!
Hi Umar, Great video and enjoyed thorughly but i have one question.why are we using the approach of sum(grad1+grad2+....+gradN), why cant we use Avg of Gradients.
Of course you can (but you don't have to) use the average of the gradients. Actually, people usually take the average of the gradients. The reason we use the average is because we want the loss to be (more of less) the same as the non-distributed model, so you can compare the plots of the two. I don't know if PyTorch internally automatically takes the average of the gradients, I'd have to check the documentation/source.
Wouldn't the accumulated gradient need to be divided by the total number of individual gradients summed (or the learning rate needs to be divided by this value) to make it equivalent?
Yes, if you want to treat the "cumulative gradient" as a big batch, then you'd usually divide it by the number of items to keep it equivalent to the single-item setup. But it's not mandatory: as a matter of fact, loss functions on PyTorch have a "reduction" parameter, which is usually set to "mean" (so dividing the loss by the number of items) but can also be set to "sum". One reason we usually calculate the "mean" loss is because we want to make comparisons between models with different hyperparameters (batch size), so the loss should not depend on the batch size. But remember that mathematically you don't have to
In my understanding, yes the loss is accumulated for one batch theoretically, and the gradients are computed based on this accumulated loss too. But in the parallel implementation, both the loss calculated in the feedforward process, and the gradients calculated in the back propagation process executed in a parallel way. Here @umarjamilai use a for loop to illustrate the de facto parallel mechanism.
Amazing video. Ideal video of how a lecture on a video should be
That's an amazing resource! It's great to see you sharing such detailed information on a complex topic. Your effort to explain everything clearly will really help others understand and apply these concepts. Keep up the great work!
Great video, thanks for creating this. I have use DDP quite a lot but seeing the visualizations for communication overlap helped me build a very good mental model.
Would love to see more content around distributed training - Deepspeed ZeRO, Megatron DP + TP + PP
Starting to watch my 3rd video on this channel, after transformer from scratch and quantization. Thank you for the great content and also for the code and notes to look back again. Thank you.
This is second video Ive watched from this channel after "quantization". And frankly wanted to express my gratitude towards your work as it is very easy to follow and the level of abstractions is tenable to understand concepts holistically.
I really love your vidoes. you have a natural talent on simplifying logic and code. in same capacity as Andrej
Dang. Never thought learning DDP would be this easy. Another great content from Umar. Looking forward for FSDP
Super high quality lecture. You have a gift of teaching, man. Thank you!
absolutely amazing! You made these concepts so accessible!
this channel is hidden gem
Incredible content, Umar! Well done! 🎉
Great work! thank you !
Another great video, Umar. Nice work
Really impressive!
Amazing content! Thanks for your sharing
The video was very interesting and useful. Please make a similar video on DeepSpeed functionality. And in general, how to train large models (for example LLaMa SFT) on distributed systems (Multi-Server) when GPUs are located on different PCs.
You deserve many more likes and subscribers!
Thankyou so much for this amazing video. It is really informative.
well explained each and every detail, Great work Great Explanation👍
can you make this type of detailed video on distributed training through tensor parallelism? it would be very helpful. Thank you!
Thank you very much for your wonderful video. Can you teach a video on how to use the accelerate library with dpp?
Amazing learning stuff ! Very Thanks !~ 🥰🥰🥰
very nice and informative video. Thanks
thanks for the video!! can you cover the Tensor, Sequence and Pipeline Parallel and D using Dtensors in Pytorch next?
you teach soooooooo good
Thank you so much for this
You rock always 😂
Awesome video. Please make tutorial on FSDP as well
Great video
Great intro video. Do you have any plans to also cover other parallelism: Model, Pipeline, Tensor, etc.
In broadcast , if we are sending the copy of file from rank 0 and rank 4 node to other node. How is the total time still 10 second. Because still I am having same internet speed of 1MB/s.
Could anyone explain? I am bit confused.
Also what happens if I am having odd numbers of nodes
Thanks!
谢谢你！我们在领英connect吧
Federated learning basics please.❤
great video
fantastic
Great!
Please create a video on model parallelism and FSDP.
If time permits for you, Please make an video for entire GPU and TPU and how to them effectively and most of us donno .
please create a playlist for pytorch for beginners and intermediates.
Thanks for reading.
please do some thing related to audio large models like conformers,quartznet ,etc
I wish i could like it twice
You can share it on social media. That's the best way to thank me 😇
@@umarjamilai not sure if it’s in your plans, but if you are open to suggestions, I would love to watch a video on multimodal models. Again, awesome work!
Check my latest video!
I'm always confused with DP and DDP. Can you please tell me the difference between them? While both of them belong to data parallelism method.
DP only works on a single machine, while DDP can work on multiple machines. However, PyTorch now recommends using DDP also for single-machine setup.
@@umarjamilai thank you for your reply
SUUUPERRRR
another banger
could provide another videos with respect to model parallel and pipeline parallel ? thanks..
Working with fsdp and megatron now and I really want to figure this out from scratch haha, it sounds fun but a big headache
Hi Umar, Great video and enjoyed thorughly but i have one question.why are we using the approach of sum(grad1+grad2+....+gradN), why cant we use Avg of Gradients.
Of course you can (but you don't have to) use the average of the gradients. Actually, people usually take the average of the gradients. The reason we use the average is because we want the loss to be (more of less) the same as the non-distributed model, so you can compare the plots of the two. I don't know if PyTorch internally automatically takes the average of the gradients, I'd have to check the documentation/source.
@@umarjamilaithanks for the info.
How to do in Kubernetes? Please explain it.
Wouldn't the accumulated gradient need to be divided by the total number of individual gradients summed (or the learning rate needs to be divided by this value) to make it equivalent?
Yes, if you want to treat the "cumulative gradient" as a big batch, then you'd usually divide it by the number of items to keep it equivalent to the single-item setup. But it's not mandatory: as a matter of fact, loss functions on PyTorch have a "reduction" parameter, which is usually set to "mean" (so dividing the loss by the number of items) but can also be set to "sum".
One reason we usually calculate the "mean" loss is because we want to make comparisons between models with different hyperparameters (batch size), so the loss should not depend on the batch size.
But remember that mathematically you don't have to
shouldnt loss be accumulated ? loss += (y_pred - y_actual)^0.5
In my understanding, yes the loss is accumulated for one batch theoretically, and the gradients are computed based on this accumulated loss too. But in the parallel implementation, both the loss calculated in the feedforward process, and the gradients calculated in the back propagation process executed in a parallel way. Here @umarjamilai use a for loop to illustrate the de facto parallel mechanism.
do you have a discord channel?
21:01
Always great to watch your video, excellent work