HI Krish.. dL/dW'11= should be [dL/dO21. dO21/dO11. dO11/dW'11] + [dL/dO21. dO21/dO12. dO12/dW'11] as per the last chain rule illustration. Please confirm
I have been taking a well-known world-class course on AI and ML since the past 2 years and none of the lecturers have made me so interested in any topic as much as you have in this video. This is probably the first time I have sat through a 15-minute lecture without distracting myself. What I realise now is that I didn't lack motivation or interest, nor that I was lazy - I just did not have lecturers whose teaching inspired me enough to take interest in the topics, yours did. You have explained the vanishing gradient problem so very well and clear. It shows how strong your concepts are and how knowledgeable you are. Thank you for putting out your content here and sharing your knowledge with us. I am so glad I found your channel. Subscribed forever.
many years ago in the college I was enjoy watching videos from IIT - before the mooc area, India had and still have many good teachers ! It brings me joy to see that again. Seems Indians have a gene of pedagogy
Sir or As my Indian Friends say, "Sar", you are a very good teacher and thank you for explaining this topic. It makes a lot of sense. I can also see that you're very passionate however, the passion kind of makes you speed up the explanation a bit making it a bit hard to understand sometimes. I am also very guilty of this when I try to explain things that I love. Regardless, thank you very much for this and the playlist. I'm subscribed ✅
Very nice way to explain. Learned from this video- 1. Getting the error (Actual Output - Model Output)^2 2. Now We have to reduce an error i.e Backpropagation, We have to find a new weight or a new variable 3. Finding New Weight = Old weight x Changes in the weight 4. Change in the Weight = Learning rate x d(error / old weight) 5. After getting a new weight is as equals to old weight due to derivate of Sigmoid ranging between 0 to 0.25 so there is no update in a new weight 6. This is a vanishing gradient
Kudos to your genuine efforts. One needs sincere efforts to ensure that the viewers are able to understand things clearly and those efforts are visible in your videos. Kudos!!! :)
Appreciate your way of teaching which answers fundamental questions.. This "derivative of sigmoid ranging from 0 to 0.25" concept was nowhere mentioned.. thanks for clearing the basics...
I must say this, normally I am kinda person who prefers to study on own and crack it. Never used to listen to any of the lectures till date because I just don't understand and I dislike the way they explain without passion(not all though). But, you are a gem and I can see the passion in your lectures. You are the best Krish Naik. I appreciate it and thank you.
Very nice now i understand why weights doesn't update in RNN. The main point is derivative of sigmoid is between 0 and 0.25. Vanishing gradient is associated with only sigmoid function. 👋👋👋👋👋👋👋👋👋👋👋👋
I just want to add this mathematically, the derivative of the sigmoid function can be defined as: *derSigmoid = x * (1-x)* As Krish Naik well said, we have our maximum when *x=0.5*, giving us back: *derSigmoid = 0.5 * (1-0.5) --------> derSigmoid = 0.25* That's the reason the derivative of the sigmoid function can't be higher than 0.25
should be: derSigmoid(x) = Sigmoid(x)[1-Sigmoid(x)], and we know it reaches maximum at x=0. Plugging in: Sigmoid(0)=1/(1+e^(-0))=1/2=0.5, thus derSigmoid(0)=0.5*[1-0.5]=0.25
Sir thank u for teaching us all the concepts from basics but just one request is that if there is a mistake in ur videos then pls rectify it as it confuses a lot of people who watch these videos as not everyone sees the comment section and they just blindly belive what u say. Therefore pls look into this.
Derivative of loss with respect to w11 dash you specified incorrectly, u missed derivative of loss with respect to o21 in the equation. Please correct me if iam wrong.
@@krishnaik06Hey!, U dnt hv to apologise, on the contrary u r dng us a favour by uploading these useful videos, I was a bit confused and wanted to clear my doubt that all, thank you for the videos... Keep up the good work!!
I understood it. Thanks for the great tutorial! My query is: weight vanishes when respect to more layers. When new weight ~= old weight result becomes useless. what would the O/P of that model look like (or) will we even achieve global minima??
Hello sir, why the chain rule explained in this video is different from the very last chain rule video. kindly clearly me and thanks for such an amazing series on deep learning.
Understood completely! If weights hardly change, no point in training and training. But I have got a question, where can I use this knowledge and understanding I just acquired ?
Very well explained. Vanishing gradient problem as per my understanding is that, it is not able to perform the optimizer job (to reduce the loss) as old weight and new weights will be almost equal. Please correct me, if i am wrong. Thanks!!
Gradient Descent will be applied on Cost function right ?-1/m Σ (Y*log(y_pred) + (1-y)* log(1-y_pred))... in this case if they had applied on the activation function, how the algo will come to global minima.
it is because when you want to control or fix the output of a given neuron within a certain range, for example, if the neuron is always giving inputs between 9 and 10, you can put a bias =-9 so as to make the neuron output between 0 and 1
Great video, one question, when you calculate the new weights using the old weight - learning rate x derivative of loss with respect to weight, the derivative of loss wrt weight is that the sigmoid function ?
Sir i'm really confusing about the actual y value please can you tell about that. i thought it would be our input value but here input value is so many with one predicted output
Thanks krish .Video was superb but I am having apprehension I might get lost somewhere .Please provide some reading reference regrading this topic considering as a beginner.Cheers
you meant that the derivative of the sigmoid is between 0 and 0.25, right? I wanted to clarify about that range written in red color. The sigmoid of z would be between 0 and 1, from what I understood. Any reply will be appreciated :)
In red color you've written that sigmoid of z is between 0 and 0.25 whereas you meant to say that the derivative of sigmoid of z is between 0 and 0.25.
you talk about changing weights on back propagation by derivation and then suddenly talks about derivating activation function, i dont understand that part, does function triggered during back propagation?
so if we have 2 layers and as we know 1 forward and back step is 1 epoch. If we now have 100 epochs the derivative is going to get smaller every time? Or the vanishing problem is due to many hidden layers and not depended on the number of epochs?
HI Krish.. dL/dW'11= should be [dL/dO21. dO21/dO11. dO11/dW'11] +
[dL/dO21. dO21/dO12. dO12/dW'11] as per the last chain rule illustration. Please confirm
...but O12 is independent of W11,in that case won't the 2nd term be zero?
wrong bruh
we don't
have the second term
Can anyone clarify this? I too have this question.
@@Ajamitjain dL/dW'11= should be [dL/dO21. dO21/dO11. dO11/dW'11]
I have been taking a well-known world-class course on AI and ML since the past 2 years and none of the lecturers have made me so interested in any topic as much as you have in this video. This is probably the first time I have sat through a 15-minute lecture without distracting myself. What I realise now is that I didn't lack motivation or interest, nor that I was lazy - I just did not have lecturers whose teaching inspired me enough to take interest in the topics, yours did.
You have explained the vanishing gradient problem so very well and clear. It shows how strong your concepts are and how knowledgeable you are.
Thank you for putting out your content here and sharing your knowledge with us. I am so glad I found your channel. Subscribed forever.
many years ago in the college I was enjoy watching videos from IIT - before the mooc area, India had and still have many good teachers ! It brings me joy to see that again. Seems Indians have a gene of pedagogy
I hardly comment on videos, but this is a gem. One of the best videos explaining vanishing gradients problems.
Sir or As my Indian Friends say, "Sar", you are a very good teacher and thank you for explaining this topic. It makes a lot of sense. I can also see that you're very passionate however, the passion kind of makes you speed up the explanation a bit making it a bit hard to understand sometimes. I am also very guilty of this when I try to explain things that I love. Regardless, thank you very much for this and the playlist. I'm subscribed ✅
Consider reducing playback speed.
Thank you sir for making this misleading concept crystal clear. Your knowledge is GOD level 🙌
so far best explanation about vanishing gradient.
Very nice way to explain.
Learned from this video-
1. Getting the error (Actual Output - Model Output)^2
2. Now We have to reduce an error i.e Backpropagation, We have to find a new weight or a new variable
3. Finding New Weight = Old weight x Changes in the weight
4. Change in the Weight = Learning rate x d(error / old weight)
5. After getting a new weight is as equals to old weight due to derivate of Sigmoid ranging between 0 to 0.25 so there is no update in a new weight
6. This is a vanishing gradient
I am amazed by the level of energy you have! Thank you :)
Kudos to your genuine efforts. One needs sincere efforts to ensure that the viewers are able to understand things clearly and those efforts are visible in your videos. Kudos!!! :)
Appreciate your way of teaching which answers fundamental questions.. This "derivative of sigmoid ranging from 0 to 0.25" concept was nowhere mentioned.. thanks for clearing the basics...
Look for Mathematics for Deep Learning. It will help
One of the best vedio on clarifying Vanishing Gradient problem..Thank you sir..
So happy I found this channel! I would have cried if I found it and it was given in Hindi (or any other language than English)!!!!!
Kudos sir ,am working as data analyst read lots of blogs , watched videos but today i cleared the concept . Thanks for The all stuff
I must say this, normally I am kinda person who prefers to study on own and crack it. Never used to listen to any of the lectures till date because I just don't understand and I dislike the way they explain without passion(not all though). But, you are a gem and I can see the passion in your lectures. You are the best Krish Naik. I appreciate it and thank you.
Thank you very much, I was wandering around the internet to find such an explanatory video.
One of the best explanations of vanishing gradient problem. Thank you so much @KrishNaik
I'm lucky to see this wonderful class.. Tq..
Marana mass explanation🔥🔥. Simple and very clearly said.
Nice presentation..so much helpful...
Krish.. You are earning a lot of Good Karmas by posting such excellent videos. Good work!
Overall got the idea, that you are trying to convey. Great work
You should get Oscar for your teaching skills.
Thank you so much. The amount of effort you put is commendable.
very simple and nice explanation . I understand it in first time only
Very nice now i understand why weights doesn't update in RNN. The main point is derivative of sigmoid is between 0 and 0.25. Vanishing gradient is associated with only sigmoid function. 👋👋👋👋👋👋👋👋👋👋👋👋
Great teaching skills !!!
Thank you so much, great quality content.
I just want to add this mathematically, the derivative of the sigmoid function can be defined as:
*derSigmoid = x * (1-x)*
As Krish Naik well said, we have our maximum when *x=0.5*, giving us back:
*derSigmoid = 0.5 * (1-0.5) --------> derSigmoid = 0.25*
That's the reason the derivative of the sigmoid function can't be higher than 0.25
COOL
cool
should be: derSigmoid(x) = Sigmoid(x)[1-Sigmoid(x)], and we know it reaches maximum at x=0. Plugging in: Sigmoid(0)=1/(1+e^(-0))=1/2=0.5, thus derSigmoid(0)=0.5*[1-0.5]=0.25
@@tvfamily6210 Thank you!
I'm still confused. The weight w should be in here somewhere. This seems to be missing w.
I like the way you explain things, making them easy to understand.
Thank you sir for your amazing video. that was great for me.
Thank you, Krish SIr. Nice explanation.
Very well explained. I can't thank you enough for clearing all my doubts!
crystal clear explanation !
Love your videos, I have watched and taken many courses but no one is as good as you
hats off to you sir,Your explanation is top level, THnak you so much for guiding us...
do u completed his full playlist?
Thank you for all the effort you put into your explanations, they are very clear!
Sir thank u for teaching us all the concepts from basics but just one request is that if there is a mistake in ur videos then pls rectify it as it confuses a lot of people who watch these videos as not everyone sees the comment section and they just blindly belive what u say. Therefore pls look into this.
Krish...you rock brother!! Keep up the amazing work!
Nice video Krish.Please make practicle based video on gradient decent,CNN,RNN.
Very nice video sir , you explained very well the inner intricacies of this problem
Derivative of loss with respect to w11 dash you specified incorrectly, u missed derivative of loss with respect to o21 in the equation. Please correct me if iam wrong.
Please reply
Evn I hv this doubt
Apologies for the delay...I just checked the video and yes I have missed that part.
@@krishnaik06Hey!,
U dnt hv to apologise, on the contrary u r dng us a favour by uploading these useful videos, I was a bit confused and wanted to clear my doubt that all, thank you for the videos... Keep up the good work!!
@@krishnaik06 I think you have also missed the w12 part in the derivative. Please correct me if I am wrong
Great Lecture
I understood it. Thanks for the great tutorial!
My query is:
weight vanishes when respect to more layers. When new weight ~= old weight result becomes useless.
what would the O/P of that model look like (or) will we even achieve global minima??
that was very well explained
you are legend nayak sir
you sir made neural network so much fun!
Very nice series... 👍
Great efforts Sir
Hello sir, why the chain rule explained in this video is different from the very last chain rule video. kindly clearly me and thanks for such an amazing series on deep learning.
Great explanation, Thank you!
great video! thank you so much!
You just earned a +1 subscriber ^_^
Thank you very much for the clear and educative video
Understood completely! If weights hardly change, no point in training and training. But I have got a question, where can I use this knowledge and understanding I just acquired ?
you teach better than ivy league professors. what a waste of money spending $$$ on college.
Thank you so much for this
Good job bro as usual... Keep up the good work.. I had a request of making a video on implementing back propagation. Please make a video for it.
Already the video has been made.please have a look on my deep learning playlist
@@krishnaik06 I have seen that video but it's not implemented in python.. If you have a notebook you can refer me to pls
With respect to implementation with python please wait till I upload some more videos
Great tutorial man! Thank you!
excellent video
nice explanation.
your videos are very helpful ,good job and good work keep it up...
Very interesting
Nice expalnation sir
Guruvar ko pranam🙏
nice explanation
May god bless you ..
Very well explained. Vanishing gradient problem as per my understanding is that, it is not able to perform the optimizer job (to reduce the loss) as old weight and new weights will be almost equal. Please correct me, if i am wrong. Thanks!!
As usual extremely good outstanding...
And a small request can expect this DP in coding(python) in future??
Yes definitely
Thank you so much
Hats Off Brother
great video and great explanation
This video is amazing and you are amazing teacher thanks for sharing such amazing information
Btw where are you from banglore?
very nice explanation,,great :)
excellent explanation sir
Gradient Descent will be applied on Cost function right ?-1/m Σ (Y*log(y_pred) + (1-y)* log(1-y_pred))... in this case if they had applied on the activation function, how the algo will come to global minima.
You are just amazing. Thnx
best explanation. Thanks man
super video...extremely well explained.
This is an interesting fact to know. Makes me curious to see how ReLU overcame this problem
Thank you !!
very good explanation.
Could you please explain why bias is needed in neural networks along with weights?
it is because when you want to control or fix the output of a given neuron within a certain range, for example, if the neuron is always giving inputs between 9 and 10, you can put a bias =-9 so as to make the neuron output between 0 and 1
Great-
Excellent 👌
Why cant they just make the activation function curve aggressively spike up at x=0?
Great video, one question, when you calculate the new weights using the old weight - learning rate x derivative of loss with respect to weight, the derivative of loss wrt weight is that the sigmoid function ?
Sir i'm really confusing about the actual y value please can you tell about that. i thought it would be our input value but here input value is so many with one predicted
output
Should we again add bias to the product of the output from the hidden layer O11, O12 and weights W4, W5?
Thanks krish .Video was superb but I am having apprehension I might get lost somewhere .Please provide some reading reference regrading this topic considering as a beginner.Cheers
Dear Sir Splendid tutorial indeed. Please share the reference book link that you are following.
nice explaination
you meant that the derivative of the sigmoid is between 0 and 0.25, right? I wanted to clarify about that range written in red color. The sigmoid of z would be between 0 and 1, from what I understood. Any reply will be appreciated :)
if u ask me which one is more understanding between andrew Ng and krish naik , i think this would be my choice
In red color you've written that sigmoid of z is between 0 and 0.25 whereas you meant to say that the derivative of sigmoid of z is between 0 and 0.25.
yes
There is a very big mistake at 4:14 sir, you didn't applied the chain rule correctly, check the equation.
I have the same doubt
you talk about changing weights on back propagation by derivation and then suddenly talks about derivating activation function, i dont understand that part, does function triggered during back propagation?
Thank you !
Sir GANs ky bhi tutorials bnae plz
Hi Krish, Thanks for the good videos..God may bless you and your family..
Do we have merial for ML and DL.If you have how to get it.
so if we have 2 layers and as we know 1 forward and back step is 1 epoch. If we now have 100 epochs the derivative is going to get smaller every time? Or the vanishing problem is due to many hidden layers and not
depended on the number of epochs?