Prof...stop ...stop...it's already dead! Oh BP you thought you were this tough complex thing and then you met Prof. Justin Johnson who ended you once and for all! The internet is 99.99% garbage but content like this makes me so glad that it exists. What a masterclass! What a man!
Sir, you are amazing! I've wasted hours reading and watching internet gurus on this topic, and they could not explain it at all, but your lecture worked!
At 58:56 prof Johnson tells something huge imho , the final equation is not formed by jacobians , finally I got it..simply the best explanation on the backprop .Thank you prof Johnson
I don't get how back propagation tutorials by 3B1B, StatQuest, etc, get so much praise, but neither of them are as succinct as you were in those first two examples. Fuck that was simple.
19:38 Shouldn't 0.39 be 0.4 and 0.59 be 0.6 -- not sure where the rounding errors have creeped in. 49:45 would it not be much easier to use Einstein index notation?
Great lecture thank you. I have a question, would be great if anyone could clarify. When you first introduce vector valued backpropagation, you have the example showing 2 inputs to the node, each input is a vector of DIFFERENT dimension - when would this be the case in a real scenario? I thought the vector formulation was so that we could compute the gradient for a batch of data (e.g. 100 training points) rather than running backprop 100x. In that case the input vectors and output vectors would always be of the same dimension (100). Thanks!
Can anyone explain 1:08:05? dL/dx1 should be next to dL/dL, not L when it is subject to function f2'. Thereby back propagation cannot connect fs and f's.
Our upstream gradient was -0.53 right? And now we need the local gradient of e^-x which is -e^-x and -e^-(-1)= -0.36. So upstreamgrad(-0.53) multiplied with local grad (-0.36) is 0.1949 which is approximately 0.2. So 0.2 is not local grad it is local multiplied with upstream
I don't think beaver's answer is quite right. The upstream gradient is -0.53. But the local gradient comes from the function e^x not e^-x. The derivative of e^x is e^x. Now we plug in the input which is -1 and we get e^-1 as the local gradient. This is approximately 0.37. Now that we have the local gradient we just multiply it with the upstream gradient -0.53 which results in approximately -0.20.
No, the operation is not relu, its calculation of the downstream gradient. since last row of jacobian is 0 meaning that changes in that value does not affect the output, so 0.
For the point of passing or killing the value of the upstream matrix, you have to decide pass or kill by looking at the input matrix, here that is [ 1, -2, 3, -1] so looking at -1, we will kill that value from the upstream matrix, so 0.
It is actually muchhhhhh more simpler than the way he used to explain. I believe he was redundant and too many symbols that hides the beauty of the underneath reason of the algorithm and the math behind it. It all could have been explained in less amount of time.
![KRK - ชนิสรา Ft.WanMai , N/A [Official MV] Prod. By Sakarin](http://i.ytimg.com/vi/C-WMwwYobcU/mqdefault.jpg)
Prof...stop ...stop...it's already dead! Oh BP you thought you were this tough complex thing and then you met Prof. Justin Johnson who ended you once and for all!
The internet is 99.99% garbage but content like this makes me so glad that it exists. What a masterclass! What a man!
this lecture is an example of a perfect technical lecture
I 've just watched 30 minutes, but I 'm so excited to comment here that it's definately the best course for back propagation!!!!
Amazing! One of the best Backprop explanation out there!
I work in ML and am doing review for interviews, this lecture is extremely thorough!
Sir, you are amazing! I've wasted hours reading and watching internet gurus on this topic, and they could not explain it at all, but your lecture worked!
Dr. JJ, you sly sun of a gun. This is one of the best things ever. 47:39, the way he asks if it is clear. It is damn clear man. Well Done!
Best lecture ever on explanation of backpropagation in math
Finally!! I understood how to apply backpropagation. Thank you sir! Thank you!
Future reference for anybody, but I think there's a typo @ 50:24. It should be dz/dx * dL/dz when using chain rule to find dL/dx
finally some coverage on backprop with tensors
At 58:56 prof Johnson tells something huge imho , the final equation is not formed by jacobians , finally I got it..simply the best explanation on the backprop .Thank you prof Johnson
10:02. Dr. Johnson means, "right to left" not "left to right"
I don't get how back propagation tutorials by 3B1B, StatQuest, etc, get so much praise, but neither of them are as succinct as you were in those first two examples. Fuck that was simple.
You earned a like, a comment and a subscriber ... what an explanation .
wooooowww.. what a superb lecture on backpropagation. simply amazing.
Such an amazing lecture with easy-to-understand examples!
Such awesome and intuitive explaination!
Thank you very much! I really enjoy this lecture! Hello from Russia with love :)
22:22 the local gradient should be "[1-sigma(1.00)]*sigma(1.00)" where 1.00 is the input to the sigmoid-fcn block
The good thing about these lectures is that finally Dr.Johnson has more time to speak compared to cs231n !
how come u are getting the value of e^x as -0.20. Could u explain
Amazing courses!
Top notch content
this is extremly hard. but this is a great lecture for sure. you are awesome Mr Johnson
I finally understand backprop!
Beautifully done!
Masterpiece!!!!
@49:44 - Mistake in dL/dx formula - 2nd operand should be dL/dz (not dL/dx)
At around 18:20 shouldn't the original equation have a w_2 term that gets added to w_0*x_0+w_1*x_1?
w2 is a bias
THANK YOU SO MUCH! finally not shallow and excellent explanation.
19:38 Shouldn't 0.39 be 0.4 and 0.59 be 0.6 -- not sure where the rounding errors have creeped in.
49:45 would it not be much easier to use Einstein index notation?
Great lecture thank you. I have a question, would be great if anyone could clarify. When you first introduce vector valued backpropagation, you have the example showing 2 inputs to the node, each input is a vector of DIFFERENT dimension - when would this be the case in a real scenario? I thought the vector formulation was so that we could compute the gradient for a batch of data (e.g. 100 training points) rather than running backprop 100x. In that case the input vectors and output vectors would always be of the same dimension (100). Thanks!
1:03:00 should the dimension of grad x3 / x2 be D2 x D3?
Awesome explanation of Backpropagation! Amazing slides! Much better than CS231n.
great video.thanks for posting it
Can anyone explain 1:08:05? dL/dx1 should be next to dL/dL, not L when it is subject to function f2'. Thereby back propagation cannot connect fs and f's.
Amazing!
19:00 where did w2 come from?
19:14 Can someone explain computing the local gradient of exponential function. I mean how the result -0.2 comes? I'm lost there!!!
Our upstream gradient was -0.53 right? And now we need the local gradient of e^-x which is -e^-x and -e^-(-1)= -0.36. So upstreamgrad(-0.53) multiplied with local grad (-0.36) is 0.1949 which is approximately 0.2. So 0.2 is not local grad it is local multiplied with upstream
@@beaverknight5011 got it, thank you so much!
@@zainbaloch5541 you are welcome, good luck with your work
I don't think beaver's answer is quite right. The upstream gradient is -0.53. But the local gradient comes from the function e^x not e^-x. The derivative of e^x is e^x. Now we plug in the input which is -1 and we get e^-1 as the local gradient. This is approximately 0.37. Now that we have the local gradient we just multiply it with the upstream gradient -0.53 which results in approximately -0.20.
Can anyone clarify the computation of hessian matrix in detail ?
46:16, shouldn't dl/dx be 4, 0, 5, 9 instead of 4, 0, 5, 0?
No, the operation is not relu, its calculation of the downstream gradient. since last row of jacobian is 0 meaning that changes in that value does not affect the output, so 0.
For the point of passing or killing the value of the upstream matrix, you have to decide pass or kill by looking at the input matrix, here that is [ 1, -2, 3, -1] so looking at -1, we will kill that value from the upstream matrix, so 0.
54:51 when my cd player gets stuck on a old eminem track
thanks, good god, best wish to you.
👍👍👍👍
45:00 Jacobean matrix does not have to be diagonal right?
Correct. That was unique to the ReLU function.
The "local gradient slices" in his discussion at 53:00 are slices of a more complex Jacobian.
Why is he calculating derivatives relative to the inputs?
It is actually muchhhhhh more simpler than the way he used to explain. I believe he was redundant and too many symbols that hides the beauty of the underneath reason of the algorithm and the math behind it. It all could have been explained in less amount of time.
do you have any resources that explain this better?
terrible sound quality !
it seems the coughing guy got the china virus at that time
I'm pretty sure you'd be called out as racist back in the days of your comment. Now that it's almost proven to be a china virus...