WooW. After I watched 5 other videos about ResNets, I was still lost. Now I got this video and it cleared my misunderstandings out of my mind. Super cool!
Finally got it! Sometimes it is hard for me to grasp even the basic concepts if I don't know what I should understand and take from the topic. Thank You very much for these videos that will at first tell you what problem you are trying to solve is and why you should solve it and then clearly explains the soulutions. Thumbs up! :)
I think we could use the ResNet concept to improve Dropout, creating a "shutdown" regularization: Select a layer (or rather nodes from that layer) that ought to be shutdown and instead only act on the cost function, by adding a cost relative to that layer not being an identity layer. Then the network is free to gradually adapt itself (hopefully by reducing train-set overfit and generalizing) as to push that layer into being evermore so of an identity one. Then if that layer manages to be an identity, it can be permanently shutdown. This could be a way to reduce the network size, and maybe could automatically be applied on high variance with low bias. As far as linear Z functions go, one way for a layer to be an identity is if it has the same amount of nodes as inputs, and if you make a cost for each node[j] so that only their weight[j] is 1 while all other weights are 0, so this would be similar to a "identity" Z layer. But I think that trying to make the activation function also an identity is a hassle, but even ignoring the activation function, if you could still manage to just shutdown the Z function nodes and stack the posterior activation back into the previous activation, that would already be a network simplification. Edit: We also could try to simplify the activation functions if we generalize them and re-parametrize them. Eg. for a ReLU activation function, we could turn it into a leaky ReLU where the leaky-side parameter starts at zero (so it's just like normal ReLU), then we add a cost of that parameter being zero and we let backprop start pushing it towards 1, in which case that previously ReLU activation has turned into the identity activation, which can them be gracefully shutdown.
Just a question for those out there - has anyone been able to use techniques from ResNets to improve the convergence speed of deep fully connected networks? Usually people use skip connections in the context of convolutional neural nets but I haven't seen much gain in performance with fully connected ResNets, so just wondering if there's something else I may be missing.
Hello everyone. In this video at time 5.20 you have mentioned that as number of layers increase in plain network , training error gets increased in practice. Could you please explain me or Share me some references why does this actually occurs? One reason,i found that vanishing gradient problem and this can be addressed using ReLU. Thus, one can use ReLU in plain network. Then why does ResNet is very traditional?
You still calculate them, you just keep a copy of the original a_l value and plug it into the network before calculating a[l+2]. Why would you even do that? It's explained in the next video
@@IvanOrsolic If we plug this value to the network, then what is the benefit of skipping the layers? still we are going through the layers to calculate a[l+2], we just added one more term here but how it helps in skipping the connection ?
Hi Andrew , I have a question, when we are calculating al+2 we need both al and z[l+2] . But z[l+2] can only be calculated by calculating a[l+1] , so how will we get that ? ....Many thanks:-)
Theoretically, the error should go down. But in practice, I think the exploding gradients for a network with a large number of layers increases the error.
Yes there is. I am not sure why Andrew Ng didn't touch up on this. Basically once you add the skip connection you are including an additive term inside the non-linearity. The additive term can only increase the function space (the range of the function www.intmath.com/functions-and-graphs/2a-domain-and-range.php) as it is inside the original function (the theory of nesting functions). Hence, you allow the network to have more approximating/predictive capacity in each layer. You can visit the D2L lectures about this: d2l.ai/chapter_convolutional-modern/resnet.html?highlight=resnet
I find a ResNet behaves as a shallower net. It gives a solution that resembles that of a four to six (roughly) layered net when being eight laters. ResNets are out for me.
Kavorka Games ResNets are now that state of the art for image recognition. Every new architecture uses it and doesn’t make sense anymore to use plain networks.
@@okktok but isn't the actual layers in play = total layers/2 as we are providing a shortcut? so, on a broader note, they are just like plain networks which looks bigger?
![[Classic] Deep Residual Learning for Image Recognition (Paper Explained)](http://i.ytimg.com/vi/GWt6Fu05voI/mqdefault.jpg)
your videos are iconic and should be preserved in the national library
Internet is a nice global digital library right?
*international library
Hi Andrew, You are my first professor who taught me ML. I studied your course at Coursera nice seeing you again.
Periyar oru porukki.. avan nijamaana naathigane kidaiyaadhu.. avar berum hindukkal manathai punpaduthara maathiriye pesuvar..
Andrew os simply the best instrutor in neural networks subject out there. Helped me a Lot.
I am doing the DL specialization after finishing your old ML course. By far you are the best teacher out there. Thank you so much for this.
WooW. After I watched 5 other videos about ResNets, I was still lost. Now I got this video and it cleared my misunderstandings out of my mind. Super cool!
Finally got it! Sometimes it is hard for me to grasp even the basic concepts if I don't know what I should understand and take from the topic. Thank You very much for these videos that will at first tell you what problem you are trying to solve is and why you should solve it and then clearly explains the soulutions. Thumbs up! :)
never seen tutorial videos which such a clear explanation. He is the best
Your explanation is very clear! Thank you for the lecture.
Me after listening to most people explaining Resnet: "What? Why? Why do you do this?"
Me after listening to Andrew: "Makes sense. Easy peasy."
You are the best, Andrew
I'm trying to understand the components behind Semantic Segmentation and your videos really helped!
Excellent both theoretically and technically
Sir, very nice explanation as always, thanks a lot.
Thank you, it was a very brief and simplified explanation, loved it.
I think we could use the ResNet concept to improve Dropout, creating a "shutdown" regularization:
Select a layer (or rather nodes from that layer) that ought to be shutdown and instead only act on the cost function, by adding a cost relative to that layer not being an identity layer. Then the network is free to gradually adapt itself (hopefully by reducing train-set overfit and generalizing) as to push that layer into being evermore so of an identity one. Then if that layer manages to be an identity, it can be permanently shutdown.
This could be a way to reduce the network size, and maybe could automatically be applied on high variance with low bias.
As far as linear Z functions go, one way for a layer to be an identity is if it has the same amount of nodes as inputs, and if you make a cost for each node[j] so that only their weight[j] is 1 while all other weights are 0, so this would be similar to a "identity" Z layer. But I think that trying to make the activation function also an identity is a hassle, but even ignoring the activation function, if you could still manage to just shutdown the Z function nodes and stack the posterior activation back into the previous activation, that would already be a network simplification.
Edit: We also could try to simplify the activation functions if we generalize them and re-parametrize them. Eg. for a ReLU activation function, we could turn it into a leaky ReLU where the leaky-side parameter starts at zero (so it's just like normal ReLU), then we add a cost of that parameter being zero and we let backprop start pushing it towards 1, in which case that previously ReLU activation has turned into the identity activation, which can them be gracefully shutdown.
"man pain" 2:10 😂
omg he is really writing that :D
Thank you!
So the skipped connections don't literally skip layers but rather add the original input onto the output of the 'skipped' layers?
I think so too, at least it is the only explanation that I understand
Great
+1 this is great
Why training error is increasing in reality as opposed to theory in plain model??
what is the benefit of adding identity blocks and skip it? Instead of skipping it why then we are adding?
Do you broadcast a[l] to make it match the dimensionality of a[l+2]?
Good
Just a question for those out there - has anyone been able to use techniques from ResNets to improve the convergence speed of deep fully connected networks? Usually people use skip connections in the context of convolutional neural nets but I haven't seen much gain in performance with fully connected ResNets, so just wondering if there's something else I may be missing.
Wait!!!!! Z_{l+1} should normally be equal to W_{l}a_{l} + b_{l+1}??
I have a very silly doubt if the skip layers/connections exist isn't the real layers in play = total layers/2?
Hello everyone. In this video at time 5.20 you have mentioned that as number of layers increase in plain network , training error gets increased in practice. Could you please explain me or Share me some references why does this actually occurs? One reason,i found that vanishing gradient problem and this can be addressed using ReLU. Thus, one can use ReLU in plain network. Then why does ResNet is very traditional?
I think the title of the paper is wrong. The correct one is "Deep residual learning for image recognition".
If a_l skips two layers to directly enter the final ReLU, how do we get the z_(l+2) in the final equation a_(l+2) =g(z_(l+2) + a_(l))? Thanks!
You still calculate them, you just keep a copy of the original a_l value and plug it into the network before calculating a[l+2].
Why would you even do that? It's explained in the next video
@@IvanOrsolic If we plug this value to the network, then what is the benefit of skipping the layers? still we are going through the layers to calculate a[l+2], we just added one more term here but how it helps in skipping the connection ?
there's some high-frequency noise in many of this specialization's videos which is hurting my ears.
use a low pass filter. Lol
how are the dimensions handled, I mean that dimension of a[l] could happen to be unequal to a[l + 2];
Padding
padding with zeros or using an Conv 1x1 inside skip connection
Meet the ML god
Sounds like terminator..
Hi Andrew , I have a question, when we are calculating al+2 we need both al and z[l+2] . But z[l+2] can only be calculated by calculating a[l+1] , so how will we get that ? ....Many thanks:-)
If I understand your question correctly, a[I+2] = activation of z[I+2] ......
just like in formula a[l+1]=g(z[l+1])...
Yep this confuses me as well. How can we have both z[l+2] and a[l+1] if a[l+1] skips a layer?
is there any theoretical motivation justifying the increasing error of a deep plain network during training?
Theoretically, the error should go down. But in practice, I think the exploding gradients for a network with a large number of layers increases the error.
Yes there is. I am not sure why Andrew Ng didn't touch up on this. Basically once you add the skip connection you are including an additive term inside the non-linearity. The additive term can only increase the function space (the range of the function www.intmath.com/functions-and-graphs/2a-domain-and-range.php) as it is inside the original function (the theory of nesting functions). Hence, you allow the network to have more approximating/predictive capacity in each layer. You can visit the D2L lectures about this:
d2l.ai/chapter_convolutional-modern/resnet.html?highlight=resnet
I find a ResNet behaves as a shallower net. It gives a solution that resembles that of a four to six (roughly) layered net when being eight laters. ResNets are out for me.
Kavorka Games ResNets are now that state of the art for image recognition. Every new architecture uses it and doesn’t make sense anymore to use plain networks.
@@okktok but isn't the actual layers in play = total layers/2 as we are providing a shortcut?
so, on a broader note, they are just like plain networks which looks bigger?
If we use resnets, are we losing the information in between the layers?
no, we're not losing them - we just add x after a amount of layers (in this example 2 layers) - this is our ResNet block
Thank you !