I have noted that my models would not converge nicely (last assignment from C1W4, 3 ReLU + 1 sigmoid layers) when comparing to a notebook reference that I'm following. If I just initialized my weights from a normal distribution, the cost would get stuck at a high value. I've tried scaling the weights, changing to a uniform distribution, changing the learning rate to various values, nothing worked. Then following your code, I saw that if I divided the weights for each layer according to the sqrt of the number of features to that layer, then it would start converging beautifully. Would be interesting to know why! Thanks for your lessons!
If you use tanh activation function you have an even bigger problem - the gradients will always be equal to zero, and no learning is feasible (not even a disabled - all weights go in the same direction - learning).
@@byteofcnn3519 how would you learn what the perfect learning rate is? It makes sense to initialize it at 0.01 as that rate is similar to the pace of learning in humans (tiny changes over time).
Since we are using leaky ReLU for most cases now, should we initiate weights as extreme as possible so when back propagation take place, they will have higher chance to land in different local extremas?
kiryu nil, well I think the best way to initialize the weights is by using xaveir initializer, from my observations, its the best way, i think that is why the default initializer in tensorflow is this
@@dhirajupadhyay01 , In short, it helps signals reach deep into the network. If the weights in a network start too small, then the signal shrinks as it passes through each layer until it’s too tiny to be useful. If the weights in a network start too large, then the signal grows as it passes through each layer until it’s too massive to be useful. Xavier initialization makes sure the weights are ‘just right’, keeping the signal in a reasonable range of values through many layers.
As you can see here www.desmos.com/calculator/hzsiwhfmdw When x is too large sigmoid(x) is ~flat. thus, the derivative~=0 and when we have a very small gradient/derivative we're going to make very small steps towards the minimum which means a slow learning
I have noted that my models would not converge nicely (last assignment from C1W4, 3 ReLU + 1 sigmoid layers) when comparing to a notebook reference that I'm following.
If I just initialized my weights from a normal distribution, the cost would get stuck at a high value. I've tried scaling the weights, changing to a uniform distribution, changing the learning rate to various values, nothing worked.
Then following your code, I saw that if I divided the weights for each layer according to the sqrt of the number of features to that layer, then it would start converging beautifully. Would be interesting to know why!
Thanks for your lessons!
It seems like the most general statement of the solution is that the coefficients must form full rank matrices.
from where can we access the practice questions?
If you use tanh activation function you have an even bigger problem - the gradients will always be equal to zero, and no learning is feasible (not even a disabled - all weights go in the same direction - learning).
That random korean subtitle tho...
what is the best choice for learning rate(alpha)...?
@Amey Paranjape can the learning rate be learned?or is it meaningful to do so?
@@byteofcnn3519 how would you learn what the perfect learning rate is? It makes sense to initialize it at 0.01 as that rate is similar to the pace of learning in humans (tiny changes over time).
Since we are using leaky ReLU for most cases now, should we initiate weights as extreme as possible so when back propagation take place, they will have higher chance to land in different local extremas?
kiryu nil, what do u mean by 'as extreme'??
Using tf.random_normal, to set high standard deviation*
kiryu nil, well I think the best way to initialize the weights is by using xaveir initializer, from my observations, its the best way, i think that is why the default initializer in tensorflow is this
@@wolfisraging But why? (if you could explain)
@@dhirajupadhyay01 , In short, it helps signals reach deep into the network.
If the weights in a network start too small, then the signal shrinks as it passes through each layer until it’s too tiny to be useful.
If the weights in a network start too large, then the signal grows as it passes through each layer until it’s too massive to be useful.
Xavier initialization makes sure the weights are ‘just right’, keeping the signal in a reasonable range of values through many layers.
**UNSTABLE EQUILIBRIUM**
If W = 0, B = 0, then A = 0. Similarly all vectors should be zero. Isn't it?
I also think all nodes should be equal to zero. Interestingly, though, Andrew never mentions that property.
Yes. But the symmetry can also explain why we cannot initialize all weights to an nonzero same value.
I would say, that it depends on the chosen activation function
can anyone explain why gradient descent study slow when the slope is 0 (flat)? arent we are trying to find the max and min in this function? thanks
As you can see here www.desmos.com/calculator/hzsiwhfmdw
When x is too large sigmoid(x) is ~flat. thus, the derivative~=0
and when we have a very small gradient/derivative we're going to make very small steps towards the minimum which means a slow learning
Best activation function???
ReLu
Could have been a shorter video....