Good to hear that you found it helpful! Thank you for the feedback; this is covered in some of the previous videos (part of this playlist: th-cam.com/video/U-XYvDhGQ2c/w-d-xo.html&pp=gAQB)
How often are these methods even used in practise because from what I have realized you would need to retrain neural network as many times as there are combinations/configurations?
Great question! Actually, these methods are commonly adopted in practice to optimize the performance of the problem at hand. You indeed have to retrain the neural network as many times as there are configurations. This is expensive but can be worked around by testing the configurations on a subset of the data. Also, it is possible to use successive halving to make the hyperparameter search more efficient.
Great question! LogUniform sampling is especially useful when the range of the hyperparameter of interest spans multiple orders of magnitude. For example, suppose we want to find a good value for the learning rate within a range [0.01, 100]. Because the interval [10,100] is larger than [1, 10], uniform sampling is more likely to sample more points in the interval [10,100]. However, what if we want to sample uniformly across the different orders of magnitude? By sampling LogUniformly, we can achieve this, since the distance between [1,2] (corresponding to [10,100]) is equal to the distance [0,1] (corresponding to [1, 10]). If you'd like to know more, this is a good blog post (answer was inspired by this): towardsdatascience.com/why-is-the-log-uniform-distribution-useful-for-hyperparameter-tuning-63c8d331698
Thank you for the explanation, I found it very clear :)
thank you so much for the video!
Great explanation. Would be helpful if Hyperparameter Tuning was also included in the vid along with other background information.
Good to hear that you found it helpful! Thank you for the feedback; this is covered in some of the previous videos (part of this playlist: th-cam.com/video/U-XYvDhGQ2c/w-d-xo.html&pp=gAQB)
How often are these methods even used in practise because from what I have realized you would need to retrain neural network as many times as there are combinations/configurations?
Great question! Actually, these methods are commonly adopted in practice to optimize the performance of the problem at hand. You indeed have to retrain the neural network as many times as there are configurations. This is expensive but can be worked around by testing the configurations on a subset of the data. Also, it is possible to use successive halving to make the hyperparameter search more efficient.
Why are we using logUniform and not just uniform? I didnt quite get that.
Great question! LogUniform sampling is especially useful when the range of the hyperparameter of interest spans multiple orders of magnitude. For example, suppose we want to find a good value for the learning rate within a range [0.01, 100]. Because the interval [10,100] is larger than [1, 10], uniform sampling is more likely to sample more points in the interval [10,100]. However, what if we want to sample uniformly across the different orders of magnitude? By sampling LogUniformly, we can achieve this, since the distance between [1,2] (corresponding to [10,100]) is equal to the distance [0,1] (corresponding to [1, 10]). If you'd like to know more, this is a good blog post (answer was inspired by this): towardsdatascience.com/why-is-the-log-uniform-distribution-useful-for-hyperparameter-tuning-63c8d331698
@@aixplained4763 Amazing! I get it now, thank you for this and the vid.
great