This is showing that the quality and value of a video is not depending on how fancy the animations are, but how expert and pedagogue the speaker is. Really brilliant! I assume you spent a lot of time designing that course, so thank you for this!
"Now that we understand the REASON we're doing this, let's get into the math." The world would be a better place if more abstract math concepts were approached this way, thank you.
This is awesome! Lots of machine learning books or online courses don't bother explaining the reason behind Ridge regression, you helped me a lot by pulling out the algebraic and linear algebra proofs to show the reason WHY IT IS THIS! Thanks!
I was searching for ridge regression on the whole internet and stumbled upon this is a video which is by far the best explanation you can find anywhere thanks.
Excellent video! One more thing to add - if you're primarily interested in causal inference, like estimating the effect of daily exercise on blood pressure while controlling for other variables, then you want an unbiased estimate of the exercise coefficient and standard OLS is appropriate. If you're more interested in minimizing error on blood pressure predictions and aren't concerned with coefficients, then ridge regression is better. Also left out is how we choose the optimal value of lambda by using cross-validation on a selection of lambda values (don't think there's a closed form expression for solving for lambda, correct me if I'm wrong).
Superb. Thanks for such a concise video. It saved a lot of time for me. Also, subject was discussed in a fluent manner and it was clearly understandable.
These explanations are by far the best ones I have seen so far on youtube ... would really love to watch more videos on the intuitions behind more complicated regression models
Fantastic! It's like getting the Cliff's Notes for Machine Learning. These videos are a great supplement/refresher for concepts I need to knock the rust off of. I think he takes about 4 shots of espresso before each recording though :)
Hi and thanks fr the video. Can you explain briefly why when the m_i and t_i variables are highly correlated , then the estimators β0 and β1 are going to have very big variance? Thanks a lot in advance!
You should add in that all the variables (dependent and independent) need to be normalized prior to doing a ridge regression. This is because betas can vary in regular OLS depending on the scale of the predictors and a ridge regression would penalize those predictors that must take on a large beta due to the scale of the predictor itself. Once you normalize the variables, your A^t*A matrix being a correlation matrix of the predictors. The regression is called "ridge" regression because you add (lambda*I + A^t*A ) which is adding the lambda value to the diagonal of the correlation matrix, which is like a ridge. Great video overall though to start understanding this regression.
Excellent approach to discuss Lasso and Ridge regression. It could have been better if you have discussed how Lasso yields sparse solutions! Anyway, nice discussion.
We start out by adding a constraint that beta 1 squared + beta 2 squared must be less than c squared, where c is some number we choose. But then after choosing lamda, we minimize F and c ends up having no effect at all on our choice of the betas. I may be wrong but it doesn't seem like c has any effect on our choice of lambda either. I find it strange that we start out with the criteria that beta 1 squared + beta 2 squared must be less than c squared, but the choice of c is irrelevant. If someone can help me un-boggle my mind that would be great.
Good question - I think it has to do with using the method of Lagrange multipliers to solve the constrained OLS optimization problem. The lambda gets multiplied by the expression in the parentheses at 11:17, which includes the c squared term. So whatever c squared value you choose, it's going to be changed anyways when you multiply by the lambda.
It is unintuitive that we are constraining weights(betas) within value c^2, yet the regularization expression does not include the c but rather sum of squared weights. Certainly I am missing something here. Alternatively, why adding a sum of squared betas(or weights) to the cost function help optimize beta that stays within constraint so that betas don't become large and vary across datasets?
Is the reason to not choose big LAMBDA because we maight get underfitting? If we choose big LAMBDA we get small W and then the output function (hypothesis) won’t reflect our data and we might see underfitting.
Can someone explain why does Ridge regression leads to shrinkage of co-efficients but not entirely zero co-efficients, whereas Lasso causes some co-efficients to become zero entirely.
Sir, a question about 4:54: I understand that in tax/income example the VARIANCE of the beta0-beta1's is high, since there's an additional beta2 effecting things. However, the MEAN in the population should be the same, even with high variance, isn't it so? Thanks in advance!
Hey Ritvik, I have a question about this one, I don't really know why we are choosing the point that is far from the origin point. So which direction does the gradient descent and why? Please help me out here, thank you so much!
Can anyone explain the statement "The efficient property of any estimator says that the estimator is the minimum variance unbiased estimator", so what is minimum variance denotes here.
Hi, OLS is unbiased. So also for data with multicollinearity, OLS generates an unbiased model (training data) but with high variance (test data). Ridge regression adds bias to the model (training data) to reduce the variance and make it more accurate (test data). I many real-world applications not only the model accuracy but also the "true" Betas (test set) are relevant to make accurate statements regarding the prediction and the importance of the variables. I tried the ridge regression in a simulation. However, the Betas were far away from the "true" Betas. Are the shrunken Betas are more like the "true" Betas (test set) or are there only less variance (e.g. a lower RMSE) than the Betas from the training data.
@ritvik when you said that the estimated coefficients has small variance does that implies the tendency of obtaining different estimate values of those coefficients ? I tend to confuse this term 'variance ' with the statistic Variance (spread of the data!).
Variance is the change in prediction accuracy of ML model between training data and test data. Simply what it means is that if a ML model is predicting with an accuracy of "x" on training data and its prediction accuracy on test data is "y" Variance = x - y A smaller variance would thus mean the model is fitting less noise on the training data, reducing overfitting. this definition was taken from: datascience.stackexchange.com/questions/37345/what-is-the-meaning-of-term-variance-in-machine-learning-model Hope this helps.
This is showing that the quality and value of a video is not depending on how fancy the animations are, but how expert and pedagogue the speaker is. Really brilliant! I assume you spent a lot of time designing that course, so thank you for this!
Wow, thanks!
Totally agree. I learn a lot from his short videos. Precise, concise, enough math, enough ludic examples. True professor mind.
This is the best explanation of Ridge regression that I have ever heard! Fantastic! Hats off!
"Now that we understand the REASON we're doing this, let's get into the math."
The world would be a better place if more abstract math concepts were approached this way, thank you.
good point
Watched these 5 years ago to understand the concept and I passed an exam. Coming back to it now to refresh my memory, still very well explained!
Nice! Happy to help!
This is awesome! Lots of machine learning books or online courses don't bother explaining the reason behind Ridge regression, you helped me a lot by pulling out the algebraic and linear algebra proofs to show the reason WHY IT IS THIS! Thanks!
I was searching for ridge regression on the whole internet and stumbled upon this is a video which is by far the best explanation you can find anywhere thanks.
It's so inspiring to see how you get rid of the c^2! I learned Ridge but didn't know why! Thank you for making this video!
This is, by far, the best explanation of Ridge Regression that I could find on TH-cam. Thanks a lot!
Your data science videos are the best I have seen on TH-cam till now. :)
Waiting to see more
I appreciate it!
Excellent video! One more thing to add - if you're primarily interested in causal inference, like estimating the effect of daily exercise on blood pressure while controlling for other variables, then you want an unbiased estimate of the exercise coefficient and standard OLS is appropriate. If you're more interested in minimizing error on blood pressure predictions and aren't concerned with coefficients, then ridge regression is better.
Also left out is how we choose the optimal value of lambda by using cross-validation on a selection of lambda values (don't think there's a closed form expression for solving for lambda, correct me if I'm wrong).
You are the best of all.... you explained all the things,,, so nobody is gonna have problems understanding them.
Amazing video, you really explained why we do things which is what really helps me!
Thanks a lot.. I watched many videos and read blogs before this but none of them clarified at this depth
I think its explained very fast, but still very clear, for my level of understanding its just perfect !
best explanation of any topic i've ever watched , respect to you sir
Thank you soooo much!!! You explain everything so clear!! and there is no way I couldn't understand!
This really helps me! Definitely the best ridge and lasso regression explanation videos on TH-cam. Thanks for sharing! :D
This really is gold, amazing!
You, Ritvik, are simply amazing. Thank you!
This really helped a lot. A big thanks to you Ritvik!
I subscribed just after watching this. Great foundation for ML basics
Superb. Thanks for such a concise video. It saved a lot of time for me. Also, subject was discussed in a fluent manner and it was clearly understandable.
It's just awesome. Thanks for this amazing explanation. Settled in mind forever.
I'm impressed by your explanation. Great job
Thanks! That means a lot
Excellent explanation! Could you please do a similar video for Elastic-net?
This is literally the best video on ridge regression
These explanations are by far the best ones I have seen so far on youtube ... would really love to watch more videos on the intuitions behind more complicated regression models
This is gold. Thank you so much!
So so so very helpful! Thanks so much for this genuinely insightful explanation.
Fantastic! It's like getting the Cliff's Notes for Machine Learning. These videos are a great supplement/refresher for concepts I need to knock the rust off of. I think he takes about 4 shots of espresso before each recording though :)
The explanation is so clear!! Thank you so much!!
Anyone else get anxiety when he wrote with the marker?? Just me?
Felt like he was going to run out of space 😂
Thank you so much thoo, very helpful :)
Brilliant! Just found your channel and can't wait to watch them all!!!
The best ridge regression lecture ever.
I was looking for the math behind the algorithm. Thank you for explaining it.
No problem!
Stunning! Absolute gold!
seriously!!!
Very good explanation in an easy way!
Thank you. I make the comment because I know I will never need to watch it again! Clearly explained..
Glad it was helpful!
Hi and thanks fr the video. Can you explain briefly why when the m_i and t_i variables are highly correlated , then the estimators β0 and β1 are going to have very big variance? Thanks a lot in advance!
Hi same question here😶🌫
Awesome, Thanks a Million for great video! Searching you have done video on LASSO regression :-)
best explanation on ridge reg. so far
Your explanation is extremely good!
I don't have money to pay him so leaving a comment instead for the algo. He is the best.
You should add in that all the variables (dependent and independent) need to be normalized prior to doing a ridge regression. This is because betas can vary in regular OLS depending on the scale of the predictors and a ridge regression would penalize those predictors that must take on a large beta due to the scale of the predictor itself. Once you normalize the variables, your A^t*A matrix being a correlation matrix of the predictors. The regression is called "ridge" regression because you add (lambda*I + A^t*A ) which is adding the lambda value to the diagonal of the correlation matrix, which is like a ridge. Great video overall though to start understanding this regression.
SUPER !!! You have to become a professor and replace all those other ones !!
Excellent approach to discuss Lasso and Ridge regression. It could have been better if you have discussed how Lasso yields sparse solutions! Anyway, nice discussion.
Brilliant! Could you make more videos about Cross validation, RIC, BIC, and model selection.
We start out by adding a constraint that beta 1 squared + beta 2 squared must be less than c squared, where c is some number we choose. But then after choosing lamda, we minimize F and c ends up having no effect at all on our choice of the betas. I may be wrong but it doesn't seem like c has any effect on our choice of lambda either. I find it strange that we start out with the criteria that beta 1 squared + beta 2 squared must be less than c squared, but the choice of c is irrelevant. If someone can help me un-boggle my mind that would be great.
Good question - I think it has to do with using the method of Lagrange multipliers to solve the constrained OLS optimization problem. The lambda gets multiplied by the expression in the parentheses at 11:17, which includes the c squared term. So whatever c squared value you choose, it's going to be changed anyways when you multiply by the lambda.
simple and effective video, thank you!
Brilliant simplification of this topic. No need for fancy presentation to explain the essence of an idea!!
THIS IS ONE HELL OF A VIDEO !!!!
I think it's the best video ever made
Excellent explanation, thanks!
I would trade diamonds for this explanation (well, allegorically! :) ) Thank you!!
Finally, someone who talks quickly.
great video, the explanation is really clear!
Awesome video! Very intuitive and easy to understand. Are you going to make a video using the probit link?
Thank you! Your explaining is really good, Sir. Do you have time to make a video explaining the adaptive lasso too?
It is unintuitive that we are constraining weights(betas) within value c^2, yet the regularization expression does not include the c but rather sum of squared weights. Certainly I am missing something here. Alternatively, why adding a sum of squared betas(or weights) to the cost function help optimize beta that stays within constraint so that betas don't become large and vary across datasets?
great video, brief and clear.
Amazingly helpful. Thank you.
best explanation ever!
Excellent explanation .
you are the man, keep doing what you're doing
Stunning!! Need more access to your coursework
Is the reason to not choose big LAMBDA because we maight get underfitting? If we choose big LAMBDA we get small W and then the output function (hypothesis) won’t reflect our data and we might see underfitting.
Thanks! why lamba cannot be negative? What if to improve variance it is need to increase the slope and not decrease?
awesome video, thank you very much!
Shouldn't the radius of the Circle be c instead of c^2 (at time around 7:00)?
Beautiful explanation
Dude ! Hats off 🙏🏻
I love this video, really informative! Thanks a lot
Thanks for this really helpful video!
Could you explain why the independent variables in A should be standardized for Ridge and Lasso Regression?
excellent video! Keep up the great work!
very good explanation in an easy way!!
great video! thank you very much.
Very clear. Thank you!
It is a brilliant video. Great
thanks for the nice explanation
great video - thanks
This is gold indeed!
Great video. A (very minor) question: isn't it c instead of c^2 when you draw the radius vector of the circle for \beta restriction?
think of it as an equation of a circle with center (0,0)
Great videos thanks for making it
Huge thanks!
nice video , I have a question: lambda depends on c, isnt it?
excellent video! thank you!
You are awesome!
Can someone explain why does Ridge regression leads to shrinkage of co-efficients but not entirely zero co-efficients, whereas Lasso causes some co-efficients to become zero entirely.
Sir, a question about 4:54: I understand that in tax/income example the VARIANCE of the beta0-beta1's is high, since there's an additional beta2 effecting things. However, the MEAN in the population should be the same, even with high variance, isn't it so? Thanks in advance!
hey, great video and excellent job
Hey Ritvik, I have a question about this one, I don't really know why we are choosing the point that is far from the origin point. So which direction does the gradient descent and why? Please help me out here, thank you so much!
Can anyone explain the statement "The efficient property of any estimator says that the estimator is the minimum variance unbiased estimator", so what is minimum variance denotes here.
Hi,
OLS is unbiased. So also for data with multicollinearity, OLS generates an unbiased model (training data) but with high variance (test data). Ridge regression adds bias to the model (training data) to reduce the variance and make it more accurate (test data). I many real-world applications not only the model accuracy but also the "true" Betas (test set) are relevant to make accurate statements regarding the prediction and the importance of the variables. I tried the ridge regression in a simulation. However, the Betas were far away from the "true" Betas. Are the shrunken Betas are more like the "true" Betas (test set) or are there only less variance (e.g. a lower RMSE) than the Betas from the training data.
excellent video, thanks.
I was taught that the name Ridge Regression comes from the lambda I matrix. It looks like a ridged staircase shape.
@ritvik when you said that the estimated coefficients has small variance does that implies the tendency of obtaining different estimate values of those coefficients ? I tend to confuse this term 'variance ' with the statistic Variance (spread of the data!).
Variance is the change in prediction accuracy of ML model between training data and test data.
Simply what it means is that if a ML model is predicting with an accuracy of "x" on training data and its prediction accuracy on test data is "y"
Variance = x - y
A smaller variance would thus mean the model is fitting less noise on the training data, reducing overfitting.
this definition was taken from: datascience.stackexchange.com/questions/37345/what-is-the-meaning-of-term-variance-in-machine-learning-model
Hope this helps.
@@benxneo thanks mate!
excellent video!!!!
this is great stuff
very helpful, thanks
What is the intuition behind putting the constraint on the size of the Beta coefficient rather than the standard errors of the Beta coefficient?