Thank you very much! I am from China, not only can you give us good tutorial but also your English is very cleary so that I can improve my English listening level. Thank you very much! 唯一真神。
In case anyone's wondering why increasing the learning rate decreased the cost, it's because the number of iterations is very low in order for gradient descent to converge. So if lower learning rate gives you higher cost, you should increase the number of iterations.
Hello, I was wondering why at minute 12:25, when calculating the dw parameters you don't use np.sum if it includes the sum in the formula. Also is there a reason its included with parameter db?
It's bc the dot product already performs the summation so the np.sum command would be redundant. Since db only uses for vector subtraction, it needs the np.sum command
Good work with detailed explanations. Always happy when I am struggling with a concept to see you have a video out even remotely related to what I am working on.
Excellent work! To evaluate our model we could also use the coefficient of determination (R**2) which is not affected by the scale of our measurements like MSE does.
Hi. Let's say we multiply two matrices A and B. Then the number of columns of A must be number of rows of B: The size formula is: A(lxm) * B(mxn) = C(lxn) In our example: np.dot(X, self.weights) --> (80,1) * (1,) = (80,) -->for every sample we get one value (the 1 is because we have 1 feature) and in the transposed case: np.dot(X.T, (y_predicted - y)) --> (1, 80) * (80,) = (1,) --> for every feature we get one value
Hello; while this video is mostly very clear and easy to understand, I do have one question, namely, doesn't line 23 perform the same action for every element in self.weights? if this is the case, and since every element in self.weights starts as 0, then why do you use an array instead of a constant?
@@patloeber ya, I sometimes watch you after class at night when the lights are low & I was blown away. lol. I'm over it, but that was a blast! Thank you for the work you put into this :)
This helped me so much, you explain it so clearly and your code is really neat and easy to understand! Thanks so much! Subbed, liked and will be using your videos as my ML bible from now on XD
N is not necessarily a constant. It depends on the data points you have. If you remove N form the equation, you cannot apply the package to different datasets
Bro, I really really really need you. Please make more videos on Machine Learning and please make more frequently. You take 1 week to make 1 video which is not fair. Please 2-3 videos in a week. Please it's my humble request.
I am hearing your request, and I try to increase my frequency at some point in the future! But I still have a normal day job and it takes time and effort for machine learning videos :)
Very good explanation! When I try the Boston Housing dataset from sklearn on this regression module. It throw me a runtime warning on the self.weight -=self.learning_rate * dw. When I reduce the iter from 1000 to 100, it works but with terrible MSE
Hmmm...I wonder why mse is so high, the graph looks prefectly fine. Anyway, great tutorials, thank you so much! Definitely waiting for gradient boosting tutorials!
Thank you very much! I think it's so high because the range is not normalized (e.g. y ranges from -160 to 200), So when you sum over all the test points and also square it, it gets high. MSE is always sample dependent.
Awesome vid. I would love to access that exact notebook with the actual math equations but I don't see it in your github. Is it possible to access that?
great videos! keep doing it. Just a quick question. With packages like Pytorch and Tensorflow making a lot of these things very simple, what is the thought process for implementing in Python. Is it for better understanding or do you think it's better way to learn?
Thanks for the vid. Great work. One question; however, what if you have more than two weights/model parameters, w, to optimize? Couldn’t you just use the normal equation, ((X.T.dot(X))**-1)*(X.T.dot(y)), to solve for w…or np.linalg.solve(X.T.dot(X), X.T.dot(y))?
Thanks. This code worked for most of the datasets but returns nan for boston dataset and returned different weights for diabetes datasets. But using Closed Form Normal equation instead of gradient descent gives correct weights and bias. why so.....??
My code is not optimized at all and might not always produce good results. Also try applying feature normalization before passing it to the algorithm. This usually improves performance here a lot
Why is there no 2 in the code for df/dw? Also why use Transpose for one and dot when calculating prediction,and what about this www.statisticshowto.com/probability-and-statistics/regression-analysis/find-a-linear-regression-equation/ , is this a different way to solve linear regression?
1. Sometimes people multiple MSE * 1/2 so it cancels out when you take the derivative (datascience.stackexchange.com/questions/52157/why-do-we-have-to-divide-by-2-in-the-ml-squared-error-cost-function) 2. For this simple case you can use LSE as a different approach to optimization methods like Gradient Descent (like shown in video) towardsdatascience.com/https-medium-com-chayankathuria-optimization-ordinary-least-squares-gradient-descent-from-scratch-8b48151ba756
@@patloeber After a few iterations the value of weight becomes an array of nan. And so does bias. Same thing works fine with sklearn's linear regression. Tried fixing it but couldn't. So I had to ask you at last. Thanks for responding.
Sorry I missed your response earlier, but yes as Damian said it is very likely that your dataset has invalid or incorrect data. My algorithm is not optimized and does not include error checks like in sklearn. You may also try applying a standard scaler before fitting the data.
@@patloeber Hey I have made some changes to both linear regression and knn to make it compatible with both dataframes and sklearn's data. Can I make a pull request ?
You can also copy my repo from github and try to run this with the exact same folder structure. Or maybe this is helpful: stackoverflow.com/questions/4142151/how-to-import-the-class-within-the-same-directory-or-sub-directory
regressor = LinearRegression() regressor.fit(X_train, y_train) predicted = regressor.predict(X_test) def mse(y_true, predicted): return np.mean((y_true - predicted)**2) mse_value=mse(y_test, predicted) def mse(y_true, predicted): ---> 11 return np.mean((y_true - predicted)**2) 12 mse_value=mse(y_test, predicted) 13 print(mse_value) ValueError: operands could not be broadcast together with shapes (20,) (20,80) i dont know why i can run this code, i follow excactly what him did. Can anybody help me. tks
Linear algebraic method: /////////////////////////////////////////////////////////////////////////// # This is linear least squares b = np.linalg.inv( np.dot( X.T, X ) ).dot( X.T ).dot( y ) # predict using the coefficients y_predicted = X.dot( b ) /////////////////////////////////////////////////////////////////////////// Then you have your solution. I was trying to get the videos implementation to have a better overall mean square error than the linear algebra formulation but could not do it. Anyway, this is just another way if anyone is looking. Great video series btw! I have enjoyed going through it.
Thank you very much! I am from China, not only can you give us good tutorial but also your English is very cleary so that I can improve my English listening level. Thank you very much! 唯一真神。
this is a really nice summary of the fundamentals. perfect for brushing up after learning about ML from 5 years ago. really solid.
Sir you deserve more than you have right now! you made a 13 year old guy teach those high level concepts. Loads of love from India!
This is nice to hear :) Thanks!
In case anyone's wondering why increasing the learning rate decreased the cost, it's because the number of iterations is very low in order for gradient descent to converge. So if lower learning rate gives you higher cost, you should increase the number of iterations.
thank you sir you just saved me from my assignment this semester. the instructions were very clear and visual. please keep producing videos!
Hello, I was wondering why at minute 12:25, when calculating the dw parameters you don't use np.sum if it includes the sum in the formula. Also is there a reason its included with parameter db?
It's bc the dot product already performs the summation so the np.sum command would be redundant. Since db only uses for vector subtraction, it needs the np.sum command
Good work with detailed explanations. Always happy when I am struggling with a concept to see you have a video out even remotely related to what I am working on.
That’s nice to hear :)
Excellent work!
To evaluate our model we could also use the coefficient of determination (R**2) which is not affected by the scale of our measurements like MSE does.
such a nice content , it made me so satisfied. love from China
Couldn't have done this without your help! And I needed this, thanks man
glad it was helpful!
Short and sweet :), extremely helpful.
Not just the content, I love your accent. :D
thank you so much for thoses great tut, can u please re-explain the line 20 from fit method i cant understand why we transpose the X matrix...
Hi. Let's say we multiply two matrices A and B. Then the number of columns of A must be number of rows of B:
The size formula is: A(lxm) * B(mxn) = C(lxn)
In our example:
np.dot(X, self.weights) --> (80,1) * (1,) = (80,) -->for every sample we get one value (the 1 is because we have 1 feature)
and in the transposed case:
np.dot(X.T, (y_predicted - y)) --> (1, 80) * (80,) = (1,) --> for every feature we get one value
@@patloeber Thank you now that's make sense to me
can you please explain what you mean by features and samples in the linear regression part?
13:15 Why using X.T and not X?
I have a question, how can we add mse in class method, LinearRegression
Thank you very much sir❤❤
I have seen MSE calculated as (1/2n)... Whats the difference between using (1/n) and (1/2n)?
Hello; while this video is mostly very clear and easy to understand, I do have one question, namely, doesn't line 23 perform the same action for every element in self.weights? if this is the case, and since every element in self.weights starts as 0, then why do you use an array instead of a constant?
hey man, did you figure out the answer?
Beautiful explaination!
hey I have a question what is the 'predicted' in mse_value = mse(y_test, predicted)???
Nice video dude! Can you tell me the font you use in your Editor please?!
in this video it's the monokai theme, now I use Nightowl theme
does this not work with datasets that have more than 3 features...the code is giving me a nan array as the prediction
Thanks for your amazing explaination about all ml algorithms, which helped me a lot!
Glad it is helpful :)
Dude. That switching dark to *BRIGHT* mode 😳
you mean from jupyter notebook to VS Code and vice versa? :D
@@patloeber ya, I sometimes watch you after class at night when the lights are low & I was blown away. lol. I'm over it, but that was a blast! Thank you for the work you put into this :)
Super helpful thanks a lot!
Patrik, I was wondering if scikit learn use gradient descent for their Linear Regression? Any comments on that?
sklearn Linear Regression uses Ordinary Least Squares solver, but they also have a SGDRegressor that uses gradient descent
In linear_regression_test.py what are noise and random_state ?
why 2 is missing in bias formula ???🤔🤔
can you please help me to understand ?
Subbed, thanks for the video! A bit over my head (for now) but i'm getting there.
Thanks, and great to hear you're getting into ML! No worries, I know it can take some time :)
This helped me so much, you explain it so clearly and your code is really neat and easy to understand! Thanks so much! Subbed, liked and will be using your videos as my ML bible from now on XD
Awesome, thank you!
Is this ANN example?
I don't see the use of m1 and m2 at the end when you plot the regression line
Thank you so much for the video..keep up the good work !
thanks :)
Why is there no 2 in the code for df/dw?
N is also a constant while calculating db and dw like 2......So can we omit N while calculating dw?
N is not necessarily a constant. It depends on the data points you have. If you remove N form the equation, you cannot apply the package to different datasets
why dJ/dw = dw in 4:48?
dw should just be short for dJ/dw so I can use this expression in the code
thank god , wonderful explanation
Thanks!
Bro, I really really really need you. Please make more videos on Machine Learning and please make more frequently. You take 1 week to make 1 video which is not fair. Please 2-3 videos in a week. Please it's my humble request.
I am hearing your request, and I try to increase my frequency at some point in the future! But I still have a normal day job and it takes time and effort for machine learning videos :)
@@patloeber Thank you so much and please continue to use OPPS concept to build model I want to learn so many things from you.
really good. I find the really knowledgeable material
Thanks!
Thank you very much for such wonderful explanations....!! I hope you make more such videos...
I will :)
Thanks for your amazing explaination
Glad you like it!
where can I find jupyter notebook explained
ML Notebooks are available on Patreon
Why didn't you used np.sum in dw?
because we need the dot product for the vectors (which includes a sum). db (bias) is only a single value so no need for the dot product
@@patloeber thanks! Now it makes sense.
Wonderful video, thank you!
Glad you like it
That was great, thanks!
Glad you liked it!
Very good explanation! When I try the Boston Housing dataset from sklearn on this regression module. It throw me a runtime warning on the self.weight -=self.learning_rate * dw. When I reduce the iter from 1000 to 100, it works but with terrible MSE
What warning do you get ? Maybe you should scale the dataset and clip values that are too large or close to 0
Hmmm...I wonder why mse is so high, the graph looks prefectly fine. Anyway, great tutorials, thank you so much! Definitely waiting for gradient boosting tutorials!
Thank you very much! I think it's so high because the range is not normalized (e.g. y ranges from -160 to 200), So when you sum over all the test points and also square it, it gets high. MSE is always sample dependent.
Awesome vid. I would love to access that exact notebook with the actual math equations but I don't see it in your github. Is it possible to access that?
not yet. I am planning to release them on my website
He sounds like Janice from FRIENDS 😂
Wonderful.
great videos! keep doing it. Just a quick question. With packages like Pytorch and Tensorflow making a lot of these things very simple, what is the thought process for implementing in Python. Is it for better understanding or do you think it's better way to learn?
Implementing from scratch is a great way to really understand the concepts behind those algorithms!
Thanks for the vid. Great work. One question; however, what if you have more than two weights/model parameters, w, to optimize? Couldn’t you just use the normal equation, ((X.T.dot(X))**-1)*(X.T.dot(y)), to solve for w…or np.linalg.solve(X.T.dot(X), X.T.dot(y))?
Thanks. This code worked for most of the datasets but returns nan for boston dataset and returned different weights for diabetes datasets. But using Closed Form Normal equation instead of gradient descent gives correct weights and bias. why so.....??
My code is not optimized at all and might not always produce good results. Also try applying feature normalization before passing it to the algorithm. This usually improves performance here a lot
@@patloeber thanks for reply
Yes...I tried boston dataset..It didnt work!!Did you find any method for that to work?
@@vikramm4967 use the closed form solution, which have no parameters to tweak and hence always converges and gives right solution
@@sanjibyadav7973 But that has more time complexity right?
Actually gradient is the steepest ascent, so to descent to minima we go in opposite direction of the gradient.
Hi, I cant find the I python notebook in the github link .
It' not yet on Github, but maybe I add it in the future
@@patloeber Can you please make a video on Gradient boosting as well ? from scratch ?
Thanks :)
danke schön!
Why is there no 2 in the code for df/dw? Also why use Transpose for one and dot when calculating prediction,and what about this www.statisticshowto.com/probability-and-statistics/regression-analysis/find-a-linear-regression-equation/ , is this a different way to solve linear regression?
1. Sometimes people multiple MSE * 1/2 so it cancels out when you take the derivative (datascience.stackexchange.com/questions/52157/why-do-we-have-to-divide-by-2-in-the-ml-squared-error-cost-function)
2. For this simple case you can use LSE as a different approach to optimization methods like Gradient Descent (like shown in video) towardsdatascience.com/https-medium-com-chayankathuria-optimization-ordinary-least-squares-gradient-descent-from-scratch-8b48151ba756
This is gold
thanks!
Hi. Is it possible to share the jupyter notebooks?
Hi. I'm planning to release them on my website when I find the time :)
which editor is best for python scripts?
VS Code or PyCharm
i think that you forgot to add termination criterion to your for loop
that you must add a break condition if |new_mse - old_mse| < epsilon
This worked really well on this dataset, but it gives error on pandas dataframe even after i have converted them to numpy arrays.
Which error ? Common errors with other datasets are incorrect types or incorrect shape of your data
@@patloeber After a few iterations the value of weight becomes an array of nan. And so does bias. Same thing works fine with sklearn's linear regression. Tried fixing it but couldn't. So I had to ask you at last. Thanks for responding.
Faraz Khan does your pandas dataframe contain nans?
Sorry I missed your response earlier, but yes as Damian said it is very likely that your dataset has invalid or incorrect data. My algorithm is not optimized and does not include error checks like in sklearn. You may also try applying a standard scaler before fitting the data.
@@patloeber Hey I have made some changes to both linear regression and knn to make it compatible with both dataframes and sklearn's data. Can I make a pull request ?
ModuleNotFoundError: no module named linear_regression. It always shows like this.
your file must be named linear_regression.py and it must be in the same folder
@@patloeber It is the same. Everything is good. I don't understand what's happening sir.
@@dhananjaykansal8097 Try to write "from .logistic_regression import ...", so prefix it with a dot "."
You can also copy my repo from github and try to run this with the exact same folder structure. Or maybe this is helpful: stackoverflow.com/questions/4142151/how-to-import-the-class-within-the-same-directory-or-sub-directory
@@patloeber Alright sir. I shall try that tomorrow for sure. I'm so glad that you're replying
Very Helpful. Keep it up
thank you!
CANT WE EQUATE THE DERIVATIVE DIRECTLY TO ZERO AND CALACULATE MINIMUM. CAN ANYBODY ANSWER THIS
See my comment in logistic regression video.
regressor = LinearRegression()
regressor.fit(X_train, y_train)
predicted = regressor.predict(X_test)
def mse(y_true, predicted):
return np.mean((y_true - predicted)**2)
mse_value=mse(y_test, predicted)
def mse(y_true, predicted):
---> 11 return np.mean((y_true - predicted)**2)
12 mse_value=mse(y_test, predicted)
13 print(mse_value)
ValueError: operands could not be broadcast together with shapes (20,) (20,80)
i dont know why i can run this code, i follow excactly what him did. Can anybody help me. tks
The second derivate (df/db) is wrong because d/db(sum_i=1toN(b))=N
man great but plzz make it in one tab its vry confusing to see which function you have called
thanks for the feedback!
i think you have applied stochastic gradient
what if we have more features
Hi, the code should work for more features, too :) I just used one dimension so that we can have a simple plot of the results
J'(w,b) instead ofJ'(m,b)
Correct, thanks for the hint!
very nice tutorial here. Please can you post the code also?
I already did. You can find the link in the video description :)
Wow
Linear algebraic method:
///////////////////////////////////////////////////////////////////////////
# This is linear least squares
b = np.linalg.inv( np.dot( X.T, X ) ).dot( X.T ).dot( y )
# predict using the coefficients
y_predicted = X.dot( b )
///////////////////////////////////////////////////////////////////////////
Then you have your solution. I was trying to get the videos implementation to have a better overall mean square error than the linear algebra formulation but could not do it. Anyway, this is just another way if anyone is looking. Great video series btw! I have enjoyed going through it.