Linear Regression in Python - Machine Learning From Scratch 02 - Python Tutorial
ฝัง
- เผยแพร่เมื่อ 29 ก.ย. 2024
- Get my Free NumPy Handbook:
www.python-eng...
In this Machine Learning from Scratch Tutorial, we are going to implement the Linear Regression algorithm, using only built-in Python modules and numpy. We will also learn about the concept and the math behind this popular ML algorithm.
~~~~~~~~~~~~~~ GREAT PLUGINS FOR YOUR CODE EDITOR ~~~~~~~~~~~~~~
✅ Write cleaner code with Sourcery: sourcery.ai/?u... *
📓 Notebooks available on Patreon:
/ patrickloeber
⭐ Join Our Discord : / discord
If you enjoyed this video, please subscribe to the channel!
The code can be found here:
github.com/pat...
Further readings:
ml-cheatsheet....
ml-cheatsheet....
You can find me here:
Website: www.python-eng...
Twitter: / patloeber
GitHub: github.com/pat...
#Python #MachineLearning
----------------------------------------------------------------------------------------------------------
* This is a sponsored link. By clicking on it you will not have any additional costs, instead you will support me and my project. Thank you so much for the support! 🙏
In case anyone's wondering why increasing the learning rate decreased the cost, it's because the number of iterations is very low in order for gradient descent to converge. So if lower learning rate gives you higher cost, you should increase the number of iterations.
Very good explanation! When I try the Boston Housing dataset from sklearn on this regression module. It throw me a runtime warning on the self.weight -=self.learning_rate * dw. When I reduce the iter from 1000 to 100, it works but with terrible MSE
What warning do you get ? Maybe you should scale the dataset and clip values that are too large or close to 0
Linear algebraic method:
///////////////////////////////////////////////////////////////////////////
# This is linear least squares
b = np.linalg.inv( np.dot( X.T, X ) ).dot( X.T ).dot( y )
# predict using the coefficients
y_predicted = X.dot( b )
///////////////////////////////////////////////////////////////////////////
Then you have your solution. I was trying to get the videos implementation to have a better overall mean square error than the linear algebra formulation but could not do it. Anyway, this is just another way if anyone is looking. Great video series btw! I have enjoyed going through it.
this is a really nice summary of the fundamentals. perfect for brushing up after learning about ML from 5 years ago. really solid.
He sounds like Janice from FRIENDS 😂
Thank you very much! I am from China, not only can you give us good tutorial but also your English is very cleary so that I can improve my English listening level. Thank you very much! 唯一真神。
Hello, I was wondering why at minute 12:25, when calculating the dw parameters you don't use np.sum if it includes the sum in the formula. Also is there a reason its included with parameter db?
It's bc the dot product already performs the summation so the np.sum command would be redundant. Since db only uses for vector subtraction, it needs the np.sum command
Thank you very much sir❤❤
regressor = LinearRegression()
regressor.fit(X_train, y_train)
predicted = regressor.predict(X_test)
def mse(y_true, predicted):
return np.mean((y_true - predicted)**2)
mse_value=mse(y_test, predicted)
def mse(y_true, predicted):
---> 11 return np.mean((y_true - predicted)**2)
12 mse_value=mse(y_test, predicted)
13 print(mse_value)
ValueError: operands could not be broadcast together with shapes (20,) (20,80)
i dont know why i can run this code, i follow excactly what him did. Can anybody help me. tks
Sir you deserve more than you have right now! you made a 13 year old guy teach those high level concepts. Loads of love from India!
This is nice to hear :) Thanks!
The second derivate (df/db) is wrong because d/db(sum_i=1toN(b))=N
Why didn't you used np.sum in dw?
because we need the dot product for the vectors (which includes a sum). db (bias) is only a single value so no need for the dot product
@@patloeber thanks! Now it makes sense.
thank you so much for thoses great tut, can u please re-explain the line 20 from fit method i cant understand why we transpose the X matrix...
Hi. Let's say we multiply two matrices A and B. Then the number of columns of A must be number of rows of B:
The size formula is: A(lxm) * B(mxn) = C(lxn)
In our example:
np.dot(X, self.weights) --> (80,1) * (1,) = (80,) -->for every sample we get one value (the 1 is because we have 1 feature)
and in the transposed case:
np.dot(X.T, (y_predicted - y)) --> (1, 80) * (80,) = (1,) --> for every feature we get one value
@@patloeber Thank you now that's make sense to me
Why is there no 2 in the code for df/dw? Also why use Transpose for one and dot when calculating prediction,and what about this www.statisticshowto.com/probability-and-statistics/regression-analysis/find-a-linear-regression-equation/ , is this a different way to solve linear regression?
1. Sometimes people multiple MSE * 1/2 so it cancels out when you take the derivative (datascience.stackexchange.com/questions/52157/why-do-we-have-to-divide-by-2-in-the-ml-squared-error-cost-function)
2. For this simple case you can use LSE as a different approach to optimization methods like Gradient Descent (like shown in video) towardsdatascience.com/https-medium-com-chayankathuria-optimization-ordinary-least-squares-gradient-descent-from-scratch-8b48151ba756
Hmmm...I wonder why mse is so high, the graph looks prefectly fine. Anyway, great tutorials, thank you so much! Definitely waiting for gradient boosting tutorials!
Thank you very much! I think it's so high because the range is not normalized (e.g. y ranges from -160 to 200), So when you sum over all the test points and also square it, it gets high. MSE is always sample dependent.
I don't see the use of m1 and m2 at the end when you plot the regression line
i think you have applied stochastic gradient
Hello; while this video is mostly very clear and easy to understand, I do have one question, namely, doesn't line 23 perform the same action for every element in self.weights? if this is the case, and since every element in self.weights starts as 0, then why do you use an array instead of a constant?
hey man, did you figure out the answer?
J'(w,b) instead ofJ'(m,b)
Correct, thanks for the hint!
Nice video dude! Can you tell me the font you use in your Editor please?!
in this video it's the monokai theme, now I use Nightowl theme
I have seen MSE calculated as (1/2n)... Whats the difference between using (1/n) and (1/2n)?
why 2 is missing in bias formula ???🤔🤔
can you please help me to understand ?
i think that you forgot to add termination criterion to your for loop
that you must add a break condition if |new_mse - old_mse| < epsilon
hey I have a question what is the 'predicted' in mse_value = mse(y_test, predicted)???
does this not work with datasets that have more than 3 features...the code is giving me a nan array as the prediction
I have a question, how can we add mse in class method, LinearRegression
Actually gradient is the steepest ascent, so to descent to minima we go in opposite direction of the gradient.
Patrik, I was wondering if scikit learn use gradient descent for their Linear Regression? Any comments on that?
sklearn Linear Regression uses Ordinary Least Squares solver, but they also have a SGDRegressor that uses gradient descent
can you please explain what you mean by features and samples in the linear regression part?
Subbed, thanks for the video! A bit over my head (for now) but i'm getting there.
Thanks, and great to hear you're getting into ML! No worries, I know it can take some time :)
Excellent work!
To evaluate our model we could also use the coefficient of determination (R**2) which is not affected by the scale of our measurements like MSE does.
Is this ANN example?
where can I find jupyter notebook explained
ML Notebooks are available on Patreon
such a nice content , it made me so satisfied. love from China
ModuleNotFoundError: no module named linear_regression. It always shows like this.
your file must be named linear_regression.py and it must be in the same folder
@@patloeber It is the same. Everything is good. I don't understand what's happening sir.
@@dhananjaykansal8097 Try to write "from .logistic_regression import ...", so prefix it with a dot "."
You can also copy my repo from github and try to run this with the exact same folder structure. Or maybe this is helpful: stackoverflow.com/questions/4142151/how-to-import-the-class-within-the-same-directory-or-sub-directory
@@patloeber Alright sir. I shall try that tomorrow for sure. I'm so glad that you're replying
Wonderful.
In linear_regression_test.py what are noise and random_state ?
Not just the content, I love your accent. :D
Why is there no 2 in the code for df/dw?
N is also a constant while calculating db and dw like 2......So can we omit N while calculating dw?
N is not necessarily a constant. It depends on the data points you have. If you remove N form the equation, you cannot apply the package to different datasets
man great but plzz make it in one tab its vry confusing to see which function you have called
thanks for the feedback!
Super helpful thanks a lot!
Awesome vid. I would love to access that exact notebook with the actual math equations but I don't see it in your github. Is it possible to access that?
not yet. I am planning to release them on my website
CANT WE EQUATE THE DERIVATIVE DIRECTLY TO ZERO AND CALACULATE MINIMUM. CAN ANYBODY ANSWER THIS
See my comment in logistic regression video.
great videos! keep doing it. Just a quick question. With packages like Pytorch and Tensorflow making a lot of these things very simple, what is the thought process for implementing in Python. Is it for better understanding or do you think it's better way to learn?
Implementing from scratch is a great way to really understand the concepts behind those algorithms!
Thanks for the vid. Great work. One question; however, what if you have more than two weights/model parameters, w, to optimize? Couldn’t you just use the normal equation, ((X.T.dot(X))**-1)*(X.T.dot(y)), to solve for w…or np.linalg.solve(X.T.dot(X), X.T.dot(y))?
13:15 Why using X.T and not X?
Short and sweet :), extremely helpful.
thank you sir you just saved me from my assignment this semester. the instructions were very clear and visual. please keep producing videos!
Thank you very much for such wonderful explanations....!! I hope you make more such videos...
I will :)
Bro, I really really really need you. Please make more videos on Machine Learning and please make more frequently. You take 1 week to make 1 video which is not fair. Please 2-3 videos in a week. Please it's my humble request.
I am hearing your request, and I try to increase my frequency at some point in the future! But I still have a normal day job and it takes time and effort for machine learning videos :)
@@patloeber Thank you so much and please continue to use OPPS concept to build model I want to learn so many things from you.
Dude. That switching dark to *BRIGHT* mode 😳
you mean from jupyter notebook to VS Code and vice versa? :D
@@patloeber ya, I sometimes watch you after class at night when the lights are low & I was blown away. lol. I'm over it, but that was a blast! Thank you for the work you put into this :)
Beautiful explaination!
Hi. Is it possible to share the jupyter notebooks?
Hi. I'm planning to release them on my website when I find the time :)
thank god , wonderful explanation
Thanks!
This worked really well on this dataset, but it gives error on pandas dataframe even after i have converted them to numpy arrays.
Which error ? Common errors with other datasets are incorrect types or incorrect shape of your data
@@patloeber After a few iterations the value of weight becomes an array of nan. And so does bias. Same thing works fine with sklearn's linear regression. Tried fixing it but couldn't. So I had to ask you at last. Thanks for responding.
Faraz Khan does your pandas dataframe contain nans?
Sorry I missed your response earlier, but yes as Damian said it is very likely that your dataset has invalid or incorrect data. My algorithm is not optimized and does not include error checks like in sklearn. You may also try applying a standard scaler before fitting the data.
@@patloeber Hey I have made some changes to both linear regression and knn to make it compatible with both dataframes and sklearn's data. Can I make a pull request ?
This helped me so much, you explain it so clearly and your code is really neat and easy to understand! Thanks so much! Subbed, liked and will be using your videos as my ML bible from now on XD
Awesome, thank you!
Thanks for your amazing explaination
Glad you like it!
Thank you so much for the video..keep up the good work !
thanks :)
Couldn't have done this without your help! And I needed this, thanks man
glad it was helpful!
which editor is best for python scripts?
VS Code or PyCharm
why dJ/dw = dw in 4:48?
dw should just be short for dJ/dw so I can use this expression in the code
Good work with detailed explanations. Always happy when I am struggling with a concept to see you have a video out even remotely related to what I am working on.
That’s nice to hear :)
very nice tutorial here. Please can you post the code also?
I already did. You can find the link in the video description :)
Hi, I cant find the I python notebook in the github link .
It' not yet on Github, but maybe I add it in the future
@@patloeber Can you please make a video on Gradient boosting as well ? from scratch ?
Thanks :)
Thanks for your amazing explaination about all ml algorithms, which helped me a lot!
Glad it is helpful :)
really good. I find the really knowledgeable material
Thanks!
Wonderful video, thank you!
Glad you like it
Thanks. This code worked for most of the datasets but returns nan for boston dataset and returned different weights for diabetes datasets. But using Closed Form Normal equation instead of gradient descent gives correct weights and bias. why so.....??
My code is not optimized at all and might not always produce good results. Also try applying feature normalization before passing it to the algorithm. This usually improves performance here a lot
@@patloeber thanks for reply
Yes...I tried boston dataset..It didnt work!!Did you find any method for that to work?
@@vikramm4967 use the closed form solution, which have no parameters to tweak and hence always converges and gives right solution
@@sanjibyadav7973 But that has more time complexity right?
danke schön!
Wow
what if we have more features
Hi, the code should work for more features, too :) I just used one dimension so that we can have a simple plot of the results
That was great, thanks!
Glad you liked it!
This is gold
thanks!
Very Helpful. Keep it up
thank you!