The Kernel Trick - THE MATH YOU SHOULD KNOW!
ฝัง
- เผยแพร่เมื่อ 2 มิ.ย. 2024
- Some parametric methods, like polynomial regression and Support Vector Machines stand out as being very versatile. This is due to a concept called "Kernelization".
In this video, we are going to kernelize linear regression. And show how they can be incorporated in other Algorithms to solve complex problems.
If you like this video, hit that like button. If you’re new here, hit that SUBSCRIBE button and ring that bell for notifications!
FOLLOW ME
Quora: www.quora.com/profile/Ajay-Ha...
REFERENCES
[1] The Kernel Trick: people.eecs.berkeley.edu/~jor...
[2] Positive Definite Kernels: en.wikipedia.org/wiki/Positiv...
Going to make a video on SVM and how it uses this kernel trick. So if you want to understand the math behind one of the most common Machine learning Algorithms, *subscribe* to keep an eye out for it ;)
Few of my friends wanted to get into ML at some point, but when they realized ML is just maths they went back to web dev. I love math and your videos make me love ML even more. Keep up!
very good explanation. also in depth. Most other videos are just explaining without formula, which are too simple
This video is awesome, thanks for the lecture!
I think at 2:50 the suffix to y and x should be i.
Thank you for the great explanation.
Amazing! Elegant explanation.
Thank you a lot! It really soole my problems about kernal tricks as I felt extremely puzzled in our professor's lecture.
Thank you so much for this excellent Video
Very nice presentation...like it.
Thanks a lot, that cleared many doubts !!
At about 2:56 you need to explain to customers the logic here: w* is not the solution because alpha depends on w. Also, at about 4:16 you cancel K, although K is singular. At lest you need some discussions of the range and null space of the kernel.
I was wondering about how w* was the solution yet contained w in it. What is it supposed to be?
Damn Dude!!, you really deserve a lot more views and subscriptions.Keep doing the great work.
Thanks for the kind words homie!
@@CodeEmporium But polynomial regression is an example of a nonlinear function generally right? Unless itnjust has several linear variables..
Great explanation.
Great Video
Just to made myself clear with your development. In minute 4:13, is the symmetry of the Kernel Matrix (K = t(K)) that allows you to have [t(y)* K * alpha] instead of [t(alpha) * K * y]? where "t" is transpose.
At 3:37, isn't the Kernel matrix K of size (m,m) rather than (m,n) since we do a dot product of every observation (from X1 to Xm) with every other one, so doing a square matrix (as you mentioned yourself just before), this gives a matrix K of size m * m, and not m * n unless I missed stgh.
So the last element of the first row for instance should be Phi(X1)t * Phi(Xm), and not Phi(X1)t * Phi(Xn).
Correct ?
i was wondering about this too. It should be m x m or at least he shouldve stated m = n. Annotations is really annoying when it is not explained
Can you make a video like this for simple linear regression without regularization?
I have a few related questions. First, the regression equations are overdetermined, i.e. the number N of data points is greater than the number M of basis functions, right? And this is why we need regularization (ridge regression, lambda>0), right? If N>M and K is NXN, it has rank (at most) M so K^{-1} does not exist, but (K+lambda I)^{-1} does. That is OK, and I suppose you can let lambda go to zero to minimize the amount of regularization. But then, if you use a Gaussian radial basis function, this is infinite dimensional (M goes to infinity), and the regularization is no longer needed. Does all this seem correct?
If you go to the section on prediction, you'll see that the size of M (even if M = infinity) is irrelevant, and what's required is the inversion of K + lambda I, which is "just" inverting an
N x N matrix. I don't think M has anything to do with the degree of overfitting. So yes even for a Gaussian kernel (when M = infinity) you still want to regularize.
Man, you are the best
Thank you for your videos! I love the fact that you show equations.
Glad you like my style
Thank you :)
Thanks for uploading this video, having kernel trick to get around phi phi transpose is a good solution, however, at the end we have the inverse of K+lambda I which is a big matrix. Do you have any solution for that?
To my knowledge the two practical methods that exist to avoid dealing with the often huge matrix K are "Random Features" and "Nystrom Method". But in general the huge matrix K and related issues (like inversion) are really why deep learning is often used when lots of data is available.
Dude thanks a lot ❤
omg I watched plethora of videos and read so many articles trying to explain me what kernel method actually gains, and I finally understand not only conceptually but also down to the technical level
Haha I made this video so long ago I thought I explained it in a real complex way. That said, super glad this was helpful
can w use the same techniques with svm , as in svm we have to use the lagrangian to get to the dual form
Thank you! Great explanation.
Thanks for the compliments
Thanks a lot!
How to get the vector 'k' in the last slide?
It would be a great help to understand this better if can you please make a video that takes a dataset as a m samples and n features and how would we calculate this K matrix and use this for transforming the features.
I think there might be a mistake in your equation for J(alhpa) at 3:21, should be J(w*) instead.
incredible explaination, thank you very much.
Thanks so much for watching and commenting! Glad it is useful
Man you are a champion. Thank you very much.
I'd love to be one someday. Thanks a ton :)
sorry I don’t understand the last part. In prediction, how to calculate k(x)? Say we have all training points so that we have the K, but for testing point x what does it mean? Thanks for help bro
Where does the $1/\lambda$ come from in the derivative at 2:47?
Damn! Damn! Damn! Couldn't have been better. Thanks a lot! A lot! As simple as it can get!
Thanks a lot!
Please make more videos on GAN
why do we get the 1/lambda term after derivative in w* ?? @2:52
Dear author, how can we obtain the w* at 2:58? Do we obtain it by setting the derivatives of J(w) to zero? Could you please explain it? Thanks.
Yes. The idea is to find the weight vector that minimize the cost.
@@CodeEmporium Thank you very much for your reply.
@@CodeEmporium Can you so the same but now without regularization?
It is indeed an awesome video but viewers should have some background knowledge so that is easy to follow. What is nice is that ties in so many concepts in a single 7 min video. A good warm up video for those who have to go out and develop some code. I have resisted watching Code Emporium for a long time , now I'm a subscriber.
You're right. I made this video while in grad school. So it was meant to serve as a refresher to me before exams :) That's why it's a lil hard to follow. Maybe if i had my audience more in mind at the time, this video may have been more accessible
@@CodeEmporium ضص
There are grand errors in the equations that you solved. But, yes the explanation is on point.
I understand data scientists might want to shy away from Hilbert spaces, but this stuff is so much clearer if you just use the Finite Representer Theorem to reformulate Ridge regression as a simple regression problem involving the kernel matrix K. :) Just my opinion.
I can't really get the difference between polynomial regression and nonlinear regression. Please can you help me with that?
You could probably answer that question without even knowledge of linear algebra or regression. After all, what is relationship between a polynomial function and a non-linear function? A non-linear function is simply any function that isn't of the form y = mx + b (or y-hat = b1x1 + b2x2 + ... bnxn for multiple linear regression). A polynomial function is of the form y = [polynomial expression here]. Recall that a polynomial expression is one that only involves terms of the form cx^n + cx^(n-1) + ... cx^0. Examples include a linear function, but also quadratics, cubics, and so forth.
In short, a polynomial function can be linear (as in the case of y=mx+b) or non-linear (as in the case of, say, y = 3x^2 + x + 2.
In linear regression, you are fitting a line to a data. In a non-linear regression, you are fitting a curve (and it would have to be a curve, not a line since it's *non-linear*) to a set of data. BUT that curve doesn't have to be polynomial in nature (though it certainly can be). Whether that curve is defined by a polynomial function (not of order 1) OR something else is up to the circumstance! It might, for example, be an exponential function or a sinusoidal one. Recall that neither of the latter are polynomial functions because they are not of the form y = c1x^n + c2x^(n-1) + ... cnx^0.
Hope that was a satisfying answer!
How did you solve for the optimal w* from J(w)? I'm new to matrix calculation, would be great if you can give an explanation. Thank you.
use Lagrange multiplier
Great Video. Thank you. However, there is a mistake at 6:30, y_pred = w'*phi(x) (w was phi'*(K+lambda)^-1*y ). Hence, w' = y'*(K+lambda)^-1*phi and y_pred = y'(K+lambda)^-1*phi*phi(x). But you wrote phi'*phi(x) and it's a inner product! It is not Kernel. phi is a nxm matrix, m can be infinite (phi' * phi > mxm covariance matrix and phi*phi' > n*n Kernel matrix) We know Kernel matrix. It cannot be infinetexinfinete dimensions. It should be nxn matrix. There is also a problem with the notation you wrote at 3:32. If phi = [phi(x1)' ; phi(x2)'; ... ; phi(xn)' ]nxm then, phi(x)*phi(x)' can be Kernel matrix as phi(x)' = [phi(x1) phi(x2) ... phi(xn)] and phi(x)*phi(x)' = [phi(x1)'phi(x1) phi(x1)'phi(x2) ... phi(x1)'phi(xn); ..... ; phi(xn)'phi(x1) phi(xn)phi(x2) ... phi(xn)phi(xn)]nxn. But you wrote m*n matrix at 3:32. This cannot be possible. if m is not equal to n, then it cannot be symmetric.
It is hard to explain this with plain text, sorry. In short, m is the number of dimensions and n is the number of samples. At 3:32 K is a mxn matrix. Then, K cannot be symmetric if m is not equal to n. K must be symmetric as you said. Hence, phi is nxm matrix and phi^T is a mxn matrix and phi*phi^T is a nxn matrix.
2:53 why after solve for w, you still have w in left and right parts of equation?
can someone explain the where the 1/lambda comes from in the derivative?
I think there might be a mistake in your equation for alpha_n at ~3:08. The summation shouldn't be there
You're right. My bad. I'll pin your comment for now (until I you/someone else points out some more mistakes). Thanks for the heads up!
@@CodeEmporium Under summation everything should be indexed by i not by n, and the summation is ok.
can u please help me solving one problem on this topic
Thats fine you make predictions without phi, but in order to make predictions you need to compute w*. In your kernelized version, you still need phi transpose, in addition to K, in order to estimate w*. Can you explain that better? How to get rid of phi transpose in the kernelized version of w*?
He does this in the last section ("prediction"). You really want the actual prediction function y, and he shows the formula for it just in terms of K and not with phi floating around anywhere.
What kind of math is this vector calculus ?thanks
what does w* represent? thank you
thanks！
Subscribed
Good speed in video. But I should say perhaps it is a bit too under-detailed. Maybe, one should be able to get through if he went through all your videos. I would request for more details in derivations in future videos.
Damn this is so beatiful
Thanks! ;)
But why did you write \sum_{n=1}^{N} ||w||^2 when there's no n term in that part of the ridge regression equation? Very confusing, to say the least....
2:57 you mean \alpha_n = \frac{1}{\lambda} (y_n - w^T\phi(x_n)), without the \sum operator. Had me confused for a while.
at 6.33 how is it possible to shift "y" before the kernel matrix without transposing as compared to at 4.45
@CodeEmporium, its a genuine question. Please do reply.
How is 𝜑^𝑇 𝜑 variance matrix ?
can someone tell me what is gram matrix ?
Did you hire someone else to do the voice over for your video?
maaaaaatrix
Finally, I understood!
Dense but informative.
What is the point of the summations in minute three if you don't even use the index variable? Why y_n and x_n and not y_i and x_i? Also, would have been better if you credited the analytics vidhya article you took this information from (Same with your support vector machine video)
several typos for sure but good video!
Yea. I have tried to get better about this over the years. Thanks for watching!
Maahtrix? It may not all be one world, but it does overlap. Matrix!
Math Trix
@@CodeEmporium I am glad you have a sense of humor.
Aayega
still didnt get it !
ughhhhhhhh
..but do I need to know that math to apply kernalisation? Really do I?
from sklearn.kernel_ridge import KernelRidge
mattrix
कोष थिता nhi aayega bro
hey, make it easy, we came here to understand it easily, not hard, damm it
This is amazing but, there are the errors I'm getting implementing this into python :( Anyone one could help?
i am even more confused now.
user name does not check out.
Wrong equations confuse people, pls correct it
Going to riding fhor fhirsht vi teck no
bro is the mahtrix contrarian
Mathy trix
What a fuck boy way to say maatrix 😂, great video though
perfect definition of How Not To Teach Machine Learning Concepts.
Bouncer.
Galt he
You're pronouncing "matrix" incorrectly.
No, you are, because of the vowel shift in the English language. A German person, for example, would pronounce it as he does.
who gives a f**k. you pronounce chicago as "shikago" and chimes as chimes. there are many other stupidity filled in this language.
2 minutes in and understood absolutely nothing. waste of time
maybe you need some prior course
sooooo, nobody is going to mention the ridiculous pronunciation of the word "matrix"?
Mathtrix
You are just news reading the topic not teaching..-_-
Matrix is pronounced May-Trix.
I'm partly from India andthe states, so my pronunciations and metrics are all over the place. Ill be more consistent
What does it matter you idiot. Why is chicago pronounced as "shikago" and chimes as chimes. English is a faulty language. No wonder why NASA scientists researched that english as a language is not suitable for artificial intelligence-nlp but Sanskrit is.
@@CodeEmporium this really annoys me of these native english speaker picking up on non native english speakers.you dont need to justify how you say a matrix. Can native english speakers or any other native language speakers pronounce indian languages like Sanskrit and/or tamil properly? off course not. dont be apologetic.
how do you justify magic pronounced as "may-gic" and matrix as "matrix"
Oh my God thank you. I always find it hard to pronounced it easily.
Too many gross math mistakes (some of them pointed out in other comments). You need to read more/better material. Please don't post any more misleading and incorrect videos.
Why not pronouncing "/ˈmātriks/"????
Cuz phonics and I never got along. I have wartime flashbacks from the 3rd grade