Gaussian Processes
ฝัง
- เผยแพร่เมื่อ 16 พ.ค. 2024
- The machine learning consultancy: truetheta.io
Want to work together? See here: truetheta.io/about/#want-to-w...
For Machine Learning, Gaussian Processes enable flexible models with the richest output you could ask for - an entire predictive distribution (rather than a single number). In this video, I break down what they are, how they work and how to model with them. My intention is this will help you join the large group of people successfully applying GPs to real world problems.
SOCIAL MEDIA
LinkedIn : / dj-rich-90b91753
Twitter : / duanejrich
Enjoy learning this way? Want me to make more videos? Consider supporting me on Patreon: / mutualinformation
SOURCES
Chapter 17 from [2] is the most significance reference for this video. That's where I discovered the Bayesian Linear Regression to GP generalization, the list of valid ways to adjust a kernel and the Empirical Bayes approach to hyperparameter optimization. Also, it's where I get most of the notation. (In fact, for all my videos, Kevin Murphy's notation is what I follow most closely.)
[1] is a very thorough practical and theoretical analysis of GPs. When I first modeled with GPs, this book was a frequent reference. It offers a lot of practical advice for designing kernels, hyperparameter optimization and interpreting results.
[5] offers a useful tutorial on how to design kernels. I attribute this source for my intuitive understanding of how to combine kernels.
Neil's talks ([4]) on GPs were also influential. They've helped me develop much of my intuition on how GPs work.
[3] is an beautiful tutorial on GPs. I'd recommend it to anyone learning about GPs for the first time.
---------------------------
[1] C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine Learning. MIT Press, 2006.
[2] K. P. Murphy. Probabilistic Machine Learning (Second Edition), MIT Press, 2021
[3] J. Görtler, et al., "A Visual Exploration of Gaussian Processes", Distill, 2019. distill.pub/2019/visual-explo...
[4] N. Lawrence, Gaussian Processes talks on MLSS Africa, • Neil Lawrence - Gaussi... , • Neil Lawrence Gaussian...
[5] D. K. Duvenaud, The Kernel Cookbook: Advice on Covariance Functions, University of Cambridge, www.cs.toronto.edu/~duvenaud/...
[6] K. Weinberger, "Gaussian Processes", Cornell University, • Machine Learning Lectu... and • Machine Learning Lectu...
RESOURCES
GPyTorch provides an extensive suite of PyTorch based tools for GP modeling. They have efficient handling of tensors, fast variance calculations, multi-task learning tools, integrations with Pyro, and Deep Kernel Learning, among other things. Exploring this as a toolset is a great way to become a competent GP modeler. Link : gpytorch.ai/
Also, I'd recommend source [5] for getting familiar with how to model with GPs. Understanding the kernel space to function space relationship takes time, but it takes less with this guide. Also, it links to Duvenaud's PhD Thesis, which is a very deep dive on the subject (though don't ask me about it - I didn't read it!).
EXTRA
Why is it OK to act as though a sample from a multiplied kernel comes from multiplying the function samples from the two component kernels?
The problem comes from the fact that if x1 is a sample from a Multivariate Normal with mean zero and covariance matrix S1 and the same is true for x2 and S2, then the element-wise product x1*x2 is not distributed as a multivariate Normal. However, whatever distribution x1*x2 has, it still has a covariance of S1*S2 (I've verified this experimentally). That means it wiggles similarly to a sample from the product kernel.
The background here is, I accidentally thought it was true for quite a while and it was helpful for modeling. I certainly could never tell it wasn't true. When creating this video, I discovered it wasn't in fact true, but merely a useful approximation.
Wallpaper: github.com/Duane321/mutual_in...
Timestamps
0:00 Pros of GPs
1:06 Bayesian Linear Regression to GPs
3:52 Controlling the GP
7:31 Modeling by Combining Kernels
8:52 Modeling Example
11:55 The Math behind GPs
18:42 Hyperparameter Selection
21:58 Cons of GPs
22:58 Resourcing for Learning More
![ตรงที่เดิม - ฝน พรสุดา [ต้นฉบับ :เบส ขวางหวัน]](http://i.ytimg.com/vi/c5YnbGEztI8/mqdefault.jpg)
Literally the best explanation on the internet for GP Regression Models. He's not trying to be cool, but genuinely trying to explain the concepts
Thank you my man. And yes, my risk of being cool is zero lol
Agreed, I've been trying to understand GPs for a task at work and this is the easiest to understand explanation I've found, liked and subbed!
Absolutely excellent but I like to suggest a minor correction at 1:54. The linear regression DOES account for the uncertainly of the line; the linear regression prediction interval would produce intervals that widen as you move away from the data just like the Bayesian ones. (Many textbooks give the approximate formula for prediction intervals that don't widen but the actual non-approximate formula will give widening bands.)
My postgrad supervisor literally told me to watch this a few times just so I can explain it clearly to Human Sciences people in my research proposal. Thanks for all your effort making it!
Great reference video, I'm sure I will come back to it again and again. The level of detail in all the simulations you do is just incredible. Do you make all your animations in manim?
Thanks brother! And i don’t use manim actually. I like representing data with Altair, which is like a better version of matplotlib. So I have a small library which turns Altair pics into vids.
Excellent video on this topic. Brief and elegant explanations!
Great video. I've seen GPs mentioned a few times in papers and always glossed over it. Thanks for the great explanation!
Props for explaining such a complex model in a friendly way
I learn each time I rewatched the video. So much better than sitting lectures where you only listen once.
The way you motivate the problem really adds insights for understanding.
Absolutely love your exposition. So good!
Best video for GP I have seen! Thank you so much!
The visualizations are the catch. Just excellent 😊
The production value of this is insane
Another great video! Love seeing each one come up
I know how hard it is to explain this topic, so simply and comprehensively. I am extremely thankful for your efforts.
Great balance between technical depth and intuition for me right now. I love how you say that multiplication "is like", but still is not. This gives intuition, but provides a warning for the day when we have come further in our understanding. 🤗
Ha yea glad these details aren’t unnoticed. It’s a careful game making sure I never say anything *technically* wrong.
Yes sir sometimes to make a point you need to recontextualize the matter to specific to make things easier to understand@@Mutual_Information
Excellent explanations and visualizations. Helped me a lot, thank you!
What a smooth way to explain such complex math , thank you
Love the level you've pitched this video at.
A great video! Thank's. I used GP at work many years ago and enjoyed the framework a lot.
Awesome Explanation. Thank you.
Thank you for creating and sharing this great material. All of your videos I watched so far are incredibly informative and well edited.
Glad you're getting something out of it!
I just discovered your videos yesterday and now they're popping up on my YT home screen and I feel a bit like a little boy in a toyshop. How have these high quality fantastic tutorials evaded me for so long, when I spend so much time looking at technical content on TH-cam? Seriously impressive! I'll definitely be one to share your videos when the opportunity arises.
Much appreciated! I got some really cool stuff coming in May. If you like this stuff, you'll *love* what's coming. Thanks again!
It looks like you have optimized the hyperparameters of making an awesome video. So concise, but still a sprinkle of humor here and there. Awesome visualizations, so appreciated.
haha I thought that was gonna be a nitpick, pleasantly surprised - thank you!
By far this is the best video I have seen on this subject! Thank you very much!
I did understand just a few things, but still I watched this video till the very end - the production value is insane! And maybe I’ll need GPs in the future? :D
You definitely deserve much more subscribers, your videos are great!
Phenomenal video. I genuinely can't thank you enough for how accessible this was. I'm sure I'll come back and reference it, or your other work, as I continue preparing for my upcoming internship working on physics-informed neural networks.
That's awesome! Glad it helped
the first time I watched this a month or so a go I had no idea what was happening. However I have recently needed to use a GP and after a lot of reading up on them and coming back to this video, I can appreciate it a lot more with some understanding:)
Yea my topics require some prerequisite 😅 but with a little getting used to on the notation and basics of probability/stats, I think it should be fairly digestible. Glad you got something out of it.
Truly amazing how you turned such a complex topic into an accessible explanation, thanks a lot!
Glad you liked it :)
Straight forward and explained well thank you
Such a clear and intuitive explanation of GPs! Great work!
Thanks Minh :)
Man, you have some beautiful explanations, and the way you explain the details is somehow very simple to understand, thank you so much!
Glad you liked it!
This video is really nice. Thank you so much for creating this content material.
a really really hard-core video... thanks D.J
What the hell that's a great channel I'm so glad I found you. Production quality is spot on, thank you for taking such care !
Happy to have you! Welcome!
Such a clear and intuitive explanation of GPs! Great work!. Excellent video on this topic. Brief and elegant explanations!.
Thank you - that’s what I’m going for!
Great video! Just dived into GPs by learning about their application in system identification techniques. In fact I'm learning for my examn right now and looked for a video that nicely sums up this topic and gives some intuition. This video matches my needs 100%, thank you very much.
Excellent - you're the ideal viewer!
Absolutely perfect! I heard of GPs and was wondering what they were exactly, wanted a bit of intuition of how and why they work, how to use them, just as a quick intro or motivation before learning them later on.
This video answered all of this in a duration that is absolutely perfect: not too long so that it can be watched "leisurely", and not too short so that you still give enough information that I don't have the impression that I learnt nothing.
Didn't know your channel, will definitely check the rest out!
Thanks a lot! That's exactly what I'm going for. Relatively short and dense with useful info. Glad it worked for you.
Thank you! I've been researching paper dedicated to the gaussian approach to time series prediction(as a task in a lab), and I really struggled with it. But after your video, everything has been sorted out in my head, and i finally have understood it!
Exactly what I've hoped to do - happy it helped!
Another great video. Keep up the good work!
Brilliant video! loved the graphics.
As I wrote you on LinkedIn, this is probably the best video on GPs out there! I know it takes a long time to put together something of such high quality, but I hope I will see more of your videos in the future! 😊
thanks, means a lot - and it's coming. This one has been taking awhile, but it'll be out soon :)
The best explanation of Kernel so far!
Agreed! 😎
I read the distill article and came back to watch the whole video again for the second time. Now it's crystal clear! Thanks so much!!
Distill is an epic educational source :)
@@Mutual_Information It's sad that they're in hiatus since last year :(
Hopefully they'll come back some day
holy sh*t, you have unlocked the secret of GP and Bayesian stuff... I have struggled so hard to understand what is even GP as it is so abstract. Thank you so much for your great work!
Happy to help my friend ;)
A truly fantastic explanation to them! The visuals were instructive and well presented, thank you for making this!
and thank you for watching ;)
This is truly a great explanation that helps me to connect all the dots together!! Thanks a lot!!!!
I'm glad it help. When I was studying GPs, these are the ideas that floated in my head - happy to share htem.
Quite literally
Badum tssch
Amazing video on GP's .
This is brilliant. Thank you.
I will have to watch your video several times to understand (if I can) everything but undoubtly your video is professional and very very well done !! congratulations
Thanks Xavier!
Super quality content! Thank you so much: I subscribed and I hope your number of subscribers increases more and more to motivate you to keep going!
Thank you - I hope so too!
Perfect! - research, delivery, production, duration, pictorial intuitiveness, mathematical rigor, naïve friendly 👏🏽
Thank you Sourav - I'm trying!
Super fricking impresed! Bravo
Really good explanation, the animations help so much. Thank you, I really appreciate it.
You're very welcome - Glad to hear it's landing as intended!
Keep this up! It really helps
This is amazing, thanks
Dude has named his channel mutual information so when we look for the concept of mutual information, all his videos will pop up 🤣 genius!
In min 3:00 I saw a smile coming out of my mouth, just how happy I was when I was listening to you!
This is a masterpiece work! Really thank you :)
Thank you very much - glad it's getting some love :)
Wooow! Excellent quality video!
You are the best man! Thank you for your videos, you 're helping a lot of students, because your explanations are so clear and intuitive.
I Hope the best for you.
Thank you - you improved my Friday
Thanks a lot for this vid man it literally saved my life, you're really one hell of a teacher
Thank you - I'm getting a little better over time, but it's a work in progress.
If you love what I'm doing, one thing that would be *huge* for me, is if you tell anyone you think might be interested. This channel is pretty small and it'll be easier to work on it if it gets a little more attention : )
@Mutual Information best of luck man 🫡
What a great video! Very helpful, thanks!
You're very welcome!
What an Explanation! Become fan in seconds.
Now you make me love math again. Thanks.
Nice video - love it
What an excellent video! Just thinking about the amount of effort that must have gone into this gives me anxiety
Nice visualizations man. Just discovered your channel.
Great video ! ❤
Excellent
A really good explanation! Though I wasn't able to understand everything, I would keep coming to this video until I do. ;D
Thank you. Happy to answer any questions too
sooo helpful
Great video
Thank you.
Thank you back.
Love these vids. Can you do a video about normalizing flows in the future?
I plan on making one. It’s a very interesting idea. In the meantime, there is already an excellent explanation : th-cam.com/video/i7LjDvsLWCg/w-d-xo.html
Great video - I subbed!
you might want to look into probabilistic numeric, cheers great video you made there!
Thanks a lot Amazing video
You are very welcome
I would be really helped by putting variable definitions on screen while they're in use! I find myself forgetting what f and f* are for example as I mull it over and watch the explanation. Amazing video! I'm a fan.
Thanks Graham! It's always a balance thinking about what does/doesn't go on screen. More recently, I'm biasing towards *less* on screen, b/c I've gotten feedback that what's on screen can be overwhelming.
But, if you have some question about what may be confusing, ask here and I may be able to help
@@Mutual_Information I'm thinking what was hard for me is that everything was defined and then they were used? the viewer needs to remember what each things means before they can give it the context, and context allows us to combine things and save on short term memory?
great!!!
Really well made explanation :)
Thanks, glad the effort is appreciated!
I built a model years ago that I never realized is perhaps a GP model. I only learned about GP models a weeks ago. It doesn't use any real-valued data; only binary vectors. The similarity kernel is Hamming distance. Other than that, it's basically what he described here.
Your observation of the product of two normally distributed variables is true for the following reason: given independent scalar random variables X,Y, we have Var(XY) = Var(X)Var(Y) + Var(X) (E(Y))^2 + Var(Y) (E(X))^2. Given two multivariate random normals U,V with mean zero, we may choose to work in a basis (possibly different for the two distributions) where the covariance matrices are diagonal. The all components of each vector are independent and so Cov(U) Cov(V) = Cov(UV) by working element-wise. Since this is true in one basis, it must therefore be true in every basis.
So far beyond my abilities. Like Frankenstein's monster, I am soothed by its music.
Missed a lot of math, will get back later!
Damn. Just Damn. This is great! Like: really really really great.
Thank you Ian - more good stuff cooking!
Hi, great video for GP! I have quite new to this topic, is it possible to use GP to model multiple output? e.g. my "input" data is time, and the output data is 2D coordinates (x(t), y(t)). If it is possible, how to setup the covariance function? Thanks so much
Thank you. Do you care about uncertainty in your output?
@@Mutual_Information Hi, thanks for the reply 😃 Yes I need to obtain the uncertainty as well. I think if x(t) and y(t) are "independent", then fitting independent GPR with uncertainty bounds are applicable. However, if they are dependent (like coordinates of an ant moving along a circle), then x and y are correlated. What do you think?
This is cool stuff! There is something I want to understand from the similarity heat map of the linear kernel. If the function samples are dissimilar as they get further apart (according to the lines), should the heat map not be brighter at (0,0) and fade as it approach (10,10)? I am trying to get the picture in my head.
Thanks! I think you're thinking about it from a difficult angle. It's not function *samples* that are similar/dissimilar, it's specific *inputs* across samples. So, for the linear kernel, for inputs that are very similar (like input=0, so heatmap is high, which means similar), the outputs are virtually the same spot. For inputs closer to 10, the inputs are dissimilar and outputs are far apart. Since all function samples are lines, this will manifest as two lines which intersection at input=0 but are far apart at 10. Make sense?
@@Mutual_Information I do not totally understand. It is not your explanation that is bad. I simply need to get to know GP better. Thank you for trying to explain!
Hey, that's a good video, I enjoyed it a lot and you earned a sub :) However, wouldn't weget something very similar to GPs when we allowed for different basis functions in the Bayesian regression example? These functions don't have to be straight lines, so if I choose some polynomials, sines or exponents as my basis functions and I perform Bayesian linear regression, wouldn't I get basically the same things that GPs offer?
Yes you would! Bayesian linear regression with basis functions gives you a GP. But a GP is more general. You can input any similarity via the kernel. Given a kernel, it can be hard to determine the basis functions you’d need to use to recreate it using Bayesian linear regression.
thank you for your understandable video!
I'm still wander what is the point of "similar y for similar x", is it make sure the function is smooth, or other usage?
looking forward to your reply!
The goal of the problem is predict y for a given incoming x.. and we can learn to do this by observing many pairs of (x_i, y_i)'s. So we make an assumption: "If x1 is similar to x2 (that is K(x1, x2) is large/positive)), then we expect y1 and y2 to be close". With that assumption, we can form a prediction for y when given an x.. and that basically is formed by determining "which y value would best work with our similar-y's for similar x's given the x's and y's observed?" and then you can form your prediction that way. The GP does all this hard work for you and allows for noise and whatnot.
Not sure if others are also interested, but I think a coding example with GPyTorch could be interesting.
This is something I'm working on! I'd like to make code samples available alongside my videos. They aren't currently available b/c the modeling code is intertwined with the animation code, so it would make for a terribly difficult to decipher code if released as-is. My plan is.. once my video production workflow is a little more streamlined, I'll pair these video with code snippets.
The best video about GP I have ever seen! Thank you for sharing. I would like to reproduce the graphs that you created in a script, but unfortunately I cannot see any code about it on you github page! It is possible to access to those scripts? with the examples that you produced?
Thank Matteo - I appreciate it!
Unfortunately, the code for this one was heavily intertwined with the animation code, so I didn't make it public. But I wasn't doing anything you can't learn from reading the GPyTorch docs
If you were using a chi-distribution as a kernel could you combine kernel-a and kernel-b multiplicatively? If I recall, gaussian distributions are linear, ie: their sum is a gaussian, however there product is not. Chi-distributed variables on the other hand, when you multiply their products, you get an f-distribution, which is tractable.
Really cool video! You definitely have an INSANE amount of material to make more videos on! Definitely subscribing!
Very interesting idea.. maybe there is a very special choice of kernel such that the multiplication kernel-and-sampled-function-distributions holds exactly, just like a sum does. I really don't know!
If I had to guess, I'd say there is no such kernel. The problem arises from the multivariate normal, which is always operating in a GP, regardless of the kernel. And that problem is.. if you sample two vectors from a multivariate normal.. and multiply them together element wise.. the distribution of that thing is NOT some other multivariate normal. The kernel can only change the covariance of those two vectors, but that problem doesn't depend on those covariances.
And thanks for the subscription!
@@Mutual_Information Ah that makes sense; it's called a *gaussian process* not a *insert-random-pdf* process after all !
Thanks for the reply!
Hey, great video! In practice in my machine learning class we did both GPR and GPC I found it very difficult to scale it to more than 10k samples. It seems that despite the advantages it has, it is not useful for a lot of practical problems. Can you maybe show show video on how to invert a matrix with less than O(n^3) complexity and which software someone could use for GPR/GPC for larger data?
Yea, so that's a big component of GP research. Getting the cost down. A dominate approach are inducing point methods, where you try to summarize a large dataset with a smaller data set of "inducing points". It's a popular approach, but introduces another source of uncertainty.
In my experience, I tend to use GPs with smaller datasets.
Hey, love the videos. What software do you use to create your visuals?
I use a very dope, though static, python plotting library called Altair. And then I have a personal library that turns many of them into videos.
Hi,
I am trying to implement a GPR myself and am struggling with writing code for the parameter updating process. I am using gradient descent to maximize the log density function, w.r.t. the terms: sigma_n, sigma_f, etc.
My output predictions fit decently to the data, but only works for univariate input. Also, my noise prediction is just constant or zero throughout, despite the evident noise present. Please advise me on how to approach this.
Might it be possible to host a virtual meeting with someone who can help?
Entertaining and informative video, thank you! Out of curiousity, have you looked into neural network gaussian procceses at all? I feel like you'd dig em
I haven't looked at them much. All my experience with GPs has been via this handcrafted kernel approach, but it would be nice to get NNs in the mix. GPyTorch makes that pretty easy in fact.. I probably should..
@@Mutual_Information is that what you use (GPyTorch) when modeling gaussian processes? ( instead of GPFlow, PyMC3. Etc.)
@@MeshRoun yep it’s been my go-to so far
I am already GP expert 😃
So 23'47" is an upper bound on the time needed to explain your monitor backgrouds! :-)
But what is the underlying data and GP's hyperparameters?
And also, where does the "process" in "GP" come from?
Ah! you speak to a trade-off I was concerned about. In the original script, I actually listed the hyperparameter values of the fitted models.. but it made the explanation wordy and put a good deal extra notation on screen. Removing it was a push towards simplicity.. but yea, the cost is, I create room for questions.
Also, regarding the datasets. For the first dataset presented, that's just data generated according to the linear regression assumptions, where the coefficient is small (so it's a week signal). For all other datasets, I used a GP itself to generate the data.
And "Process" comes from the fact that the technical definition of a Gaussian Process is it's a type of stochastic process. Specifically, a GP is an infinite collection of random variables where any finite subset has a multivariate normal. "Process" refers to the "infinite collection of RVs". It's a pretty theoretical idea and I didn't think it was *crucial* to understanding the ML models using GPs, so I avoided it. Just thought that discussing an infinite collection of RVs could be avoided without sacrificing much.
@@Mutual_Information Thanks a lot for these further details! ;-)
Sir i am using matlab regreession learner toolbox i understood that kernel functions tries to find how similar or far apart 2 input data points are,can you tell what are basis functions then there are 3 basis functions zero,constant and linear?
Awesome video!. Only one question. Minute 09:20. A linear kernel does not imply that the realizations of the random process must be linear, does it?. Thanks!!
Thanks Jesus. And regarding your Q, in the broader model, no a linear kernel doesn't imply the realizations need to be linear, since there is a noise component in the overall kernel. That allows points along a sample to be different in a nonlinear way.
Thanks for nice video.
Can you please explain why GPs are called non-parametric when they have mean and covariance matrix and they rely on that for doing any predictions?
glad you liked it. To answer your question, non-parametric doesn't "no parameters". It means the parameters grow with the size of the data, and that's what happening here. A GP on 100 data points has a lot more parameters than one on 1000 data points.