I have been playing with NODEs for a few weeks now. The video is really helpful and intuitive. Probably it is the clearest explanation I have heard so far. Thank you, Professor.
Thanks Dr. Brunton for making a video on Neural ODE. Came across this paper as soon as it came out back in 2018. Still goes over my head particularly the introduction of the 2nd differential equation/ adjoint sensitivity method. Would really appreciate if you explain it in detail.
So basically rising awareness that there are better approximations to "residual" integration. Thanks for the reminder. From my course on numerical computation, using better integrators is actually better than making smaller time steps, rising the possible accuracy given some limited amount of bits for your floating point numbers.
So if I understand correctly, ODE networks fit a vector field as a function of x by optimizing the entire trajectory along that field simultaneously, whereas the residual network optimizes one step of the trajectory at a time?
I study neural ode for quite a long time, and found it is good for initial value problem, however, for external input problem, it is really hard to train.
Nice video, but I really miss the connection point between the NNs and the math part. I have a PhD in physics and I've worked a lot with the math you're talking about. Also I've worked a few years as a data scientist and I kinda understand how it goes with the neural networks. But I really miss the point how you make these two work together. Sorry if I sound dumb here.
I would vote for more details on the adjoint part. It is not very clear to me how to use AD for df/dx(t) now that x changes continuously (or do we select a clever integrator during training?) .
@Eigensteve is the nth order runge kutta integrator not just what a UNet is, after its being properly trained. The structure appears the same and the coefficients would be learned.
Why is it implicit that x(k+1)=x(k)+f(x) is Euler integration ? Can be any integrator depending on how you build f(x), Runge Kutta for example f is f(x) =h/6*(k1+2*k2+2*k3+k4).
I couldn't understand what a problem the NODE solves. What is the source data and what is the goal? Perhaps, you are trying to approximate a dynamical system (the rhs function of it) with a NN (i.e. you approximate the rhs as a composition of activation and linear functions), s.t. trajectories of the synthetic system look like the source data. Is this correct? Is it like an alternative to HMM?
Nice presentation Steve! I just gave a very similar presentation on Neural ODE-s just a week prior. Would like to see it one day to be used for audio compression. Keep up the content!
If you only have access to the x-data and numerically differentiate to obtain dxdt to train the Neural ODE. How does this noise propagate in the final solution? Does it acts as regularisation?
I suspect these are not the type of courses with defined prereqs, but def need calculus series, linear algebra series, and some computer science. To really understand it, classical mechanics and signals and systems (control theory, discrete and continuous).
This seems like you are changing your loss function not your network. Like there is some underlying field you are trying to approximate and you're not commenting on the structure of the network for that function. You are only concerning yourself with how you are evaluating that function (integrating) to compare to reality. I think it's more correct to call these ODE Loss Functions, Euler Loss Functions, or Lagrange Loss Functions for neural network evaluation.
Very confusing presentation! First Neural ODEs are presented as a continuous version of ResNets, which would imply that the integration happens in "depth" which would make them similar to fully-connected or convolutional neural networks (non-sequence models). The afterwards it is suggested that the integration actually happens in "time" which makes neural ODEs much more similar to sequence models. Even ChatGPT et al. are confused and can't answer this distinction properly. Seems like it is a quite buzzword-driven field...
![Python Symbolic Regression (PySR) [Physics Informed Machine Learning]](http://i.ytimg.com/vi/df43V4OjMVs/mqdefault.jpg)
![Python Symbolic Regression (PySR) [Physics Informed Machine Learning]](/img/tr.png)
![Residual Networks (ResNet) [Physics Informed Machine Learning]](http://i.ytimg.com/vi/w1UsKanMatM/mqdefault.jpg)
![Lagrangian Neural Network (LNN) [Physics Informed Machine Learning]](/img/n.gif)
I have been playing with NODEs for a few weeks now. The video is really helpful and intuitive. Probably it is the clearest explanation I have heard so far. Thank you, Professor.
Thanks Dr. Brunton for making a video on Neural ODE. Came across this paper as soon as it came out back in 2018. Still goes over my head particularly the introduction of the 2nd differential equation/ adjoint sensitivity method. Would really appreciate if you explain it in detail.
So basically rising awareness that there are better approximations to "residual" integration. Thanks for the reminder.
From my course on numerical computation, using better integrators is actually better than making smaller time steps, rising the possible accuracy given some limited amount of bits for your floating point numbers.
Thank you for taking me back to Engineering Class control Systems
So if I understand correctly, ODE networks fit a vector field as a function of x by optimizing the entire trajectory along that field simultaneously, whereas the residual network optimizes one step of the trajectory at a time?
Love your content ! Went through the entire complex analysis videos, and now gonna go through this one as well !
Cool summary and intro for liquid NNs.
this is great --- i think about this stuff all the time, but didn't know others did :/
Great video, I learned a lot! Piqued my interest and inspired me to do a deep dive into all the topics mentioned
I study neural ode for quite a long time, and found it is good for initial value problem, however, for external input problem, it is really hard to train.
Nice video, but I really miss the connection point between the NNs and the math part. I have a PhD in physics and I've worked a lot with the math you're talking about. Also I've worked a few years as a data scientist and I kinda understand how it goes with the neural networks.
But I really miss the point how you make these two work together. Sorry if I sound dumb here.
Fantastic video! Do you have any references for the mathematics behind the continuous adjoint method?
Awesome video. One question I'm asking myself is: Why isn't everybody using NODEs instead of resnets if they are so much better?
very interesting course, love such great video...
Awesome video and very helpful. Thanks
I would vote for more details on the adjoint part. It is not very clear to me how to use AD for df/dx(t) now that x changes continuously (or do we select a clever integrator during training?) .
Can you please teach latent neural ode in detail?
@Eigensteve is the nth order runge kutta integrator not just what a UNet is, after its being properly trained. The structure appears the same and the coefficients would be learned.
Amazing review. Engaging and sharp
10:10 or just say: NeuralODE models the vector field itself instead of the discretised increment like Residual connection.
Euler integration got dumped on so hard in this video
Interesting...
Very interesting
Very cool!
I love it great video
Why is it implicit that x(k+1)=x(k)+f(x) is Euler integration ? Can be any integrator depending on how you build f(x), Runge Kutta for example f is
f(x) =h/6*(k1+2*k2+2*k3+k4).
I couldn't understand what a problem the NODE solves. What is the source data and what is the goal? Perhaps, you are trying to approximate a dynamical system (the rhs function of it) with a NN (i.e. you approximate the rhs as a composition of activation and linear functions), s.t. trajectories of the synthetic system look like the source data. Is this correct?
Is it like an alternative to HMM?
Nice presentation Steve! I just gave a very similar presentation on Neural ODE-s just a week prior. Would like to see it one day to be used for audio compression. Keep up the content!
What about periodic functions? Is there a way to get nice approximations with neural networks?
If you only have access to the x-data and numerically differentiate to obtain dxdt to train the Neural ODE. How does this noise propagate in the final solution? Does it acts as regularisation?
Is there a model that can do integro-differential equations?
Does some one have a example repository or libary so i can plaz with it
Maybe in the Neural ODE paper?
Hi Professor , What are some prerequisites for this course?
I suspect these are not the type of courses with defined prereqs, but def need calculus series, linear algebra series, and some computer science. To really understand it, classical mechanics and signals and systems (control theory, discrete and continuous).
Damn I'm still going through the ode and dynamical systems course, this new material seems interesting AF though
This seems like you are changing your loss function not your network. Like there is some underlying field you are trying to approximate and you're not commenting on the structure of the network for that function. You are only concerning yourself with how you are evaluating that function (integrating) to compare to reality.
I think it's more correct to call these ODE Loss Functions, Euler Loss Functions, or Lagrange Loss Functions for neural network evaluation.
Very confusing presentation! First Neural ODEs are presented as a continuous version of ResNets, which would imply that the integration happens in "depth" which would make them similar to fully-connected or convolutional neural networks (non-sequence models). The afterwards it is suggested that the integration actually happens in "time" which makes neural ODEs much more similar to sequence models. Even ChatGPT et al. are confused and can't answer this distinction properly. Seems like it is a quite buzzword-driven field...
multi flashbacks