KAN: Kolmogorov-Arnold Networks

A Google Algorithms Seminar TechTalk, presented by Ziming Liu, 2024-06-04
ABSTRACT: Inspired by the Kolmogorov-Arnold representation theorem, we propose Kolmogorov-Arnold Networks (KANs) as promising alternatives to Multi-Layer Perceptrons (MLPs). While MLPs have fixed activation functions on nodes ("neurons"), KANs have learnable activation functions on edges ("weights"). KANs have no linear weights at all -- every weight parameter is replaced by a univariate function parametrized as a spline. We show that this seemingly simple change makes KANs outperform MLPs in terms of accuracy and interpretability. For accuracy, much smaller KANs can achieve comparable or better accuracy than much larger MLPs in data fitting and PDE solving. Theoretically and empirically, KANs possess faster neural scaling laws than MLPs. For interpretability, KANs can be intuitively visualized and can easily interact with human users. Through two examples in mathematics and physics, KANs are shown to be useful collaborators helping scientists (re)discover mathematical and physical laws. In summary, KANs are promising alternatives for MLPs, opening opportunities for further improving today's deep learning models which rely heavily on MLPs.
ABOUT THE SPEAKER: Ziming Liu is a fourth-year PhD student at MIT & IAIFI, advised by Prof. Max Tegmark. His research interests lie in the intersection of AI and physics (science in general):
Physics of AI: “AI as simple as physics”
Physics for AI: “AI as natural as physics”
AI for physics: “AI as powerful as physicists”
He publishes papers both in top physics journals and AI conferences. He serves as a reviewer for Physcial Reviews, NeurIPS, ICLR, IEEE, etc. He co-organized the AI4Science workshops. His research have strong interdisciplinary nature, e.g., Kolmogorov-Arnold networks (Math for AI), Poisson Flow Generative Models (Physics for AI), Brain-inspired modular training (Neuroscience for AI), understanding Grokking (physics of AI), conservation laws and symmetries (AI for physics).

[@athanatic]

Amazing! Can't wait to see all the applications!

[@bologcom]

Now I can understand KAN more clearly. Thank you!

[@braineaterzombie3981]

Google started working on it this fast. Thats crazy

[@jmirodg7094]

Very interesting thanks

[@AlgoNudger]

Thanks.

[@bhaskarpandey8586]

Welcome

[@user-dg9cr6wu9i]

This architecture is not compatible with current hardware due to the need to compute many additional and diverse nonlinear functions.

[@xba2007]

Not really, the bsplines are just simple multiplications / additions. In the end it's exactly the same type of operations.

[@jks234]

Okay. Rewriting.
My intuition on this is now, this is MLPs, but with nonlinear terms attahced to the weights and no nonlinear activation layer.
In my reflections on this, it sounds like the nonlinear terms are selected by the trainer.
Hm.
I don’t know what this will bring. I feel that introducing the nonlinear terms is almost like biasing the model before training.
Whereas linear terms are much less biased.
But I’m not sure.

[@mulderbm]

Such interesting stuff and not so much time to do anything with it it should have been my bread and butter haha

[@tianhao_harryzhang]

Has it been integrated into common AI framework like PyTorch or tensorflow?

[@clivedsouza6213]

how is the activation selection done? Don't you need a lookup/domain of functions to choose from?

[@clivedsouza6213]

splines, got it

[@movsessaryan1262]

Do KANs require fewer GPUs to achieve the same results for certain problems ?

[@leosmi1]

KANs Pros and Cons
Pros
- Accuracy
- Interpretability
- Faster neural scaling laws (achieve comparable or better outcomes with fewer parameters)
Cons
- Speed and efficiency (10x slower than MLPs given the same number of parameters)
- Scaling

[@aabbcc12411]

Since the "activation function" of each edges are different, the current implementation of KAN doesn't work well with GPU but it should be possible to be accelerated by specially designed chips

[@movsessaryan1262]

Thanks for clarifying!

[@meguellatiyounes8659]

Mlp in disguise.

[@VijayEranti]

Imagine llm agent interacting with kan to do above. We can let it run autonomously

[@mawkuri5496]

why the blue guy blurred? is he wanted by the FBI?