Vladimir Vapnik: Predicates, Invariants, and the Essence of Intelligence | Lex Fridman Podcast #71
ฝัง
- เผยแพร่เมื่อ 9 ก.ค. 2024
- Vladimir Vapnik is the co-inventor of support vector machines, support vector clustering, VC theory, and many foundational ideas in statistical learning. He was born in the Soviet Union, worked at the Institute of Control Sciences in Moscow, then in the US, worked at AT&T, NEC Labs, Facebook AI Research, and now is a professor at Columbia University. His work has been cited over 200,000 times.
The associate lecture that Vladimir gave as part of the MIT Deep Learning series can be viewed here: • Complete Statistical T...
This episode is presented by Cash App. Download it & use code "LexPodcast":
Cash App (App Store): apple.co/2sPrUHe
Cash App (Google Play): bit.ly/2MlvP5w
PODCAST INFO:
Podcast website:
lexfridman.com/podcast
Apple Podcasts:
apple.co/2lwqZIr
Spotify:
spoti.fi/2nEwCF8
RSS:
lexfridman.com/feed/podcast/
Full episodes playlist:
• Lex Fridman Podcast
Clips playlist:
• Lex Fridman Podcast Clips
OUTLINE:
0:00 - Introduction
2:55 - Alan Turing: science and engineering of intelligence
9:09 - What is a predicate?
14:22 - Plato's world of ideas and world of things
21:06 - Strong and weak convergence
28:37 - Deep learning and the essence of intelligence
50:36 - Symbolic AI and logic-based systems
54:31 - How hard is 2D image understanding?
1:00:23 - Data
1:06:39 - Language
1:14:54 - Beautiful idea in statistical theory of learning
1:19:28 - Intelligence and heuristics
1:22:23 - Reasoning
1:25:11 - Role of philosophy in learning theory
1:31:40 - Music (speaking in Russian)
1:35:08 - Mortality
CONNECT:
- Subscribe to this TH-cam channel
- Twitter: / lexfridman
- LinkedIn: / lexfridman
- Facebook: / lexfridmanpage
- Instagram: / lexfridman
- Medium: / lexfridman
- Support on Patreon: / lexfridman - วิทยาศาสตร์และเทคโนโลยี
I really enjoyed this conversation with Vladimir. Here's the outline:
0:00 - Introduction
2:55 - Alan Turing: science and engineering of intelligence
9:09 - What is a predicate?
14:22 - Plato's world of ideas and world of things
21:06 - Strong and weak convergence
28:37 - Deep learning and the essence of intelligence
50:36 - Symbolic AI and logic-based systems
54:31 - How hard is 2D image understanding?
1:00:23 - Data
1:06:39 - Language
1:14:54 - Beautiful idea in statistical theory of learning
1:19:28 - Intelligence and heuristics
1:22:23 - Reasoning
1:25:11 - Role of philosophy in learning theory
1:31:40 - Music (speaking in Russian)
1:35:08 - Mortality
This talk will be legendary as it gets on record one of the last living of a breed of masters.. Well done!
Lex: What's the meaning of Life?
Vladimir Vapnik: Let me first talk about digit recognition.
PS: Really loved the talk Lex! Keep up the great work 👍
I can't consume Lex's content fast enough. The ideas presented in these podcasts are so helpful, its like a patch for your brain's software.
training data for your neural net
with high vc dimension
Same for me. But now I am addicted.
Smishi 😂hilarious bro
I love the format of the video and timestamps and the fact that it is also available in iTunes. I recently did my project on SVM and your video couldn't be more wonderful. Respect and 🙇 🙇 🙇 Love from Nepalese student in Kathmandu. x0x0.
Loved the russian conversation part 1:31:40
I loved your username. Especially that written correctly. Are you an uyghur person, if I may?
Thank you Lex for another beautiful experience.
For me, the ideas of weak and strong convergence on functions over Hilbert space are a good way of thinking about evolution generally. And both of your ideas are important.
Evolution has worked on both, in an involuntary sort of way over both classes of "spaces" (physical and function, recurs as deeply as one is able).
Early on in evolutionary history, the physical space in which replication could occur was small, as was the space of possible replicators and strategies (functions).
Natural selection has increased both levels of "space".
By the time our particular evolutionary sequence gave rise to language and to the conceptual spaces that the exploration of the space of all possible symbols and logics and topologies made possible, we were already very complex machines, with huge sets of functions (sorted, selected, and optimised to some degree) for both weak and strong convergence over the sets of spaces experienced by our ancestors (all of them, simultaneously, over multi-generational time-spans).
So depending upon how you view them, you can call them functions or predicates or heuristics, they can be any and all, to different degrees in different contexts.
There does not appear to be any limit to the depth of context - it seems to be infinitely extensible.
Functions and predicates that worked in one set of spaces will not necessarily perform well in another set of spaces, and they are often a good place to start.
If a processor is fully loaded, then random search is the most efficient possible search algorithm (all indexing forms use more processor cycles).
So if one is pushing the space of spaces (as Wolfram does with NKS) and one uses the functions that Yudkowski explored in AI to Zombies as generally useful approaches to limiting the function space; and one uses as abstract a set of representations possible that still retain useful mapping to what science seems to be indicating actually exists (which includes QM and GR and biochemistry and psychology and culture ...), then the picture is very clearly that these ideas are both critical; and the strategy space of evolution is fundamental to understanding.
One has to get that all new levels of complexity in evolved systems are predicated on cooperation, and that rapidly gets mind numbingly complex at every level.
Our current societal preoccupation with the very simple idea of evolution being competition is actually imposing existential level risk upon all of us. AI in a competitive context is not survivable.
AI in a cooperative context is necessary for our long term survival (it is the only access possible to solution spaces for a large class of already well characterised existential level risks).
More people have got to start seeing these twin realities.
Lex, this podcast is so bad ass. It's so inspiring to hear from the great minds in the world today. Thank you for your work in doing this. I encourage you to expand into all aspects of science and the world that surrounds us. Keep it up.
Few people can talk about #AI, the #arts and #philosophy.
Lex Fridman always does, and in this legendary episode with Prof. Vladimir Vapnik, the triple conversation develops at the same level.
Chapeau!
Ps. The part in Russian on Johann Sebastian Bach’s music predicates is touching, also for not speakers like me.
Vladimir is my favourite Lex guest yet. He points us at handwriting recognition and suggests we start our search for intelligence there. If you need more than 60 samples for your deep learning algorithms then they are not intelligent in the sense that that we humans can converge upon a generalised model with just a handful of examples.
Our visual cortex needed hundreds of millions of years to evolve from a simple light receptor, a lot of people forget that.
Such a great job Lex. These interviews are opening new worlds to me.
Thanks so much for this Lex! Vapnik is a legend!
Dude this podcast is so underrated! Great talk with Vladimir Vapnik!
You are doing a very great job. I know you have done very hard work for your every video. Thanks a lot for your whole team for the back to back remarkable interviews.
What an outstanding conversation! Было очень приятно услышать ваш маленький русский разговор тоже🙂 Потрясающие люди оба
Thanks a lot for this conversation, Lex. I always admired Vapnik, and his original work on Statistical Learning and VC dimension was the first pathway into AI in the late 1990s that made sense to me since my training was in mechanics, mathematical analysis, and control theory.
On the other hand, your broad and exploratory approach to AI appeals to me as a means of synthesizing the multitude of opinions and results.
The counter play between platonic rigor and a sort of scientific empiricism is *so* valuable and yet rare.
Thank you for doing these Lex!
That is literally the best video I’ve seen on TH-cam.
I’m now rewriting my PhD thesis after listening to this. Love mnist again :)
One of the best podcast on this channel so far
My two cents on Vapnik's astute observation about symmetry: brains inherently recognize symmetry because of the natural noisy connections they're born with, able to relate any aspect of perception to any other, virtually invariant of sensory mode or locality within a sensory mode. As it stands, creatures possessed of vision are able to learn all about symmetry early in their development - about when they are able to recognize anything at all - because their visual cortex has learned to encode whenever any part of their vision corresponds with any other part, in color, shade, saturation, texture, motion, etcetera. This lends itself extremely well to the recognition of simple symbols (albeit not very simple in terms of machine-learning). Experiencing everything, to my mind, is what defines the predicates that enable us to recognize digits, letters, voices, faces, shapes, and sounds, and even to have volition predicates for controlling not just our bodily/motor actions, but our own attention, decision making, and thought processes.
In other words, the predicates (or at least some of them, or an aspect of them) is inherent to the structure of reality - and all things that have meaning, to us and our fellow creatures at least. Imagine a creature trying to learn how to see if its optic nerve was feeding its visual cortices something like this, for instance: th-cam.com/video/gv0KbAWu0Hk/w-d-xo.html
There was a study involving raising cats in environments textured with horizontal or vertical lines exclusively. The result? They couldn't conceive of heights or jumping up/down to things if they lived in the vertical-line "world", while those that grew up in the horizontal-line "world" could at least see heights, and jump up to them (or down from them) but would bump right into anything vertical as though completely blind to it. www.psychologytoday.com/us/blog/brain-food/201404/the-cat-nobel-prize-part-ii
The predicates are out there, and I believe we can define a system that procedurally generates them via "experience".
Thank you very much for this, Lex!
Thank you so much. Keep up the good work !!!
Vapnik is legend. Very patient.
Lex thanks !this has opened a new world for me
Nice discussion thanks for different thinking!
wow such an inspiring and great podcast
Awesome! I find these predicate ideas fundamental if we want to advance further from approximation by throwing a bunch of matrices (useful, but we are still here since the 20th century) on top of an image towards true predicate based "intelligence", even though it's an overwhelming task for humanity or a non-existant thing
Good one!!!
« Let’s focus on digits recognition »
Awesome
When something is difficult to under listening to it more than once until you understand.
Огромное спасибище!
Getting through this was an intense experience.
did I get the end right?
when solving a problem of interest do not solve a more general problem as an intermediate step. ?
I was thinking of the MNIST challenge:
I would first convert the handwritten digits in xy coordinates in order to treat the digits as very simple circuits of Grand Prix motorcycle racing
.
Next, the predicates I would define to represent the different racing tracks would be:
- infinity: trajectories with no end and no start (e.g. 0, 8, or part of 9 and 6)
- straightness: straight trajectories with a beginning and an end (e.g. 1, part of 7)
- symmetry: because Dr Vladimir asked for it
- curveness: ...
QUESTIONS FOR YOU:
1. which other predicates would you use to describe racing circuits?
2. how can we project an handwritten image into a xy coordinates?
Brilliant
Here some concepts of image predicates based on photography:
Is image sharp
Is image color
Is image black and white
Is image contains objects
Is image has subject
Is image conveys the story
Is image over exposed
Is image under exposed
Etc...
These all very general concepts which could be applied to judge quality of the image to compare images.
Love the conceptual ideas from Vladimir. That is the level of abstractions Vladimir is referring to.
I think he actually outlined how to identify conditions for selecting admissible functions in formal way which could be formalised.
I haven’t heard Lex speaking Russian before. Wow, his accent is so hard.
That acecent is aewsome
Лекс люблю тебя!
Lex: Bathroom break?
Vlad: I don't know about that, I am here for digit recognition ...
Subtitles would certainly be helpful for the russian part...
Vladimir Vapnik, what a magical mind
Maestro Vladimir, your smile is also brilliant like your mind.
I've always thought about things this way but never had the language to communicate it effectively.
Wonderful interview. Could even the greatest Russian novelist dream up a such conversation about music, poetry, philosophy, and statistics? Thank you.
My comment: My favorite Mayakovsky poem: “Cloud in Trousers,” and the lines something like, “I will be tender, not a man, but a cloud in trousers.”
Living legend
Lex I would love to see you interview Yuval Noah Harari.
I found it so interesting that he wasn't fully convinced a machine would/ could find the predicates.
Mr. Vapnik's accent is so strong so I think I would understand more if he'd speak russian ;)))
Guest Suggestions: Robert Sapolsky, Andrew Ng, Richard Dawkins, Nick Bostrom, Andrej Karpathy
Just a silly question: Is your middle name Luther?
To me sometimes 'predicates' in Vapniks use seemss to constraints in constraint optimization (no?), i suppose they are different, but how exactly? Other times the concept seems to be close to the features learned in Convnets, and still other times it seems asociated with the kernel-functions (say Rbf), used in the 'kernel-trick'. Are they all these things at once? If yes, then how was these things united which were separate before...? Or if no, what should be my basis for understanding 'predicates' in an intutive but mathematical sence?
"When solving a problem of interest, do not solve a more general problem as an intermediate step" Vladimir Vapnik
I do nothing else
@@MS-il3ht Me too, I only realized that when it was formulated like this. Most of our time can be sometimes wasted trying to solve irrelevant tasks.
@@kyrgyzsanjar well, being a philosopher entails that I think a bit differently about this... :-)
1:31:50 great scene
So I know this is a rather late comment to come for this video, but even though it also seems that in any place so far in which I mentioned this specific kind of a generalization of symmetry (of which there is many different kinds of generalizations of course) that I came up with myself about 10 years ago as I applied to the Technical University of Munich to study mathematics, no one seemed to have been interested in it to generalize it further and build up on it or investigate it further, since at least symmetry seemed to have played an important role in this discussion, I'd like to communicate and explain this concept (which so far I also haven't seen coming up elsewhere already, but I haven't searched for that long for it either), and who knows, maybe it can be helpful:
For the simple case of functions that map from |R (or subsets of it) to |R, instead of only considering axial symmetries, like how the function g(x)=x draws a 45° axis with respect to which the function f(x)=1/x is axially symmetric to, I wanted to extend this to symmetry curves, rather than just axes. And actually for this specific example, the family of functions h_{c}(x)=sqrt(c+x^2), each with any constant c in |R to pick (as long as the term is well-defined, of course), constitutes all symmetry curves (or functions) with respect to each of which the function f(x)=1/x is symmetric to, in the following sense (as example by which I'll try to explain the general concept): For any point on the graph of any of the functions h_{c}, if one takes the (required to exist) tangent to the function through this point (by means of the 1st function derivative) and then based on this tangent constructs the unique to this tangent line orthogonal linear (though technically affine) function that also goes through this point, then if this function intersects the function f of which one wants to investigate what it's symmetric to, then for every intersection point on 1 side of it, there should always be exactly 1 unique corresponding intersection point with f on the other side of it such that the distance of either intersection point to the specified point on the graph of h_{c} should be the same as for the other intersection point.
Building neural networks can give some ideas about how brainy neural nets work and so guide neuroscience research to look into that and if confirmed then we have proof that that direction is good because it works.
1:39:42 when he taps the table lmao
👍 ❤
I want to be able to understand what professor Vapnik is talking about. What should I read and study? What are the predicates? What does it precisely mean that CNN is a single predicate? What is the functional variability in this context?
rich sutton would be a great guest
Спасибо, Lex for your podcasts. Very interesting. Keep up the good work. We in nowadays Russia are living through tough times. Let's hope that the darkest hour is just before the dawn
Who is the Jan Lacomb in reference to Convolutional Neural Networks? around 32:00
It's Yan LeCun he's referring to.
34:25 ...let's talk not on emotional level but on mathematical level...
OK, I'm half an hour in and maybe he'll say something to change my mind later but I just have to say this guy could benefit from studying semiotics, of which I recommend Umberto Eco's books as a starting point.
Vladimir talks much about invariance. Is this as simple as translational invariance, where moving the object around in space does not change the result, or is it something much deeper?
Having a formal idea of whatever you perceive as *ideal* (predicate(unit)), which is abstract representation of *ideal* implementation detail for a given function would already in itself be ultimate solution. This structure wouldn’t necessarily have anything to do with intelligence since it’s solution is entirely based on human reasoning and it can not reason for itself
I think his theory is exactly what our minds are doing by trying to accomodate opposite set of ideas into one while tryint to form a unified theory of universe. Similar what hegelian philosophy says. Its jst that he has proved it in mathematical terms.
We need translation for 1:31:50
In an alternate universe, John Wick went to MIT, studied machine learning, runs a successful podcast and kept his original name, Lex Fridman.
the Levi-Strauss moment was painful)
I didnt totaly get the difference between an heuristic and a predicate.
They seem to do he same, reduce the amount of possible explantions to a problem.
@38:20 Savage
Nice to see my former supervisor again...
That's right Lex. I love you enough to listen to you shoot the shit in Russian.
children under 2 aren't affected by the predicate of counting past 2 and actually understanding what 2 is compared to 1
First layer of CNN is about symmetry decomposition (brkoen symmetry detection). Later layers are more and more constructs of the first layer. The professor didnt say anything new. When he say predicate i think "feature", again, anything new. a weak convergence thats what NN optimizing for.
Hes 100% rational like a math machine
I enjoy all these conversation even though my brain just whines as it tries to accommodate all the things being said and my consciousness is tries to mock me for how little I know, and what is worse, for how little I can reasonably ever know given all my limitations compared to the people speaking on these podcasts or to Lex himself.
:-o
You may surprise yourself over time. ;)
ｐｒｅｄｉｃａｔｅ
Russians like to think deeply.
That's wonderful, isn't it?
Replace the word functions with solutions then math analogies work better
I wouldn't like to work for Vladimir. I feel like he would push me to work on things he believes is the way to go without caring about if I share the same view or not.
Hi All, Anybody who can reference a good book, blog or video resource and dare I say dumbed down versions of understanding predicate, invariance, admissible functions and function itself. Something which could ease me into this video?
Does Lex speak russian? Just curious.
difficulty in communicate with high iQ and knowledgeable person is clearly seen here ,,interviewer is like a grad student to understand what he is saying ,he goes to emotions , professor bring back to reality again and again .... looool
this is a great interview of lex and i fully enjoyed ,it took more than 5 replys to fully understand the conversation and am still contemplating on it . see more interviews like this with masters of our time .thank you very much lex
I don't believe life has a purpose. It's just something self-replicating. The most basic system I can think of that is capable of imposing its own structure on matter around itself is growing crystals.
Life is definitely complex and hard to understand the beginnings of, but when you keep things like the lack of information from the earliest life-forms in mind, I don't think that there has to be more to it than that. It's just something self-replicating that kept self-replicating and evolving, and now we are here.
If something stops replicating for some reason, they will just disappear, and those who keep on going will keep history looking the same (life evolved and kept evolving), although the species involved will differ. If the most basic life is very simple, I don't see why there has to be anything deeper to it than randomness. I do not expect the most basic life-forms to be around today, since competition over billions of years means that life has gotten very good at eating other life, and the earliest didn't have to defend itself or fight for resources, not to mention that the environment back then was very different.
Just because the tracks were lost and we don't know where it started, that doesn't mean that there has to be a point. Life just happened, and it keeps going. Now it's up to you to make life worth it.
Lex, a scientist who is willing to exhibit some soul! Hold on to that!
The discussion around digits and admissible functions is rather repetitive. The conversation is stuck in some small region of Hilbert space.
I like yore Flag.. Point is...! Hoho Kiss
lol
To anybody not in the Prolog world: Hilbert Space is the space of Prolog terms, which is comparable to the space of possible JSON objects. Predicates from logic are implemented in Prolog as multi-valued search algorithms. This is the AI of symbolic computation.
I get the notion that Vapnik is interested in having learning use Hilbert Space predicates as their tools, in place of or in addition to linear algebra in the content of neural networks. I am not clear how he intends these worlds to interact. From the start, Hilbert space computations are not really GPU friendly but on the other hand if better predicates than linear equations can be employed you might get fast convergence with less iteration.
abj there is also a formal mathematical definition of a Hilbert space of functions - basically requires that you include enough functions such that sequences of functions are convergent in the set and that there is an inner product that is defined for any two functions.
Depending on what the functions are, it may or may not be amenable to GPU computation. Linear functions, for instance, form a Hilbert space.
Don't let hillary know about this guy!......
hamsters
2am.. at first glance of the name I thought holy shit he got the president of Russia.
I feel like Vladimir purposely doesn't say clearly the point of his argument. I hear a lot of mumbling instead of to the point statements. Is he not confident in his ideas, or what could be the cause of me feeling this way?
This talk will be legendary as it gets on record one of the last living of a breed of masters.. Well done!