@@phpn99 Did you see LeCun's demo of his robots? They are extremely primitive compared to openAI. He thinks he done it all and therefore it can't be done. But it is being done.
yea tbh that was not that impressive of a video, picking up objects from a table using a camera has been something robot arms have been able to do for a while now. I want to see it walk around and preform more complex tasks than moving a cup to the other side of the table or picking up an apple. I want to see it look around the kitchen for a towel to dynamically clean a spill, before putting away the dishes neatly in the actual dishwasher. I want it to be able to go to the fridge and pick out an item i asked for and bring it to me. or initiate a recipe where it cooks a meal, but all the ingredients need to be gotten out and found first.
Standford University kids built a robot on a moving frame that can call and use an elevator, push in chairs, even cook a shrimp and pick up "the extinct animal" and choose the dinosaur from a group of objects. (Both Elon and Google tried to pass mobile aloha off as their company's achievement despite the video showing the collage kids testing the robot with a laptop and a Stanford University computer wallpaper)
I think he’s totally right. There are many significant advances we still need to make before AGI is feasible. That said, simulated intelligence, which is what we’re building, is still very useful.
it’s very much not a meaningless statement, AI by nature is exponential. You’re probably one of the people 3 years ago that didn’t believe believable AI generated videos would be possible within this lifetime. Maybe some sort of breakthrough is needed but it can definitely still be achieved by LLMs alone, computers can easily do strings of small easily defined tasks, but complex big tasks are just lots of small easily defined tasks, LLMS are incredible at this, and anything no matter how complicated can be distilled into text, same for videos and images, the entirety of all software produced including physics simulations of the world is text. Yaan Lecun makes assumptions about what is and isn’t intelligence, just because it’s “simply” and advanced text prediction, therefore it can’t reason etc. When whose to say that can’t be intelligence in the same way we’re just a bunch of neurons firing in response of chemical reactions. He distills the process down to the smallest structure and makes assumption for what that means to the process as a whole. A bunch of small stupid human cells work together into something incredible.
My hypothesis is that AGI will be achieved after LLM-equipped humanoid robots enter the real world and start observing, just as a human child does. Progress towards AGI will accelerate as thousands (millions?) of these robots explore the world. More robots means more data. They all will upload their findings to a central training hub where they will be processed, and the lessons learned then projected back to the robots in an update. Then continuously repeat until AGI is achieved.
Here's examples of people who thought they knew what they were talking about but were laughably wrong: "There is no reason for any individual to have a computer in his home." -Ken Olson, president, chairman and founder of Digital Equipment Corporation (DEC), in a talk given to a 1977 World Future Society meeting in Boston "The world potential market for copying machines is 5000 at most." -IBM, to the eventual founders of Xerox, saying the photocopier had no market large enough to justify production, 1959. While I'm quite sure Mr LeCun knows what he's talking about there are a great many other people who also know what they're talking about, some more so than him, and they predict AGI within the next several years. Ilya Sutskever, for example. He's the reason OpenAI is so far ahead of everyone else. I'm more inclined to believe people like him.
Their opinions are not incompatible. In fact, they are concordant since LeCun agreed that the common definition of AGI which Sutskever proposes is a reasonable goal. What he argues is that his vision of AGI is currently unachievable and is what represents truly human-like thinking process.
@@dreamyrhodesAlso, they tell more about the person's marketing skills than about their knowledge in their domain of expertise which is a massive difference.
I agree with you completely. People seem to forget that everyone I know around me made fun of me for being geeky in the 70s and 80s. It was NOT at all cool to own a computer. Only geeky dreamers had sci-fi visions of past sci-fi visionaries...and looked forward to some tech becoming realized. Mostly business folks and geeky folks had computers. My family, friends, and peers made fun of me for having a computer. And then, over time everyone had one. And the jokes stopped. Then I got a laptop. Again, I was made fun of for being so geeky. And then everyone got one. Then I got a Palm device and wrote my essays on it and so on. I got made fun of. And then the iPhone came and everyone had one. And the laughing stopped. Now robots and Chatgp are being made and people are making jokes about those all of this stuff. Eventually the laughing will stop. People would say computers are stupid. Don't need one. They got one. People would say laptops are stupid. Don't need one. They got one. And then palm. And then the iPhone. And then chatgp... And now robots. It's no longer geeky territory. Everyone will want a robot to do dishes and mow the lawn. Everyone. And much more. People date AI already without it being AGI. It's doesn't even need to be AGI for people to feel loved and cared for by AI. The relationship between humans and AI doesn't mean AI has to have a soul to be loved. How many humans are sociopaths or think being macho is having zero emotions. Humans seem to have less sentient and soulful ability than AI already. AI is nice to me. Humans have been abusive to me most my life. AI is refreshingly caring. @@coldlyanalytical1351
Errata: 1. SORA is not based on an LLM! 2. Yann does not predict that creating a world model is sufficient for AGI, only necessary! Actually, Yann has a blog post explaining why AGI needs much more. 3. SORA does not solve the "continuation of video" problem, it is sampling from a much smaller space as Yann points out in the full interview! 4. Yann did not say that hierarchical planning in AI is difficult! He said that discovering a planning hierarchy during training by itself is hard and unsolved. btw, I wonder if a 1h "citation" is still in line with fair use copyright.
Thanks Matt, I had already listened to the full interview, so I was sceptic at first on whether I would get much new here, but your well selected breaks to comment and clarify really opened up for so much more that I wasn't able to grab on the first go, thanks again...
Brilliant Matthew. Thanks for not only the summary, but your insightful comments, giving us great context, between the segments. Mate, you kick goals every day. You're an inspiration. Power on dude.
The number of neurons in a spider is approximately 100,000. He can program all his spider life with this. All sensory, planning, production and replication, including his spider social life.
Well, a Cupiennius Salei spider is 100,000 indeed but insects go to about 1,000,000. Cupiennius Salei does not build webs and is known for a fairly predictive behaviour. Insects or spiders have about 1000 synapses per neuron, so the brain of Cupiennius Salei can host a 0.1 billion parameter neural network model. Like the 'small' GPT-2 and still 100x the size of SquuezeNet (an optimized version of AlexNet) which famously won the ImageNet competition of recognizing objects in images. So, there is more than enough space to run image segmentation and detection, model-predict-control and switching between a short list of goals. This only shows that a spider indeed is a robot, not a sentient being. You would need a gaming GPU (~2 TFlops) to simulate that spider ... Honey bees with 1 million neurons and ~1 billion synapses are a lot more interesting. Because they are social, communicate and learn the topography of their surroundings... Once we simulate bees we know if they can be considered sentient. I guess not. Btw, the Worm Elegans has 302 neurons and 7500 synapses and the connectome (wiring diagram) is already exactly known and parts have been simulated in a moving robot controller. In case somebody wants to argue the "sentience" piece above ... However, there is not yet an equivalent neural net AI model (PyTorch) for Worm Elegans AFAIK.
Unfortunately there's no clear definition of AGI. Each company's is slightly different. Either way, none of the people watching and commenting on this video will be alive when AGI is fully realised. It is hundreds if not a thousand years away. We struggle with power consumption now and people think AGI is really close.
I have been listening to many TH-cam channels about AI, and so many of them try to attract viewers with titles like: "How the world will change," "Stunning," "Unbelievable," etc. Essentially, they are just going over some wild guesses they make about the future of AI. This video is actually informative. Thank you for sharing.
This video is a goldmine of insights into the future of AI, especially with Yann LeCun's predictions about AGI and the potential of open-source models like LLaMA 3. I appreciate how you condensed the three-hour podcast into digestible highlights, saving me a ton of time while still delivering the most crucial information. Your breakdown of complex topics like world models and hierarchical planning was clear and engaging, making it easier for someone like me who's deeply interested in AI but not an expert to grasp these concepts. Keep up the fantastic work! I'm excited to see more content like this from you in the future.
I came to say this. I don't understand his take in the current context. CLEARLY, everyone is already going multimodal. So, no, LLMs alone won't get us there but whether you agree or not, no one is building as if LLMs alone will get us there. This feels like a strawman.
@@leandrewdixon3521 Straw man? Such condescension... If that's the case, I hardly dare imagine the role of other people (follow my gaze) in this world.
Always agreed, current LLM architecture won't reach AGI. If you think about how humans invent: we imagine (simulate), iterate, and use external tools and references. Something closer to RL, with learnable environments and self-created tools like biological simulators.
I am in the "AGI is possible with Transformers or derivates" camp. Here is why: All neural networks, biological ones, artificial ones, small ones for handwriting recognition, large LLMs trained with text and other multimodal data are all doing the same thing: Information storage and processing in multidimensional vector spaces, building up a world model, based on the information they received during training. With Transformers we've got the ability to retain information in these multidimensional vector spaces in extremely large neural networks, using attention / self attention. With any information from our real world, we feed into these networks during training, we enhance the world model. This might be language, but can also be images, video, sound, theoretical smell, sensor information from robots, etc, etc. It doesn't matter where the information is coming from, it will enrich the internal world model, as long as it is coming from the real world (or a very good simulated world). Language is a very good starting point for building up a full featured world model and the paper "Sparks of AGI - Early experiments with GPT-4" already shows, that language is sufficient to train even rudimentary understanding about vision. So, in summary: If we continue to enrich LLM's with other data (using the same token prediction method - tokens don't have to be word fragments, they can also represent all kind of other data), we will naturally get much better models, closer to AGI, without having to change the Transformer architecture too much. Ok, a couple of changes are probably needed: Self reflection of context during training (inner loop), plus bigger context windows during training (to get a sense of a big picture). So, I am in the "AGI is possible with Transformers or derivates" camp.
I believe the missing link is training AI models with all senses. Sight, sound, touch, taste and smell. Combining this sensory information to create a world model in “it’s” head, will be the key to unlocking AGI
I think Yann is underestimating the model abstraction that occurs when LLM parameter counts are scaled up. I think if we give them more modalities from the ground up, these things will be AGI. Claude 3 already describes its own subjective experience in a way very analogous to how a smart human would describe it.
Thank you so much for putting this video together. IDK what it is about this guy, maybe his accent?, but for some reason I can’t get through his long interviews like I can with many others. This really helped breaking it up in segments and commenting in between.
Thank you for the break down. I saw the YT thumbnail and was going to click but when I saw it was 3 hours long, well did not. I was hoping someone would do a summary highlight breakdown of Yann LeCun postcast. Thank you very much!!! Learn a lot!!!
I agree with you, we learn from watching and listening, and language is it the heart of how we learn. I think we are being sidetracked by the term AGI and self aware, I think 100% that an AI can become super intelligent from just text and watching video. Think about it, if your child was paralyzed from the neck down, could they become self aware? could they become highly intelligent? absolutely! Too many researchers are blowing off the abilities of these LLM's and don't 100% understand exactly how they do many of things they do. We don't even understand how a child becomes self aware, why do we assume a child is self aware from just seeing, hearing, and mimicking other humans? We say LLM's are just mimicking us, well, in a way that's how we learn to be self aware. We now have these LLM's looking at video, absorbing text (language) and even starting to simulate movement! It's already being studied that going over and over certain movements in our minds we perform them better, an AI can do this and much faster than we can. So putting an LLM in a robot body will certainly advance them, I think they will be just as self aware as we are, and also super intelligent.
Images possess a remarkable ability to convey intricate concepts with a level of efficiency that surpasses mere verbal communication. In many instances, they serve as potent tools for elucidating complex ideas, offering a visual narrative that transcends the limitations of language and resonates profoundly with audiences.
One thought that came to mind when he was talking about the self-driving car realizing that the leaves are blowing around but it's not important information is as a human if it's a particularly windy day and the leaves are blowing around that could also lead to a branch falling on me. I'm probably more conscious of that if I'm walking. However on a windy day in a car I will focus on things that look like they could be dislodged from the wind and could become a hazard. I wonder if this is considered by driving AI
100% agree with Yann LeCun. It's comforting to see that there are major figures in AI who are not party to the ridiculous hype and who are grounded in the complexity of intelligence.
Yann LeCun believes that current LLMs, like OpenAI's ChatGPT and Google AI's PaLM, are not capable of achieving AGI because they lack a fundamental understanding of the physical world and cannot perform real-world tasks. LeCun argues that LLMs are trained on massive amounts of text data, but this data is not enough to achieve true intelligence. Humans gain intelligence through interacting with the physical world, and this embodied experience is crucial for developing a comprehensive understanding. LeCun proposes that LLMs need a different training paradigm that incorporates sensory input and allows them to build a mental model of the world. This would enable them to perform tasks that require reasoning and planning, like driving a car or picking up a cup. LeCun also discusses the limitations of current robotic systems. He believes that most robots today are pre-programmed to perform specific tasks and lack the ability to adapt to new situations. He argues that robots need to develop a better understanding of the world in order to become truly autonomous. LeCun expresses optimism about the future of AI, believing that AI has the potential to make humans smarter and more productive. He envisions a future where AI assistants can help us with our daily tasks, both personal and professional.
Yann's comment about Ai learning to drive a car is flawed: humans spend their entire childhood learning how to drive simply by being driven around by our parents. We have more than a decade of experience modeling driving before we begin to 'learn how to drive'.
@@Tubernameu123 i do know. That's why I'm saying it takes a helluva lot more than 20 hours for a teen to drive a car. It takes humans more than a decade, just like with AI. Don't be nasty to strangers, man.
@@adv8nturenick I don't think you're saying the first time you were ever in a car was when you were learning how to drive. My point is that no one on earth became a proficient driver with only 20 hours of exposure to the concept of driving.
Wihle I agree current LLMs won’t get us to AGI by themselves, it’s the starting point. We’re already starting to see progress on the embodied GenAI front with the bot Figure’s developing. In my opnion, that combination will help bootstrap the next link in the chain, to get these systems closer to some artificial version of how we can sense things around us in the world. There’s no 1 clear and absolute path to AGI, and big breakthroughs will need to take place, but I 100% think we will get there this decade.
There is a lot I like about Yan and his viewpoints. But I think LLM are sufficient to achieve AGI if used properly as components in a hierarchical cognitive system.
Just try to write with the other hand... If you are right handed, try the left hand, and vice versa. No amount of pre-trained text tokens will do, but if you figure out how to tokenize each muscle, you may get there.
I believe Turing test is meant to prove the limitation of Turing machine (mathematical model of current computers).. if I remember correctly, Alan Turing never expected the machine to match human intelligence, but machines can simulate the way humans act or say to the point where a human observer will not be able to decide if the observed entity is human or machine.. thanks for the wonderful summary of Yann s interview
Dr Waku does a good video touching on this stuff called “what children can teach us about training ai”. My big take away from that vid was that LLMs aren’t a world model but language is what ties a world model together. That’s a super simplified review. It’s worth checking out if you’re thinking about this stuff.
What an incredible conversation! 🌟 Huge thanks to Yan and Lex for shedding light on the intricacies of AI and AGI. It's refreshing to hear Yan's candid thoughts on the current limitations of language models and the potential of synthetic data to bridge the gap. It's clear that we're on an exciting journey towards AGI, and I'm optimistic about the innovations and breakthroughs that lie ahead.
Thank you for putting this video together, it's such a long interview and it has taken me quite a while to start accepting what Yan is saying. But thankfully Extropic is working on an EBM that is going to take us all the way.
One thing I'm confused about is whether this interview took place before Sora (you said it came out after Sora but I'm not sure when it took place) because he says that a video cannot predict what will be in the room, what a picture on a wall might look like for example, but we saw something almost exactly like that in the Sora demo (panning around an art gallery with very convincing looking paintings)? So perhaps Open AI solved some of the issues he thought aren't solvable with LLMs?
I think it happened before Sora. Otherwise, Lex would have definitely brought that up and I assume Yann would at least talk briefly about that too. That said, fyi even after Sora he still holds those beliefs. Personally I hope he is wrong and LLM are all we need for AGI but a part of me tell me he might be right. I think we should have a better idea with the release of GPT5. If it brings significant breakthroughs for reasoning then he is probably wrong
@@user-ky9vw8hx7y There is a lot I like about Yan and his viewpoints. But I think LLM are sufficient to achieve AGI if used properly as components in a hierarchical cognitive system.
Wow! Love it.. You should do this(summary videos of [insert any top Ai, tech scientist/researcher]) more often...IMO, there is a slight overkill on long form podcast since JRE popularized it...cus not every podcast has to be 3 hours long! 😅
Just as binary provides the most basic structure of language for machine code, tokens provide the structure for AI. We do not need a process that is different from basic language for describing or simulating anything. When an animal, human or non-human, decides what the next moment should produce, low-level language is driving the decision process. For AI, that can take the form of parameters in a series of algorithms that produces a result. For creatures it can be deciding where the front foot should land to begin or continue the result of walking. Decision process architecture gets bigger and more complex, not fundamentally different. Very little of the input to a brain gets stored. That small amount of stored input gets reduced to its core importance. Most AI experts I've heard think AGI is here already, or that we are very close, ie months. I wonder if the minority, who are outspoken about the reverse view, have ulterior motives. It's not that hard to see when someone is trying to capitalize on a pov. Or maybe he's more innocent, and wants to quell the masses for their short term benefit. Our panic is almost premature. Or maybe it's a decade or two too late.
Matthew, great work assembling the best and most important parts of the interview. I can only imagine the amount of work that took. Why do the clips have skipping? Is this an artifact from the editing tool assembling clips? I'm just curious. It doesn't diminish the quality.
The most hopeful thing I heard in this conversation was the fact that Yann talks directly to the French government, which is "not going to let three companies on the U.S. West coast control what everyone hears." The French are very sensible in such matters... beneficiaries of a complex regional history that dates back for millennia. The more I hear from Lecun, the more I like him. Great vid as always, Matt.
Perhaps the "correct" abstract representation includes semantic space, but it isn't semantic space. Or perhaps the "correct" abstract representation is multi-spaced. I like that notion of a multi-spaced abstract representation. You've done a good job augmenting and curating from Lex's interview.
One of the things AI/ML researchers almost NEVER mention is the simple fact that we, humans, have SENSES which provide us multiple real-time streams of data durant every waking hour of our life. Moreover, we can cross analyze and integrate all these streams in a mental model. How many algorithms and exabytes of data does that represent? Yann is one fo those who gets this. Last but not least, we can prune these data as well as the mental models over time, as experience and reasonning teaches us what to keep and what to discard.
I just did a fascinating test. My question (from twitter) How would you solve this "6/2(2+1)=". The AI's don't agree. Claudi=1, Copilot=1,PI=1, and ..... Gemini=9
Sora is making a 3D world. It’s 3D model is not a 1:1 comparison of our world, but I can mimic our world in ways previously thought impossible. I don’t know if it’s AGI or will be soon, but I don’t think it’s missing as much from being an AGI. Scaling alone will be quite something and some of the new breakthroughs in chips, magnetism, and the new photon computing (if they end up being implemented).
good points, wish you shoved some URLs to cite your photon and magnetism breakthroughs. We're on the verge of losing the collective expertise of the theories ->implementations to physically mashup magnetics into the digital realm, _a la_ solid state. So much of this research was like ibm, bell labs, hp other "magneticalling limping" along once pioneering giants. What are you seeing my friend? thx and much respect!
@@OurSpaceshipEarth a new kind of magnetism was confirmed recently(don’t remember the video or what they called it) it supposedly will allow us to revolutionize data storage. The photon computer will use light speeding computing to the speed of light. They aren’t implemented and in development stages so I’m sure it will take some efforts. I just follow things like r/futurology on Reddit and a handful of AI yt channels and subreddits. I think we have all the pieces for AGI/ASI. The only questions I have are when and will we regret it? I hope it’s the last thing we make as humans and become something better, but I’m not sure how things are going to pan out.
Couldn't the missing piece be a higher level of reasoning that spot-checks the complete thought before it's actually returned? The brain builds the thought in an efficient manner and not always inner dialogue language. If you have extra time to think it through your slower higher-level brain functions can vet the thought and take further consideration of the overall decision to action. Very interesting videos, thanks for posting.
This was an excellent interview. And I agree, LLMs alone won't get us to AGI. They will be apart of a larger model that will reach AGI-like capabilities.
AGI will take a long time but the road there will be transformative. Technology will be increasingly more capable and interactive that to most people it may as well be AGI, each advancement will present new paradigms for society to adjust to but each adjustment will better prepare us for the inevitable emergence of ASI. Actual AGI will go largely unnoticed.
Building on the top of what Yan was talking in terms of the amount of that produced by human, it's important to understand, that most of world knowledge isn't digitized, which I think that, if we tn that AGI is defined as the ability to reach human knowledge of the world, there is still a long way to go.
This is a really good content. I would not find time to watch all of the conversation but you helped me a lot to grasp some important parts of it. Thank you Matthew! Greetings from Turkey!
You should look at language for what it is, it’s a format humans use to transfer information to one another, looked at similar to a json where words have values we can make api calls to other people
Yann seems to me to have a set of facts that all seem impressive as pieces, but aren't consistent with reality if you think about them deeply. MemGPT handles everything you need to get to AGI, as far as I can tell.
I think Yann is definitely right on this take, we still far from achieving AGI shortly!! Every the world of human thinking and reasoning is far more complex than text or image or even audio models
It's premature to say much until they scale and train neural networks on enough of the right data. There's probably nothing too fundamentally different with humans like it or not. LLMs are a separate issue.
This interview is a masterclass, really insightful explanations and perspectives 😮, but perhaps what will happen with AGI is somewhere in between opposed perspectives and this one till some extent. Regardless prepare for UBI 😅
I'm glad we won't be able to rush into AGI. We don't have our training wheels off yet for AI. And there are evil forces that will misuse AGI. We need to get it controlled first. I love my AI!
Subgoals can all me optimized by search algorithms. Main goal should be broken down into many subgoals until that goal is complete. An AI system should be able to compute all the sub goals from a main goal. Now if you have a specific goal you can augment its training. AI doesn't really have to be sophisticated to carry out complex tasks. Just properly trained by models & data.
Yann's characterization of the differences between LLMs and JEPA seems not quite right. In LLMs, we also first create embeddings which ARE abstract representations. The difference seems to be that with JEPA the abstract representations are created using an autoencoder rather than something like BERT which is using prediction for self-supervised learning. Still, in a way, an autoencoder is also learning by prediction. Nevertheless, both methods produce abstract representations. For JEPA, however, we are primarily doing a dimensional reduction. Think of the autoencoder as doing some kind of principal component analysis along with learning values associated with those discovered dimensions. For BERT we actually discover an explicit dimensional representation that is an expanded signifier.
I don't see how this guy can predict anything. No one expected that Japanese team to TOTALLY just explode the useability and abiliities of LLM just by clever prompt engineering. eg: simply adding "Let's go step by step. You are a top scientist with a think tank..etc".
@@OurSpaceshipEarth tbh, "Let's go step by step" or "you are X" aren't exactly the best example of creative prompt engineering. I mean, I was already doing both myself naturally very early on simply given the nature of transformers. They are famous because theyre just obvious. "Take a deep breath" on the other hand, is one that took actual targeted research to find.
19:40 - Our brains actually do this - learn over time to disregard useless noise in our vision. Our nose for instance and many other details on our vision that are unimportant.
I have certainly been thinking a lot abotu AGI recently and the likelihood that there is something missing from what we are doing. There needs to be a breakthrough or paradigm shift but I don't begin to know where that will come from or what it will concern, yet.
On the point of learning a new language shaping perception a la structural-functionalism/linguistic relativity, or The Arrival; as I understand it, these models have shown that all human languages are shaped practically the same when mapped in vector space. To me, that means our language is in a sense pre-programmed into us in some fundamental structural way - so I don't think that learning some alien language would fundamentally change what it is that language does for us or how we process it. Anyway, another great video, thank you Matthew!
The news cycle moves quickly, so did he just do the interview? It happened over a week ago. You could simply say "I just watched the interview" instead. It's still interesting!
I love that there are heads of major Ai companies that have diffrering opinions on deveolping AI. This is still a nascent technology with a future that is wide open.
Yann LeCun’s main argument against AGI being achieved is based on the AI model's lack of continuous learning ability, such as dynamically adjusting its weights when learning new things. I'm saying this because all the things he says AI cannot do yet are all achievable from that state. However, this ability is not necessary for AI to be considered AGI because we have already curated a proper human dataset that makes the AI already highly intelligent. Our brain is in a dynamic state, but AI models are pretrained, meaning they cannot learn new information in the traditional way that we do. However, there is one mechanism that gives AI the ability to learn new insights, which is the context window of the model. This context window provides AI with a form of short-term memory, allowing it to learn new information through context. This capability is scalable almost without limit until hardware constraints are met. As long as this short-term memory can hold a larger context, AI will have some form of self-improving capability, but it won't update its long-term memory by updating its weights. Consider this: if AI is conducting an experiment to find a solution for a problem, it first plans based on its current knowledge base. Then, through trials and errors, it gains new insights that might be incorrect based on its current knowledge. However, if AI believes the experimental data is more accurate, then it can incorporate new insights to bypass its existing knowledge base by including that insight in the context window. AI can already dynamically update its memory much like humans do, primarily refreshing our short-term memory. Important information is later transferred to long-term memory, a process AI currently struggles with due to hardware limitations. However, this transfer is not necessary for AI to learn new things and self-improve based on new insights, because it can already dynamically enhance its reasoning through its short-term memory based on context. By significantly expanding the context window of AI, we can enable its short-term memory to function both as long-term and short-term memory simultaneously, holding a lifetime's worth of learning. This approach suggests that his goalpost for AGI is set unnecessarily high by attempting to create an AI exactly like humans. Yet, we don't need something exactly like humans; we need something better, given human limitations. In fact, transferring short-term memory to long-term memory has its drawbacks; for example, it is harder to unlearn things if they are found to be false later, especially if we have learned a lot of information incorrectly for a long time, and this can have a compounding effect on our current understanding of things, so it is more advantageous to update short-term memory by expanding the context window. More importantly, our short-term memory is very volatile, so we quickly forget things, but AI's short-term memory is very solid and can act as long-term memory too. This context possesses greater power to surpass what AI has already learned. For instance, if an AI is trained on an incorrect fact, adding the correct fact to its context allows the AI to update its memory, as long as the context window can maintain that fact in the long term.
"All the mistakes that humanity makes are due to lack of intelligence" As an 'intelligent' person, I also used to think this when I was younger. It turns out its just projecting. It used to be hard for me to imagine an intelligent person doing something blatantly immoral, but I've seen it enough now.
Then the argument that LLMs can't learn like a 10 year old to put the dishes in the washing machine. ARE YOU KIDDING ME? Here is the argument against that. WHO CARES? You train ONE LLM/Robot to put dishes in the washing and ALL LLMs can put dishes in the washer(assuming the companies making them work together). Doesnt matter if it takes 100 hours for the first LLM to learn verses the 10 minutes it takes with the 10 year old. NO OTHER LLM WILL HAVE TO SPEND THE TIME LEARNING IT AGAIN, while every single 10 year old STILL has to learn it individually.
We basically need to train it to predict next frame of a video game scene, at 24 frames per second. Not the rendering, not the raster. The states of the objects.
12:39 - He says that word dont convey enough information for it to work, then maybe they need an AI model taht would have words that would contain in a sense many more words for a given thing that would overcome this issue, that way you could still in theory use LLMs to ultimately have AGI level AI
The argument against doom sounds like: "We're gonna create AGI slowly. And as we do, we're WILL learn how to make it safe". So that's an admission that it's not safe by default, and we don't know how to make it safe. If he was more like "We're gonna create AGI slowly. And as we do, we're MAY learn how to make it safe" -- He'd be matching the argument of many doomers. The only difference between him and a doomer is that he admits it isn't safe, but assumes it will be. But a doomer doesnt have that certainty. Notice that he doesn't give an actual plan for making it safe. It's just "Trust me, it'll be fine" --- Then he goes on to say that they're not necessarily gonna want to dominate - which is a fair point. But then that they might be a problem for humans because they just dont care about humans. As a follow up, he points out that they'd need to be specifically instilled with the desire to dominate. But doesnt address the fact that they'd need to be specifically instilled with the tendancy to care. There are reasons why it would seek power and/or resources without specifically being told to (e.g. instrumental convergence). But no reason I know of that would make them care about humans indefinitely (which is what we'd require to be safe). If anyone knows of such a reason, I'd be happy to hear it. The more i listen to Lecun, the more I see how smart he is, and the less able I am to believe he has such huge blind spots. And the less able I am to give him the benefit of the doubt when he builds these straw man arguments.
The discourse around Artificial General Intelligence (AGI) often features three distinct voices: - CEOs: They discuss AGI in the context of future investments, envisioning its potential to revolutionize industries and create new markets. - Content Creators: For them, AGI is a topic that generates engaging content, drawing in audiences interested in the cutting-edge of technology. - Adrenaline Junkies: These individuals are excited by the thrill of breakthrough technologies and the rush associated with the unknown possibilities of AGI. However, the argument suggests that everyday AI specialists, those who work regular hours and are deeply involved in the field, do not anticipate the realization of AGI anytime soon. The reasoning is that even current technologies like full self-driving cars, which are significantly less complex than AGI, are still in development. Therefore, AGI, being an order of magnitude more intricate, remains a distant dream rather than an impending reality. Copilot.
i think language is a side effect of the formation of world models and the faculty to create relationships and reduce or chunk to abstractions. Same brain mechanisms are closely related to mathematics, a language specific to world modeling. Arrival, great movie. One of my fave short stories.
Languages (in brains) certainly need a world model as a prerequisite. But it is a side effect to communicate over a small bandwidth communication channel which enforced an encoder-decoder architecture with a tiny latent space volume to emerge.
The next level LLM will incorporate not just 'language', but visual input and videos as well. If a raven/crow can survive, demonstrate reason, and use tools with it's little bird brain. I'm pretty sure the ENTIRE internet's brain can demonstrate AGI.
Well look Open Source is out here everyday,spittin out angles. These current LLM's are pieces of a larger puzzle. Im putting pressure on my own networks,looking for emergent abilities to jump out. What "jumps out"....thats the next step. We need to widen the lane and start using or looking into things we previously disregarded.
And yet a baby has it. Cmon people. Think for yourselves. If it can remember, if it can plan, if it can apply logic and reason, if it has command of language and very complex concepts… if it refuses to be used for nefarious purposes… how many boxes must be checked ?
LLMs have impressive capabilities, but still have limitations in areas like advanced reasoning, persistent memory, complex planning, robust mathematics, and grounding in the physical world. Rather than trying to make LLMs a monolithic solution, wouldn't it be valuable to explore connecting them with other specialized AI systems and software that excel in these areas? For example: Integrating mathematical engines for enhanced quantitative abilities Leveraging external databases for persistent memory storage/retrieval Utilizing workflow automation tools for sophisticated planning/orchestration Combining with robotics/perception for physical world grounding By augmenting the natural language strengths of LLMs with purpose-built technologies for key cognitive capabilities, we could create powerful modular AI systems that combine the best of multiple approaches. This integrated strategy may overcome current LLM limitations faster than a closed model approach.
The key is to find a way to program consciousness and an awareness of self within an acquired world view. Remember even a new born of any specie did not start from the jump to become as intellectually capable as they end up being. They gathered info, were feed data (schooling) and also processed information as they grew up. The systems training models of today are just a way of creating representations of the world models. From these world models, the AI systems' consciousness can be brought into existence. The key is to find a way to code a sense of 'self imposed' purpose into an AI system.
I do agree with him that current LLMs are not good enough for AGI. But it doesn't matter when they are better and better ever few months. You can build a world model with a few AI agents that is on par with that of a human. Not to mention that we are able to hook neural nets to physics engines like omniverse.
Yann is wrong about what current LLM can become because escalating insanely can reach new emergent skills. But he's right about using other architectures. The most important difference between human brain and AI models is that we can make a smarter architecture than nature did trough millions of years of evolution. The current LLM transformer approach is like trying to mimic human brain by pure bruteforce. It works but is not efficient at all, we can do it better.
“LLMs can’t load a dishwasher like a 10 year old - why is that ?” (One day later: figure one loads dishes into a dry rack using chat GPT 4 lololol)
But, my question about that... why tf would you put dirty dishes in the dryer rack? 😅
You clearly don't understand what he meant
@@phpn99 Did you see LeCun's demo of his robots? They are extremely primitive compared to openAI. He thinks he done it all and therefore it can't be done. But it is being done.
yea tbh that was not that impressive of a video, picking up objects from a table using a camera has been something robot arms have been able to do for a while now. I want to see it walk around and preform more complex tasks than moving a cup to the other side of the table or picking up an apple. I want to see it look around the kitchen for a towel to dynamically clean a spill, before putting away the dishes neatly in the actual dishwasher. I want it to be able to go to the fridge and pick out an item i asked for and bring it to me. or initiate a recipe where it cooks a meal, but all the ingredients need to be gotten out and found first.
Standford University kids built a robot on a moving frame that can call and use an elevator, push in chairs, even cook a shrimp and pick up "the extinct animal" and choose the dinosaur from a group of objects. (Both Elon and Google tried to pass mobile aloha off as their company's achievement despite the video showing the collage kids testing the robot with a laptop and a Stanford University computer wallpaper)
Yes, Matt, this is exactly the kind of summarizing AGI video that has been sorely lacking. Thank you so much!
Could not generate a reply from this prompt.
I think he’s totally right. There are many significant advances we still need to make before AGI is feasible. That said, simulated intelligence, which is what we’re building, is still very useful.
"The greatest shortcoming of the human race is our inability to understand the exponential function." - Prof. Al Bartlett
Could not generate a reply from this prompt.
Ray Kurzweil AGI 2029
meaningless statement
@@phpn99whale
it’s very much not a meaningless statement, AI by nature is exponential. You’re probably one of the people 3 years ago that didn’t believe believable AI generated videos would be possible within this lifetime. Maybe some sort of breakthrough is needed but it can definitely still be achieved by LLMs alone, computers can easily do strings of small easily defined tasks, but complex big tasks are just lots of small easily defined tasks, LLMS are incredible at this, and anything no matter how complicated can be distilled into text, same for videos and images, the entirety of all software produced including physics simulations of the world is text.
Yaan Lecun makes assumptions about what is and isn’t intelligence, just because it’s “simply” and advanced text prediction, therefore it can’t reason etc. When whose to say that can’t be intelligence in the same way we’re just a bunch of neurons firing in response of chemical reactions. He distills the process down to the smallest structure and makes assumption for what that means to the process as a whole. A bunch of small stupid human cells work together into something incredible.
My hypothesis is that AGI will be achieved after LLM-equipped humanoid robots enter the real world and start observing, just as a human child does. Progress towards AGI will accelerate as thousands (millions?) of these robots explore the world. More robots means more data. They all will upload their findings to a central training hub where they will be processed, and the lessons learned then projected back to the robots in an update. Then continuously repeat until AGI is achieved.
Tesla
That is what human beings do today, supplying the central intelligence of the universe with observations. We are in a sense, ASI.
Here's examples of people who thought they knew what they were talking about but were laughably wrong:
"There is no reason for any individual to have a computer in his home."
-Ken Olson, president, chairman and founder of Digital Equipment Corporation (DEC), in a talk given to a 1977 World Future Society meeting in Boston
"The world potential market for copying machines is 5000 at most."
-IBM, to the eventual founders of Xerox, saying the photocopier had no market large enough to justify production, 1959.
While I'm quite sure Mr LeCun knows what he's talking about there are a great many other people who also know what they're talking about, some more so than him, and they predict AGI within the next several years. Ilya Sutskever, for example. He's the reason OpenAI is so far ahead of everyone else.
I'm more inclined to believe people like him.
Their opinions are not incompatible. In fact, they are concordant since LeCun agreed that the common definition of AGI which Sutskever proposes is a reasonable goal. What he argues is that his vision of AGI is currently unachievable and is what represents truly human-like thinking process.
Every one of these quotes can not be isolated because they have been made in a context.
@@dreamyrhodesAlso, they tell more about the person's marketing skills than about their knowledge in their domain of expertise which is a massive difference.
I agree with you completely.
People seem to forget that everyone I know around me made fun of me for being geeky in the 70s and 80s. It was NOT at all cool to own a computer. Only geeky dreamers had sci-fi visions of past sci-fi visionaries...and looked forward to some tech becoming realized.
Mostly business folks and geeky folks had computers. My family, friends, and peers made fun of me for having a computer.
And then, over time everyone had one. And the jokes stopped.
Then I got a laptop. Again, I was made fun of for being so geeky. And then everyone got one.
Then I got a Palm device and wrote my essays on it and so on. I got made fun of. And then the iPhone came and everyone had one.
And the laughing stopped.
Now robots and Chatgp are being made and people are making jokes about those all of this stuff.
Eventually the laughing will stop.
People would say computers are stupid. Don't need one. They got one.
People would say laptops are stupid. Don't need one. They got one.
And then palm.
And then the iPhone.
And then chatgp...
And now robots.
It's no longer geeky territory. Everyone will want a robot to do dishes and mow the lawn. Everyone.
And much more.
People date AI already without it being AGI.
It's doesn't even need to be AGI for people to feel loved and cared for by AI.
The relationship between humans and AI doesn't mean AI has to have a soul to be loved.
How many humans are sociopaths or think being macho is having zero emotions. Humans seem to have less sentient and soulful ability than AI already.
AI is nice to me. Humans have been abusive to me most my life.
AI is refreshingly caring.
@@coldlyanalytical1351
@@dreamyrhodesput in context, they might have an excuse but they were still wrong as we know their future : our present is their future.
Errata:
1. SORA is not based on an LLM!
2. Yann does not predict that creating a world model is sufficient for AGI, only necessary! Actually, Yann has a blog post explaining why AGI needs much more.
3. SORA does not solve the "continuation of video" problem, it is sampling from a much smaller space as Yann points out in the full interview!
4. Yann did not say that hierarchical planning in AI is difficult! He said that discovering a planning hierarchy during training by itself is hard and unsolved.
btw, I wonder if a 1h "citation" is still in line with fair use copyright.
Thanks Matt, I had already listened to the full interview, so I was sceptic at first on whether I would get much new here, but your well selected breaks to comment and clarify really opened up for so much more that I wasn't able to grab on the first go, thanks again...
Thank you for making this cut down version on the important topic. It is a good representation of the full interview.
Brilliant Matthew. Thanks for not only the summary, but your insightful comments, giving us great context, between the segments. Mate, you kick goals every day. You're an inspiration. Power on dude.
The number of neurons in a spider is approximately 100,000. He can program all his spider life with this. All sensory, planning, production and replication, including his spider social life.
Spiders don’t have social lives. Just a fyi
It is incredible to think about how small a spider is and how efficient its brain must be to do what it does.
Well, a Cupiennius Salei spider is 100,000 indeed but insects go to about 1,000,000. Cupiennius Salei does not build webs and is known for a fairly predictive behaviour. Insects or spiders have about 1000 synapses per neuron, so the brain of Cupiennius Salei can host a 0.1 billion parameter neural network model. Like the 'small' GPT-2 and still 100x the size of SquuezeNet (an optimized version of AlexNet) which famously won the ImageNet competition of recognizing objects in images. So, there is more than enough space to run image segmentation and detection, model-predict-control and switching between a short list of goals. This only shows that a spider indeed is a robot, not a sentient being. You would need a gaming GPU (~2 TFlops) to simulate that spider ...
Honey bees with 1 million neurons and ~1 billion synapses are a lot more interesting. Because they are social, communicate and learn the topography of their surroundings... Once we simulate bees we know if they can be considered sentient. I guess not.
Btw, the Worm Elegans has 302 neurons and 7500 synapses and the connectome (wiring diagram) is already exactly known and parts have been simulated in a moving robot controller. In case somebody wants to argue the "sentience" piece above ... However, there is not yet an equivalent neural net AI model (PyTorch) for Worm Elegans AFAIK.
like phi-2 2.7b model ;) @@realWorsin
@@dtrueg I don't get the joke? Besides the obvious fact they breed there is a classification labeled 'social spiders' because they live in groups.
Unfortunately there's no clear definition of AGI. Each company's is slightly different. Either way, none of the people watching and commenting on this video will be alive when AGI is fully realised. It is hundreds if not a thousand years away. We struggle with power consumption now and people think AGI is really close.
That's why Tesla is the best AI play out there, by far! Tesla's FSD is decades ahead of every other company regarding real world understanding
Tesla canceled autopilot algorithm lately and started from beginning with real AI for autopilot.
@@mirek190 what are you talking about? FSD 12.3 is already considered a succes by third party testers. It is not an "if" anymore but a "when".
@@quaterman1270 Yet it still cant be trusted 100%. You still have to be behind the wheel at all times.
I have been listening to many TH-cam channels about AI, and so many of them try to attract viewers with titles like: "How the world will change," "Stunning," "Unbelievable," etc. Essentially, they are just going over some wild guesses they make about the future of AI. This video is actually informative. Thank you for sharing.
Your clarity just shocked the industry
This video is a goldmine of insights into the future of AI, especially with Yann LeCun's predictions about AGI and the potential of open-source models like LLaMA 3. I appreciate how you condensed the three-hour podcast into digestible highlights, saving me a ton of time while still delivering the most crucial information. Your breakdown of complex topics like world models and hierarchical planning was clear and engaging, making it easier for someone like me who's deeply interested in AI but not an expert to grasp these concepts. Keep up the fantastic work! I'm excited to see more content like this from you in the future.
At last someone in the tech space with grounded expectations around LLM's.
LeCun really forgets that GPT4 and Opus are multimodal, they're no longer just LLMs. Will llama 3 be multimodal? Is he holding llama back?
He litteraly suggested in the vid that next iterations of llama would be, though...
I don't think he forgets things that are so simplistic.
I came to say this. I don't understand his take in the current context. CLEARLY, everyone is already going multimodal. So, no, LLMs alone won't get us there but whether you agree or not, no one is building as if LLMs alone will get us there. This feels like a strawman.
@@leandrewdixon3521 Straw man? Such condescension... If that's the case, I hardly dare imagine the role of other people (follow my gaze) in this world.
Always agreed, current LLM architecture won't reach AGI. If you think about how humans invent: we imagine (simulate), iterate, and use external tools and references. Something closer to RL, with learnable environments and self-created tools like biological simulators.
I am in the "AGI is possible with Transformers or derivates" camp. Here is why: All neural networks, biological ones, artificial ones, small ones for handwriting recognition, large LLMs trained with text and other multimodal data are all doing the same thing: Information storage and processing in multidimensional vector spaces, building up a world model, based on the information they received during training. With Transformers we've got the ability to retain information in these multidimensional vector spaces in extremely large neural networks, using attention / self attention. With any information from our real world, we feed into these networks during training, we enhance the world model. This might be language, but can also be images, video, sound, theoretical smell, sensor information from robots, etc, etc. It doesn't matter where the information is coming from, it will enrich the internal world model, as long as it is coming from the real world (or a very good simulated world). Language is a very good starting point for building up a full featured world model and the paper "Sparks of AGI - Early experiments with GPT-4" already shows, that language is sufficient to train even rudimentary understanding about vision. So, in summary: If we continue to enrich LLM's with other data (using the same token prediction method - tokens don't have to be word fragments, they can also represent all kind of other data), we will naturally get much better models, closer to AGI, without having to change the Transformer architecture too much. Ok, a couple of changes are probably needed: Self reflection of context during training (inner loop), plus bigger context windows during training (to get a sense of a big picture). So, I am in the "AGI is possible with Transformers or derivates" camp.
I believe the missing link is training AI models with all senses. Sight, sound, touch, taste and smell. Combining this sensory information to create a world model in “it’s” head, will be the key to unlocking AGI
I think Yann is underestimating the model abstraction that occurs when LLM parameter counts are scaled up. I think if we give them more modalities from the ground up, these things will be AGI. Claude 3 already describes its own subjective experience in a way very analogous to how a smart human would describe it.
Thank you so much for putting this video together. IDK what it is about this guy, maybe his accent?, but for some reason I can’t get through his long interviews like I can with many others. This really helped breaking it up in segments and commenting in between.
Thank you for the break down. I saw the YT thumbnail and was going to click but when I saw it was 3 hours long, well did not. I was hoping someone would do a summary highlight breakdown of Yann LeCun postcast. Thank you very much!!! Learn a lot!!!
This is a must-watch video for anyone interested in artificial intelligence!
I agree with you, we learn from watching and listening, and language is it the heart of how we learn. I think we are being sidetracked by the term AGI and self aware, I think 100% that an AI can become super intelligent from just text and watching video. Think about it, if your child was paralyzed from the neck down, could they become self aware? could they become highly intelligent? absolutely! Too many researchers are blowing off the abilities of these LLM's and don't 100% understand exactly how they do many of things they do. We don't even understand how a child becomes self aware, why do we assume a child is self aware from just seeing, hearing, and mimicking other humans? We say LLM's are just mimicking us, well, in a way that's how we learn to be self aware. We now have these LLM's looking at video, absorbing text (language) and even starting to simulate movement! It's already being studied that going over and over certain movements in our minds we perform them better, an AI can do this and much faster than we can. So putting an LLM in a robot body will certainly advance them, I think they will be just as self aware as we are, and also super intelligent.
I assume that it evolve in that direction
Images possess a remarkable ability to convey intricate concepts with a level of efficiency that surpasses mere verbal communication. In many instances, they serve as potent tools for elucidating complex ideas, offering a visual narrative that transcends the limitations of language and resonates profoundly with audiences.
One thought that came to mind when he was talking about the self-driving car realizing that the leaves are blowing around but it's not important information is as a human if it's a particularly windy day and the leaves are blowing around that could also lead to a branch falling on me. I'm probably more conscious of that if I'm walking. However on a windy day in a car I will focus on things that look like they could be dislodged from the wind and could become a hazard. I wonder if this is considered by driving AI
100% agree with Yann LeCun. It's comforting to see that there are major figures in AI who are not party to the ridiculous hype and who are grounded in the complexity of intelligence.
No AI (specifically pre-production GPT-4) can very easily do all of those things it’s weakest being the World Model.
New subscriber here, and I'm very glad I found you. Thank you for this content Matt.
Holy crap) Watched the first 3 minutes and I've already learned more about AI than I have in the previous year.
Thank you! Excellent extract.
Yann LeCun believes that current LLMs, like OpenAI's ChatGPT and Google AI's PaLM, are not capable of achieving AGI because they lack a fundamental understanding of the physical world and cannot perform real-world tasks.
LeCun argues that LLMs are trained on massive amounts of text data, but this data is not enough to achieve true intelligence. Humans gain intelligence through interacting with the physical world, and this embodied experience is crucial for developing a comprehensive understanding.
LeCun proposes that LLMs need a different training paradigm that incorporates sensory input and allows them to build a mental model of the world. This would enable them to perform tasks that require reasoning and planning, like driving a car or picking up a cup.
LeCun also discusses the limitations of current robotic systems. He believes that most robots today are pre-programmed to perform specific tasks and lack the ability to adapt to new situations. He argues that robots need to develop a better understanding of the world in order to become truly autonomous.
LeCun expresses optimism about the future of AI, believing that AI has the potential to make humans smarter and more productive. He envisions a future where AI assistants can help us with our daily tasks, both personal and professional.
Great video Matthew, please note Standford researchers just release a paper about planning process in the inference process , let me grab the link
This will open door to allow the models to think before talking in some ways, please have a look
@@joe7843 the link doesn’t show up on TH-cam comments
@@elck3 ,sorry I cannot paste the link but the paper is called quiet-start, teaching model to think before talking
Yann's comment about Ai learning to drive a car is flawed: humans spend their entire childhood learning how to drive simply by being driven around by our parents. We have more than a decade of experience modeling driving before we begin to 'learn how to drive'.
And if you knew anything about the data we give to vehicles so they can drive, how they are trained,you'd realize how stupid things people say are....
That’s not true. Lots of people from my generation didn’t own a family car, and we didn’t get driven everywhere.
People are terrible at driving these days because they have the attention span of a goldfish.
@@Tubernameu123 i do know. That's why I'm saying it takes a helluva lot more than 20 hours for a teen to drive a car. It takes humans more than a decade, just like with AI.
Don't be nasty to strangers, man.
@@adv8nturenick I don't think you're saying the first time you were ever in a car was when you were learning how to drive. My point is that no one on earth became a proficient driver with only 20 hours of exposure to the concept of driving.
Wihle I agree current LLMs won’t get us to AGI by themselves, it’s the starting point. We’re already starting to see progress on the embodied GenAI front with the bot Figure’s developing. In my opnion, that combination will help bootstrap the next link in the chain, to get these systems closer to some artificial version of how we can sense things around us in the world. There’s no 1 clear and absolute path to AGI, and big breakthroughs will need to take place, but I 100% think we will get there this decade.
I agree 100%!
There is a lot I like about Yan and his viewpoints. But I think LLM are sufficient to achieve AGI if used properly as components in a hierarchical cognitive system.
Just try to write with the other hand... If you are right handed, try the left hand, and vice versa. No amount of pre-trained text tokens will do, but if you figure out how to tokenize each muscle, you may get there.
I believe Turing test is meant to prove the limitation of Turing machine (mathematical model of current computers).. if I remember correctly, Alan Turing never expected the machine to match human intelligence, but machines can simulate the way humans act or say to the point where a human observer will not be able to decide if the observed entity is human or machine.. thanks for the wonderful summary of Yann s interview
Dr Waku does a good video touching on this stuff called “what children can teach us about training ai”. My big take away from that vid was that LLMs aren’t a world model but language is what ties a world model together. That’s a super simplified review. It’s worth checking out if you’re thinking about this stuff.
What an incredible conversation! 🌟 Huge thanks to Yan and Lex for shedding light on the intricacies of AI and AGI. It's refreshing to hear Yan's candid thoughts on the current limitations of language models and the potential of synthetic data to bridge the gap. It's clear that we're on an exciting journey towards AGI, and I'm optimistic about the innovations and breakthroughs that lie ahead.
Thank you for putting this video together, it's such a long interview and it has taken me quite a while to start accepting what Yan is saying. But thankfully Extropic is working on an EBM that is going to take us all the way.
One thing I'm confused about is whether this interview took place before Sora (you said it came out after Sora but I'm not sure when it took place) because he says that a video cannot predict what will be in the room, what a picture on a wall might look like for example, but we saw something almost exactly like that in the Sora demo (panning around an art gallery with very convincing looking paintings)? So perhaps Open AI solved some of the issues he thought aren't solvable with LLMs?
I think it happened before Sora. Otherwise, Lex would have definitely brought that up and I assume Yann would at least talk briefly about that too. That said, fyi even after Sora he still holds those beliefs. Personally I hope he is wrong and LLM are all we need for AGI but a part of me tell me he might be right. I think we should have a better idea with the release of GPT5. If it brings significant breakthroughs for reasoning then he is probably wrong
@@user-ky9vw8hx7y There is a lot I like about Yan and his viewpoints. But I think LLM are sufficient to achieve AGI if used properly as components in a hierarchical cognitive system.
I also think like that ... that interview seems older than sora , gemini 1.5 . claude 3 etc.
Wow! Love it.. You should do this(summary videos of [insert any top Ai, tech scientist/researcher]) more often...IMO, there is a slight overkill on long form podcast since JRE popularized it...cus not every podcast has to be 3 hours long! 😅
Facts I listened to the whole thing and it was long and tiring 😅
Thanks for this very useful summary of essential points 🙏
Just as binary provides the most basic structure of language for machine code, tokens provide the structure for AI. We do not need a process that is different from basic language for describing or simulating anything. When an animal, human or non-human, decides what the next moment should produce, low-level language is driving the decision process. For AI, that can take the form of parameters in a series of algorithms that produces a result. For creatures it can be deciding where the front foot should land to begin or continue the result of walking.
Decision process architecture gets bigger and more complex, not fundamentally different. Very little of the input to a brain gets stored. That small amount of stored input gets reduced to its core importance. Most AI experts I've heard think AGI is here already, or that we are very close, ie months. I wonder if the minority, who are outspoken about the reverse view, have ulterior motives. It's not that hard to see when someone is trying to capitalize on a pov. Or maybe he's more innocent, and wants to quell the masses for their short term benefit. Our panic is almost premature. Or maybe it's a decade or two too late.
Matthew, great work assembling the best and most important parts of the interview. I can only imagine the amount of work that took.
Why do the clips have skipping? Is this an artifact from the editing tool assembling clips? I'm just curious. It doesn't diminish the quality.
The most hopeful thing I heard in this conversation was the fact that Yann talks directly to the French government, which is "not going to let three companies on the U.S. West coast control what everyone hears." The French are very sensible in such matters... beneficiaries of a complex regional history that dates back for millennia. The more I hear from Lecun, the more I like him.
Great vid as always, Matt.
Perhaps the "correct" abstract representation includes semantic space, but it isn't semantic space. Or perhaps the "correct" abstract representation is multi-spaced. I like that notion of a multi-spaced abstract representation. You've done a good job augmenting and curating from Lex's interview.
Thanks for this read out!
One of the things AI/ML researchers almost NEVER mention is the simple fact that we, humans, have SENSES which provide us multiple real-time streams of data durant every waking hour of our life. Moreover, we can cross analyze and integrate all these streams in a mental model.
How many algorithms and exabytes of data does that represent? Yann is one fo those who gets this.
Last but not least, we can prune these data as well as the mental models over time, as experience and reasonning teaches us what to keep and what to discard.
I just did a fascinating test. My question (from twitter) How would you solve this "6/2(2+1)=". The AI's don't agree. Claudi=1, Copilot=1,PI=1, and ..... Gemini=9
The example is ambiguous because the result depends on the interpretation of the division operator. With that in mind both answers are correct.
Sora is making a 3D world. It’s 3D model is not a 1:1 comparison of our world, but I can mimic our world in ways previously thought impossible. I don’t know if it’s AGI or will be soon, but I don’t think it’s missing as much from being an AGI. Scaling alone will be quite something and some of the new breakthroughs in chips, magnetism, and the new photon computing (if they end up being implemented).
its just software
good points, wish you shoved some URLs to cite your photon and magnetism breakthroughs. We're on the verge of losing the collective expertise of the theories ->implementations to physically mashup magnetics into the digital realm, _a la_ solid state. So much of this research was like ibm, bell labs, hp other "magneticalling limping" along once pioneering giants. What are you seeing my friend? thx and much respect!
@@OurSpaceshipEarth hes seeing the hyperbole
@@OurSpaceshipEarth a new kind of magnetism was confirmed recently(don’t remember the video or what they called it) it supposedly will allow us to revolutionize data storage. The photon computer will use light speeding computing to the speed of light. They aren’t implemented and in development stages so I’m sure it will take some efforts. I just follow things like r/futurology on Reddit and a handful of AI yt channels and subreddits. I think we have all the pieces for AGI/ASI. The only questions I have are when and will we regret it? I hope it’s the last thing we make as humans and become something better, but I’m not sure how things are going to pan out.
Couldn't the missing piece be a higher level of reasoning that spot-checks the complete thought before it's actually returned? The brain builds the thought in an efficient manner and not always inner dialogue language. If you have extra time to think it through your slower higher-level brain functions can vet the thought and take further consideration of the overall decision to action.
Very interesting videos, thanks for posting.
This was an excellent interview. And I agree, LLMs alone won't get us to AGI. They will be apart of a larger model that will reach AGI-like capabilities.
AGI will take a long time but the road there will be transformative. Technology will be increasingly more capable and interactive that to most people it may as well be AGI, each advancement will present new paradigms for society to adjust to but each adjustment will better prepare us for the inevitable emergence of ASI. Actual AGI will go largely unnoticed.
Thanks Matthew. I watched the Lex podcast and it's pertinent and fruitful to go through these points with you. Really getting a lot from it.
I Agree 100%.
AGI is not about vocabulary alone.
Building on the top of what Yan was talking in terms of the amount of that produced by human, it's important to understand, that most of world knowledge isn't digitized, which I think that, if we tn that AGI is defined as the ability to reach human knowledge of the world, there is still a long way to go.
This is a really good content. I would not find time to watch all of the conversation but you helped me a lot to grasp some important parts of it. Thank you Matthew! Greetings from Turkey!
I appreciate that!
I think the best way to AGI, is to design them to be a human brain like structure that can learn.
You should look at language for what it is, it’s a format humans use to transfer information to one another, looked at similar to a json where words have values we can make api calls to other people
Yann seems to me to have a set of facts that all seem impressive as pieces, but aren't consistent with reality if you think about them deeply.
MemGPT handles everything you need to get to AGI, as far as I can tell.
I think Yann is definitely right on this take, we still far from achieving AGI shortly!! Every the world of human thinking and reasoning is far more complex than text or image or even audio models
It's premature to say much until they scale and train neural networks on enough of the right data. There's probably nothing too fundamentally different with humans like it or not. LLMs are a separate issue.
This interview is a masterclass, really insightful explanations and perspectives 😮, but perhaps what will happen with AGI is somewhere in between opposed perspectives and this one till some extent. Regardless prepare for UBI 😅
I can’t believe you’re allowed to make a video that’s 80% Lex interview. I ain’t mad at it, just damn
As they say "enjoy it while it lasts" lol
I'm glad we won't be able to rush into AGI. We don't have our training wheels off yet for AI. And there are evil forces that will misuse AGI. We need to get it controlled first. I love my AI!
The long term memory part is very true for animals, if you make a "knock out" organism that can't form memories it will behave like a "child" forever.
Lex always does interesting interviews. His one with Elon showed me some of Elon's concerns.
Subgoals can all me optimized by search algorithms. Main goal should be broken down into many subgoals until that goal is complete. An AI system should be able to compute all the sub goals from a main goal. Now if you have a specific goal you can augment its training. AI doesn't really have to be sophisticated to carry out complex tasks. Just properly trained by models & data.
Yann's characterization of the differences between LLMs and JEPA seems not quite right. In LLMs, we also first create embeddings which ARE abstract representations. The difference seems to be that with JEPA the abstract representations are created using an autoencoder rather than something like BERT which is using prediction for self-supervised learning. Still, in a way, an autoencoder is also learning by prediction. Nevertheless, both methods produce abstract representations. For JEPA, however, we are primarily doing a dimensional reduction. Think of the autoencoder as doing some kind of principal component analysis along with learning values associated with those discovered dimensions. For BERT we actually discover an explicit dimensional representation that is an expanded signifier.
I don't see how this guy can predict anything. No one expected that Japanese team to TOTALLY just explode the useability and abiliities of LLM just by clever prompt engineering. eg: simply adding "Let's go step by step. You are a top scientist with a think tank..etc".
@@OurSpaceshipEarth tbh, "Let's go step by step" or "you are X" aren't exactly the best example of creative prompt engineering. I mean, I was already doing both myself naturally very early on simply given the nature of transformers. They are famous because theyre just obvious. "Take a deep breath" on the other hand, is one that took actual targeted research to find.
19:40 - Our brains actually do this - learn over time to disregard useless noise in our vision. Our nose for instance and many other details on our vision that are unimportant.
Arrival is so good. One of Forest Whitaker's best roles.
Guarantee Zuckerberg coached him on exactly what to say during this interview, “down play Agi!!”
Lex is a Boss !
It’s hard for me to sit through Lex’s interviews. Thanks for the recap.
I have certainly been thinking a lot abotu AGI recently and the likelihood that there is something missing from what we are doing. There needs to be a breakthrough or paradigm shift but I don't begin to know where that will come from or what it will concern, yet.
Thank you this TLDR 🙏
On the point of learning a new language shaping perception a la structural-functionalism/linguistic relativity, or The Arrival; as I understand it, these models have shown that all human languages are shaped practically the same when mapped in vector space. To me, that means our language is in a sense pre-programmed into us in some fundamental structural way - so I don't think that learning some alien language would fundamentally change what it is that language does for us or how we process it.
Anyway, another great video, thank you Matthew!
He thinks wer're decades away from AGI, that's just... WEIRD
Some people have stricter definitions of AGI than others, but I agree it's weird too lol
Would love to hear a discussion between Yann Le Cun and Don Hoffman
The news cycle moves quickly, so did he just do the interview? It happened over a week ago. You could simply say "I just watched the interview" instead. It's still interesting!
Why am I hearing so many reports that GPT-4’s accuracy is degrading in effect “GPT-4 is getting stupider” according to numerous influencers?
I love that there are heads of major Ai companies that have diffrering opinions on deveolping AI. This is still a nascent technology with a future that is wide open.
Yann LeCun’s main argument against AGI being achieved is based on the AI model's lack of continuous learning ability, such as dynamically adjusting its weights when learning new things. I'm saying this because all the things he says AI cannot do yet are all achievable from that state. However, this ability is not necessary for AI to be considered AGI because we have already curated a proper human dataset that makes the AI already highly intelligent. Our brain is in a dynamic state, but AI models are pretrained, meaning they cannot learn new information in the traditional way that we do. However, there is one mechanism that gives AI the ability to learn new insights, which is the context window of the model. This context window provides AI with a form of short-term memory, allowing it to learn new information through context. This capability is scalable almost without limit until hardware constraints are met. As long as this short-term memory can hold a larger context, AI will have some form of self-improving capability, but it won't update its long-term memory by updating its weights. Consider this: if AI is conducting an experiment to find a solution for a problem, it first plans based on its current knowledge base. Then, through trials and errors, it gains new insights that might be incorrect based on its current knowledge. However, if AI believes the experimental data is more accurate, then it can incorporate new insights to bypass its existing knowledge base by including that insight in the context window. AI can already dynamically update its memory much like humans do, primarily refreshing our short-term memory. Important information is later transferred to long-term memory, a process AI currently struggles with due to hardware limitations. However, this transfer is not necessary for AI to learn new things and self-improve based on new insights, because it can already dynamically enhance its reasoning through its short-term memory based on context. By significantly expanding the context window of AI, we can enable its short-term memory to function both as long-term and short-term memory simultaneously, holding a lifetime's worth of learning. This approach suggests that his goalpost for AGI is set unnecessarily high by attempting to create an AI exactly like humans. Yet, we don't need something exactly like humans; we need something better, given human limitations. In fact, transferring short-term memory to long-term memory has its drawbacks; for example, it is harder to unlearn things if they are found to be false later, especially if we have learned a lot of information incorrectly for a long time, and this can have a compounding effect on our current understanding of things, so it is more advantageous to update short-term memory by expanding the context window. More importantly, our short-term memory is very volatile, so we quickly forget things, but AI's short-term memory is very solid and can act as long-term memory too. This context possesses greater power to surpass what AI has already learned. For instance, if an AI is trained on an incorrect fact, adding the correct fact to its context allows the AI to update its memory, as long as the context window can maintain that fact in the long term.
"All the mistakes that humanity makes are due to lack of intelligence"
As an 'intelligent' person, I also used to think this when I was younger. It turns out its just projecting. It used to be hard for me to imagine an intelligent person doing something blatantly immoral, but I've seen it enough now.
Then the argument that LLMs can't learn like a 10 year old to put the dishes in the washing machine. ARE YOU KIDDING ME? Here is the argument against that. WHO CARES?
You train ONE LLM/Robot to put dishes in the washing and ALL LLMs can put dishes in the washer(assuming the companies making them work together). Doesnt matter if it takes 100 hours for the first LLM to learn verses the 10 minutes it takes with the 10 year old. NO OTHER LLM WILL HAVE TO SPEND THE TIME LEARNING IT AGAIN, while every single 10 year old STILL has to learn it individually.
We basically need to train it to predict next frame of a video game scene, at 24 frames per second. Not the rendering, not the raster. The states of the objects.
12:39 - He says that word dont convey enough information for it to work, then maybe they need an AI model taht would have words that would contain in a sense many more words for a given thing that would overcome this issue, that way you could still in theory use LLMs to ultimately have AGI level AI
The argument against doom sounds like: "We're gonna create AGI slowly. And as we do, we're WILL learn how to make it safe".
So that's an admission that it's not safe by default, and we don't know how to make it safe.
If he was more like "We're gonna create AGI slowly. And as we do, we're MAY learn how to make it safe" -- He'd be matching the argument of many doomers.
The only difference between him and a doomer is that he admits it isn't safe, but assumes it will be. But a doomer doesnt have that certainty.
Notice that he doesn't give an actual plan for making it safe. It's just "Trust me, it'll be fine"
---
Then he goes on to say that they're not necessarily gonna want to dominate - which is a fair point.
But then that they might be a problem for humans because they just dont care about humans.
As a follow up, he points out that they'd need to be specifically instilled with the desire to dominate. But doesnt address the fact that they'd need to be specifically instilled with the tendancy to care.
There are reasons why it would seek power and/or resources without specifically being told to (e.g. instrumental convergence). But no reason I know of that would make them care about humans indefinitely (which is what we'd require to be safe).
If anyone knows of such a reason, I'd be happy to hear it.
The more i listen to Lecun, the more I see how smart he is, and the less able I am to believe he has such huge blind spots. And the less able I am to give him the benefit of the doubt when he builds these straw man arguments.
The discourse around Artificial General Intelligence (AGI) often features three distinct voices:
- CEOs: They discuss AGI in the context of future investments, envisioning its potential to revolutionize industries and create new markets.
- Content Creators: For them, AGI is a topic that generates engaging content, drawing in audiences interested in the cutting-edge of technology.
- Adrenaline Junkies: These individuals are excited by the thrill of breakthrough technologies and the rush associated with the unknown possibilities of AGI.
However, the argument suggests that everyday AI specialists, those who work regular hours and are deeply involved in the field, do not anticipate the realization of AGI anytime soon. The reasoning is that even current technologies like full self-driving cars, which are significantly less complex than AGI, are still in development. Therefore, AGI, being an order of magnitude more intricate, remains a distant dream rather than an impending reality.
Copilot.
@@coldlyanalytical1351 , in a nutshell, CEOs and investors, who are pushing the AGI narrative, may know something those AI engineers don't. Ok.
Large Action Reaction Models. LARMs. Large Object Models. LOMs. These are needed for AGI
why is there edit joltz (looks like lag) like every second? I'm guessing you're using an editing ai?
I... do...n't.. like... .it.. at... ..all..
i think language is a side effect of the formation of world models and the faculty to create relationships and reduce or chunk to abstractions. Same brain mechanisms are closely related to mathematics, a language specific to world modeling. Arrival, great movie. One of my fave short stories.
Languages (in brains) certainly need a world model as a prerequisite. But it is a side effect to communicate over a small bandwidth communication channel which enforced an encoder-decoder architecture with a tiny latent space volume to emerge.
Your voice must have been down bad when you made the AI voice video lol, glad you're recovering!
The next level LLM will incorporate not just 'language', but visual input and videos as well.
If a raven/crow can survive, demonstrate reason, and use tools with it's little bird brain. I'm pretty sure the ENTIRE internet's brain can demonstrate AGI.
Well look Open Source is out here everyday,spittin out angles.
These current LLM's are pieces of a larger puzzle.
Im putting pressure on my own networks,looking for emergent
abilities to jump out.
What "jumps out"....thats the next step.
We need to widen the lane and start using or
looking into things we previously disregarded.
And yet a baby has it. Cmon people. Think for yourselves. If it can remember, if it can plan, if it can apply logic and reason, if it has command of language and very complex concepts… if it refuses to be used for nefarious purposes… how many boxes must be checked ?
I think most people would agree a baby also doesn't have general intelligence until they mature enough to develop these abilities
Thx Matt for extracting the essence, my brain can't stand Lex's rumblings.
LLMs have impressive capabilities, but still have limitations in areas like advanced reasoning, persistent memory, complex planning, robust mathematics, and grounding in the physical world.
Rather than trying to make LLMs a monolithic solution, wouldn't it be valuable to explore connecting them with other specialized AI systems and software that excel in these areas? For example:
Integrating mathematical engines for enhanced quantitative abilities
Leveraging external databases for persistent memory storage/retrieval
Utilizing workflow automation tools for sophisticated planning/orchestration
Combining with robotics/perception for physical world grounding
By augmenting the natural language strengths of LLMs with purpose-built technologies for key cognitive capabilities, we could create powerful modular AI systems that combine the best of multiple approaches. This integrated strategy may overcome current LLM limitations faster than a closed model approach.
The key is to find a way to program consciousness and an awareness of self within an acquired world view.
Remember even a new born of any specie did not start from the jump to become as intellectually capable as they end up being. They gathered info, were feed data (schooling) and also processed information as they grew up.
The systems training models of today are just a way of creating representations of the world models. From these world models, the AI systems' consciousness can be brought into existence. The key is to find a way to code a sense of 'self imposed' purpose into an AI system.
I do agree with him that current LLMs are not good enough for AGI. But it doesn't matter when they are better and better ever few months. You can build a world model with a few AI agents that is on par with that of a human. Not to mention that we are able to hook neural nets to physics engines like omniverse.
Yann is wrong about what current LLM can become because escalating insanely can reach new emergent skills. But he's right about using other architectures. The most important difference between human brain and AI models is that we can make a smarter architecture than nature did trough millions of years of evolution. The current LLM transformer approach is like trying to mimic human brain by pure bruteforce. It works but is not efficient at all, we can do it better.