Indeed, that chart must be one of the worst charts I could imagine for making any kind of argument about scaling. It's a flawed benchmark, and his "expected trend line" goes to 140% or so.
@@IvanBudiselic He also admitted he just drew on the lines himself instead of projecting from actual datapoints.. and he uses this as an argument for them slowing down. His other arguments don't stand because we don't have GPT-5 yet, he's just asserting GPT-4 turbo is supposed to be GPT-5 and not a distilled GPT-4. He also compared a bunch of LLM startups starting from nothing to GPT-4. So everything rests on him drawing a line in mspaint based off his feelings.
@@VoloBonja I doubt it will be hallucination free or close to AGI, but for 2 years in the making at the pace OpenAI innovates, I expect something much more than just CoT baked into responses for it's reasoning capabilities
He put up a graph of models since GPT-4 from other companies and startups that were starting from nothing and catching up to OpenAI and then asserted that this is proof that GPT-4 is the largest scale possible. Anyone with intellectual honestly in his position would have said "look, clearly I was wrong and GPT-3 had much more room to scale. i need to recalibrate if I don't want to be wrong again when GPT-5 comes out". Instead he comes up with these insane copes full of mental gymnastics and hand drawn graphs he created in mspaint.
He's now comparing GPT4T (a distilled model of GPT-4) to GPT-4 and expecting exponential improvements in what I guarantee is a prompting issue. This is embarrassing.
AI has room to scale, but the underlying design of the model is extremely important to using that scale. If they could throw 50T parameters at GPT-4 and get ASI they would do it. Current model types tend to get diminishing returns from more parameters, hence why the small versions are still decent despite being much smaller.
I get the skepticism he heavily leans into, but I do also think that even 4o/claude models are far more superior in comparison to previous smaller-breadth models. I think they're (esp 4o) currently handicapped to keep an even keel with the marketspace. At least from a business standpoint, there would be no point issuing frontier models/capability as they're produced. Yeah it would be good for all of us, but with capitalism it probably will never happen that way.
@@jacobnunya808 Small models are horrible by default. Go back to llama 1 and you won't be able to use it for anything practical. Scale isn't all you need to achieve AGI either, but it's all we needed to achieve the trivial tasks gary marcus mentioned in the past. He needs to admit his mistakes is he wants to be taken seriously as a critic.
The most ironic thing in the world, is Gary Marcus pointing out all the errors made my A.I. systems, then proceeding to draw a line that goes above 100% on a benchmark graph. I guess he just disproved his own general intelligence...
I wouldn't trust employees who can't consistently look up the answers to simple questions. I wouldn't trust an employee who can multiply/divide/compute basic math consistently. I wouldn't trust an employee that writes broken, untested code but claims it is working. I wouldn't trust an employee who makes up information that misleads stakeholders and customers. I wouldn't trust an employee that confabulates information rather than saying "I don't know". --- This is the current state of LLMs, so unless they fix these issues the tech will fizzle out and, at best, become just another doo-dad embedded in apps to make your life 5% easier.
Language models as part of agentic workflows can extract entities and relationships from text such as transcripts from conversations, and then merge this knowledge into graph representations. People can select parts of their conversations that they want to merge into a global shared graph representation. A global platform can be scaled up to merge selected parts of millions of simultaneous conversations with people around the world into a shared world model, a collective human and digital intelligence, by the end of this year. Global collective intelligence would be of immense value even if it's not AGI.
There are papers showing that LLMs cannot accurately extract entities and relationships from text. If you ask an LLM to extract the entities in the text "I travelled to Barcelona to watch them play Real Madrid", it will extract the sentence subject, the city of Barcelona, and the football club Real Madrid, but it will fail to extract the 'FC Barcelona' entity that is implicit in the text.
@@dungeon_architectGood point. That's a good reason to keep the human in the loop, to catch and correct those kinds of mistakes. We are talking about collective human and digital intelligence.
@@dungeon_architect That's interesting. Can you provide a reference to this paper? Maybe it's an issue of prompting, or they were testing older models. GPT4o seems to have no issue: "Hi GPT4, Consider the following sentence: "I travelled to Barcelona to watch them play Real Madrid" Can you list all the entities that are being referred to in this sentence, either explicitly or implicitly?" GPT4o: Sure! Let's break down the sentence and identify the entities being referred to, either explicitly or implicitly: 1. **"I"** - The speaker or writer, an individual traveling to Barcelona. 2. **"Barcelona"** - A city in Spain, which is the destination of the speaker. 3. **"them"** - This refers to a specific team, implicitly understood as FC Barcelona, given the context of playing Real Madrid. 4. **"Real Madrid"** - A professional football team from Madrid, Spain. Implicitly, there are some additional references: - **FC Barcelona** - Though not directly mentioned by name, "them" refers to FC Barcelona as the opposing team of Real Madrid. - **A football/soccer match** - The context of the sentence suggests that the speaker is referring to a football match between FC Barcelona and Real Madrid. - **Football stadium** - The match would implicitly take place in a football stadium, possibly referring to the Camp Nou in Barcelona. So the entities are: 1. The speaker ("I") 2. Barcelona (the city) 3. FC Barcelona (implied by "them") 4. Real Madrid (the team) 5. The football match (implicit) 6. The football stadium (implicit, possibly Camp Nou)
Couldn't seem to reproduce what @@dungeon_architect was talking about in GPT-3.5, 4, 4o or llama 3.1. Tried switching up the names to test the abstraction in case it was overfitting on the test, still nothing. Wouldn't mind betting it's a skill issue.
Does anyone know if there's a presentation of Ben Goertzel available? I'm following his work from ~2005, and I like what he did with OpenCog architecture.
9:53 "these things are not actually physical laws like laws of the universe they're just generalizations we've seen for a while they don't necessarily hold” There are two very strange things about this statement. 1. So, while AGI itself hasn't arisen from scaling between 2021 and 2024, the scaling “laws” have definitely held. Even if there’s no “law” that says the scaling trends will “necessarily hold,” there’s yet to be any evidence that says they won’t. There is instead lots of evidence pointing towards a continued trend where larger models trained on more data better predict that data - along predictable, consistent curves. 2. This statement is quite ironic. There aren’t actually “physical laws of the universe” because the universe doesn’t “follow” laws. All known “laws of physics” are simply models we’ve developed and refined over time to match our observations of reality. Many “known” “laws” have been wrong for different reasons, and many break down at different scales. *Both* scaling laws and physical laws are "just generalizations we've seen for a while" that won't "necessarily hold."
1. the total amount of human produced data on the internet up until now is finite, and it doesn't increase exponentially, at least not at a rate that can support scaling laws; training a model on 10x more data costs ~10x more money. perhaps "scaling laws" would continue indefinitely if we had near infinite data and resources. but we don't. 2. universe does follow rules. Otherwise, there'd be only chaos, or nothing at all. The only thing is, our understanding of the law of the universe has always been incomplete.
@@sehbanomer8151 1. Who told you that Deep Learning models can only be trained on data on the internet? The data on the internet is the bootstrapping phase as we embody AI's eventually they will train on data from the natural world.
Jfc the AI Overviews thing. There's a snopes article debunking this. Anti-AI bros created a template for people to type in whatever they wanted to pretend it said. The LLM wasn't even generating the answers in the first place. It was just selecting a snippet from pages that were poorly ranked like they always have been. The only thing the LLM did was reword it.
Or consider this: he was paid by some people to (the masses are experiencing FUD) suppress the trajectory since governments are legislating against AI, preventing development. They need to compete with China (and don't have "annoying" legislators) to achieve AGI.
Alternative title for the video: "the guy who spelled the last decade prediction that Deep learning has hit a wall, but this time for sure, is predicting that AI has hit a wall now"
There's this thing with these people (and this goes for all AI "gurus"), that once they've made their view public, and especially wrote books about and and their entire career depends on this initial view being right, they will do anything to find examples supporting their view and anything to dismiss evidence against it. I tried all his examples of LLM failures, and ChatGPT handles them perfectly well, so they're cherry picked by him from a time in the past or a worse competitor. When you make arguments against AI, you should do so against the state of the art, anything else makes no sense. Second, OpenAI and others almost certainly have (or could train) much better models in-house, but the inference is just not economical at this point. Most work in the past year has gone towards optimizing GPT4 level models so that you can actually deploy them, that doesn't mean you can't improve on it. The same goes for grounding etc, it's just a matter of time before models are not only trained on all the text on the internet but all the videos and images too, which gives it another level of grounding. Just because it's not economical yet doesn't mean it won't work. Same goes for world models (where he clearly doesn't understand LeCun's work on JEPA). There is just so much left to do, it's way too early to call a slow down. He's a complete clown.
Your comment would be very witty... if not for all the scientific evidence that Deep Learning has indeed hit a wall (and a lot of the "progress" is smoke and mirrors provided by extensive RLHF bootstrapping and clunky "multi-modal" architectures that leverage more traditional algorithms.) Deep Learning can still be immensely valuable and relevant without making silly over-claims and declaring itself the holy grail of AI.
I just realized that this talk is a month old and OpenAI o1 model is roughly 3 weeks old... Well thankfully, it only took a week for OpenAI to respond to Gary Marcus with their new reasoning model. Pretty cool
There may be too much money being spent on scaling LLMs, but agents are getting more and more development money. 2024 has been the name of nascent agents. 2025 will be the year when agents really enter the workplace filling jobs in place of people.
A wonderful talk. Pointing out a lot of the bullshit behind some AI hype (like the blind faith on scaling up everything to achieve AGI). We really need hybrid systems and start building cognitive models of the world.
As a hobbyist programmer and mathematics enthusiast I don’t think a machine learning model making mistake is bad it’s normal . Remember it’s a statistical distribution output. To the model right answer is just as valid as wrong answers and what we need to do instead is to have a layer of logic that can apply some sort of light logic to its output to filter it through a real life lense. I don’t agree with shitting on AI and I also don’t agree with companies overhyping a statistical distribution machine . So far these models have some use as customer service bot trained on company info and maybe some stuff outside of that . But I don’t think it’s ready for more real life stuff
I broadly agree with your statement, though the use cases for the more recent models goes way beyond just chat bots. The problem is that they are being oversold to some degree and under explained. They may also have been needlessly sanitised for the sake of marketability. As for misinformation spreading - yes if the consumers of the information have useless filtration systems in place already, which is not really the fault of AI/ML developers.
Yep - these things are working as designed. But the sellers of them are making absurd claims that they are "conscious" - "miniAGI" - "capable of reasoning" etc. Purely coincidentally (I am sure) they stand to make immense amounts of money from these claims.
A sane argument in a sea of hype. Yes there are some niche usees but the market is pumping money in expecting we are going to have personal robo super intelligent terminators in a few years. Very little practical reliable use of this stuff the bubble will burst. I also studied this stuff in grad school, it's neat but we won't have full self driving, thinking machines or displace that many workers over it.
Gary is right. People is comments listen to too many podcasts where AGI is already here and Altman is a king. Human level AGI is coming any time soon, not 5, not 10, not 20 years. We needto work harder to make it happen. And it's not about scaling, it's about new explanations
Gary is not right. He’s been wrong about most things and will continue to be proven wrong going forward. That has nothing to do with AGI being here or Altman being king. It’s just an empirical observation about his track record and how many things he’s been wrong about.
@@VoloBonja If you want people to spend time giving you proper arguments, maybe you should put time into your own arguments and not just say "Gary is right" and make baseless assertions like "it's not happening anytime soon" with zero evidence. We have no idea if LLM performance is slowing down until the next generation of models comes out i.e. GPT-5. Gary is making disingenuous comparisons between models that don't represent the same step change, for example GPT3 -> 4 vs GPT4 -> 4o. Also, at 26:40 he makes an extremely silly error showing benchmark performance as a percent and claims that there's slow down because the performance gains are getting smaller on the graph as they get closer to 100%... Duh... that's how percentages work. You can't get past 100%. It's asymptotic. 10% to 90% looks huge on the graph and 99% to 99.9% looks tiny, despite the latter change being larger. Even worse, he has a hand drawn line that projects to 140% which is what he expects without slow down... he notices this error at 26:40. You should re-watch this part as you clearly weren't paying attention the first time... Gary Marcus has predicted slow down time and time again, and he's been wrong every time. You and Gary on currently on the anti A.I. hype, hype train...
I think he should really make it really clear, he is simply saying pure llms as they are cannot solve some problems but llms might be one piece to the large puzzle of AGI. AGI may not be this single thing trained with deep learning , I don't think most AI researchers believe that including ilya, AGI might just be a whole system of these different architectures working in unison. Also about grok, grok uses flux to generate the images so if flux doesn't have the image of the current prime minister of italy, we cant blame grok.
When dooms day argument and quantum immortality collides that when you get the constant streak of ai winters. Wherever you get to the crossroad you always go to the line that keeps agi non-invented.
i think we can all agree, that the human brain has agi. and that it is not neuro-symbolic. only neuro, no symbols. or at least, if there are symbols, they are also made of neurons. so the symbolic part is only because we want to build something that flies, not a bird with all its organs and intestines. what we currently have is a paper plane, but we want to build a fighter jet. we dont want to build a build a bird with feathers and a stomach. and i think we can all agree, that there are at least some similarities between how a human brain works and an llm. so the hope is, that if we scale this up, an exact match of the human brain will emerge. but a scaled up paper plane is still not a jet fighter. there are things missing. we have to figure out, what is missing, and then we have a chance, to mimic the human brain. the hardware is there i think. i dont think we need a 100 times more gpus. what is missing is embodiment, continious learning from real world, real time data. multimodal, neurons in neurons out. i dont think we are missing something special, that we havent discovered yet, like glia cells that are important for reasoning or something like that.
The logic is flawed, the definition of a bursting bubble is when a practice is generally adopted and the whole sector rests heavily on it.. were not there yet for AI.
I enjoyed the talk, and except for the graph, everything made sense to me. Perhaps I need to read more of his work to understand the criticisms in the comments!!
Nick Bostrom bases his simulation hypothesis as well as his 'solved world' idea on AI having solved physics, chemistry, evolution, psychology, everything.
The elephant image generation example is idiotic. Grok isn't generating images. It's just passing on your prompt to Flux via an API. It's widely known that they aren't trained on negative words for a reason. Instead of saying "without an elephant" he could have just put "elephant" into the negative prompt and got the image he wanted. This is entirely a skill issue. Error between keyboard and chair.
@@deadeaded No, they have "not words" censored. You can't expect it to magically receive information that isn't there. That's like talking to your toaster in another room and blaming your neighbour's alexa for not playing the music and claiming this proves it doesn't understand english. The place to input your not words is in the negative conditioner. If you input it somewhere else, you simply aren't communicating those words to it. It literally has no access to the information. If I put this reply comment to you on some obscure reddit thread instead of here, who's fault is that if you can't read what I say? I also really don't buy this argument that it should just magically see the words like we do and respond because that's implying we expect it to be conscious, which nobody in the ml community believes or even thinks is necessary.
@@SchmelvinMoyville I actually have friends close to ben goertzel, but it's all open source code, publicly available literature, and self-evident when you simply use the tools. I'd prefer to spend my time building things instead of posturing about my own tweets for 2 and a half hours
@@SchmelvinMoyville I pointed out that the boomer needs to put his negative prompts in the input that says "negative prompts" for it to hear him. I gotta tell ya, I'm not seeing the equivalence to giving a 2 hour speech about your own skill issues.
He made a comment about Microsoft Tay being dangerous. And that it was a good decision to take it off the market. Tay, a pre-LLM cleverbot like chatbot from 2015. This is the most prominent figure promoting crippling regulations on AI, by the way. Oh and he's simultaneously telling us that AI is just a fad, not very useful and that it won't improve. Pick one, this is just inconsistent.
i honestly think gpt-4/4o is a decent AGI already. it is generally pretty intelligent. it can answer a huge number of things on a huge number of topics. in my opinion that is general and that is intelligent. I think the main problem is that in some aspects its super human smart, and some aspects its less intelligent than a child. This just means its intelligence is different than ours. But it still comes down to how you define intelligence, and depending on the point the person is trying to make they can slide the scale quite a bit in either direction. The other thing is that alot of this is emergent behaviors, and emergence is more like sudden leaps.
Especially when he's not. The man made a bunch of specific claims about tasks language models would never be able to do. That any model nowadays does easily. I specifically remember him making a big deal of GPT3 giving inaccurate advice on what to do in a bear attack. That was easily fixed just with a better prompt.
if he's right he's right, should he just shut up just to make LLM's fanboys happy? He gave tons of examples: the current tech is unreliable, useful for some use cases, not applicable in mission critical tasks, may create huge embarrassments to businesses if applied in menial tasks and it's certainly light years away from AGI.
He's not doing it to be nasty or to simply be right. Huge amounts of money is being invested in LLMs with vague promises of AGI. That money is not being invested in more promising areas. When the investors see that their losses aren't going to turn into profits anytime soon, they'll move their money somewhere else, and we'll enter another AI winter.
Ask Dalle3 for a picture of a beach without an elephant, and you get a…beach without an elephant. I still think there is a grain of insight in his argument, but he needs to refine it. A more mature version of it may be that we can never reach fully automated AGI without humans involved somehow, but that with humans + AI we have reached early stage AGI long ago.
Yes, there is no cure to hallucinations; the same as there is no cure to unqualified human workers. Some agents extrapolate and jump into conclusions. It has to be solved. Both for AI and humans
if you think that, given the current trajectory of ignoring safety, AI will be beneficial for humanity, you are truly an idiot. Maybe an LLM can reason better than you.
Yes, stable diffusion is not an AGI, no serious person has ever said it is, only this guy I gave this video an honest try of 40 minutes, felt like listening tosome one who is trying to sell me a time share. Verbal trickry and sraw man arguments..
I see gary macus. I watch the video. Simple as that. I rather listening to wrong but interesting talk than correct but boring. The thing is gary isn't even completely wrong or have no good point that worth listening to. He just stated different way of viewing 'AGI'. LLM fanboys sure like to view things at capability level not mechanism level, while gary views the thing at mechanism level. Go learn proper science, math and logic. This is how science work unless you want to work like alchemy ofc. Then, suit yourself. But please don't go out saying people wrong (and use that to shut down any opposite view) when you guys don't even know how the who damn thing called LLM actually works.
The "mechanistic view" gary marcus has been posturing about is the "stochastic parrots hypothesis" proposed by 2 ethicists with no background in machine learning that's been debunked from 1000 different angles by mech interp researchers that have literally just looked inside the model. The reason people don't believe these pseuds is because they give you feel good narratives about how special/magical/irreplaceable functions of the human brain are with nothing to back it up, and ml researchers give you interactive models you can use to engineer the outputs you want using the abstract features they found. They don't ask you to deny the stochastic parrots argument, their predictions just work well enough to create tools for you to falsify that for yourself. If your theory can't predict the future or be used to engineer anything, it's as useful as religion.
Nonsense, if I want to listen to "listen to wrong but interesting" my kids can easily fill that void. Gary Marcus, has too much recognition and can't even admit when he gets things wrong. As for proper science oh please, Some concepts are to complicated for just using logic and science and you have to use alchemy/Engineering to get where you are going, Nature itself uses alchemy on a absolutely massive to scale to build organisms and ultimately us. The human brain itself is scale! I think most people whom watch this channel have a fair idea how LLM's function.
VC business model: get people to put money on a good sounding idea, make 2% on money invested/year, 20% on exit. So for every $100M invested, VCs make $4M/year. Do the maths.
He seems like a useful irritant and counterbalance. As an industry outsider, just a regular user who works in bioscience and education, I can’t understand all the hate in the comments.
A useful counterbalance actually makes great points. He's not useful in the slightest, he has no intellectual honesty, twists facts and data in the most stupid way simply to support a long dead narrative that has to do with nothing but his own ego. That's why he gets hate
@@fburton8 Don't bother with the haters. There's a reason their comments are all ad-homs like "he's immature", "he has an ego", "he's dishonest", "he's operating in bad faith". They have no substantive critique of his arguments. If they did they'd say it.
It’s not simply a person asking an LLM (Large Language Model) a question and getting a full answer. There can be a conversation that goes on between the person and the LLM. The human creates a prompt, LLM replies, person reviews answer. Person refines the prompt with requesting specific types of data, inputting other data and various examples, LLM replies again and so forth. RAG (Retrieval Augmented Generation) is another AI method. Amazing and complex answers are the result whether it’s coding, technical design, learning assist program, help desk, etc. AI is not overhyped and not in a bubble.
but when you train a big mamoth with tons of money and can't even reply simple general knowledge question on text vs image. that is something to think about.
The core problem of AI is not the accuracy, ability to reason, or passing the Turing Test. The problem of AI is systems development. AI algorithms are programming human behavior in ways we don't want by feeding individualized streams to each person, directing their thinking, their interests, their purchases, their life. This is the only problem we should be concerned about, and we should make it illegal.
You're missing maybe an even greater problem with chat based AI. It is extremely resource heavy and uses a ton of energy. Every more simple question asked in chat gpt uses like 10 times more energy than a Google search..more complex requests can be ridiculously energy intensive. So one of the biggest questions right now should be about making this type of AI sustainable. Because right now it's not.
@@northernswedenstories1028 While I agree this is an issue, it's a self-capping issue. When AI reaches the energy threshold, they will have to find other ways to improve their model efficiencies. The problem is what the models are already doing and have been doing for the past decade. The problem is not LLMs. The problem is the algorithmic mycelium network created by giant players in a game that 99.9 % of human beings do not want to play, while the 0.1% who think life has become too boring and seem to need a greater thrill are willing to put 99.9% of people in a perpetual screen blackout while they take people's hard-earned wealth and flush it down the toilet for shits and giggles.
i do not believe in this AGI thing, but 1:12:16 I do not even know what it means to solve physics. Yes AI can solve physics. Like structural calculations of FEM way back in the 60s. But it can also and has already proven to be useful to find new physical laws. Actually these topic are the ones which give real progress. Not this AGI prompts feed by twitter and tick-tock accounts. Where is one difference. The data for physics can be infinitive and of high quality.
I love your channel, but this has to be the lowest caliber content you’ve ever published. The dude talked for an hour by referencing his own tweets, making stuff up and producing self-contradicting and emotional arguments that frankly shows he doesn’t understand anything about AI and should be embarrassed by producing a self-own like this talk
@@VoloBonja 26:40 is an example of where he's apparently wrong. ~There are other examples. ~Also, Gary doesn't contribute to the Ai field currently. His prior contributions of old are also reasonably highly questionable.
@@godbennett the point still holds. You cant take 100 times the internet data, without making AI eat output of itself. Do you think GPTs skills grow exponentially? I don’t. And GPT4o was a proof, GPT5 will be another. His contribution since early 2000s still valid, hallucinations still with us, garbage in garbage out is still with us. People jump on Gary, while time shows most AI leaders of today are clowns on paycheck, while people still live in hype. When is AGI coming btw?
19:43 the image stuff is made by Black Forest Labs, a whole different company than makes the language model; and if it did include the face? he had just complained it could do real people. Is it a stupidity issue or just intentionally held-out data?
@Gary Marcus - you are smart but some of your statements lack the ground truth. If you want to have a valid and sound statement - try to get more precise
Remember when this pseud claimed LLMs would never have any practical or scientific applications? And now they're advancing medical science, and we see headlines like 75% of knowledge workers use them and stackoverflow showing 76% of professional devs use them
As far as i know llms specifically are not used in like any hard science fields, no? Like llms just operate on text, they arent good at any type of modeling because they werent designed for that
@@tomma1431 I can't link papers on yt but GenSLM by Nvidia predicted characteristics of new covid variants. Some great examples are "Harnessing large language models (LLMs) for candidate gene prioritization and selection" and "Large language models identify causal genes in complex trait GWAS" which you should be able to google. I've literally seen hundreds of these. Nvidia is using ChipNeMo for designing new architectures, CoScientist came out shortly after the release of GPT-4 showing it easily handless end to end novel prize winning chemical experiments testing it's own hypotheses with lab robots it controls, and Sakana AI just announced an LLM for generating novel neurips papers to accelerate ml research. I would be here all day trying to list just the ones I personally know about.
@@fburton8 CoScientist was conducting it's own experiments by controlling lab robots weeks after GPT-4 release. Paper is "Autonomous chemical research with large language models". GenSLM predicted covid variants. You can find countless papers like "Harnessing large language models (LLMs) for candidate gene prioritization and selection". Sakana AI just released their code for an agent that produces neurips papers to accelerate ml research with a paper called "The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery". I would be here for days listing just the ones I know about.
It is not fair to say you have no AGI in case of errors. Humans make horrible errors and we are GI. This is an absolutely wrong measure. I agree that the nature of error is concerning but your thought process is completely flawed
Gary Marcus is still peddling his views after all these years. I listened to him 7 to 8 years ago and thought he might have a good point. But over the years, evidence has been completely stacked against his view but yet he stubbornly refused to change his mind. I stopped listening to him.
It seems as though there are a number of academics who also get to ride on the tail winds of AI hype. Rather a lot of opinions, backed up with some light weight reasoning, prefaced with some amount of 'told you so' stories. Low water mark of contemporary public AI commentary.
I always supported the idea to use LLMs only where necessary, and combine it is with other forms of AI or CLI programs. Especially for people trying to get their own things done. The problem with Gary Marcus seems to be that he is hostile towards trying out scaling based on Deep Learning. He is making it about security, social issues, politics and about himself. He also uses strawman arguments based on advertisement for videos feom people he slanders as grifters. While their advertisement might point to the quality of service of something, he acts like their claim is about the amount of training data, which is even not a benchmark for quality in itself. That said, I think lowering expectations is better for AI development than doomer fear-mongering, so I don't really care.
Interesting talk! In general he seems biased against AI and makes very big claims, but does not really provide much evidence for his claims. Most of the examples are cherry picked: for instance that picture of an elephant in front of the beach is just one instance, is good for an entertaining and non-technical talk. However, pretty much all the metrics that are used in AI or any scientific field really are concerned with an average over a larger sample size not just one hand-picked example.
I have a simple rule. If anyone in an intellectual space mentions the word woke, I skip. You can’t be intelligent and mix it with really infantile rhetoric like that
I'm as harsh on Gary as anyone and 'Woke' is obviously made up, but if it applies to anything it's this. They literally just hardcoded in 'diversity' to every prompt to the point it broke the model entirely. That's not the same as black people existing in middle earth. Obviously a corporate PR move that has nothing to do with AI or 'leftist political bias', but this is the most rational thing I've heard Gary say.
Gary Marcus often provides critiques of generative AI that many in the field find lacking in substance and overly reliant on rehashed arguments and generalizations. His tendency to revise his stance in response to breakthroughs in AI research appears more opportunistic than insightful. A critical assessment of his work raises the question: where is the concrete evidence to support his claims? Beyond the slides and graphs, there seems to be little substance or actionable insight. The current state of AI, especially as we approach 2025, reflects a field on the brink of transformative advancements, with AGI closer than ever. Leading researchers and labs are fully aware of the momentum building within the field. Marcus' views, often dismissed by AI experts, fail to resonate with the community due to their lack of depth and credibility. Instead of dwelling on such critiques, the focus should remain on the unprecedented progress being made. The coming years will undoubtedly usher in a paradigm shift, underscoring how far the field has come and how much potential lies ahead. STOP taking this person seriously. They are merely trying to capitalize on the generative AI trend by criticizing it. A TOTAL CLOWN!
The only thing yoy really learn from realists/pessimists is that they were right every few years , and that things are much worse than popular opinion.
Hey Gary, the pandemic is still with us, more now than ever. Please use your platform to let people know we need to invest in air. COVID risk >> AI risk
I believe this guy missed the point of AI. Even a 10 year old will understand AI still in its early stages and it’s gonna get better with fine tuning and reinforcement learning, and human oversight and evaluation is needed to improve these models. I don’t understand if he is acting ignorant but to point out the AI mistakes at this early stages is like saying look how this 5 year old can’t understand math. How come he doesn’t understand little progress is better than no progress? This what happens when you seek perfection before you even learn how to walk.
his criticism is how the entire field of AI is betting all the resources on this one class of models (transformers, or more generally end-to-end deep learning models) instead of exploring alternatives. The major justification for this trend is: "they work." Performance & capability is an important metric, but there are other equally important metrics that are largely overlooked.
Your argument is based on a flawed premise. When I taught my dogs to sit, I didn't intuit that if I just kept training them enough, one day I would be able to teach them to read and write. There may be hard upper bounds on some approaches to AGI - Gary thinks this is the case for pure LLMs (hence arguing for a hybrid approach).
@@calmhorizons Dogs do not write text. You are using inequivalent comparison. There is a cap_limit on what could be thought to a dog, but is there a scientific proof that there is a cap_limit of LLM model capabilities? He have ZERO proof LLMs will get good or bad. In fact, this area of research is evolving so fast that no one can predict what is going to happen for the next year. OpenAI now are working on reasoning models. Many small fine-tuned models now are outperforming many generalist LLMs. NO ONE COULD PREDICT THIS LAST YEAR. I don't have an issue with his argument, but to claim that the discovery of application of transformer as if it was nothing and downgrading baby steps advancement in AI, and learning from our mistakes is what I disagree with him.
I normally really like this channel. So, I just wanted to say it is irresponsible to start talking about the AI bubble as if AI was some sort of "fad". (Not even going to waste my time on the actual video).
Marcus definitely isn't talking about AI as a fad. He's written books and had an AI company. One thing many people don't understand about the AI bubble is that it is almost entirely about money. Investors are losing money right now because AI costs more to make and run than it brings in. Investors will only put up with that for a time before they pull out. It is not about whether the technology is good or bad. If the bubble bursts, the tech doesn't go away. That said, this particular bubble also has a tech component. The AI companies are allowing fanboys to think that AGI is going to inevitably come out of LLMs. They know it isn't but use the lie to keep the bubble inflated. LLMs have useful applications but they are far, far from AGI.
I was just about to say… this guy is really full of himself… it is such a turn off. The minute he goes into all this awesomeness, I just stop the video. Eww.
Precisely. He has an agenda. AI regulation + big C narrative lines up. At the very least anybody still repeating the big C narrative is not a critical thinker and not worth time to spend listening to.
main points of talk: complain, complain, complain more and a lot, big massive complaining 😂😂 end with handwaving solutions that chat gpt would generate 😂😂😂
@@VoloBonja Virtual nobody in the field thinks scale is all you need. Even altman says he thinks it will need something else. They're simply scaling because it's a reliable way to increase intelligence.
Note the closed vs open source models slide originated from huggingface.co/blog/mlabonne/sft-llama3 (Maxime Labonne/Liquid AI)
26:40 that was pretty embarrassing. his slide proved him wrong and he realized it mid talk.
Indeed, that chart must be one of the worst charts I could imagine for making any kind of argument about scaling. It's a flawed benchmark, and his "expected trend line" goes to 140% or so.
@@IvanBudiselic He also admitted he just drew on the lines himself instead of projecting from actual datapoints.. and he uses this as an argument for them slowing down.
His other arguments don't stand because we don't have GPT-5 yet, he's just asserting GPT-4 turbo is supposed to be GPT-5 and not a distilled GPT-4. He also compared a bunch of LLM startups starting from nothing to GPT-4. So everything rests on him drawing a line in mspaint based off his feelings.
@@steve_jabz I wonder how good GPT-5 will be. no halucinaiton, reasoning, almost AGI, but better
@@VoloBonja I doubt it will be hallucination free or close to AGI, but for 2 years in the making at the pace OpenAI innovates, I expect something much more than just CoT baked into responses for it's reasoning capabilities
If he's the standard for intelligence, AGI is already achieved lol
He put up a graph of models since GPT-4 from other companies and startups that were starting from nothing and catching up to OpenAI and then asserted that this is proof that GPT-4 is the largest scale possible. Anyone with intellectual honestly in his position would have said "look, clearly I was wrong and GPT-3 had much more room to scale. i need to recalibrate if I don't want to be wrong again when GPT-5 comes out". Instead he comes up with these insane copes full of mental gymnastics and hand drawn graphs he created in mspaint.
He's now comparing GPT4T (a distilled model of GPT-4) to GPT-4 and expecting exponential improvements in what I guarantee is a prompting issue. This is embarrassing.
I don't think extrapolating based on literally 2 datapoints in either direction is a very reasonable thing to do.
AI has room to scale, but the underlying design of the model is extremely important to using that scale. If they could throw 50T parameters at GPT-4 and get ASI they would do it. Current model types tend to get diminishing returns from more parameters, hence why the small versions are still decent despite being much smaller.
I get the skepticism he heavily leans into, but I do also think that even 4o/claude models are far more superior in comparison to previous smaller-breadth models. I think they're (esp 4o) currently handicapped to keep an even keel with the marketspace. At least from a business standpoint, there would be no point issuing frontier models/capability as they're produced. Yeah it would be good for all of us, but with capitalism it probably will never happen that way.
@@jacobnunya808 Small models are horrible by default. Go back to llama 1 and you won't be able to use it for anything practical. Scale isn't all you need to achieve AGI either, but it's all we needed to achieve the trivial tasks gary marcus mentioned in the past. He needs to admit his mistakes is he wants to be taken seriously as a critic.
The most ironic thing in the world, is Gary Marcus pointing out all the errors made my A.I. systems, then proceeding to draw a line that goes above 100% on a benchmark graph. I guess he just disproved his own general intelligence...
I wouldn't trust employees who can't consistently look up the answers to simple questions.
I wouldn't trust an employee who can multiply/divide/compute basic math consistently.
I wouldn't trust an employee that writes broken, untested code but claims it is working.
I wouldn't trust an employee who makes up information that misleads stakeholders and customers.
I wouldn't trust an employee that confabulates information rather than saying "I don't know".
---
This is the current state of LLMs, so unless they fix these issues the tech will fizzle out and, at best, become just another doo-dad embedded in apps to make your life 5% easier.
Its not an employee. Its a tool.
Language models as part of agentic workflows can extract entities and relationships from text such as transcripts from conversations, and then merge this knowledge into graph representations. People can select parts of their conversations that they want to merge into a global shared graph representation. A global platform can be scaled up to merge selected parts of millions of simultaneous conversations with people around the world into a shared world model, a collective human and digital intelligence, by the end of this year.
Global collective intelligence would be of immense value even if it's not AGI.
There are papers showing that LLMs cannot accurately extract entities and relationships from text. If you ask an LLM to extract the entities in the text "I travelled to Barcelona to watch them play Real Madrid", it will extract the sentence subject, the city of Barcelona, and the football club Real Madrid, but it will fail to extract the 'FC Barcelona' entity that is implicit in the text.
@@dungeon_architectGood point. That's a good reason to keep the human in the loop, to catch and correct those kinds of mistakes. We are talking about collective human and digital intelligence.
@@dungeon_architect That's interesting. Can you provide a reference to this paper? Maybe it's an issue of prompting, or they were testing older models. GPT4o seems to have no issue: "Hi GPT4,
Consider the following sentence: "I travelled to Barcelona to watch them play Real Madrid"
Can you list all the entities that are being referred to in this sentence, either explicitly or implicitly?"
GPT4o: Sure! Let's break down the sentence and identify the entities being referred to, either explicitly or implicitly:
1. **"I"** - The speaker or writer, an individual traveling to Barcelona.
2. **"Barcelona"** - A city in Spain, which is the destination of the speaker.
3. **"them"** - This refers to a specific team, implicitly understood as FC Barcelona, given the context of playing Real Madrid.
4. **"Real Madrid"** - A professional football team from Madrid, Spain.
Implicitly, there are some additional references:
- **FC Barcelona** - Though not directly mentioned by name, "them" refers to FC Barcelona as the opposing team of Real Madrid.
- **A football/soccer match** - The context of the sentence suggests that the speaker is referring to a football match between FC Barcelona and Real Madrid.
- **Football stadium** - The match would implicitly take place in a football stadium, possibly referring to the Camp Nou in Barcelona.
So the entities are:
1. The speaker ("I")
2. Barcelona (the city)
3. FC Barcelona (implied by "them")
4. Real Madrid (the team)
5. The football match (implicit)
6. The football stadium (implicit, possibly Camp Nou)
@@dungeon_architect
Someone should try hard to fine-tune a model doing that.
Couldn't seem to reproduce what @@dungeon_architect was talking about in GPT-3.5, 4, 4o or llama 3.1. Tried switching up the names to test the abstraction in case it was overfitting on the test, still nothing. Wouldn't mind betting it's a skill issue.
AI is not only about GPT or LLM. Many things are going on for example application of AI in physics, medical, robotics, design, etc.
Probablistic ML can be a game changer.
Does anyone know if there's a presentation of Ben Goertzel available? I'm following his work from ~2005, and I like what he did with OpenCog architecture.
Did you search on TH-cam?! He certainly did some talks and interviews.
@@NoidoDev This is a video from a conference that just happened. Ben had a presentation there. So I asked.
@@alpha007org
Ah, sorry, my bad. I overlooked that the question was specifically about that conference.
@@NoidoDev I didn't specifically write it, but it was a big conference, so I thought it's obvious. But re-reading what I wrote, it's not clear.
9:53
"these things are not actually physical laws like laws of the universe they're just generalizations we've seen for a while they don't necessarily hold”
There are two very strange things about this statement.
1. So, while AGI itself hasn't arisen from scaling between 2021 and 2024, the scaling “laws” have definitely held. Even if there’s no “law” that says the scaling trends will “necessarily hold,” there’s yet to be any evidence that says they won’t. There is instead lots of evidence pointing towards a continued trend where larger models trained on more data better predict that data - along predictable, consistent curves.
2. This statement is quite ironic. There aren’t actually “physical laws of the universe” because the universe doesn’t “follow” laws. All known “laws of physics” are simply models we’ve developed and refined over time to match our observations of reality. Many “known” “laws” have been wrong for different reasons, and many break down at different scales. *Both* scaling laws and physical laws are "just generalizations we've seen for a while" that won't "necessarily hold."
1. the total amount of human produced data on the internet up until now is finite, and it doesn't increase exponentially, at least not at a rate that can support scaling laws; training a model on 10x more data costs ~10x more money. perhaps "scaling laws" would continue indefinitely if we had near infinite data and resources. but we don't.
2. universe does follow rules. Otherwise, there'd be only chaos, or nothing at all. The only thing is, our understanding of the law of the universe has always been incomplete.
@@sehbanomer8151 1. Who told you that Deep Learning models can only be trained on data on the internet? The data on the internet is the bootstrapping phase as we embody AI's eventually they will train on data from the natural world.
@@sehbanomer8151not true on second point. Universe follows laws’ is just our assumption and so far it worked. OP is right.
@@zyzhang1130 "universe doesn't follow any law" is also an assumption, and as you said, the opposite assumption has worked so far.
@@sehbanomer8151 so does the empirical scaling law right?
Jfc the AI Overviews thing. There's a snopes article debunking this. Anti-AI bros created a template for people to type in whatever they wanted to pretend it said. The LLM wasn't even generating the answers in the first place. It was just selecting a snippet from pages that were poorly ranked like they always have been. The only thing the LLM did was reword it.
Or consider this: he was paid by some people to (the masses are experiencing FUD) suppress the trajectory since governments are legislating against AI, preventing development. They need to compete with China (and don't have "annoying" legislators) to achieve AGI.
I really enjoyed the production of the Q and A. Thank you
Alternative title for the video: "the guy who spelled the last decade prediction that Deep learning has hit a wall, but this time for sure, is predicting that AI has hit a wall now"
There's this thing with these people (and this goes for all AI "gurus"), that once they've made their view public, and especially wrote books about and and their entire career depends on this initial view being right, they will do anything to find examples supporting their view and anything to dismiss evidence against it.
I tried all his examples of LLM failures, and ChatGPT handles them perfectly well, so they're cherry picked by him from a time in the past or a worse competitor. When you make arguments against AI, you should do so against the state of the art, anything else makes no sense. Second, OpenAI and others almost certainly have (or could train) much better models in-house, but the inference is just not economical at this point. Most work in the past year has gone towards optimizing GPT4 level models so that you can actually deploy them, that doesn't mean you can't improve on it. The same goes for grounding etc, it's just a matter of time before models are not only trained on all the text on the internet but all the videos and images too, which gives it another level of grounding. Just because it's not economical yet doesn't mean it won't work. Same goes for world models (where he clearly doesn't understand LeCun's work on JEPA). There is just so much left to do, it's way too early to call a slow down. He's a complete clown.
Your comment would be very witty... if not for all the scientific evidence that Deep Learning has indeed hit a wall (and a lot of the "progress" is smoke and mirrors provided by extensive RLHF bootstrapping and clunky "multi-modal" architectures that leverage more traditional algorithms.)
Deep Learning can still be immensely valuable and relevant without making silly over-claims and declaring itself the holy grail of AI.
I just realized that this talk is a month old and OpenAI o1 model is roughly 3 weeks old... Well thankfully, it only took a week for OpenAI to respond to Gary Marcus with their new reasoning model. Pretty cool
Have you tried o1? I use it often, it’s not the answer
There may be too much money being spent on scaling LLMs, but agents are getting more and more development money. 2024 has been the name of nascent agents. 2025 will be the year when agents really enter the workplace filling jobs in place of people.
A wonderful talk. Pointing out a lot of the bullshit behind some AI hype (like the blind faith on scaling up everything to achieve AGI). We really need hybrid systems and start building cognitive models of the world.
We need better benchmarks to look at than contaminated full of errors MMLU
As a hobbyist programmer and mathematics enthusiast I don’t think a machine learning model making mistake is bad it’s normal . Remember it’s a statistical distribution output. To the model right answer is just as valid as wrong answers and what we need to do instead is to have a layer of logic that can apply some sort of light logic to its output to filter it through a real life lense. I don’t agree with shitting on AI and I also don’t agree with companies overhyping a statistical distribution machine . So far these models have some use as customer service bot trained on company info and maybe some stuff outside of that . But I don’t think it’s ready for more real life stuff
I think he's shitting on people/ companies who are overhyping LLM and making false promises.
@@sehbanomer8151100% what he’s doing; shitting on Sam Altman type characters promising AGI will ‘solve physics’ in the next decade.
I broadly agree with your statement, though the use cases for the more recent models goes way beyond just chat bots. The problem is that they are being oversold to some degree and under explained. They may also have been needlessly sanitised for the sake of marketability. As for misinformation spreading - yes if the consumers of the information have useless filtration systems in place already, which is not really the fault of AI/ML developers.
Yep - these things are working as designed. But the sellers of them are making absurd claims that they are "conscious" - "miniAGI" - "capable of reasoning" etc.
Purely coincidentally (I am sure) they stand to make immense amounts of money from these claims.
A sane argument in a sea of hype. Yes there are some niche usees but the market is pumping money in expecting we are going to have personal robo super intelligent terminators in a few years. Very little practical reliable use of this stuff the bubble will burst. I also studied this stuff in grad school, it's neat but we won't have full self driving, thinking machines or displace that many workers over it.
Gary is right. People is comments listen to too many podcasts where AGI is already here and Altman is a king.
Human level AGI is coming any time soon, not 5, not 10, not 20 years. We needto work harder to make it happen. And it's not about scaling, it's about new explanations
Gary is not right. He’s been wrong about most things and will continue to be proven wrong going forward. That has nothing to do with AGI being here or Altman being king. It’s just an empirical observation about his track record and how many things he’s been wrong about.
You very clearly have no idea what you're talking about...
@@generichuman_ dismissed without arguments
@@generichuman_ Seriously. People are so clueless on AI, and pseudo “experts” like Gary make it a hundred times worse.
@@VoloBonja If you want people to spend time giving you proper arguments, maybe you should put time into your own arguments and not just say "Gary is right" and make baseless assertions like "it's not happening anytime soon" with zero evidence.
We have no idea if LLM performance is slowing down until the next generation of models comes out i.e. GPT-5. Gary is making disingenuous comparisons between models that don't represent the same step change, for example GPT3 -> 4 vs GPT4 -> 4o. Also, at 26:40 he makes an extremely silly error showing benchmark performance as a percent and claims that there's slow down because the performance gains are getting smaller on the graph as they get closer to 100%... Duh... that's how percentages work. You can't get past 100%. It's asymptotic. 10% to 90% looks huge on the graph and 99% to 99.9% looks tiny, despite the latter change being larger. Even worse, he has a hand drawn line that projects to 140% which is what he expects without slow down... he notices this error at 26:40. You should re-watch this part as you clearly weren't paying attention the first time...
Gary Marcus has predicted slow down time and time again, and he's been wrong every time. You and Gary on currently on the anti A.I. hype, hype train...
I think he should really make it really clear, he is simply saying pure llms as they are cannot solve some problems but llms might be one piece to the large puzzle of AGI. AGI may not be this single thing trained with deep learning , I don't think most AI researchers believe that including ilya, AGI might just be a whole system of these different architectures working in unison.
Also about grok, grok uses flux to generate the images so if flux doesn't have the image of the current prime minister of italy, we cant blame grok.
When dooms day argument and quantum immortality collides that when you get the constant streak of ai winters. Wherever you get to the crossroad you always go to the line that keeps agi non-invented.
surprised by this comment section. Lots of LLM fanboys out today I guess
What do you call people who use terms like “fanboy”?
Yup. This field is rotten regarding its philosophical foundations.
@@haldanesghost 🤓👆
I guess you're a Gary Marcus fanboy then, which is a very bad sign for your intelligence
"Don't you dare criticize my favorite toy 😤"
i think we can all agree, that the human brain has agi. and that it is not neuro-symbolic. only neuro, no symbols. or at least, if there are symbols, they are also made of neurons. so the symbolic part is only because we want to build something that flies, not a bird with all its organs and intestines. what we currently have is a paper plane, but we want to build a fighter jet. we dont want to build a build a bird with feathers and a stomach. and i think we can all agree, that there are at least some similarities between how a human brain works and an llm. so the hope is, that if we scale this up, an exact match of the human brain will emerge. but a scaled up paper plane is still not a jet fighter. there are things missing. we have to figure out, what is missing, and then we have a chance, to mimic the human brain. the hardware is there i think. i dont think we need a 100 times more gpus. what is missing is embodiment, continious learning from real world, real time data. multimodal, neurons in neurons out. i dont think we are missing something special, that we havent discovered yet, like glia cells that are important for reasoning or something like that.
The logic is flawed, the definition of a bursting bubble is when a practice is generally adopted and the whole sector rests heavily on it.. were not there yet for AI.
I enjoyed the talk, and except for the graph, everything made sense to me. Perhaps I need to read more of his work to understand the criticisms in the comments!!
the comment section on these is always filled with cultish nonsense
Nick Bostrom bases his simulation hypothesis as well as his 'solved world' idea on AI having solved physics, chemistry, evolution, psychology, everything.
The elephant image generation example is idiotic. Grok isn't generating images. It's just passing on your prompt to Flux via an API. It's widely known that they aren't trained on negative words for a reason. Instead of saying "without an elephant" he could have just put "elephant" into the negative prompt and got the image he wanted. This is entirely a skill issue. Error between keyboard and chair.
@@deadeaded No, they have "not words" censored. You can't expect it to magically receive information that isn't there. That's like talking to your toaster in another room and blaming your neighbour's alexa for not playing the music and claiming this proves it doesn't understand english. The place to input your not words is in the negative conditioner. If you input it somewhere else, you simply aren't communicating those words to it. It literally has no access to the information.
If I put this reply comment to you on some obscure reddit thread instead of here, who's fault is that if you can't read what I say?
I also really don't buy this argument that it should just magically see the words like we do and respond because that's implying we expect it to be conscious, which nobody in the ml community believes or even thinks is necessary.
@@steve_jabzthat’s cool man you should do a presentation like this guy did so you can grace us with your knowledge
@@SchmelvinMoyville I actually have friends close to ben goertzel, but it's all open source code, publicly available literature, and self-evident when you simply use the tools. I'd prefer to spend my time building things instead of posturing about my own tweets for 2 and a half hours
@@steve_jabz thats really interesting considering youre doing the same shit but in a youtube comment but thats cool do your thing
@@SchmelvinMoyville I pointed out that the boomer needs to put his negative prompts in the input that says "negative prompts" for it to hear him. I gotta tell ya, I'm not seeing the equivalence to giving a 2 hour speech about your own skill issues.
He isn't operating in good faith - everyone knows this. Why would you contribute to his rabid quixotic low quality tirades..
Everyone?
Bwahahahaha
Oh really? His take seems to be quite accurate from my point of view.
He made a comment about Microsoft Tay being dangerous. And that it was a good decision to take it off the market. Tay, a pre-LLM cleverbot like chatbot from 2015. This is the most prominent figure promoting crippling regulations on AI, by the way.
Oh and he's simultaneously telling us that AI is just a fad, not very useful and that it won't improve. Pick one, this is just inconsistent.
Exactly! Bad Faith Gary
Exactly
i honestly think gpt-4/4o is a decent AGI already. it is generally pretty intelligent. it can answer a huge number of things on a huge number of topics. in my opinion that is general and that is intelligent. I think the main problem is that in some aspects its super human smart, and some aspects its less intelligent than a child. This just means its intelligence is different than ours. But it still comes down to how you define intelligence, and depending on the point the person is trying to make they can slide the scale quite a bit in either direction. The other thing is that alot of this is emergent behaviors, and emergence is more like sudden leaps.
A bit juvenile making most of the talk about how you were right and other people are wrong
We may soon know who is RIGHT :)
Especially when he's not. The man made a bunch of specific claims about tasks language models would never be able to do. That any model nowadays does easily. I specifically remember him making a big deal of GPT3 giving inaccurate advice on what to do in a bear attack. That was easily fixed just with a better prompt.
if he's right he's right, should he just shut up just to make LLM's fanboys happy?
He gave tons of examples: the current tech is unreliable, useful for some use cases, not applicable in mission critical tasks, may create huge embarrassments to businesses if applied in menial tasks and it's certainly light years away from AGI.
That’s Gary for you: warped perceptions, maladaptive cognitive schemas, and a gizzard neck that’s begging for a 2 horsepower CPAP machine
He's not doing it to be nasty or to simply be right. Huge amounts of money is being invested in LLMs with vague promises of AGI. That money is not being invested in more promising areas. When the investors see that their losses aren't going to turn into profits anytime soon, they'll move their money somewhere else, and we'll enter another AI winter.
Ask Dalle3 for a picture of a beach without an elephant, and you get a…beach without an elephant.
I still think there is a grain of insight in his argument, but he needs to refine it.
A more mature version of it may be that we can never reach fully automated AGI without humans involved somehow, but that with humans + AI we have reached early stage AGI long ago.
The govt cant regulate ai in the us, its free speech. Stop being worried though it wont amount to much.
Bars, finally where hearing from people who are seeing through the hype and claims of exponential growth!, something, something, something, AGI!
If you thought Gary was on point here, you should take your brain back and ask for a refund...
@@generichuman_ lol
Yes, there is no cure to hallucinations; the same as there is no cure to unqualified human workers. Some agents extrapolate and jump into conclusions. It has to be solved. Both for AI and humans
He is secretely an alien trying to stop humanity from developing AI
if you think that, given the current trajectory of ignoring safety, AI will be beneficial for humanity, you are truly an idiot. Maybe an LLM can reason better than you.
Cruise had not that many operators you are speaking about. If they would have - there wouldn’t be shit that happen
Yes, stable diffusion is not an AGI, no serious person has ever said it is, only this guy
I gave this video an honest try of 40 minutes, felt like listening tosome one who is trying to sell me a time share.
Verbal trickry and sraw man arguments..
I see gary macus. I watch the video. Simple as that. I rather listening to wrong but interesting talk than correct but boring. The thing is gary isn't even completely wrong or have no good point that worth listening to. He just stated different way of viewing 'AGI'. LLM fanboys sure like to view things at capability level not mechanism level, while gary views the thing at mechanism level. Go learn proper science, math and logic. This is how science work unless you want to work like alchemy ofc. Then, suit yourself. But please don't go out saying people wrong (and use that to shut down any opposite view) when you guys don't even know how the who damn thing called LLM actually works.
Exaxtly. there are very few rationalists in the LLM industry, except for mech-interp people like Chris Olah
The "mechanistic view" gary marcus has been posturing about is the "stochastic parrots hypothesis" proposed by 2 ethicists with no background in machine learning that's been debunked from 1000 different angles by mech interp researchers that have literally just looked inside the model. The reason people don't believe these pseuds is because they give you feel good narratives about how special/magical/irreplaceable functions of the human brain are with nothing to back it up, and ml researchers give you interactive models you can use to engineer the outputs you want using the abstract features they found. They don't ask you to deny the stochastic parrots argument, their predictions just work well enough to create tools for you to falsify that for yourself. If your theory can't predict the future or be used to engineer anything, it's as useful as religion.
Nonsense, if I want to listen to "listen to wrong but interesting" my kids can easily fill that void.
Gary Marcus, has too much recognition and can't even admit when he gets things wrong.
As for proper science oh please, Some concepts are to complicated for just using logic and science and you have to use alchemy/Engineering to get where you are going, Nature itself uses alchemy on a absolutely massive to scale to build organisms and ultimately us.
The human brain itself is scale!
I think most people whom watch this channel have a fair idea how LLM's function.
There is just not much new substance. We hear these exact same speculations from everyone.
VC business model: get people to put money on a good sounding idea, make 2% on money invested/year, 20% on exit. So for every $100M invested, VCs make $4M/year. Do the maths.
Oh, Gary, enough already. Good lord.
He seems like a useful irritant and counterbalance. As an industry outsider, just a regular user who works in bioscience and education, I can’t understand all the hate in the comments.
A useful counterbalance actually makes great points. He's not useful in the slightest, he has no intellectual honesty, twists facts and data in the most stupid way simply to support a long dead narrative that has to do with nothing but his own ego. That's why he gets hate
@@2CSST2 Hmm, okay, but I thought he made some good points.
@@fburton8 Don't bother with the haters. There's a reason their comments are all ad-homs like "he's immature", "he has an ego", "he's dishonest", "he's operating in bad faith". They have no substantive critique of his arguments. If they did they'd say it.
The openai bubble might pop but sams been maneuvering for government integration to continue his grift.
AI isnt a bubble though.
Yea if there is a shortage of radiologist it's definitely because of Hinton...
It’s not simply a person asking an LLM (Large Language Model) a question and getting a full answer. There can be a conversation that goes on between the person and the LLM. The human creates a prompt, LLM replies, person reviews answer. Person refines the prompt with requesting specific types of data, inputting other data and various examples, LLM replies again and so forth. RAG (Retrieval Augmented Generation) is another AI method. Amazing and complex answers are the result whether it’s coding, technical design, learning assist program, help desk, etc. AI is not overhyped and not in a bubble.
but when you train a big mamoth with tons of money and can't even reply simple general knowledge question on text vs image. that is something to think about.
The core problem of AI is not the accuracy, ability to reason, or passing the Turing Test. The problem of AI is systems development. AI algorithms are programming human behavior in ways we don't want by feeding individualized streams to each person, directing their thinking, their interests, their purchases, their life. This is the only problem we should be concerned about, and we should make it illegal.
You're missing maybe an even greater problem with chat based AI. It is extremely resource heavy and uses a ton of energy. Every more simple question asked in chat gpt uses like 10 times more energy than a Google search..more complex requests can be ridiculously energy intensive. So one of the biggest questions right now should be about making this type of AI sustainable. Because right now it's not.
@@northernswedenstories1028 While I agree this is an issue, it's a self-capping issue. When AI reaches the energy threshold, they will have to find other ways to improve their model efficiencies. The problem is what the models are already doing and have been doing for the past decade. The problem is not LLMs. The problem is the algorithmic mycelium network created by giant players in a game that 99.9 % of human beings do not want to play, while the 0.1% who think life has become too boring and seem to need a greater thrill are willing to put 99.9% of people in a perpetual screen blackout while they take people's hard-earned wealth and flush it down the toilet for shits and giggles.
i do not believe in this AGI thing, but 1:12:16 I do not even know what it means to solve physics. Yes AI can solve physics. Like structural calculations of FEM way back in the 60s. But it can also and has already proven to be useful to find new physical laws. Actually these topic are the ones which give real progress. Not this AGI prompts feed by twitter and tick-tock accounts. Where is one difference. The data for physics can be infinitive and of high quality.
Josh Batch has some good ideas
I love your channel, but this has to be the lowest caliber content you’ve ever published.
The dude talked for an hour by referencing his own tweets, making stuff up and producing self-contradicting and emotional arguments that frankly shows he doesn’t understand anything about AI and should be embarrassed by producing a self-own like this talk
He understands that generating text/images/video is narrow domain, and not going to magically result in a General Intelligence. Seems relevant to me.
Give few examples:
1. "making stuff up" - what exactly?
2. "he doesn’t understand anything about AI" go ahead and prove this. Where is he wrong?
@@VoloBonja 26:40 is an example of where he's apparently wrong.
~There are other examples.
~Also, Gary doesn't contribute to the Ai field currently. His prior contributions of old are also reasonably highly questionable.
@@godbennett the point still holds. You cant take 100 times the internet data, without making AI eat output of itself.
Do you think GPTs skills grow exponentially? I don’t. And GPT4o was a proof, GPT5 will be another.
His contribution since early 2000s still valid, hallucinations still with us, garbage in garbage out is still with us.
People jump on Gary, while time shows most AI leaders of today are clowns on paycheck, while people still live in hype.
When is AGI coming btw?
@@VoloBonja 26:40 that's where he's wrong, and he realizes it in real time...
Don’t mind me. Here to get some product for my salt mining company.
I lol'd... But I am not entirely sure I get the joke
@@hugooc I also came here for salt but found gold instead it seems
For lack of response to criticism, go for unoriginal cringe I guess
"There is a good chance that if you give your answers in African American English ..."
What is "African American English"?
19:43 the image stuff is made by Black Forest Labs, a whole different company than makes the language model; and if it did include the face? he had just complained it could do real people. Is it a stupidity issue or just intentionally held-out data?
Watching this again after the o1 release: Ahahahahahahahaha
LLMs scale, but they don't generalize. You can't solve a fundamentals issue by just throwing more money at the problem.
@Gary Marcus - you are smart but some of your statements lack the ground truth. If you want to have a valid and sound statement - try to get more precise
Remember when this pseud claimed LLMs would never have any practical or scientific applications?
And now they're advancing medical science, and we see headlines like 75% of knowledge workers use them and stackoverflow showing 76% of professional devs use them
As far as i know llms specifically are not used in like any hard science fields, no? Like llms just operate on text, they arent good at any type of modeling because they werent designed for that
@@tomma1431 I can't link papers on yt but GenSLM by Nvidia predicted characteristics of new covid variants. Some great examples are "Harnessing large language models (LLMs) for candidate gene prioritization and selection" and "Large language models identify causal genes in complex trait GWAS" which you should be able to google. I've literally seen hundreds of these. Nvidia is using ChipNeMo for designing new architectures, CoScientist came out shortly after the release of GPT-4 showing it easily handless end to end novel prize winning chemical experiments testing it's own hypotheses with lab robots it controls, and Sakana AI just announced an LLM for generating novel neurips papers to accelerate ml research. I would be here all day trying to list just the ones I personally know about.
Where can I find scientific applications?
@@fburton8 CoScientist was conducting it's own experiments by controlling lab robots weeks after GPT-4 release. Paper is "Autonomous chemical research with large language models".
GenSLM predicted covid variants. You can find countless papers like "Harnessing large language models (LLMs) for candidate gene prioritization and selection".
Sakana AI just released their code for an agent that produces neurips papers to accelerate ml research with a paper called "The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery".
I would be here for days listing just the ones I know about.
@@fburton8 Google "Sakana Ai scientist".
He lost me early on with “Covid vaccine was very effective” 😂
who is ben g?
Were there no women at the conference?
Who cares?!
why were there no tech bros at the international makeup conference
Common. The same as many other humans - I don’t know the US map as well and your conclusions trigger too many red flags to draw your conclusions
It's too bad. I agree with a lot of what Gary usually says but his political biases leak all over the place, staining everything in its path.
It is not fair to say you have no AGI in case of errors. Humans make horrible errors and we are GI. This is an absolutely wrong measure. I agree that the nature of error is concerning but your thought process is completely flawed
It seems like those speeches saying China downfall is coming. It's been over 50 years😂
Gary Marcus is still peddling his views after all these years. I listened to him 7 to 8 years ago and thought he might have a good point. But over the years, evidence has been completely stacked against his view but yet he stubbornly refused to change his mind. I stopped listening to him.
A nonsense cherry picked talk by a narcissist. MLST you can do better than this.
It seems as though there are a number of academics who also get to ride on the tail winds of AI hype.
Rather a lot of opinions, backed up with some light weight reasoning, prefaced with some amount of 'told you so' stories. Low water mark of contemporary public AI commentary.
Poor Gary... God, I wouldn't wanna be glued to his broken narrative.
I always supported the idea to use LLMs only where necessary, and combine it is with other forms of AI or CLI programs. Especially for people trying to get their own things done.
The problem with Gary Marcus seems to be that he is hostile towards trying out scaling based on Deep Learning. He is making it about security, social issues, politics and about himself.
He also uses strawman arguments based on advertisement for videos feom people he slanders as grifters. While their advertisement might point to the quality of service of something, he acts like their claim is about the amount of training data, which is even not a benchmark for quality in itself.
That said, I think lowering expectations is better for AI development than doomer fear-mongering, so I don't really care.
Gary Marcus has been hitting the wall for decades now but he just keeps bouncing back.
Space, time, causality.
Excellent talk!
You don't need to be a professor to know that.
sleazy individual!
25:00 Super BS chart, makes absolutely no sense.
Interesting talk! In general he seems biased against AI and makes very big claims, but does not really provide much evidence for his claims. Most of the examples are cherry picked: for instance that picture of an elephant in front of the beach is just one instance, is good for an entertaining and non-technical talk. However, pretty much all the metrics that are used in AI or any scientific field really are concerned with an average over a larger sample size not just one hand-picked example.
Lmao
I have a simple rule. If anyone in an intellectual space mentions the word woke, I skip. You can’t be intelligent and mix it with really infantile rhetoric like that
I'm as harsh on Gary as anyone and 'Woke' is obviously made up, but if it applies to anything it's this. They literally just hardcoded in 'diversity' to every prompt to the point it broke the model entirely.
That's not the same as black people existing in middle earth. Obviously a corporate PR move that has nothing to do with AI or 'leftist political bias', but this is the most rational thing I've heard Gary say.
Oh, so there is an AGI society, guess they are fantasizing for an unforeseeable futurate... futurist, what a great cause indeed?
Thumbs down for bad info
Gary gets way more hate then he deserves. I see a lot of people discussing the wrong graph he showed, but what about other points.
If he would stick to advocating something he wants to do, and just doing it, there wouldn't be much "hate".
We've addressed all this points. They're as silly as his hand drawn graphs.
Gary Marcus often provides critiques of generative AI that many in the field find lacking in substance and overly reliant on rehashed arguments and generalizations. His tendency to revise his stance in response to breakthroughs in AI research appears more opportunistic than insightful. A critical assessment of his work raises the question: where is the concrete evidence to support his claims? Beyond the slides and graphs, there seems to be little substance or actionable insight.
The current state of AI, especially as we approach 2025, reflects a field on the brink of transformative advancements, with AGI closer than ever. Leading researchers and labs are fully aware of the momentum building within the field. Marcus' views, often dismissed by AI experts, fail to resonate with the community due to their lack of depth and credibility. Instead of dwelling on such critiques, the focus should remain on the unprecedented progress being made. The coming years will undoubtedly usher in a paradigm shift, underscoring how far the field has come and how much potential lies ahead. STOP taking this person seriously. They are merely trying to capitalize on the generative AI trend by criticizing it. A TOTAL CLOWN!
nothing ever happens.
The only thing yoy really learn from realists/pessimists is that they were right every few years , and that things are much worse than popular opinion.
Yup and now most ignore the real risk, another world war. That went from 0 percent a few years ago to I'd say 10 to 20 percent now.
Hey Gary, the pandemic is still with us, more now than ever. Please use your platform to let people know we need to invest in air. COVID risk >> AI risk
Too many I
I believe this guy missed the point of AI. Even a 10 year old will understand AI still in its early stages and it’s gonna get better with fine tuning and reinforcement learning, and human oversight and evaluation is needed to improve these models. I don’t understand if he is acting ignorant but to point out the AI mistakes at this early stages is like saying look how this 5 year old can’t understand math. How come he doesn’t understand little progress is better than no progress? This what happens when you seek perfection before you even learn how to walk.
He's saying that LLMs are a dead end for developing AGI.
his criticism is how the entire field of AI is betting all the resources on this one class of models (transformers, or more generally end-to-end deep learning models) instead of exploring alternatives.
The major justification for this trend is: "they work." Performance & capability is an important metric, but there are other equally important metrics that are largely overlooked.
Your argument is based on a flawed premise.
When I taught my dogs to sit, I didn't intuit that if I just kept training them enough, one day I would be able to teach them to read and write.
There may be hard upper bounds on some approaches to AGI - Gary thinks this is the case for pure LLMs (hence arguing for a hybrid approach).
@@calmhorizons Dogs do not write text. You are using inequivalent comparison. There is a cap_limit on what could be thought to a dog, but is there a scientific proof that there is a cap_limit of LLM model capabilities? He have ZERO proof LLMs will get good or bad. In fact, this area of research is evolving so fast that no one can predict what is going to happen for the next year. OpenAI now are working on reasoning models. Many small fine-tuned models now are outperforming many generalist LLMs. NO ONE COULD PREDICT THIS LAST YEAR. I don't have an issue with his argument, but to claim that the discovery of application of transformer as if it was nothing and downgrading baby steps advancement in AI, and learning from our mistakes is what I disagree with him.
@@MohamedAddani-qc3iw Prove there isn't a cap_limit - on generative AI and I will agree with you.
I normally really like this channel. So, I just wanted to say it is irresponsible to start talking about the AI bubble as if AI was some sort of "fad". (Not even going to waste my time on the actual video).
Bwahahahaha
Marcus definitely isn't talking about AI as a fad. He's written books and had an AI company. One thing many people don't understand about the AI bubble is that it is almost entirely about money. Investors are losing money right now because AI costs more to make and run than it brings in. Investors will only put up with that for a time before they pull out. It is not about whether the technology is good or bad. If the bubble bursts, the tech doesn't go away. That said, this particular bubble also has a tech component. The AI companies are allowing fanboys to think that AGI is going to inevitably come out of LLMs. They know it isn't but use the lie to keep the bubble inflated. LLMs have useful applications but they are far, far from AGI.
@@PaulTopping1had an ai company? It was a fraud autonomous system that ran over a lady. Autopilot powered by knowledge graphs.
@@PaulTopping1 EXACTLY
@@ihoryou misspelled it, should be Igor
Blah blah blah, there are some valid arguments to be made but he's not making them. He's a clown
one of the most closed minded individuals in the world.
57:30 Gary has a large problem, because historically, Elon HAS delivered on the promises but does it late. Probably because he needs the money first.
I was just about to say… this guy is really full of himself… it is such a turn off. The minute he goes into all this awesomeness, I just stop the video. Eww.
This talk didn't age well
A very unpleasant person
This guy is a clown…
How about you?
Wtf is “African American English” 😮
Lots of AI fanboys here. ROFL !
No point in engaging with someone who's still sleepwalking through the COVID narrative.
Precisely. He has an agenda. AI regulation + big C narrative lines up. At the very least anybody still repeating the big C narrative is not a critical thinker and not worth time to spend listening to.
AI is all hype.
main points of talk: complain, complain, complain more and a lot, big massive complaining 😂😂 end with handwaving solutions that chat gpt would generate 😂😂😂
Does Gary Marcus know anything about anything? Sorry to be rude.
scale is all you need?
@@VoloBonja Virtual nobody in the field thinks scale is all you need. Even altman says he thinks it will need something else. They're simply scaling because it's a reliable way to increase intelligence.
it is too late, algorithm for AGi already exist
where? show me