Being Turing complete is nice and all, but this does not inherently mean that there will be existing solutions to certain problems inside reasonable computational limitations.
its funny, in the universal transfromer paper they show turing completeness by ignoring the decoder while focusing on the adaptive compute time layerwise instead of context-step-wise "chain of thought" .. so transformers are double-turing-complete? :D
21:08 Great! Maybe we need an AI so smart that we can ask "solve the problem but make sure you do it in a way that our puny human brains can understand". Weird world we're creating.
Contradicts your last video about Chomsky's assertion? th-cam.com/video/8lU6dGqR26s/w-d-xo.htmlsi=K26pFdgNo35-_-hu Meaning Chomsky' was correct, no? Your question at 18:00 goes into this a little
Did Turing solve the problem with the infinite tape for the Turing machine? We talk in the video about whether anything in reality (e.g., my laptop) is equivalent to a turing machine. 😅
Because Turing machines are just a mathematical theory requiring infinite resources. Infinity does not exist for practical reasons in the universe and we can use only finite resources. So in the same way in which laptops would need infinite memory to implement Turing Machines, neutral networks would need infinitely wide vectors or infinite precision.
llms can generate coherent sentences, they are probabilistic in nature, and based on randomness. randomness/random sampling of the latent space creates halucinations, because there is no reflection, no cognition of any kind hapening. a very advanced automaton, that can imitate humlan language, nothing else, pure imitation. just like a plane isnt a bird, it flies, it goes superfast, but its not a bird. similarly a language model imitates words and sentences, but there is nothing else going on.
It is true that LLMs use randomness and statistical mathematics but that does not necessarily mean they are limited to only producing random statistical samplings. The whole is not always just the sum of the parts. LLMs can succeed in some limited reasoning. For example, it has been shown that simply asking an LLM to "think step-by-step" causes it to produce objectively better answers. How could this work if the LLM were not reasoning? If it was simply reproducing patterns you might as well just ask it to "give me a smarter sounding answer." LLMs are also able to solve logic and mathematical problems. Successfully finding solutions to such problems will not happen consistently by random chance or by simply producing textual patterns that look like answers. Some level of reasoning is required. The hallucination problem occurs when you ask an LLM to produce an answer outside of its capabilities. Instead of saying, "I don't know," it will often make up something that looks like an answer. But humans do this all the time too and that does not mean we are incapable of reasoning. It's easy and reassuring to dismiss LLMs as statistical parrots because they are then clearly not a threat. I don't believe LLMs alone, at least in their current form, pose a significant direct threat to humans. The only real danger is in how humans use LLMs for good or bad purposes. We are still a long way from AGI but these are exciting times.
@@jcoffland I think ppl conflate various types of reasoning. there is a type of reasoning that is purely mechanical, true and true = true, false or true = true, true xor true = false etc. you can memorize and do a lot of "reasoning" just by applying rules or patterns mechanically, you don't even need to know why each rule is true. classical computers and simple logic gates can already carry out this type of reasoning, thats why we've had theorem provers and prolog for decades. LLMs can do something similar but in a more statistical/probabilistic/fuzzy way. they can combine truthfulness of statements and draw conclusions probabilistically, so it looks like it has some common sense reasoning, but it;s not, it's just expanding the type of logic gate reasoning to linguistic representations. The other type of reasoning people talk about is about how the actual world works and cause/effect and intentionality. For example when I say "the cat is on the table" you dont just "compute" sentence linguistically/symbolically, but also visually, you imagine a cat on a table, because of how physics works you know the can cannot also be underneath at the same time, u know the cat could jump off at any point. you are not just thinking about that scene linguistically, like an LLM that has never actually seen a cat. its as expecting a human who has never seen, touched or heard anything and only been able to interact with a screen of printed words and a keyboard to be able to understand the world. maybe this will change once ai is able to process data in real time, is embedded in a robotic body and interact with the world, until then llms are not actually able to understand anything, they only have a purely linguistic intelligence/understanding (they manipulate words and symbols statistically)
They are beyond completeness as their attention mechanism esentially gives them a self destruction switch that flips the context and frame of a problem as it relative hardness requires it. New Oracle Turing Machines could arise from the attention mechanism. Transformers should definitely be studied further by CS theorists.
![I am a Strange Dataset: Metalinguistic Tests for Language Models - Paper Explained [🔴 at ACL 2024]](http://i.ytimg.com/vi/m_nEIsQBh_c/mqdefault.jpg)
![I am a Strange Dataset: Metalinguistic Tests for Language Models – Paper Explained [🔴 at ACL 2024]](/img/tr.png)
This was super nice! I hope AI Coffee Break podcasts become a regular thing!
AI Coffee Break pods.
Thank you! It was harder to do a podcast than expected.
@@AICoffeeBreak
I should read the paper first, then watch this amazing talk again. I did not understand completely. But thank you for this precious content.
Wonderful podcast, perfect duration, great questions, keep them coming!
Great to see you, I remember you from that beautiful video you made couple of years back on the "attention is all you need" paper
Great content and format. This channel just keeps getting better.
Fascinating. Great discussion.
Being Turing complete is nice and all, but this does not inherently mean that there will be existing solutions to certain problems inside reasonable computational limitations.
great video thanks 😁
Can His tutorial on computational expressivity of Language Model be found on youtube?
I'm afraid not. Only written form and slides here: acl2024.ivia.ch/
acl2024.ivia.ch/about
its funny, in the universal transfromer paper they show turing completeness by ignoring the decoder while focusing on the adaptive compute time layerwise instead of context-step-wise "chain of thought" .. so transformers are double-turing-complete? :D
21:08 Great! Maybe we need an AI so smart that we can ask "solve the problem but make sure you do it in a way that our puny human brains can understand". Weird world we're creating.
🎉
Contradicts your last video about Chomsky's assertion? th-cam.com/video/8lU6dGqR26s/w-d-xo.htmlsi=K26pFdgNo35-_-hu Meaning Chomsky' was correct, no? Your question at 18:00 goes into this a little
Didn't watch yet - at work xD. Did they solve the infinite input width problem?
Did Turing solve the problem with the infinite tape for the Turing machine?
We talk in the video about whether anything in reality (e.g., my laptop) is equivalent to a turing machine. 😅
@@AICoffeeBreak Why should math care about my laptops memory? That argument is nonsensical.
Because Turing machines are just a mathematical theory requiring infinite resources. Infinity does not exist for practical reasons in the universe and we can use only finite resources. So in the same way in which laptops would need infinite memory to implement Turing Machines, neutral networks would need infinitely wide vectors or infinite precision.
A ticket tape machine is Turing Complete. It's a very low bar
That Transformer Encoders do not even meet. :)
llms can generate coherent sentences, they are probabilistic in nature, and based on randomness. randomness/random sampling of the latent space creates halucinations, because there is no reflection, no cognition of any kind hapening. a very advanced automaton, that can imitate humlan language, nothing else, pure imitation. just like a plane isnt a bird, it flies, it goes superfast, but its not a bird. similarly a language model imitates words and sentences, but there is nothing else going on.
It is true that LLMs use randomness and statistical mathematics but that does not necessarily mean they are limited to only producing random statistical samplings. The whole is not always just the sum of the parts. LLMs can succeed in some limited reasoning. For example, it has been shown that simply asking an LLM to "think step-by-step" causes it to produce objectively better answers. How could this work if the LLM were not reasoning? If it was simply reproducing patterns you might as well just ask it to "give me a smarter sounding answer." LLMs are also able to solve logic and mathematical problems. Successfully finding solutions to such problems will not happen consistently by random chance or by simply producing textual patterns that look like answers. Some level of reasoning is required.
The hallucination problem occurs when you ask an LLM to produce an answer outside of its capabilities. Instead of saying, "I don't know," it will often make up something that looks like an answer. But humans do this all the time too and that does not mean we are incapable of reasoning.
It's easy and reassuring to dismiss LLMs as statistical parrots because they are then clearly not a threat. I don't believe LLMs alone, at least in their current form, pose a significant direct threat to humans. The only real danger is in how humans use LLMs for good or bad purposes. We are still a long way from AGI but these are exciting times.
@@jcoffland I think ppl conflate various types of reasoning. there is a type of reasoning that is purely mechanical, true and true = true, false or true = true, true xor true = false etc. you can memorize and do a lot of "reasoning" just by applying rules or patterns mechanically, you don't even need to know why each rule is true. classical computers and simple logic gates can already carry out this type of reasoning, thats why we've had theorem provers and prolog for decades. LLMs can do something similar but in a more statistical/probabilistic/fuzzy way. they can combine truthfulness of statements and draw conclusions probabilistically, so it looks like it has some common sense reasoning, but it;s not, it's just expanding the type of logic gate reasoning to linguistic representations. The other type of reasoning people talk about is about how the actual world works and cause/effect and intentionality. For example when I say "the cat is on the table" you dont just "compute" sentence linguistically/symbolically, but also visually, you imagine a cat on a table, because of how physics works you know the can cannot also be underneath at the same time, u know the cat could jump off at any point. you are not just thinking about that scene linguistically, like an LLM that has never actually seen a cat. its as expecting a human who has never seen, touched or heard anything and only been able to interact with a screen of printed words and a keyboard to be able to understand the world. maybe this will change once ai is able to process data in real time, is embedded in a robotic body and interact with the world, until then llms are not actually able to understand anything, they only have a purely linguistic intelligence/understanding (they manipulate words and symbols statistically)
Frodo! Is that you?!
Most likely, the phoenix from Gladiator
no that's eddie from limitless
Lip stick, east european accent, and AI.
They are beyond completeness as their attention mechanism esentially gives them a self destruction switch that flips the context and frame of a problem as it relative hardness requires it.
New Oracle Turing Machines could arise from the attention mechanism. Transformers should definitely be studied further by CS theorists.