Thanks to KiwiCo for sponsoring today's video! Go to www.kiwico.com/welchlabs and use code WELCHLABS for 50% off your first month of monthly lines and/or for 20% off your first Panda Crate.
AI today is growing exponentially, just curios, do you think we will ever hit a peak where innovation on AI will start to flat out, or hit a wall, and if so where and when do you think AI will hit its peak. You kind of skimmed over this in the end, i just wanted a bit of a longer explanation.
Was really keen to sign up for a crate for my daughter, but after 10 minutes of glitches on their system I just gave up. Not your fault of course, but you might want to let them know that their system is pants.
These days non-adsense being placed some ways into video, rather than with mutual consideration at very front/back/both where plenty people and myself would still watch, is instant skip/cliff off - though I wish success
30 years ago, I used to work with an older guy who retired from IBM. I was barely out of high school, and he used to tell me that neural networks were going to change the world once people figured out how to train them properly. He didn't live to see his dream become reality unfortunately, but he was totally right.
@@spartaleonidas540 guy I knew was named David Modlin. I wonder how many folks who had their prime years in the 60's and 70's saw this coming? I wish they had been able to see it. I suppose some of them might have lived to see it. Crazy to think about.
@@EdgarVerona Hinton's most important contributions came from the 80s onwards, but he has lived to see it, for one. He was working on neural nets in the 70s as a postdoc. It's all happened well within a human lifetime, is the crazy part.
@@squamish4244 Nice, that is very cool. Glad to hear he's still kicking! The guy I knew helped create handwriting recognition software in the 70's. It is crazy to think that someone could see basically the dawn of modern computing and also its progression to this crazy time we're in.
@@EdgarVerona Ray Kurzweil did too, but he's obsessed with mind-uploading, cryonics and resurrecting digital copies of his father etc. which is distracting, and he has trouble with being challenged on the practical implications of his predictions. He was right about the computing revolution but he's also a very strange dude. Hinton was running circles around him recently in a debate when both of them were onstage.
"one way to think about this vector, is as a point in 4096 dimentional space" give me a minute, I now gotta visualise a 4096 dimentional space in my head.
high dimensional spaces are crazy. A hypercube with the sides size=2, would have absolutely enormous surface and volume in 4096 dimension. size = 1, volume: 1 size = 1.01, volume, approx 501587856585103488.
That real-time kernel activation map was life-changing. If, whilst editing these videos, you've ever questioned if the vast amounts of effort are worth what amounts to a brief, 10s clip, just know that it's these moments which have stuck with me. Easy sub
I wanted to say this too. You actually did it, you make that animation. That is an amazing thing you've done, you've really added to the sum of human knowledge. The amount of effort must have been phenomenal. Really: thank you. Nobody else has done this. I know the effort of huge, but I'd love more even on just Alexnet. Animations on creating the node activation image generation. I'd love one of Resnet
The irony being, of course, that the script popped out of ChatGPT in about three seconds, editing by submagic slightly more, and images by stable diffusion in much less. But I agree, those few moments are worth it.
Would you say it is still worth it going into the field (studying AI) even after progress is made so incredibly fast nowadays that after the maybe 3-4 years of studying everything could have already changed again?
@@TheRealMcNuggs I say, if you love it (or have a strong interest) then absolutely! It's been changing quickly since I started, but the underlying fundamentals stay the same 👍
Computers not being fast enough to make a correct algorithm practically usable reminds me of Reed-Solomon error correcting codes. They were developed in 1960 but computers were too slow for them to be practical. They went unused until 1982 when they were used in Compact Discs after computers had become fast enough.
RS codes were used on the Voyager probes in 1977. CDs were the first large scale usage. Your basic point is still true: it took a while for computers to be complex enough to use them.
Bayesian models have followed a similar path; the basic idea is so fundamental as to be trivial, but actually using it in practice requires a high level (uh, I don't know what the big-O complexity is -- quadratic? worse?) of detail and thus computation to truly harness. The parameters might be trivial (individually, or conceptually), but there are so many of them for a problem of modest scale that it's only recently we've made much use of it.
It has always been an easy decision tree. Will the interesting case fit in system memory at all? It not, wait for the next system refresh. Can I tolerate the latency? Predicting tomorrow's weather a week from now is a good example of not being able to tolerate the latency. If it fits in memory and I can tolerate the latency, am I willing to pay for the computer time? I recall hearing stories in the 1980s about a power station with an entire Vax 11/780 devoted to running an FFT kernel on generator shaft vibration. There was no legal way to ship a replacement shaft. They had barely been allowed to truck in the first one over existing roads. Hence they spent the moon looking after the one they had.
Most people think AI is a brand new technology, while in reality there have been studies on Computer Neural Networks all the way back in the 1940s, that's insane.
But the real issue is that only now has computing power become strong enough to support everything, allowing research ideas to be realized into reality, and truly transforming these ideas into technologies with such astonishing effects.
@@empathogen75 Its just a popularity phase, TH-cam paid for itself when it was rapidly gaining users, we'll have Adobe level subscriptions in the future.
Fun fact, the kernels used in vision models work pretty much the same way as how our retinas perceive objects. In a similar structure, our eyes have cells that perceive edges at certain angles, then as shapes, then as objects in increasing abstraction.
They don’t at all; you are confusing a low level explanation for how our eyes really work Humans don’t work like the kernel at all; biology is far more efficiency and works in ways we don’t even understand yet
I was working with deep neural networks at the university during the late 90s, the main issue that stopped all progress was the use of a kind of functions between layers (the sigmoid as activation function), this effectively stopped the learning backpropagating from the output layers and limiting how many layers you can use (the problem is called the vanishing gradient). Once people rediscovered ReLU (it was invented in the early 70s, I believe, but I think the inventor published it in Japanese, so it went unnoticed) deep neural networks became possible. High computation needs were only a problem if you wanted real time or low latency, those days we used to leaving the computer calculating during nighttime to get something next day.
While this video perfectly explained how the networks work during recognition, I don't understand how they are actually training all the layers. Does anyone have a similar good source about teaching neural networks / backpropagation?
@@yannickhein9788 Hi, the most common algorithm used today, backpropagation, is based on propagate the "error" (the difference between the neural network, now on nn, prediction and real value) backwards, from the output to the input. One way of seeing it is for every layer in the nn (though not all nn can be divided in layers, but lets simplify) the error at its output is transformed to an error at its input, having into account the contribution of each neuron to the result. Performing a search in YT, there are two videos on top: th-cam.com/video/Ilg3gGewQ5U/w-d-xo.html&pp=ygUZYmFja3Byb3BhZ2F0aW9uIGV4cGxhaW5lZA%3D%3D th-cam.com/video/IN2XmBhILt4/w-d-xo.html&pp=ygUZYmFja3Byb3BhZ2F0aW9uIGV4cGxhaW5lZA%3D%3D
The visualization is what takes this video from good to fantastic. It's very evident you put a lot of effort into making this visually engaging, which is very didactic!
2:40 dude this single picture right here the way you described it this way literally was like the the thing that truly helps me understand how this all worked thank you
I really appreciate how well you communicate non-verbally despite using very little A-roll. You're expressions are clear yet natural even while reading, enunciating and employing tone, and there's no fluff; you have a neutral point for your hands to signal that there's no gesture to pay attention to. I couldn't find anything to critique in your vids if I tried and this seems particularly easy to overlook. Thanks for every absolute banger!
The vocal fry is annoying. A shame, because his vids are such fantastic quality otherwise. But I've literally just noped out of his vids before because it grates me so heavily. Use your lungs, my good man!
My dad graduated around 2009. One of his teachers (that was my teacher at Computer Science too) said at the time neural networks would thrive if they find their place in practical applications, but at the time most computer work as analogue to human work, but we've been learning how to abstract everything and work from there. That was when programs designed for computers instead of digital versions of physical media got very very popular. As he said: the tools were already there, we just needed to know how to use them, and we would shift from designing computer programs from real world to design the real world around computers. This teacher is still one of the smartest people I've ever seen so far. To put that into perspective, digital document control until a few years ago was very tied to how we handled them with paper and programs for dedicated digital control were a massive change of paradigm when they got adopted. We now tie small databases with documents, link them to project files, communicate projects with attributes... What a good place for something like NNs.
It is pretty amazing that these systems consume their own output to set the next step in their "reasoning". This implies that much of the true decision of the final output is actually already made in the first pass-through. And that the extra passes are just needed for us to extract the output in a way we can process. It also implies there is a "hidden" boundary of how far the first pass through can "reason", any cycles beyond that are "improvisations" of the path the output was already set on.
Very astute observation. And it gets to my biggest concern with any kind of recent AI model I encounter whether at work or in the wild: "what was this trained on"? Much like if you get a group of highly trained but inexperienced students together, the range/frequency of potential answers to a question near their field of expertise is likely to be a lot lower/tightly clustered than if you do the same with a less trained group. That initial lens through which the question gets passed (the training) can severely limit novel outputs. There are deeper connections between concepts it theoretically can make that humans may not, which is super cool, but fundamentally it's synthesizing. Various permutations and combinations of + - , * / , powers/square root, derivatives, and vectors. Another question I always have that is much harder to answer simply (if at all): "out of the various potential modes/models tested, what was it about this one being used that made it get selected for production". Haven't gotten a good answer yet; I'm sure if I dove deep I'd find some, but at least at work our AI folks aren't capable of explaining it.
A great learning experience i had was to deep dive into bitmap format and multiply greyscale images with 3x3, 5x5 arrays with simple patterns, ie all zero with a -1 in the middle. Different array patterns highlight edges or remove edges. it was a really eyeopening experience any software person should try that shows these fundamental operations. Great video.
It's rare to find an AI video this informative and interesting. Great pacing great focus, this is wonderful. I'm a particular fan of the sort of stop-motion / sped-up physical manipulation of papers on your desk with that overhead lighting. Very clean and engaging effect. Seeing the face-detecting kernel emerge after so few blocks was also mind-blowing!
It is not that the neural networks magically “understands” what is important. It is that the information is not random, so can be synthesized into smaller chunks. The synthesis process is what creates patterns, thus understanding.
I've been studying AI for the past year and the first 2 minutes was the best explanation I have see of how Transformers and ChatGPT works so far. Ive studied everything from Andrew Ngs Coursera courses, to Andrej Karpathy and more. Thank you for this great video!
Great video! I've been subbed ever since I've watched your amazing series on imaginary numbers, and the quality hasn't dropped and even improved. Looking forward to your next videos.
This is such a good explanation of AI, and the production value is so high. I'm bookmarking this so I can show it to my friends who ask me if I think AI is developing sentience.
feature activation visualization aren't interpretable and there's papers that have addressed this issue. Even before Alexnet, researchers couldn't interpret the weights of a deep neural network. There wasn't really a moment when we stopped understanding neural networks, we never really understood them. We understand back propagation and the frameworks (tensorflow, pytorch, tinygrad), but we don't understand the weights.
thats why neural nets are a thing to begin with, manually programming things that specific and abstract is not a job for a human brain, way too complex, too many moving parts, too much trial and error. its likely impossible for a single human to ever actually understand the step by step process of a neural net after training data.
You nailed it with this one. I'd love to see how much of your video was 'effect' and how much was real computation and composition. Seeing the third layer change with the video on that angle was very impressive.
I try to "show the real thing" whenever I can, only thing that was really "effect" i think was showing the 96 kernels learning. I thought about actually doing a training run, but didn't have the time!
@@WelchLabsVideo thanks for that honest insight. I'd put this above Grant's effort, which is a rock solid series. I really enjoyed the whole pace and b roll inserts of historical research.
Been trying to learn and implement CNNs in my PhD research and work, this has been the best video for helping me visualize each step of the process in my head :) Going to be trying to replicate these visualizations for my presentations as I think they're great ways to show how these 'black box' models function. As mentioned, the real time kernel mapping is super helpful. Wonderful work.
Great video, insightful visualizations! Regarding your footnote at 6:15, though: the myth that mantis shrimp have great color vision has been debunked long ago. They're worse at it than we are. Just because they have many photoreceptor types doesn't mean they combine them in a way similar to humans or other animals. Shouldn't have been too surprising, given their lack of brainpower.
Very informative video. Thanks!! It's great to see content that actually includes in depth research and knowledge and not just enthusiastic speculation.
Fun fact: Neural Networks are based on Neurons in the brain (hence the name), which we also do not know a lot about. Theory suggests that the neurons in our brain work very similar compared to a neural network in combining millions upon millions of simple transformations into something meaningful. This is also why research in fields like Cognitive Psychology go hand in hand with AI research. Very interesting to see where both fields are headed, because the key to understanding human intelligence is in understanding the unthinkable depths of neurons.
Earned a sub for sure. The visualizations, and especially those of the real time activation maps are just incredible tools for a better understanding. Got into DL out of a hobby and now I am using it for my research in my scientific field, especially image processing. Visualizing exactly how they map data to vectors in each layer was eye opening.
Fantastic visualizations. It is very appropriate to try to think through this transformation process as you illustrate to first see how the algorithm first reorganizes info as we perceive it into info optimized for the algorithm to recursively refine. Once you see this first iteration, then "lose sight" of the next abstraction, it becomes apparent how impossible it will be for any human to identify and correct a "flaw" in an AI model. The only approach for "correcting" a flaw in "learned data" is to somehow feed the AI more data. That assumes an imperfect system WON'T become MORE imperfect by consuming more input. This defies logic.
@@backwashjoe7864 Round #1 of the example showed that the algorithm is capble of creating flawed "links" or probabilities that lead to "incorrect" information being spit out for a given set of inputs. All of the inputs processsed in round #1 aren't "right" or "wrong," they just ARE. If the solution to (data)===> (partially incorrect output) is to feed more data in, there's no reason to expect round #2 to ELIMINATE the type of probabistic mistake encontered in round #1. It might REDUCE it but NEW errors can creep in, creating new errors in output, either for the original topic or some other prompt given the system.
Hey, thanks for this presentation. This video is a great example of how to teach about neural networks and their development. From the audio to the level of detail, everything was top notch. I hope you make more videos. I wish you much success, health and knowledge.
I believe that might happen when AGI becomes possible and more widespread, and the distinction between today’s AI and the next evolution into AGI becomes necessary.
A math professor of mine actually worked on many of the papers coming out of AI lab at MIT and he also worked on AI to play Minecraft. At the time it was really interesting to me as a sophomore, many years after I can write my own GPT, how the times haves changed!
I have been having this conversation for the last 2 years. Thanks for putting it in video form and expanding the conversation past layer 1, as most people are totally lost on layer 1.
Wow, this video was amazing! It helped me understand nuances of ML I hadn't yet grasped. In particular, the explanation of the filters through their use of the dot product as similarity maps. It's one of those things that seem obvious with hindsight, but require keen insight to find and explain!
As a very young engineer I got involved in NN with the publication in the Signal Processing IEEE journal an article on the MLP by Lippman. I also worked at a small company for the president who was at Cornell name Frank Rosenblatt. It became my job to integrate NN into our product. I developed a cool way to deal with regularization and realized how critical regularization was as we had very little data. Did not have a billion images of cats. I wrote early FORTRAN code for back propagation that ran in a Sky Warrior array processor. No one knew what would become of the field and the history of Rosenblatt v Minsky. I am sad that Frank never lived long enough to see the word 'Perceptron' on t-shirts. He won. Minsky is a foot note.
Truly amazing video, really great explanations and way of telling these hard to understand concepts. This got me more exited to learn more about this than an entire year of ai at university
Great video. The only nitpick is with title: we haven't stopped understanding AI at AlexNet (and video clearly shows that we only getting better at understanding since that moment), we finally had working "AI" starting from AlexNet. All those "expert handcrafted" AIs before were no simpler to understand (if not harder) despite being handcrafted. And they largely didn't work and it was very hard to understand why.
@@Anonymous-df8it too simple and brittle to capture the real world, I think. I started working on computer vision right after deep learning started to solve problems one by one but was not yet commonly accepted. So for some time people tried to use old and new methods and every single time classic methods only worked with toy versions of the problem and broke apart in real world when anything changed that you as human don't even notice, like different lamp temperature or some reflection.
@@rotors_taker_0h Why would it be difficult to understand how they "work" or why they didn't? Also, what were the 'classic methods' and could people in the soft sciences who know programming create an image identifier or chatbot that actually thinks like us (which should work since people can do those things, and the code should be intuitive since it's our own thought processes)? I don't know about you, but I don't remember multiplying giant matrices together a bunch of times when thinking about how to respond to you (you could argue that I did but aliens wiped my memory or something, but that would be unfalsifiable), and whilst glare and monochromatic light sources would probably make it hard to see things, those are extreme cases, and I can certainly handle sunlight vs indoor lighting
incredible video. I'm working with a ResNet on a project and sometimes, focusing on minor bugs and programming headeaches, I tend to lose the perspective on the amazing tool I'm dealing with. This video was a pleasure to watch
🎯 Key points for quick navigation: 00:00 *🤖 Introduction to AI model activation spaces* - Overview of how modern AI models like AlexNet organize and make sense of information - Description of the structure of the first model that demonstrated this, AlexNet - Introduction to the concept of high-dimensional embedding spaces in AI models 02:10 *🧠 Training and capabilities of AlexNet* - Explanation of how AlexNet was trained to predict labels from images - Detailed breakdown of the convolutional layers and visual patterns learned by AlexNet - Discussion on how AlexNet maps inputs to outputs using layers of compute blocks 08:00 *🔍 Visualizing high-dimensional embedding spaces in AI models* - Exploration of the final layers of AlexNet and the creation of a high-dimensional output vector - Description of how high-dimensional spaces can be visualized through activation atlases - Insights into how deep neural networks organize visual information in embedding spaces Made with HARPA AI
Yes, I've never taken LSD, but I understood that it is similar when I saw so called deep dream images ca 2015. Holy s*** I thought, if AI can hallucinate like that it must be working like the brain.
@@nicholasn.2883which means without prior knowledge you will not understand much of it. at least with math, it can be applied pretty universally except at extremely high levels.
The explanations, visualizations, and animations in this are incredible! You really got me to feel a much more intuitive understanding of a lot of the concepts I've been reading about for years.
The more I learn about this so called "AI" (while completely amazing, don't get me wrong), the more I realize the hype is a crock of shit. It cannot reason.
What people fail to explain is, the training has 2 core chunks. The first stage is 'pre-training' when it is fed millions of words, to understand general relationships between them. No strcuture just words and letters. The second stage is secret but we can speculate this is the 'fine-tuning' stage where data is provided as a JSON file containing a question and answer parts. I mean this is how they would do it if smart. There are also other 'experts' like code maths etc....
@@rahul_siloniya Because OpenAI and Meta keep their training datasets and procedures secret. We can't learn anything meaningful about how LLaMA or GPT was trained by looking at the model, as the model is just a set of seemingly random weights with no indication of how the weights were arrived at or what the weights actually mean. Anthropic are trying to reverse engineer the weights to figure out what they mean, but that still leaves us in the dark regarding how these models were trained.
This video is a fantastic resource for anyone interested in AI. Your ability to explain the intricate workings of AlexNet and GPT is commendable. Keep up the great work!
15:58 lol man thats nuts, from a Pentium II to a couple Nvidia cards that gamers used in their home pc's a few years ago to 25,000 specialized A100 gpus. Absolutely mind-boggling
Gotta love what Moore's Law did for us while it still worked. We're walking around with true supercomputers in our pockets, yet we always think we need to go into the 'cloud' or use a 'real' computer to do anything important. Freakin' nuts.
At 16:50 you mention that ChatGPT's transformer blocks are a generalization of the convolutional compute blocks in AlexNet. Why would you say this is? I don't see how convolutions with a sliding window approach could be generalized to attention; the models seem quite fundamentally different to me. I would argue that self-attention in transformers much more naturally evolved from RNNs instead of CNNs. Or is there some nice intuitive connection between convolutions and self-attention that I am not aware of?
So the Mayan calendar predicting that there would be the start of a new age the Mayans couldn't comprehend in 2012 was, in a way, accurate? The AI age started with AlexAI in 2012?
@@DizGaAlcam Yeah, and it's certainly just a coincidence that my animal brain is seeing as a pattern, but still! Concerning for animal brain reasons. 😅
Thanks to KiwiCo for sponsoring today's video! Go to www.kiwico.com/welchlabs and use code WELCHLABS for 50% off your first month of monthly lines and/or for 20% off your first Panda Crate.
AI today is growing exponentially, just curios, do you think we will ever hit a peak where innovation on AI will start to flat out, or hit a wall, and if so where and when do you think AI will hit its peak.
You kind of skimmed over this in the end, i just wanted a bit of a longer explanation.
Oops, I farted 4o
Was really keen to sign up for a crate for my daughter, but after 10 minutes of glitches on their system I just gave up. Not your fault of course, but you might want to let them know that their system is pants.
could have done it with ASIC a long time ago.
Just living out THEIR best life possible first...
These days non-adsense being placed some ways into video, rather than with mutual consideration at very front/back/both where plenty people and myself would still watch, is instant skip/cliff off - though I wish success
30 years ago, I used to work with an older guy who retired from IBM. I was barely out of high school, and he used to tell me that neural networks were going to change the world once people figured out how to train them properly. He didn't live to see his dream become reality unfortunately, but he was totally right.
Same except the guy was at Toronto and his name was Hinton
@@spartaleonidas540 guy I knew was named David Modlin. I wonder how many folks who had their prime years in the 60's and 70's saw this coming? I wish they had been able to see it. I suppose some of them might have lived to see it. Crazy to think about.
@@EdgarVerona Hinton's most important contributions came from the 80s onwards, but he has lived to see it, for one. He was working on neural nets in the 70s as a postdoc. It's all happened well within a human lifetime, is the crazy part.
@@squamish4244 Nice, that is very cool. Glad to hear he's still kicking! The guy I knew helped create handwriting recognition software in the 70's. It is crazy to think that someone could see basically the dawn of modern computing and also its progression to this crazy time we're in.
@@EdgarVerona Ray Kurzweil did too, but he's obsessed with mind-uploading, cryonics and resurrecting digital copies of his father etc. which is distracting, and he has trouble with being challenged on the practical implications of his predictions. He was right about the computing revolution but he's also a very strange dude. Hinton was running circles around him recently in a debate when both of them were onstage.
"one way to think about this vector, is as a point in 4096 dimentional space"
give me a minute, I now gotta visualise a 4096 dimentional space in my head.
Are you done yet ? 🙂
"One way to think about a point in 4096 dimensional space, is as a vector"
high dimensional spaces are crazy. A hypercube with the sides size=2, would have absolutely enormous surface and volume in 4096 dimension.
size = 1, volume: 1
size = 1.01, volume, approx 501587856585103488.
@@adamrak7560what does volume mean for non-3d thing?
Easy, image a 3-dimensional space and pretend it's 4096 dimensions.
I mean, that's basically what the visualizations in the video are doing.
That real-time kernel activation map was life-changing.
If, whilst editing these videos, you've ever questioned if the vast amounts of effort are worth what amounts to a brief, 10s clip, just know that it's these moments which have stuck with me. Easy sub
Ikr, shows the hard work of this guy and that is something I respect.
I wanted to say this too. You actually did it, you make that animation. That is an amazing thing you've done, you've really added to the sum of human knowledge.
The amount of effort must have been phenomenal. Really: thank you. Nobody else has done this. I know the effort of huge, but I'd love more even on just Alexnet. Animations on creating the node activation image generation.
I'd love one of Resnet
The irony being, of course, that the script popped out of ChatGPT in about three seconds, editing by submagic slightly more, and images by stable diffusion in much less. But I agree, those few moments are worth it.
@@JoseJimeniz While I'm sure it took a lot of work, someone else already did most of the work for the Activation Atlas.
Same, this was truly eye opening
I've been in the field for 10 years and never had anyone describe this so clearly and visually. Brilliant, thank you!
same here (9 years)
Would you say it is still worth it going into the field (studying AI) even after progress is made so incredibly fast nowadays that after the maybe 3-4 years of studying everything could have already changed again?
@@TheRealMcNuggs I say, if you love it (or have a strong interest) then absolutely! It's been changing quickly since I started, but the underlying fundamentals stay the same 👍
3blue1brown made a whole gen ai series which goes much deeper and visualises things better, I do recommend to have a look, really interesting stuff
Im still confused 😭
Computers not being fast enough to make a correct algorithm practically usable reminds me of Reed-Solomon error correcting codes. They were developed in 1960 but computers were too slow for them to be practical. They went unused until 1982 when they were used in Compact Discs after computers had become fast enough.
RS codes were used on the Voyager probes in 1977. CDs were the first large scale usage. Your basic point is still true: it took a while for computers to be complex enough to use them.
Bayesian models have followed a similar path; the basic idea is so fundamental as to be trivial, but actually using it in practice requires a high level (uh, I don't know what the big-O complexity is -- quadratic? worse?) of detail and thus computation to truly harness. The parameters might be trivial (individually, or conceptually), but there are so many of them for a problem of modest scale that it's only recently we've made much use of it.
@@jimktrains0 I should have specified first wide-spread use.
Logix programming same prediction, eill be viable in a yeR year and I will do it ???😮❤😂🎉🇨🇭😘💶💶💶🍆🍑🍆🥑⛔⛔⛔🪬🤣😅🏳️🌈✡️💪🏾👯♂️♂️🔯✡️🔯👬🕎♀️⛔
It has always been an easy decision tree. Will the interesting case fit in system memory at all? It not, wait for the next system refresh. Can I tolerate the latency? Predicting tomorrow's weather a week from now is a good example of not being able to tolerate the latency. If it fits in memory and I can tolerate the latency, am I willing to pay for the computer time?
I recall hearing stories in the 1980s about a power station with an entire Vax 11/780 devoted to running an FFT kernel on generator shaft vibration. There was no legal way to ship a replacement shaft. They had barely been allowed to truck in the first one over existing roads. Hence they spent the moon looking after the one they had.
Most people think AI is a brand new technology, while in reality there have been studies on Computer Neural Networks all the way back in the 1940s, that's insane.
But the real issue is that only now has computing power become strong enough to support everything, allowing research ideas to be realized into reality, and truly transforming these ideas into technologies with such astonishing effects.
@@louis-dieudonne5941 makes you think, what are we studying now that will only be possible years in the future because of the lack of resources.
It’s new in the sense that neural networks are relatively inexpensive and for the first time broadly applicable to a wide range of tasks.
@@empathogen75 Its just a popularity phase, TH-cam paid for itself when it was rapidly gaining users, we'll have Adobe level subscriptions in the future.
@@louis-dieudonne5941Not just hardware, but data as well.
Fun fact, the kernels used in vision models work pretty much the same way as how our retinas perceive objects. In a similar structure, our eyes have cells that perceive edges at certain angles, then as shapes, then as objects in increasing abstraction.
only edge detection occurs in the retina, anything more complex than that happens higher up in the various visual areas of the brain
They don’t at all; you are confusing a low level explanation for how our eyes really work
Humans don’t work like the kernel at all; biology is far more efficiency and works in ways we don’t even understand yet
@@PallasTurrets Whoops I forgot to mention but yeah, more complex stuff still occurs in the brain. Thanks for correcting me
Their similarity is less than between an airplane and a bird.
@@ВалентинТ-х6цmeaning?
Do you have a more detailed understanding of human vision to share to compare and contrast ?
I was working with deep neural networks at the university during the late 90s, the main issue that stopped all progress was the use of a kind of functions between layers (the sigmoid as activation function), this effectively stopped the learning backpropagating from the output layers and limiting how many layers you can use (the problem is called the vanishing gradient). Once people rediscovered ReLU (it was invented in the early 70s, I believe, but I think the inventor published it in Japanese, so it went unnoticed) deep neural networks became possible. High computation needs were only a problem if you wanted real time or low latency, those days we used to leaving the computer calculating during nighttime to get something next day.
Thank you for all the work you did.
Thank you for all your work, cant imagine doing all this back then
While this video perfectly explained how the networks work during recognition, I don't understand how they are actually training all the layers. Does anyone have a similar good source about teaching neural networks / backpropagation?
Bro was working on a toaster 😭
@@yannickhein9788 Hi, the most common algorithm used today, backpropagation, is based on propagate the "error" (the difference between the neural network, now on nn, prediction and real value) backwards, from the output to the input. One way of seeing it is for every layer in the nn (though not all nn can be divided in layers, but lets simplify) the error at its output is transformed to an error at its input, having into account the contribution of each neuron to the result. Performing a search in YT, there are two videos on top:
th-cam.com/video/Ilg3gGewQ5U/w-d-xo.html&pp=ygUZYmFja3Byb3BhZ2F0aW9uIGV4cGxhaW5lZA%3D%3D
th-cam.com/video/IN2XmBhILt4/w-d-xo.html&pp=ygUZYmFja3Byb3BhZ2F0aW9uIGV4cGxhaW5lZA%3D%3D
Your visualisations helped a few concepts click for me around the layers and activations Ive struggled to understand for years. Thanks!
The visualization is what takes this video from good to fantastic. It's very evident you put a lot of effort into making this visually engaging, which is very didactic!
i had to search what didactic means
Awesome video! Funny how the moment we stopped understanding AI also appears to be the moment it started working lol
An astute observation.
It works like the brain. And like the brain, the moment the results are interesting is when they have enough oomph
"If the brain were so simple we could understand it, we would be so simple that we couldn't"
The same is true for AI.
AI cant verify the truth of the answers it gives. It often gives shit answers.. So saying it works is a bit of a reach
@@Sqlldude Humans can't verify the truth of the answers they give either. Both need an external source.
2:40 dude this single picture right here the way you described it this way literally was like the the thing that truly helps me understand how this all worked thank you
I really appreciate how well you communicate non-verbally despite using very little A-roll. You're expressions are clear yet natural even while reading, enunciating and employing tone, and there's no fluff; you have a neutral point for your hands to signal that there's no gesture to pay attention to.
I couldn't find anything to critique in your vids if I tried and this seems particularly easy to overlook. Thanks for every absolute banger!
@@frostebyte he is truly a master teacher we can all learn from
The vocal fry is annoying. A shame, because his vids are such fantastic quality otherwise. But I've literally just noped out of his vids before because it grates me so heavily. Use your lungs, my good man!
Half of these comments read like they were written by chatgpt lmao
@@sntslilhlpr6601I don’t know what “vocal fry” is but his voice doesn’t sound annoying to me.
Your*
Stellar video, you’re gifted at communication. Keep at it!
Thank you! Will do!
sir, thank you
Very clever and nice vizualisations! Excellent as usual.
Thank you!
Amazing intro with scissor and carboards 👏
Thank you 😁
I was also amazed by this
Fantastic presentation on the inner workings of machine learning!
Glad you enjoyed it!
I stopped understanding AI around the six minute mark.
Lol it’s wild technical stuff
98% do
What stopped you?
@@P4INKiller that's a legitimately good question
same XD
The amount of work you must put into videos is mind boggling. Thank you for making them.
The visual aid in this video is unlike I've seen anywhere else, it really helps grasp the ideas presented easily, wonderful video!
GREAT video. Your crystal clear script and visuals make a very complex topic approachable and your production values are top notch. Kudos!
My dad graduated around 2009. One of his teachers (that was my teacher at Computer Science too) said at the time neural networks would thrive if they find their place in practical applications, but at the time most computer work as analogue to human work, but we've been learning how to abstract everything and work from there. That was when programs designed for computers instead of digital versions of physical media got very very popular. As he said: the tools were already there, we just needed to know how to use them, and we would shift from designing computer programs from real world to design the real world around computers. This teacher is still one of the smartest people I've ever seen so far.
To put that into perspective, digital document control until a few years ago was very tied to how we handled them with paper and programs for dedicated digital control were a massive change of paradigm when they got adopted. We now tie small databases with documents, link them to project files, communicate projects with attributes... What a good place for something like NNs.
It is pretty amazing that these systems consume their own output to set the next step in their "reasoning".
This implies that much of the true decision of the final output is actually already made in the first pass-through.
And that the extra passes are just needed for us to extract the output in a way we can process.
It also implies there is a "hidden" boundary of how far the first pass through can "reason", any cycles beyond that are "improvisations" of the path the output was already set on.
Very astute observation. And it gets to my biggest concern with any kind of recent AI model I encounter whether at work or in the wild: "what was this trained on"? Much like if you get a group of highly trained but inexperienced students together, the range/frequency of potential answers to a question near their field of expertise is likely to be a lot lower/tightly clustered than if you do the same with a less trained group. That initial lens through which the question gets passed (the training) can severely limit novel outputs.
There are deeper connections between concepts it theoretically can make that humans may not, which is super cool, but fundamentally it's synthesizing. Various permutations and combinations of + - , * / , powers/square root, derivatives, and vectors.
Another question I always have that is much harder to answer simply (if at all): "out of the various potential modes/models tested, what was it about this one being used that made it get selected for production". Haven't gotten a good answer yet; I'm sure if I dove deep I'd find some, but at least at work our AI folks aren't capable of explaining it.
A great learning experience i had was to deep dive into bitmap format and multiply greyscale images with 3x3, 5x5 arrays with simple patterns, ie all zero with a -1 in the middle. Different array patterns highlight edges or remove edges. it was a really eyeopening experience any software person should try that shows these fundamental operations. Great video.
This was an incredible introduction in just 18 minutes. I continue to be blown away by this channel.
Woohoo!
@@WelchLabsVideothe KJV Bible is mathematically encoded by God
What a remarkably well thought out, well organised, well created video! Just stumbled upon this channel and glad i did !
It's rare to find an AI video this informative and interesting. Great pacing great focus, this is wonderful.
I'm a particular fan of the sort of stop-motion / sped-up physical manipulation of papers on your desk with that overhead lighting. Very clean and engaging effect. Seeing the face-detecting kernel emerge after so few blocks was also mind-blowing!
Amazing - thank you!
Dude your videos are amazing and that office space looks amazing.
I stopped understanding this video at about 2 minutes
I've never seen AlexNet this way with a live preview of what happens inside. I'm sure it required a lot of time and programming: great job!
I was literally talking to my roommate about this last night!! Thanks for the excellent video
The stop-motion and animation, including visualizing AlexNet's activation, were incredible!
It is not that the neural networks magically “understands” what is important. It is that the information is not random, so can be synthesized into smaller chunks. The synthesis process is what creates patterns, thus understanding.
I've been studying AI for the past year and the first 2 minutes was the best explanation I have see of how Transformers and ChatGPT works so far. Ive studied everything from Andrew Ngs Coursera courses, to Andrej Karpathy and more. Thank you for this great video!
The quality of this content is remarkable - great job! Looking forward to watching more awesome videos like this one.
1:04 the moment I stopped understanding this video
Great video! I've been subbed ever since I've watched your amazing series on imaginary numbers, and the quality hasn't dropped and even improved. Looking forward to your next videos.
This is such a good explanation of AI, and the production value is so high. I'm bookmarking this so I can show it to my friends who ask me if I think AI is developing sentience.
Best dynamic illustrations yet. Using highlights on physically printed research papers is a wonderful story telling technique.
Love this video. First one where I finally understand how gpt-4 works. Thank you.
feature activation visualization aren't interpretable and there's papers that have addressed this issue. Even before Alexnet, researchers couldn't interpret the weights of a deep neural network. There wasn't really a moment when we stopped understanding neural networks, we never really understood them.
We understand back propagation and the frameworks (tensorflow, pytorch, tinygrad), but we don't understand the weights.
thats why neural nets are a thing to begin with, manually programming things that specific and abstract is not a job for a human brain, way too complex, too many moving parts, too much trial and error. its likely impossible for a single human to ever actually understand the step by step process of a neural net after training data.
You nailed it with this one. I'd love to see how much of your video was 'effect' and how much was real computation and composition. Seeing the third layer change with the video on that angle was very impressive.
I try to "show the real thing" whenever I can, only thing that was really "effect" i think was showing the 96 kernels learning. I thought about actually doing a training run, but didn't have the time!
@@WelchLabsVideo thanks for that honest insight. I'd put this above Grant's effort, which is a rock solid series. I really enjoyed the whole pace and b roll inserts of historical research.
The amount of work that went into that visualisation i would love a behind the scenes video!
I have some on TikTok!
Been trying to learn and implement CNNs in my PhD research and work, this has been the best video for helping me visualize each step of the process in my head :) Going to be trying to replicate these visualizations for my presentations as I think they're great ways to show how these 'black box' models function. As mentioned, the real time kernel mapping is super helpful. Wonderful work.
Great video, insightful visualizations! Regarding your footnote at 6:15, though: the myth that mantis shrimp have great color vision has been debunked long ago. They're worse at it than we are. Just because they have many photoreceptor types doesn't mean they combine them in a way similar to humans or other animals. Shouldn't have been too surprising, given their lack of brainpower.
Yes. Human vision takes up a huge part of our brain.
Say that to a mantis shrimp's face, I dare you.
This is amazing! Excellent presentation, this also illuminated for me what exactly a "latent space" is. Thank you!
Wow, so much effort has been put into the animations. Subscribed.
understatement.
Very informative video. Thanks!! It's great to see content that actually includes in depth research and knowledge and not just enthusiastic speculation.
Fun fact: Neural Networks are based on Neurons in the brain (hence the name), which we also do not know a lot about. Theory suggests that the neurons in our brain work very similar compared to a neural network in combining millions upon millions of simple transformations into something meaningful. This is also why research in fields like Cognitive Psychology go hand in hand with AI research. Very interesting to see where both fields are headed, because the key to understanding human intelligence is in understanding the unthinkable depths of neurons.
I absolutely appreciate the way in which you present this information in an easy to consume and understand format. Brilliant my friend.
Earned a sub for sure. The visualizations, and especially those of the real time activation maps are just incredible tools for a better understanding. Got into DL out of a hobby and now I am using it for my research in my scientific field, especially image processing. Visualizing exactly how they map data to vectors in each layer was eye opening.
hehe, "hotdog / not hotdog".
😂😂😂 zin yaang
SEEfood
Hotdog / NotDog
You were there 7 years ago when i started my AI journey in images….back at it when i start in language. Truly great work….love your content 👏👏👏
Fantastic visualizations. It is very appropriate to try to think through this transformation process as you illustrate to first see how the algorithm first reorganizes info as we perceive it into info optimized for the algorithm to recursively refine. Once you see this first iteration, then "lose sight" of the next abstraction, it becomes apparent how impossible it will be for any human to identify and correct a "flaw" in an AI model. The only approach for "correcting" a flaw in "learned data" is to somehow feed the AI more data. That assumes an imperfect system WON'T become MORE imperfect by consuming more input. This defies logic.
How does that defy logic?
@@backwashjoe7864 Round #1 of the example showed that the algorithm is capble of creating flawed "links" or probabilities that lead to "incorrect" information being spit out for a given set of inputs. All of the inputs processsed in round #1 aren't "right" or "wrong," they just ARE. If the solution to (data)===> (partially incorrect output) is to feed more data in, there's no reason to expect round #2 to ELIMINATE the type of probabistic mistake encontered in round #1. It might REDUCE it but NEW errors can creep in, creating new errors in output, either for the original topic or some other prompt given the system.
Hey, thanks for this presentation. This video is a great example of how to teach about neural networks and their development. From the audio to the level of detail, everything was top notch. I hope you make more videos. I wish you much success, health and knowledge.
Fantastic video. I appreciate the time spent to create it
This is a fantastic video. Thanks for visualising the kernels so well. I enjoyed every minute of it. I've re-watched a couple of times now :)
There will be some point in time, when people stop call statistic models an AI, but it will not be today for sure.
I bet on 6 months after fusion solved...
I finally gave up about a year ago in trying to hold the line on the definitional shift of "AI".
@@waylandsmith yep, it's seriously tiring. And frustrating beyond belief. And depressing.
I believe that might happen when AGI becomes possible and more widespread, and the distinction between today’s AI and the next evolution into AGI becomes necessary.
I'm only 2 minutes into the video and already impressed by the best explanation I've ever seen about how models like gpt work
A math professor of mine actually worked on many of the papers coming out of AI lab at MIT and he also worked on AI to play Minecraft. At the time it was really interesting to me as a sophomore, many years after I can write my own GPT, how the times haves changed!
I have been having this conversation for the last 2 years. Thanks for putting it in video form and expanding the conversation past layer 1, as most people are totally lost on layer 1.
Very beautiful. I loved the music background also, specially at the end!!
Thank you!
The video -> layers -> activation map animation is one of the best clarifying animations I have seen describing this process.
Wow, this video was amazing! It helped me understand nuances of ML I hadn't yet grasped. In particular, the explanation of the filters through their use of the dot product as similarity maps. It's one of those things that seem obvious with hindsight, but require keen insight to find and explain!
Woohoo!
The visualization at 5:11 is absolutely amazing, kudos!!
As a very young engineer I got involved in NN with the publication in the Signal Processing IEEE journal an article on the MLP by Lippman. I also worked at a small company for the president who was at Cornell name Frank Rosenblatt. It became my job to integrate NN into our product. I developed a cool way to deal with regularization and realized how critical regularization was as we had very little data. Did not have a billion images of cats. I wrote early FORTRAN code for back propagation that ran in a Sky Warrior array processor. No one knew what would become of the field and the history of Rosenblatt v Minsky. I am sad that Frank never lived long enough to see the word 'Perceptron' on t-shirts. He won. Minsky is a foot note.
Truly amazing video, really great explanations and way of telling these hard to understand concepts. This got me more exited to learn more about this than an entire year of ai at university
Great video. The only nitpick is with title: we haven't stopped understanding AI at AlexNet (and video clearly shows that we only getting better at understanding since that moment), we finally had working "AI" starting from AlexNet. All those "expert handcrafted" AIs before were no simpler to understand (if not harder) despite being handcrafted. And they largely didn't work and it was very hard to understand why.
Why didn't they work?
@@Anonymous-df8it too simple and brittle to capture the real world, I think. I started working on computer vision right after deep learning started to solve problems one by one but was not yet commonly accepted. So for some time people tried to use old and new methods and every single time classic methods only worked with toy versions of the problem and broke apart in real world when anything changed that you as human don't even notice, like different lamp temperature or some reflection.
@@rotors_taker_0h Why would it be difficult to understand how they "work" or why they didn't? Also, what were the 'classic methods' and could people in the soft sciences who know programming create an image identifier or chatbot that actually thinks like us (which should work since people can do those things, and the code should be intuitive since it's our own thought processes)?
I don't know about you, but I don't remember multiplying giant matrices together a bunch of times when thinking about how to respond to you (you could argue that I did but aliens wiped my memory or something, but that would be unfalsifiable), and whilst glare and monochromatic light sources would probably make it hard to see things, those are extreme cases, and I can certainly handle sunlight vs indoor lighting
@@Anonymous-df8ityou didn’t do any math in your head, but your brain did all sorts of calculations behind the scenes.
@@gilbert2720 Like what?
incredible video.
I'm working with a ResNet on a project and sometimes, focusing on minor bugs and programming headeaches, I tend to lose the perspective on the amazing tool I'm dealing with. This video was a pleasure to watch
Visualization was just wonderful, but what attracted me more is the way of delivering the information.
Excellent work! Keep it up!
🎯 Key points for quick navigation:
00:00 *🤖 Introduction to AI model activation spaces*
- Overview of how modern AI models like AlexNet organize and make sense of information
- Description of the structure of the first model that demonstrated this, AlexNet
- Introduction to the concept of high-dimensional embedding spaces in AI models
02:10 *🧠 Training and capabilities of AlexNet*
- Explanation of how AlexNet was trained to predict labels from images
- Detailed breakdown of the convolutional layers and visual patterns learned by AlexNet
- Discussion on how AlexNet maps inputs to outputs using layers of compute blocks
08:00 *🔍 Visualizing high-dimensional embedding spaces in AI models*
- Exploration of the final layers of AlexNet and the creation of a high-dimensional output vector
- Description of how high-dimensional spaces can be visualized through activation atlases
- Insights into how deep neural networks organize visual information in embedding spaces
Made with HARPA AI
so basically AI sees in LSD
Yes, I've never taken LSD, but I understood that it is similar when I saw so called deep dream images ca 2015. Holy s*** I thought, if AI can hallucinate like that it must be working like the brain.
Phenomenal video!! Love the cut out approach to showing how the different elements of the algorithms come together!
The moment I stopped understanding a single word: 0:01.
"This is an activation atlas"
Understandable, have a good day.
it’s not that hard you’re not doing any math just concepts
@@nicholasn.2883which means without prior knowledge you will not understand much of it. at least with math, it can be applied pretty universally except at extremely high levels.
Instant sub. Incredible effort in visualisation and general editing.
alr why does the right poster 17:38 look like Africa
Pre continental drift heatlands
By far one of the best explanations of these mechanics in a video intended for laymen. Congrats + subscribed
The 3D visualizations of the neural network activation is incredible. What did you use to do it?
Really crappy VPython code I wrote.
@@WelchLabsVideo it's not crappy if it works!
The explanations, visualizations, and animations in this are incredible! You really got me to feel a much more intuitive understanding of a lot of the concepts I've been reading about for years.
What an insanly high production value.
2:24 no way you actually asked if it was mad 💀
absolutely amazing video, with great explenation and visuals. Keep up the good work!!
The more I learn about this so called "AI" (while completely amazing, don't get me wrong), the more I realize the hype is a crock of shit. It cannot reason.
They boost the hype to get money from stupid investors
Excellent visualisations, super easy to understand, great vid!
What people fail to explain is, the training has 2 core chunks. The first stage is 'pre-training' when it is fed millions of words, to understand general relationships between them. No strcuture just words and letters. The second stage is secret but we can speculate this is the 'fine-tuning' stage where data is provided as a JSON file containing a question and answer parts. I mean this is how they would do it if smart. There are also other 'experts' like code maths etc....
Why is it secret now? Can't we look at Llama and check what that "secret" step is?
@@rahul_siloniya Because OpenAI and Meta keep their training datasets and procedures secret. We can't learn anything meaningful about how LLaMA or GPT was trained by looking at the model, as the model is just a set of seemingly random weights with no indication of how the weights were arrived at or what the weights actually mean. Anthropic are trying to reverse engineer the weights to figure out what they mean, but that still leaves us in the dark regarding how these models were trained.
The second step probably involves virgin sacrifices, demon summonings and an assortment of scented candles from Bath & Body Works
This video is a fantastic resource for anyone interested in AI. Your ability to explain the intricate workings of AlexNet and GPT is commendable. Keep up the great work!
13:53 Curious how logic operations look so much like the brain's own neural network....
15:58 lol man thats nuts, from a Pentium II to a couple Nvidia cards that gamers used in their home pc's a few years ago to 25,000 specialized A100 gpus. Absolutely mind-boggling
Gotta love what Moore's Law did for us while it still worked. We're walking around with true supercomputers in our pockets, yet we always think we need to go into the 'cloud' or use a 'real' computer to do anything important. Freakin' nuts.
“No one told Alex what a face was, we just forced it to see millions of them over and over again, crazzzyyyyy!”
“What’s even crazier is that the math from a photo is similar to other similar photos, crazyyyyyyyy”
How can we make Alex unlock 100% of its intelligence?
brilliant didactic visualizations. I directly subscribed
Yes, Ai is literally about creating programs that are too complex for a human to understand.
At 16:50 you mention that ChatGPT's transformer blocks are a generalization of the convolutional compute blocks in AlexNet. Why would you say this is? I don't see how convolutions with a sliding window approach could be generalized to attention; the models seem quite fundamentally different to me. I would argue that self-attention in transformers much more naturally evolved from RNNs instead of CNNs. Or is there some nice intuitive connection between convolutions and self-attention that I am not aware of?
2:20 the answer is “IP theft & plagiarism”
💯
I think the most amazing things are the stories, poems, image descriptions or images, jokes, and roleplaying that AI can do.
Man im too dumb for these videos so sad
been following since the early early days, and gotta say that it's criminal that you haven't gotten an award for science communication yet.
Is the hotdog a reference to Silicon Valley?
It's also a reference to NOT hotdog 🌭
Like Shazam for food!
Earned my sub man!! That was amazing! So much work is in that vid! Love the crafting and stop motion.
So the Mayan calendar predicting that there would be the start of a new age the Mayans couldn't comprehend in 2012 was, in a way, accurate? The AI age started with AlexAI in 2012?
Ur onto smth
@@DizGaAlcam Yeah, and it's certainly just a coincidence that my animal brain is seeing as a pattern, but still! Concerning for animal brain reasons. 😅