i think you will find out that some people know that they wil be taking this technology away from the public ....... same as google glass .... as soon as america find out they cannot sread dissemination of knowledge they will cuts all internet cables !
Do not ever look at how long your videos are. Your content is perfect and you should keep explaining things step by step. You are doing a great job. I believe you will be remembered in history as one of the pillars of AI.
I think one cool thing with videos like this is once Google implements their AI into youtube anyone will be able to watch this and just start asking questions. I've been learning a lot by watching videos like this and just copying parts of the transcript into ChatGPT to ask questions when I dont understand something.
Yes a great sharer of the important information and implementation. As well as to let you know it's in your hands to make your own before the internet is closed down or restricted in your areas or your cable gets cut !! Great work ❤
Sorry , i love your videos and what you doing for me. I couldn't attend Stanford or get into openai but learning from you is blessing to me.. i would pay you back 100times in coming years. And i was watching your git repository last two months, i could see many git code push in private ,but i was confused what he is working on.. this is he was working on. To provide quality pratical knowledge to us all on youtube.
@@AndrejKarpathy The sequel "GPT The Movie" will be a old Hong Kong style "martial arts" movie about GPT getting beaten up by the Loss Function, then entering his training phase with Gradient Descent Sensei and the final showdown vs the big evaluation boss.
I am an undergraduate student. This is the lost lecture that professors never touched upon but absolutely crucial, thank you!! I especially love how you start from the basics for so many notions, and I really learned a lot.
Many Thanks to Andrej for making this tutorial available to everyone! I have never seen a clearer explanation of a nn before stumbling upon this zero to hero series. This will help all the people articulate the inner workings of neural net and help people understand deeper concepts, that is hard to understand. Looking forward to learning more with Andrej!
The longer your videos are, the better it is for humanity. I think you are such a wonderful person and providing this stuff for everyone for free, can't thank you enough.
Finally, finished watching such a long video. Thank Andrej for sharing so many details of your knowledge. Like your teaching style so much since Tesla AI day. You are the best AI teacher!
I rarely comment on videos, but I had to here I had to. Your in-depth high-quaity resources are something to talk about. You make very complicated topics easy and engaging, your provide the knowledge for anyone to learn these highly-regarded concepts. Fruthermore, you are truly advancing the general knowledge of the public by providing these powerful videos. I would just like to express my gratitude for your videos, and how they really are making a positive impact. Thank you for dedicating many hours of work to upload these videos.
Haven't been this excited about a TH-cam video since makemore! Your videos are like an antidepressant. Such a joy to watch and follow and completely send contained. It's like having Mozart explain his art note by note
The fact that this is free is so incredible. A perfect content. For those looking for something industry-specific, Immersive Translate now allows you to customize your own AI expert, it also allows translations in the technology field become more accurate and professional.
Hi Andrej, just wanted to thank you. You are a truly inspiration. Thanks for all the effort you put in this videos and all the tremendous value they offer when being publicly spread
Thank you for doing this Andrej❤ As an undergraduate student I really appreciate having access to such incredible contents. Best of luck to your startup! Also looking forward to some computer vision related videos.
This is an unbelievably high quality lecture! I always learn a ton of new things from Andrej Karpathy. Actually I believe if I have to rank the amount of knowledge I learned from a single person, Andrej would easily rank as #1.
We are living in great times, where geniuses like Karpathy offers their invaluable knowledge for free, and people are rewarding him with the sum of money they can afford 🎉
To anyone looking for the possible cause of the error in the parameter count: It is due to the vocabulary size. In GPT-1 it was around 40000, whereas in GPT-2 the vocab_size is around 50000. So, with 40K we will get 117M and with 50K we will get 124M.
Thank you ser, this is priceless. Felt sad that it had to end at some point. Please cover more topics like mech interp, fine-tuning, mixture models, etc.
This is one of the best overviews I've seen not just on LLMs, but on the entire Deep Learning process. Thank you for going into so much detail, you're expertise really shows through your explanations. Would I watch another 4 hour video from you? Absolutely, any day!
Thanks for this incredible resource. For anyone wondering about the transposes in the parameter copying from HF GPT2 model to implemented one. HF model uses nn.Conv1d for qkv projection while Andrej uses nn.Linear. The weights dimensions in Conv1d are transposed. Hence, we need to transpose some of these weights before copying them over to Andrej's model.
Thanks for the video. I like your intuition for weight decay. Since the decay is proportional to the value, it encourages the contributions to the residual stream to be spread over more neurons.
Thank you andrej! I have watch all your videos about gpt and learn a lot! As a poor college student, It's your videos that leading me to the road of llm.
You made me a bit emotional, knowing how much impact your videos will have for some people who will use this knowledge to make a living and improve their lives and the lives of those around them...Thank you!
The whole optimisation part in this video is something incredible. It is just impossible to find out these optimisation techniques on internet for DL models. Andrej doesn't have much research papers, but I believe that each one his videos is equivalent to a research paper having equal impact as of the original transformer paper.
I hope people realise how impactful what you are doing really is. This channel, this level of content, is empowering tens of thousands of smart, motivated, people in changing the World. Thank you.
This is amazing! I've learned so much practical knowledge about how to build these models and it's helped me for my Machine Learning Engineer interview
I love you andrej, this video is incredibly useful, because you show many many different parts of the model training phase in high quality and explain everything really well. Personally I've learned so much from just the first half of the video. Thank you so much and be well ❤
Thank you Andrej for the lecture, enjoyed every minute of it! I especially found the discussion about torch compile to be helpful and interesting. I had been doing some experiments on how to speed up Mistral 7B inference in huggingface and was not seeing any improvement from torch compile. This is motivating for me to go back and try to understand what might be going wrong 😅. Thanks!
Thanks Andrej, your upload always gives me some motivation to study and understand these topics properly, in addition you have very amazing mentor skills, it's always a pleasure to see a new Andrej Karpathy upload, I appreciate all these lectures, thank you very much for all of them.
Absolute gem of a video. Although one thing which became pretty evident to me whilst watching this is that if Andrej doesn't even understand some of the PyTorch docs and internals, that when you know the library is a bit of a clusterfuck and just shows how novel a lot of this new AI technology is.
Hi Andrej, Thank you so much for these videos. LLMs would be a black box without them. By any chance when you find some time please also include in this series the RLHF part of model fine tuning. Thanks 🙏
Here in the first hour of the video release. Came here while I was studying for my Security+ Exam; was looking into the Sandbox environment topic, asked ChatGPT, for set-up possibilities, realized it was down, went to Twitter and I see this video.
3 หลายเดือนก่อน
This video can open a new era and close an old one for many topics. Thanks, dude.🤗
The most impressive thing about Andrej is that unlike the rest of us, with every minute of coding he looks more refreshed and clean shaved 😂 Joking, of course this is the second most impressive thing about him!
Very in details, but with the highest level of overview. "Machine Learning", "Deep Learning Specialization" on Coursera of Andrew Ng. and this series zero-2-hero of Andrej are really special.
![[1hr Talk] Intro to Large Language Models](http://i.ytimg.com/vi/zjkBMFhNj_g/mqdefault.jpg)
![[1hr Talk] Intro to Large Language Models](/img/tr.png)
![The moment we stopped understanding AI [AlexNet]](/img/n.gif)
It’s rare to find such high-quality, free resources that make complex topics accessible and engaging!
i think you will find out that some people know that they wil be taking this technology away from the public ....... same as google glass .... as soon as america find out they cannot sread dissemination of knowledge they will cuts all internet cables !
Jesus - that's some tip!
❤
@@Handlebrake2 :)) Jesus - That's some 4hrs of brilliant content!
Dude this guy's net worth is $50M, you could've bought yourself 10 burritos at chipotle.
Do not ever look at how long your videos are. Your content is perfect and you should keep explaining things step by step. You are doing a great job. I believe you will be remembered in history as one of the pillars of AI.
I think one cool thing with videos like this is once Google implements their AI into youtube anyone will be able to watch this and just start asking questions. I've been learning a lot by watching videos like this and just copying parts of the transcript into ChatGPT to ask questions when I dont understand something.
I was excited by how long it was instead 😂😂
@@pin65371 jesus christ what a thought...and Andrej just starts talking back to you, answering your exact questions...
@@pin65371 What other videos do you recommend that have helped you?
Yes a great sharer of the important information and implementation. As well as to let you know it's in your hands to make your own before the internet is closed down or restricted in your areas or your cable gets cut !!
Great work ❤
Sorry , i love your videos and what you doing for me. I couldn't attend Stanford or get into openai but learning from you is blessing to me.. i would pay you back 100times in coming years. And i was watching your git repository last two months, i could see many git code push in private ,but i was confused what he is working on.. this is he was working on. To provide quality pratical knowledge to us all on youtube.
Hey, the webside for the GPT2 is down, is there anyway to dowload it ?
Sorry i will have look up to it ..i will do today night will reply to you after that @@shutup1209
hf@@shutup1209
@@shutup1209 you should follow the video and make it from scratch! 😀
Those are generous tips :). I wanna learn basics so I can understand this first.
My life is simple;
Andrej drops GPT-2 The Movie, I watch.
"GPT-2 The Movie" 😅
The movie and the sequel. I had to force myself to stop watching after I realized an hour had passed.
@@AndrejKarpathy The sequel "GPT The Movie" will be a old Hong Kong style "martial arts" movie about GPT getting beaten up by the Loss Function, then entering his training phase with Gradient Descent Sensei and the final showdown vs the big evaluation boss.
You know you gotta bring the popcorn
Andrej poster video on your biography@@AndrejKarpathy
Thanks! 4 hours of decoding a "Decoder-Transformer", Kudos and appreciate your existence in this field.
I am an undergraduate student. This is the lost lecture that professors never touched upon but absolutely crucial, thank you!!
I especially love how you start from the basics for so many notions, and I really learned a lot.
Which year in you are and which country
@@ppyogesh7394 I am at the University of Toronto, going to the third year this September
This guy is "the one" in the industry, who has helped me understand the LLMs. I respectfully love this man. Hats off.
You are the Excalibur of cutting through the hype. Thank you so much. Your ethics are inspiring, and your educational materials priceless.
Many Thanks to Andrej for making this tutorial available to everyone! I have never seen a clearer explanation of a nn before stumbling upon this zero to hero series. This will help all the people articulate the inner workings of neural net and help people understand deeper concepts, that is hard to understand. Looking forward to learning more with Andrej!
I like when you add comments/metaphors about your intuition for how and why it works. Thanks you.
Checkout this man here, he got the best Ai news
Thanks AK, appreciate you sharing your knowledge with the world!
Andrej is doing himself what OpenAi was supposed to do in the early days - make AI open. Thank you, Andrej!
Hello Andrej, thank you so much for the sharing and effort! Really appreciate it!
The intellectual generosity of this man is of the highest standard.
Simply the best deep learning and LLM series online! Please keep making more videos and I'd love to be part of the journey!
The longer your videos are, the better it is for humanity. I think you are such a wonderful person and providing this stuff for everyone for free, can't thank you enough.
Finally, finished watching such a long video. Thank Andrej for sharing so many details of your knowledge. Like your teaching style so much since Tesla AI day. You are the best AI teacher!
Thanks Andrej! Tons of stuff in the video explained in simple terms, I learned a lot from it.
I rarely comment on videos, but I had to here I had to. Your in-depth high-quaity resources are something to talk about. You make very complicated topics easy and engaging, your provide the knowledge for anyone to learn these highly-regarded concepts. Fruthermore, you are truly advancing the general knowledge of the public by providing these powerful videos. I would just like to express my gratitude for your videos, and how they really are making a positive impact. Thank you for dedicating many hours of work to upload these videos.
Haven't been this excited about a TH-cam video since makemore! Your videos are like an antidepressant. Such a joy to watch and follow and completely send contained. It's like having Mozart explain his art note by note
I've learned a lot from your Neural Network video playlist. Thank you
🤗What an absolutely fantastic explanation! Every minute is filled with nuggets of deep insights!
Thanks! Really appreciate the effort you put into making these high quality educational videos!
That’s cool how Andrej explains right after releasing code
The fact that this is free is so incredible. A perfect content. For those looking for something industry-specific, Immersive Translate now allows you to customize your own AI expert, it also allows translations in the technology field become more accurate and professional.
Hi Andrej, just wanted to thank you. You are a truly inspiration. Thanks for all the effort you put in this videos and all the tremendous value they offer when being publicly spread
Thank you for doing this Andrej❤ As an undergraduate student I really appreciate having access to such incredible contents. Best of luck to your startup! Also looking forward to some computer vision related videos.
Thanks very much Andrej! Your videos are real gifts to the whole world.
This is an unbelievably high quality lecture! I always learn a ton of new things from Andrej Karpathy. Actually I believe if I have to rank the amount of knowledge I learned from a single person, Andrej would easily rank as #1.
Thanks for your Zero-to-hero series!
wow you're very thankful ty! :)
We are living in great times, where geniuses like Karpathy offers their invaluable knowledge for free, and people are rewarding him with the sum of money they can afford 🎉
To anyone looking for the possible cause of the error in the parameter count: It is due to the vocabulary size. In GPT-1 it was around 40000, whereas in GPT-2 the vocab_size is around 50000. So, with 40K we will get 117M and with 50K we will get 124M.
Thank you ser, this is priceless. Felt sad that it had to end at some point. Please cover more topics like mech interp, fine-tuning, mixture models, etc.
You are the reason I got interested in neural networks, thank you for being a great teacher.
Hii bro, can i get roadmap for ML an deep learning please
This is one of the best overviews I've seen not just on LLMs, but on the entire Deep Learning process. Thank you for going into so much detail, you're expertise really shows through your explanations.
Would I watch another 4 hour video from you? Absolutely, any day!
Having to work when you just want to watch Andrej's videos is like being invited to an open buffet but you're on a diet :)
Currently, I'm just watching your videos. They makes me calm and happy. Hope to continue studying later this year.
Thanks for this incredible resource.
For anyone wondering about the transposes in the parameter copying from HF GPT2 model to implemented one.
HF model uses nn.Conv1d for qkv projection while Andrej uses nn.Linear. The weights dimensions in Conv1d are transposed. Hence, we need to transpose some of these weights before copying them over to Andrej's model.
This video is absolutely brilliant! Thank you so much Andrej for taking the time to share your knowledge with us!
Thank you for contributing to Open Source models! And not just open weights!
So thrilled to have you making stuff on here. It's the best version of what TH-cam can be.
Thanks for the video. I like your intuition for weight decay. Since the decay is proportional to the value, it encourages the contributions to the residual stream to be spread over more neurons.
Thanks for sharing your knowledge!
Kind remainder that GOOD videos are NEVER too long
I just watched your video about 'Let's Build GPT from Scratch' yesterday. You are a great teacher and clearly explain complicated concepts. Thanks!
Thank you andrej! I have watch all your videos about gpt and learn a lot! As a poor college student, It's your videos that leading me to the road of llm.
I've never seen and experienced like you teaching me making me think i can learn everything with your teaching
your explanation is really amazing, please keep fulfilling the gap >>>> build things from scratch
Huge respect for Andrej🤗. Sharing knowledge for free is incredible.
Another piece of art 🔥
You made me a bit emotional, knowing how much impact your videos will have for some people who will use this knowledge to make a living and improve their lives and the lives of those around them...Thank you!
Another epic tutorial to build models from scratch. Thank you, Andrey!
I’m so thankful and grateful that these videos are available to view for free. Thank you Andrej!
The whole optimisation part in this video is something incredible. It is just impossible to find out these optimisation techniques on internet for DL models. Andrej doesn't have much research papers, but I believe that each one his videos is equivalent to a research paper having equal impact as of the original transformer paper.
I hope people realise how impactful what you are doing really is. This channel, this level of content, is empowering tens of thousands of smart, motivated, people in changing the World. Thank you.
Thanks Andrej!
how is it Andrej, being one of those legends that will always be remembered? Your art will be viewed by generations!
OMG a 4hr movie!
As a student, you have been my biggest inspiration and the best mentor in the field of NLP and Computer vision. Will forever be grateful!
YOU are Awesome, Andrej!! 🥂🤖
Thanks for the detailed video and explanation Andrej, have really learnt a lot watching your videos and the Makemore series!
This is the best learning resource on language models bar none.
Love your content, thanks!
This is amazing! I've learned so much practical knowledge about how to build these models and it's helped me for my Machine Learning Engineer interview
@Andrej Karpathy, Awesome! You don't know how much this means to so many people around world! Thank you so much! You are our hero!
Towards the end I thought, what, is that it? But many thanks for the effort and for bringing the material in such a clear and cohesive way.
Hi Andrej, what's the difference between this one and your "Let's build GPT" video? Which one should one learn first/which one is preferred?
Was wondering the same 😅
Use the "lets build" first, then this one; it goes over the understanding of it first, like the tokenization one as well.
When I see this video, I know I need to make time for this.
There is a huge difference between watching someone work out and workout by myself !
the GOAT has returned!
I love you andrej, this video is incredibly useful, because you show many many different parts of the model training phase in high quality and explain everything really well. Personally I've learned so much from just the first half of the video. Thank you so much and be well ❤
if i know everthing taught in that tutorial with details and i am also able to apply them by myself, can i count myself as an advanced AI developer?
no, you are like a beginner+. all the data handling and data cleaning that is necessary + bug fixing u have not had to deal with
Andrej releasing such unique super high quality content and for free, I am speechless.
Thanks!
Thank you Andrej for the lecture, enjoyed every minute of it! I especially found the discussion about torch compile to be helpful and interesting. I had been doing some experiments on how to speed up Mistral 7B inference in huggingface and was not seeing any improvement from torch compile. This is motivating for me to go back and try to understand what might be going wrong 😅. Thanks!
thank you for yet another amazing video Andrej!
Never stop. Keep publishing videos like these please. What a delight.
Thanks Andrej, your upload always gives me some motivation to study and understand these topics properly, in addition you have very amazing mentor skills, it's always a pleasure to see a new Andrej Karpathy upload, I appreciate all these lectures, thank you very much for all of them.
Videos like these is the reason, I am here at YT
and rarely leave comment
Thanks Andrej for this! Massive respect.
Absolute gem of a video. Although one thing which became pretty evident to me whilst watching this is that if Andrej doesn't even understand some of the PyTorch docs and internals, that when you know the library is a bit of a clusterfuck and just shows how novel a lot of this new AI technology is.
Simply I can't believe how generous this man is...saludos gran Andrej !, you are the best human being of your generation. !
legend !!! all I can do is buy you a thank to show my appreciation. your videos changes the way I learn deep learning
Legend
Thanks. Eagerly waiting for your LLM 101 course🤟🤟
Thanks for making such a quality-content available to the every corners of the world!!!
Andrej Karpathy is like the Roger Federer of AI - makes the hardest stuff look easy!
That's just a ton of insights and tips for anyone who getting into LLMs.Thank you for your work Andrej!
Your videos are just perfect for those who wants to go deeper in this field. Are there any other guys like you in yt?
You're saving my master's thesis, I have to pay back some! 😂Thank you for doing the 'open' part of 'open'AI, it will help a lot of people!
Hi Andrej, Thank you so much for these videos. LLMs would be a black box without them. By any chance when you find some time please also include in this series the RLHF part of model fine tuning.
Thanks 🙏
Here in the first hour of the video release. Came here while I was studying for my Security+ Exam; was looking into the Sandbox environment topic, asked ChatGPT, for set-up possibilities, realized it was down, went to Twitter and I see this video.
This video can open a new era and close an old one for many topics. Thanks, dude.🤗
Learning from you is blessings to me. Thank you sir. Please upload such contents more.
The most impressive thing about Andrej is that unlike the rest of us, with every minute of coding he looks more refreshed and clean shaved 😂 Joking, of course this is the second most impressive thing about him!
Thanks from the bottom of my heart. It is amazing to see such a high quality content.
This was awesome! Was glued to the screen for the whole video.
Could you implement RLHF? I've never heard any in-depth explanation of it.
Andrej, we sorely miss the little bunch of flowers from the early, glorious days of NN-zero-to-hero..
Very in details, but with the highest level of overview. "Machine Learning", "Deep Learning Specialization" on Coursera of Andrew Ng. and this series zero-2-hero of Andrej are really special.
Andrej man, thank you, really appreciate you gifting this to the world. Great content and presentation!!!
Thanks Andrej for the incredible video. Actually all your videos are incredible. I have learnt a ton!!!
Thanks Andrej for this awesome knowledge