If someone is interested in a code tutorial converting the GPT model to Llama, I have a step-by-step guide here: github.com/rasbt/LLMs-from-scratch/blob/main/ch05/07_gpt_to_llama/converting-gpt-to-llama2.ipynb (will add it to the description)
I think the grouped-query attention is more than a trick for computational reduction. It says something deep about what is the best way to share information in a multiagent system to have the best performance. And it says something alont the lines that it is better to give little essential info and at the same time request multiple info from many sources.
That's a nice interpretation regarding multi- and grouped-query attention. Thanks for sharing! If you go by the original papers though, the intention was more computation constraints and efficiency (e.g., see arxiv.org/abs/2305.13245), but yeah, perhaps it can actually help with modeling performance as well in certain scenarios (for instance, where there is massive overfitting otherwise).
@@SebastianRaschka I have no doubt that what you say is true, and in no way I wanted to imply you missed something. Two years ago, I spent couple of months training 100M models with different architectures. I did some weird stuff like putting all the attention layers first and then a big nonlinear layer. You will be surprised with how many monstrosities can actually work without losing too much performance. The two things I got from all this is a) There is some interesting intuition in group querying (that I cant fully articulate), and it will make sense for this to be explored further, b) skip connection, where you pass the value from previous layers to the current, is not a gimmick. If you remove it the performance drops a lot, which for me implies that attention mechanism is actually applied to get only the "new" info. I think that intuitions about the architecture is not passed from the researchers to the community and It is a pity. Also it is a pity that experimenting with architecture is a rich persons hobby. Anyway, I really like your channel. I subscribed :-).
Glad you found the videos useful! Since you asked for a book recommendation: Build a Large Language Model From Scratch (amzn.to/4fqvn0D), where you build a transformer-based LLM from the ground up, implementing each single component.
Dear Sebastian Raschka, your channel caught our attention and we would like to explore advertising possibilities with you. Looking forward to discussing potential opportunities!
Good question! It would require Python knowledge. PyTorch knowledge is also good to get started quicker, but not strictly necessary. If you are new to PyTorch, you can start with Appendix A, which is a ~50 page intro to PyTorch to get you up to speed
If someone is interested in a code tutorial converting the GPT model to Llama, I have a step-by-step guide here: github.com/rasbt/LLMs-from-scratch/blob/main/ch05/07_gpt_to_llama/converting-gpt-to-llama2.ipynb (will add it to the description)
Certainly, me, me, me.
Thank you very much for all your content.
Bought your book on manning last year - one of the best book on LLM internals. Looking forward to get the print book
Thanks for the kind words, glad to hear that you've been enjoying it! The print copies started shipping and I hope you get your's soon!
Your book is a master peace, congratulations
Thanks for the kind words!
Just received the print version of your book yesterday, I look forward to reading it!
Good timing! I hope you like it and have a fun weekend ahead!
Nice round up! Thank you for this.
I think the grouped-query attention is more than a trick for computational reduction. It says something deep about what is the best way to share information in a multiagent system to have the best performance. And it says something alont the lines that it is better to give little essential info and at the same time request multiple info from many sources.
That's a nice interpretation regarding multi- and grouped-query attention. Thanks for sharing! If you go by the original papers though, the intention was more computation constraints and efficiency (e.g., see arxiv.org/abs/2305.13245), but yeah, perhaps it can actually help with modeling performance as well in certain scenarios (for instance, where there is massive overfitting otherwise).
@@SebastianRaschka I have no doubt that what you say is true, and in no way I wanted to imply you missed something. Two years ago, I spent couple of months training 100M models with different architectures. I did some weird stuff like putting all the attention layers first and then a big nonlinear layer. You will be surprised with how many monstrosities can actually work without losing too much performance. The two things I got from all this is a) There is some interesting intuition in group querying (that I cant fully articulate), and it will make sense for this to be explored further, b) skip connection, where you pass the value from previous layers to the current, is not a gimmick. If you remove it the performance drops a lot, which for me implies that attention mechanism is actually applied to get only the "new" info. I think that intuitions about the architecture is not passed from the researchers to the community and It is a pity. Also it is a pity that experimenting with architecture is a rich persons hobby. Anyway, I really like your channel. I subscribed :-).
Enjoying your book a lot :))
Thanks! Glad to hear that it was worth all the long hours and weekends!
@@SebastianRaschka 🙏
A brilliant content. Thank you.
Thanks!!
Reading your new book ! 🎉🎉
Hope you'll like it! Happy coding and reading!
my favorite YT channel
Thanks :)
Llama 3.2 just came out today
Ha yes, I wish I could insert additional slides! What's interesting is that the small model is back from RMSNorm to LayerNorm
GPT2 training did not train on 40 billion tokens, it was 40 GB of text which is equivalent to roughly 8 billion tokens or less.
Ah yes, 40 GB, you are right. Good catch!
Thanks for sharing 😊
Hey, thanks for your videos also can you suggest to me your best book on Gen AI (I want to learn about transformers in detail)
Glad you found the videos useful! Since you asked for a book recommendation: Build a Large Language Model From Scratch (amzn.to/4fqvn0D), where you build a transformer-based LLM from the ground up, implementing each single component.
Dear Sebastian Raschka, your channel caught our attention and we would like to explore advertising possibilities with you. Looking forward to discussing potential opportunities!
What is the pre-requisites knowledge required to explore your new book: `Build a Large Language Model (From Scratch)`
Good question! It would require Python knowledge. PyTorch knowledge is also good to get started quicker, but not strictly necessary. If you are new to PyTorch, you can start with Appendix A, which is a ~50 page intro to PyTorch to get you up to speed
Yes!
What do you mean by high quality annealing?
They would select a small subset of very high quality data for the final annealing stage.
What does annealing mean in the context of LLMs? Is it the same as what we mean by an annealing LR scheduler?
@@subaruhassufferredenough7892 Yes, it's basically the same
Do you know how they determined which data was high quality?
Hi Prof. Raschka, could you please attach the slides?
Explain transformers in detail
That would be a very long video :D. But you might find my book useful in that respect.
Moore Daniel Taylor Brenda Anderson Eric