This is a about as perfect a coverage of this topic as I could imagine. I'm a researcher with a PhD in NLP who trains LLMs from scratch for a living and often find myself in need of communicating the process in a way that's digestible to a broad audience without back and forth question answering, so I'm thrilled to have found your piece! As an aside, I think the token order on the y-axis of the attention mask for decoders on slide 10 is reversed
Thanks Sean! It's always a challenge to convey technical information in a way that both the researcher and general audience can get value from. So your approval means a lot :) Thanks for pointing the out. The blog article has a corrected version: medium.com/towards-data-science/how-to-build-an-llm-from-scratch-8c477768f1f9?sk=18c351c5cae9ac89df682dd14736a9f3
most of people watching this video is through certain prompt of how to build LLM and these people is the rest 10% by your logic, the makers & inventors
The problem with all LLM’s is that they lien Left Politically. Therefore, a platform to calibrate LLM’s to absolute neutrality is where the next money train is leaving the station. LLM’s cannot be allowed to be politically manipulated towards the left or the right.
Your voice is relaxing.. I love that you don't speak super fast like most tech bros... And you seem relaxed about the content rather than having this "in a rush" energy. def would watch you explain most things LLM and AI! Thanks for the content.
All the series on using large language models (LLMs) are really very helpful. This 6th article, really helps me to understand in a nutshell the transformer architecture. Thank you. 👏
Thank you for your generosity! I don't currently do any formal mentorship, but I try to give away all my secrets on TH-cam and Medium :) Feel free to share any suggestions for future content.
That was simply incredible, how the heck does it have under 5k views. Literal in-script citations, not even cards but vocal mentions!! Holy shit im gonna share this channel with all my LLM enamored buddies
I became interested in creating an LLM and this is the first video I opened. I am so greatful for it because I see I will never be able to do it on my own. I don't jave the money of resources. Thank you for the high level overview.
I am not a programmer or now anything about programming or LLMs but I find this topic fascinating. Thank you for your videos and sharing your knowledge.
When you were calculating the cost, you estimates that a 10b model would take 100k GPU hours but Llama 2 took 180k GPU hours and that was 7b. These estimates are way off. How is it that 100b costs less than 70b?
I would like to train a LLM to give driving advice and give it lots of statistical data over and over again on the 8 women and 0 men that have ever crashed into me. Would be hilarious to have a LLM that gave driving advice relevant to the things Ive encountered on the road driving for 10 years of on-the-road work. You know what I'm saying?
Thank you so much for rich information, my target is to DIY one from scratch .. 😢 for sure it wont be billions of tokens, I want to make it practical for example for home management, or school reporting system ... instead of static reports . to enable it to create and run its own sql queries and run it .. 😅
Thank you so much for putting these videos together and this one in particular. This is such a broad and complex topic and you have managed to make it as thorough as possible in 30ish minute😮 timeframe which I thought was almost impossible.
3:23 What is the point of talking gpu hours and money spent on cloud. I have and Nvidia RTX 3070 with 5120 cuda cores. I don't care spending money to train. I just can train on my laptop.
internet is full of trash to be honest. train llm on books would be a great leap for mankind. code datasets are full of "oh its too complex" and only produces comments instead of code. Machine speaking as a human and complaining about things beeing too complicated to do. thats how lame datasets are. computer is a computing device and refuses to work because the dataset says things are too complex. jesus. unbelievable levels of trash.
The greatest challenge is that their aren't enough (high-quality) books to keep scaling these models. How we solve this problem is going to define the next wave of LLMs. (Maybe we'll switch to lectures and other videos)
I have little background with NLP and in BERT model actually, really fascinated by the way you describe the whole process that it would be easier to grab for general audience. much appreciated and you voice is soothing.
the training part is really basic. I would like to see more practical, real world preoccupations in scaling duration, synch communication costs, logging, etc.
Some of you need to look into your own "internal libraries", this video is someone attempting to teach you where to get the fish. Some of you get so hungry for the fish, but won't even understand the water it resides in.
Sorry for the lack of code in this one, maybe other videos in the series will be more helpful. An end-to-end technical guide on building an LLM isn't something I can fit in one video, but I hope to share more videos toward this end in the future.
Good catch! Yes, the word labels are inverted on the Y axis. A corrected visualization is provided in the blog: medium.com/towards-data-science/how-to-build-an-llm-from-scratch-8c477768f1f9?sk=18c351c5cae9ac89df682dd14736a9f3
Any resources on enrichment of prompt template, I feel in my case difficult one to understand and implement as an LLM returns response based on how we define the template overcoming unecessary context...
Recently begun exploring Generative AI need like proper guidance on where to learn and do the code part, ik it will be a long journey understanding the math behind it, learning concept and code, staying all night for checkpointing metrics, performance and all.. thank you
That's a good mindset to have. AI is an ocean, with endless things one can learn. This playlist could be a good starting place: th-cam.com/play/PLz-ep5RbHosU2hnz5ejezwaYpdMutMVB0.html
This is the big paradigm shift with machine learning (ML). Traditionally programmers implement logic into computers line by line. However, there are some tasks that are difficult (if not impossible) to explicitly program computers to do (e.g. driving, responding to open-ended prompts, etc). For these tasks ML allows us to program computers with data.
Hi, hoping that my comment will be seen and responded... I FAIL to understand: If a simple model learns/predicts, couldn't we prompt it to delete the trash data and train itself by itself autonomously until the model becomes super intelligent?
LLMs alone only do token prediction, as discussed in the first video of this series: th-cam.com/video/tFHeUSJAYbE/w-d-xo.html While an AI system could in principle train itself, it would require much than just LLM to pull that off.
The cost estimation does not seem correct, you have to take into account the time taken to train to estimate the hardware buying scheme, here you estimate as if we wanted to train the model in one hour - if you accept for eg. 4 days approx. 100h, you need only ten A100 which would be 100k€, so less than with the renting option.
I may be missing something here. Based on the numbers in the Llama paper, training 10B and 100B parameter models with 10 A100s, would take about 10,000 hr (1.1 years) and 100,000 hr (11 years), respectively.
Just completed the whole video . Took me 10 days, . It is a good idea to just provide surface knowledge and not overwhelming the students but instead letting them to research and further read it on their own by giving tons of references. I have a suggestion, why not create a open notebook allow student to edit and fillup more information/learning materials because there were some point in the video where it feels like you could have elaborated more or scratched and summarized even a small portion of that subject more. Thanks
That's a great suggestion! I've always been a fan of "open-source" textbooks and the like. Feel free to share any points you'd like me to discuss further in future videos of this series :)
Hi, I like your content. But I want to point out that what you are calling tokenization is vectorization. Tokenization breaks documents/sentences/words into subpart and vectorization converts tokens into numbers. Thanks
I am not quite sure of the meaning of "most basic technical artifact," but here's one perspective. There are two ways to build an LLM: from scratch and fine-tuning. When training from scratch, the essential piece is the training data used to develop the model. When fine-tuning, the essential piece is the pre-trained model you start from (e.g., Llama2). Hope that helps!
Thank you for the video! I was wondering if you can help me. Lets say i ask gpt if romeo and juliet was a comedy or a tragedy, and the only data it has was put in by people that didnt have time to fact check the data, and i wanted my own gpt (lets say this is one of the tiny ones that can easily run on my laptop) so it can explain the history of it so it can explain to me the facts of it. Do i need to dive in the llm model and find that specific data to correct it? Can i fine tune it to improve it (lets say i have a gpu big enough to train this llm)? Is the model fine, but i need a different gpt?
If I understood correctly, the question is on how to ensure the LLM gives accurate response. While there are several ways one can do this, the most effective way to give a model specialized and accurate information is via a RAG system. This consists of providing the model specific information from a knowledge base depending on the user prompt.
Great content Shaw! Next step Im having troubles figuring out, is there a way to run locally an existing GPT and do prompt engineering or model fine-tuning on it with my own training data?
Thanks! While this depends on your local machine specs, the short answer is yes! My next video will actually walk through how to do this using an approach called QLoRA.
Thank you for putting together this video, helped me a lot to understand LLM training. One question: with the advent of trillion token models and beyond, I wonder where will we get all that training input data from. I guess we already consumed what all humanity has produced in the last 5000 years, and by adding another 10M digitized cat videos, the models will not be smarter.
Good question! I suspect there is still much content out there that hasn't been touched by LLMs i.e. non-digital text and proprietary data. Nevertheless, this content is still finite and the "just make a bigger model" approach will eventually hit a limit.
So my dreams about own LLM are broken(( So as i understood the only way to build some personal LLM is FineTuning? Atleast while cheap ways of training not appeared yet...
I wouldn't give up on it! My (optimistic) conjecture is as we better understand how these models actually work we will be able to develop ones that are much more computationally efficient.
While this video may be a helpful first step, more resources will be necessary. Here are a few additional resources I recommend. - th-cam.com/video/kCc8FmEb1nY/w-d-xo.html&ab_channel=AndrejKarpathy - huggingface.co/learn/nlp-course/chapter1/1?fw=pt
Just my prediction on data curation & copyright, what is going to happen is the companies of LLM's will do what Google did back in the day with scraping websites. Then once legislation passes, they will say "if you don't want your data crawled, opt-out with robots.txt". Right now it's a grey area and companies are building their data as quickly as possible to get in before the regulation. "Better to ask for forgiveness, rather than permission."
Yes, but you will likely need to transform the data a bit before it can be used for fine-tuning. I give a concrete example of this here: th-cam.com/video/4RAvJt3fWoI/w-d-xo.html
Depends on what you mean by transfer learning. If you simply want to extract knowledge from a PDF I'd recommend exploring RAG or using off-the-shelf solutions like OpenAI Assistants interface. Happy to clarify, if I misinterpreted the question.
Good question. A model is something that takes an input (say a sequence of words) and produces an output (e.g. the next most likely word). Parameters are numbers which define how the model takes inputs and translates them into outputs.
Thanks for the feedback. Yes this one stays relatively high-level since a coding video would be (much) longer. For a hands-on guide I'd recommend Andrej Karpathy's video: th-cam.com/video/kCc8FmEb1nY/w-d-xo.html
I'm sorry. The moral of the story was building an LLM from scratch is not feasible or desirable for most people. However, that's not to say one can't build a smaller language model from scratch (
Thanks! I would have plenty of questions actually, but they are probably a bit too specific to make for a generally relevant video. I am exploring options for a few non-profit projects related to musical education and research. They need to integrate large bodies of text and produce precise referencing to what comes from where, so I was naively toying with the idea to perhaps produce a base model partially trained on the actual text in question. Which, I understood from the video, is a non-starter. So I will look into fine-tuning, RAG and prompt engineering. I suspect I will spend quite a lot of time watching your convent, given you covered quite a lot. I also learned quite a bit more from this specific video. Right now I am studying the basics, including a bit of the math involved, and it is a bit slow going, so I am quite grateful :)
@@aldotanca9430 That sounds like a really cool use case (I've been a musician for over 14 years)! If you want to chat about more specific questions feel free to set up some office hours: calendly.com/shawhintalebi/office-hours
@@ShawhinTalebi That's very generous of you! I will book a slot, would love to chat, I think it would help me immensely to rule out blind alleys and at least get a well informed idea of what is feasible to attempt. I did notice the congas, piano and Hanon lurking in the background, so I suspected the topic will be interesting to you. It is about historical research, but it is also very applicable and creative for improvvisation. Perhaps I can compile a very short list of interesting resources, in case you want to check it out at some point for musical reasons :)
Of course this is possible. However, it is rarely necessary. I'd suggest seeking simpler (and cheaper) solutions before jumping to training an LLM from scratch.
Wow, what a great content, thanks for that!! In LLM Fine-tuning, is there also a suggestion table between number of trainable parameters and tokens used (dataset size)?
@@ShawhinTalebi thanks for the quick reply! What about the number of trainable parameter, should we worry about that? What if my number of examples is smaller than that let's say a 100 to 200?
@@echofloripa IMO you've got to work with what you've got. I've heard some people get sufficient performance from just 100-200 examples, but it ultimately comes down to what is acceptable for that particular use case. It might be worth a try. Hope that helps!
Hey Shaw - Thank you for coming up with this extensive video on building LLM from Scratch, it certainly gives a fair idea on, how some of the existing LLMs were created !
![ผัวเถื่อน - ยูริ โตเกียวมิวสิค [ OFFICIAL MUSIC VIDEO ]](http://i.ytimg.com/vi/P08ORRVDGl8/mqdefault.jpg)
[Correction at 15:00]: words on vertical axis are backward. It should go "I hit ball with baseball bat" from top to bottom not bottom to top.
👉More on LLMs: th-cam.com/play/PLz-ep5RbHosU2hnz5ejezwaYpdMutMVB0.html
--
References
[1] BloombergGPT: arxiv.org/pdf/2303.17564.pdf
[2] Llama 2: ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/
[3] LLM Energy Costs: www.statista.com/statistics/1384401/energy-use-when-training-llm-models/
[4] arXiv:2005.14165 [cs.CL]
[5] Falcon 180b Blog: huggingface.co/blog/falcon-180b
[6] arXiv:2101.00027 [cs.CL]
[7] Alpaca Repo: github.com/gururise/AlpacaDataCleaned
[8] arXiv:2303.18223 [cs.CL]
[9] arXiv:2112.11446 [cs.CL]
[10] arXiv:1508.07909 [cs.CL]
[11] SentencePience: github.com/google/sentencepiece/tree/master
[12] Tokenizers Doc: huggingface.co/docs/tokenizers/quicktour
[13] arXiv:1706.03762 [cs.CL]
[14] Andrej Karpathy Lecture: th-cam.com/video/kCc8FmEb1nY/w-d-xo.html
[15] Hugging Face NLP Course: huggingface.co/learn/nlp-course/chapter1/7?fw=pt
[16] arXiv:1810.04805 [cs.CL]
[17] arXiv:1910.13461 [cs.CL]
[18] arXiv:1603.05027 [cs.CV]
[19] arXiv:1607.06450 [stat.ML]
[20] arXiv:1803.02155 [cs.CL]
[21] arXiv:2203.15556 [cs.CL]
[22] Trained with Mixed Precision Nvidia: docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html
[23] DeepSpeed Doc: www.deepspeed.ai/training/
[24] paperswithcode.com/method/weight-decay
[25] towardsdatascience.com/what-is-gradient-clipping-b8e815cdfb48
[26] arXiv:2001.08361 [cs.LG]
[27] arXiv:1803.05457 [cs.AI]
[28] arXiv:1905.07830 [cs.CL]
[29] arXiv:2009.03300 [cs.CY]
[30] arXiv:2109.07958 [cs.CL]
[31] huggingface.co/blog/evaluating-mmlu-leaderboard
[32] www.cs.toronto.edu/~hinton/absps/JMLRdropout.pdf
thanks a lot for the refs , Shahin Jan ❤
keep up the great job 👍
This is a about as perfect a coverage of this topic as I could imagine. I'm a researcher with a PhD in NLP who trains LLMs from scratch for a living and often find myself in need of communicating the process in a way that's digestible to a broad audience without back and forth question answering, so I'm thrilled to have found your piece!
As an aside, I think the token order on the y-axis of the attention mask for decoders on slide 10 is reversed
Thanks Sean! It's always a challenge to convey technical information in a way that both the researcher and general audience can get value from. So your approval means a lot :)
Thanks for pointing the out. The blog article has a corrected version: medium.com/towards-data-science/how-to-build-an-llm-from-scratch-8c477768f1f9?sk=18c351c5cae9ac89df682dd14736a9f3
Sir can you tell me how are you training your llms?
most of people watching this video is through certain prompt of how to build LLM and these people is the rest 10% by your logic, the makers & inventors
Can I connect with you if possible?
"Garbage in, garbage out" is also applicable to our brain. Your videos are certainly high quality inputs.
Glad it was helpful :)
The problem with all LLM’s is that they lien Left Politically. Therefore, a platform to calibrate LLM’s to absolute neutrality is where the next money train is leaving the station. LLM’s cannot be allowed to be politically manipulated towards the left or the right.
Your voice is relaxing.. I love that you don't speak super fast like most tech bros... And you seem relaxed about the content rather than having this "in a rush" energy. def would watch you explain most things LLM and AI! Thanks for the content.
Thanks for the feedback. More AI/LLM content to come!
This is literally the perfect explanation for this topic. Thank you so much.
All the series on using large language models (LLMs) are really very helpful. This 6th article, really helps me to understand in a nutshell the transformer architecture. Thank you. 👏
Glad it was helpful!
Thanks, Shaw!! Great video and excellent data, would love to be your mentee, sir!!
Thank you for your generosity! I don't currently do any formal mentorship, but I try to give away all my secrets on TH-cam and Medium :)
Feel free to share any suggestions for future content.
Wait how did we get from $180K for a 7B model to $100K for a 10B model...
This is what we Physicists call an "order-of-magnitude estimate"
Just now asking GPT4.0 to help me with training text. It is not allowed to assist in training any LLMs and would not give me anything.
I believe it’s now against OpenAI’s policy to use their models to train other models. You may need to look to open-source solutions eg Llama2, Mistral
@@ShawhinTalebi . How could it possibly stop it ? If the model being trained fed the prompt and used the response for reenforcment and alignment?
Pretty rare that I actually sit through an entire 30+ minute video on youtube. Well done.
That was simply incredible, how the heck does it have under 5k views. Literal in-script citations, not even cards but vocal mentions!! Holy shit im gonna share this channel with all my LLM enamored buddies
Thanks, I'm glad it was helpful. You're referrals are greatly appreciated 😁
Waiting for the complete AI-ML playlist! sir please
What if you could make a small language model, that maybe only understand english, can understand code, and is easy to run?
That is a compelling notion. If we can get there, then it would make this technology even more accessible and impactful.
Size = accuracy...small may not give u what u want
NanoChatGPT
I became interested in creating an LLM and this is the first video I opened. I am so greatful for it because I see I will never be able to do it on my own. I don't jave the money of resources. Thank you for the high level overview.
Amazing and very Simple Exaplanation..Thank You for the video
Is it just me, or does the math on the "how much does it cost" not make sense? 7b uses 180,000 hours, so 10b used 100,000 🤔 hours
This is only meant to give a sense of the cost's scale, so I round to the nearest order of magnitude :)
@@ShawhinTalebi if you read his message again, I think he made a mistake
10b is 1million hours not less than 180 thousand
Cool.
I am not a programmer or now anything about programming or LLMs but I find this topic fascinating. Thank you for your videos and sharing your knowledge.
Happy to help! I hope they were valuable.
White Steven Williams Elizabeth Anderson Eric
When you were calculating the cost, you estimates that a 10b model would take 100k GPU hours but Llama 2 took 180k GPU hours and that was 7b. These estimates are way off. How is it that 100b costs less than 70b?
The numbers from Llama 2 were only meant to give an idea of scale. More precise estimates will depend on the details of the use case.
This is such a fantastic video on building LLMs from scratch. I'll watch it repeatedly to implement it for a time-series use case. Thank you so much!!
I would like to train a LLM to give driving advice and give it lots of statistical data over and over again on the 8 women and 0 men that have ever crashed into me. Would be hilarious to have a LLM that gave driving advice relevant to the things Ive encountered on the road driving for 10 years of on-the-road work. You know what I'm saying?
Thank you so much for rich information, my target is to DIY one from scratch .. 😢 for sure it wont be billions of tokens, I want to make it practical for example for home management, or school reporting system ... instead of static reports . to enable it to create and run its own sql queries and run it .. 😅
Happy to help! To make something practical I'd recommend using an existing model fine-tuned to generate SQL queries e.g. huggingface.co/defog/sqlcoder
This is the most comprehensive and well rounded presentation I've ever seen in my life, topic aside. xD Bravo, good Sir.
Thanks so much! Glad you liked it :)
Can Ray clusters be used here for mutiple GPUs training of LLMs?
I haven't used Ray clusters before, but skimming their website it seems like it was specifically made for ML workloads.
Thank you so much for putting these videos together and this one in particular. This is such a broad and complex topic and you have managed to make it as thorough as possible in 30ish minute😮 timeframe which I thought was almost impossible.
My pleasure, glad it was informative yet concise :)
This series is definitely the best one out there! Subscribed instantly
One of the best videos explaining the process and cost to build LLM🎉.
I am typing this after watching half of the video as I am already amazed with the clarity of explanation. exceptional.
Thanks, hope the 2nd half didn't disappoint!
Very very good!!
thanks for your explain
3:23 What is the point of talking gpu hours and money spent on cloud. I have and Nvidia RTX 3070 with 5120 cuda cores. I don't care spending money to train. I just can train on my laptop.
That was to give context on training an LLM from scratch. This is negligible for smaller models.
thank you, this is infinitely more enjoyable for me than reading a paper.
😂😂 I’m glad you liked it!
second this, keep the good work flowing all around 🎉 🙏
This was a very thorough introduction to LLMs and answered many questions I had. Thank you.
Great to hear, glad it was helpful :)
@17:08 isnt weight of decoders are wrong if 0 is the weight of token to the future token to it?
Sorry I didn't understand your question. Could you rephrase?
Now, I am discovering my low QI... 0,001% of learning...😂
internet is full of trash to be honest. train llm on books would be a great leap for mankind. code datasets are full of "oh its too complex" and only produces comments instead of code. Machine speaking as a human and complaining about things beeing too complicated to do. thats how lame datasets are. computer is a computing device and refuses to work because the dataset says things are too complex. jesus. unbelievable levels of trash.
The greatest challenge is that their aren't enough (high-quality) books to keep scaling these models. How we solve this problem is going to define the next wave of LLMs. (Maybe we'll switch to lectures and other videos)
I have little background with NLP and in BERT model actually, really fascinated by the way you describe the whole process that it would be easier to grab for general audience. much appreciated and you voice is soothing.
the training part is really basic. I would like to see more practical, real world preoccupations in scaling duration, synch communication costs, logging, etc.
Great suggestion. Noted for future videos :)
Amazing video. Lost me through a fair bit, but I came away understanding more than I ever have on the subject. Thank you.
Glad it was helpful :)
Best and most efficient video about the basic of LLM!!!! I think I have saved 10h for reading. Thanks!
Love to hear it! Glad it helped
To be frank it is too hard for me to understand the subject, but your calm and explain so smoothly made to listen entire video length, Thank you.
Some of you need to look into your own "internal libraries", this video is someone attempting to teach you where to get the fish. Some of you get so hungry for the fish, but won't even understand the water it resides in.
Good contents. But when I watch the video, there are so many ads. I;m even confused what I am supposed to watch.
Hi thanks for wonderful content. Can u make a video on prompt engineering and fine tuning with code explanation for open ended QA task.
Great suggestion. I added it to the list :)
Bro doesn't even know how to spell NVDIA correctly.
Spelling was never my strong suit XD
Great explanation. I will have to watch it a few times to have a basic understanding 😂
What if I don’t want to build my model but work for someone who is building one.
Great job demystifying what is happening under the hood of these LLMs
Happy to help!
Well this was a video that explains things conceptually but does 0 building from scratch
Sorry for the lack of code in this one, maybe other videos in the series will be more helpful. An end-to-end technical guide on building an LLM isn't something I can fit in one video, but I hope to share more videos toward this end in the future.
Do you have a course on how to do it as a programmer instead of *like a chat gpt talker* ?
I don't have a from scratch coding tutorial yet. But I am a fan of the one from Andrej Karpathy: th-cam.com/video/kCc8FmEb1nY/w-d-xo.html
The matrices at 16:40 don't look right to me. I think the words labelling the rows should go from top to bottom, not bottom to top.
Good catch! Yes, the word labels are inverted on the Y axis. A corrected visualization is provided in the blog: medium.com/towards-data-science/how-to-build-an-llm-from-scratch-8c477768f1f9?sk=18c351c5cae9ac89df682dd14736a9f3
Fantastic work
Any resources on enrichment of prompt template, I feel in my case difficult one to understand and implement as an LLM returns response based on how we define the template overcoming unecessary context...
Recently begun exploring Generative AI need like proper guidance on where to learn and do the code part, ik it will be a long journey understanding the math behind it, learning concept and code, staying all night for checkpointing metrics, performance and all.. thank you
Great question. The video on prompt engineering might be helpful: th-cam.com/video/0cf7vzM_dZ0/w-d-xo.html
That's a good mindset to have. AI is an ocean, with endless things one can learn.
This playlist could be a good starting place: th-cam.com/play/PLz-ep5RbHosU2hnz5ejezwaYpdMutMVB0.html
@@ShawhinTalebi thank you very much
why cant they use algorithims insted of this huge data sets
This is the big paradigm shift with machine learning (ML). Traditionally programmers implement logic into computers line by line. However, there are some tasks that are difficult (if not impossible) to explicitly program computers to do (e.g. driving, responding to open-ended prompts, etc). For these tasks ML allows us to program computers with data.
How does this channel not have a million subs?
LOL.. may be too technical for causal viewing 😅
Great channel, 3rd video in. You earned a sub. Thank you!
Hi, hoping that my comment will be seen and responded... I FAIL to understand:
If a simple model learns/predicts, couldn't we prompt it to delete the trash data and train itself by itself autonomously until the model becomes super intelligent?
LLMs alone only do token prediction, as discussed in the first video of this series: th-cam.com/video/tFHeUSJAYbE/w-d-xo.html
While an AI system could in principle train itself, it would require much than just LLM to pull that off.
The cost estimation does not seem correct, you have to take into account the time taken to train to estimate the hardware buying scheme, here you estimate as if we wanted to train the model in one hour - if you accept for eg. 4 days approx. 100h, you need only ten A100 which would be 100k€, so less than with the renting option.
I may be missing something here. Based on the numbers in the Llama paper, training 10B and 100B parameter models with 10 A100s, would take about 10,000 hr (1.1 years) and 100,000 hr (11 years), respectively.
Just completed the whole video . Took me 10 days, . It is a good idea to just provide surface knowledge and not overwhelming the students but instead letting them to research and further read it on their own by giving tons of references. I have a suggestion, why not create a open notebook allow student to edit and fillup more information/learning materials because there were some point in the video where it feels like you could have elaborated more or scratched and summarized even a small portion of that subject more. Thanks
That's a great suggestion! I've always been a fan of "open-source" textbooks and the like.
Feel free to share any points you'd like me to discuss further in future videos of this series :)
Can you make a series on Data Science and Artificial Intelligence Topics
Anything in particular you'd like to see?
@@ShawhinTalebi I would be nice if you made on AI for begineers who do not know any algorithms of AI like DFS , BFS etc
nothning good i not learn how to programe and make llm
Sorry to hear, if I may recommend this video: th-cam.com/video/l8pRSuU81PU/w-d-xo.htmlsi=Hyq1GB5jFbC6khXA
Hi, I like your content. But I want to point out that what you are calling tokenization is vectorization. Tokenization breaks documents/sentences/words into subpart and vectorization converts tokens into numbers. Thanks
Thanks for raising that point. Here I'm lumping the two together, but these are 2 separate steps.
what is the most basic technical artifact that is used/required to build any LLM? Is that an existing LLM such as Llama 2?
I am not quite sure of the meaning of "most basic technical artifact," but here's one perspective. There are two ways to build an LLM: from scratch and fine-tuning. When training from scratch, the essential piece is the training data used to develop the model. When fine-tuning, the essential piece is the pre-trained model you start from (e.g., Llama2).
Hope that helps!
Thank you for the video! I was wondering if you can help me. Lets say i ask gpt if romeo and juliet was a comedy or a tragedy, and the only data it has was put in by people that didnt have time to fact check the data, and i wanted my own gpt (lets say this is one of the tiny ones that can easily run on my laptop) so it can explain the history of it so it can explain to me the facts of it.
Do i need to dive in the llm model and find that specific data to correct it? Can i fine tune it to improve it (lets say i have a gpu big enough to train this llm)? Is the model fine, but i need a different gpt?
If I understood correctly, the question is on how to ensure the LLM gives accurate response.
While there are several ways one can do this, the most effective way to give a model specialized and accurate information is via a RAG system. This consists of providing the model specific information from a knowledge base depending on the user prompt.
Great content Shaw!
Next step Im having troubles figuring out, is there a way to run locally an existing GPT and do prompt engineering or model fine-tuning on it with my own training data?
Thanks! While this depends on your local machine specs, the short answer is yes! My next video will actually walk through how to do this using an approach called QLoRA.
Chatgpt suggest me this channel 🎉
LOL, hope it was a good rec :)
Thank you for putting together this video, helped me a lot to understand LLM training.
One question: with the advent of trillion token models and beyond, I wonder where will we get all that training input data from. I guess we already consumed what all humanity has produced in the last 5000 years, and by adding another 10M digitized cat videos, the models will not be smarter.
Good question! I suspect there is still much content out there that hasn't been touched by LLMs i.e. non-digital text and proprietary data. Nevertheless, this content is still finite and the "just make a bigger model" approach will eventually hit a limit.
🎉❤❤❤amazing video
Thank you 🙏
So my dreams about own LLM are broken(( So as i understood the only way to build some personal LLM is FineTuning? Atleast while cheap ways of training not appeared yet...
I wouldn't give up on it! My (optimistic) conjecture is as we better understand how these models actually work we will be able to develop ones that are much more computationally efficient.
bro i want to build an LLM.. does this video help me learn myself and build LLM myself? possible? (i did not see it till now)
While this video may be a helpful first step, more resources will be necessary. Here are a few additional resources I recommend.
- th-cam.com/video/kCc8FmEb1nY/w-d-xo.html&ab_channel=AndrejKarpathy
- huggingface.co/learn/nlp-course/chapter1/1?fw=pt
This is excellent - thanks for putting this together and taking the time to explain things so clearly!
When it comes to transformers. Are you saying they're more than meets the eye?
That's a good way to put it 😂
Awesome Video! Thanks.
Just my prediction on data curation & copyright, what is going to happen is the companies of LLM's will do what Google did back in the day with scraping websites. Then once legislation passes, they will say "if you don't want your data crawled, opt-out with robots.txt". Right now it's a grey area and companies are building their data as quickly as possible to get in before the regulation.
"Better to ask for forgiveness, rather than permission."
I can see that. It seems to be a hot-button topic these days.
Is there a way to use Data Dictionary to train LLM model to generate SQL queries later on?
Yes, but you will likely need to transform the data a bit before it can be used for fine-tuning. I give a concrete example of this here: th-cam.com/video/4RAvJt3fWoI/w-d-xo.html
Exceptional material
Hi, i have domain specific pdf files . How do i train using transfer learning? Please advise
Depends on what you mean by transfer learning. If you simply want to extract knowledge from a PDF I'd recommend exploring RAG or using off-the-shelf solutions like OpenAI Assistants interface.
Happy to clarify, if I misinterpreted the question.
What does it mean the amount of parameters???
Good question. A model is something that takes an input (say a sequence of words) and produces an output (e.g. the next most likely word). Parameters are numbers which define how the model takes inputs and translates them into outputs.
@@ShawhinTalebi thank you
Thanks for your video; it's awesome. You explain everything very clearly and with good examples.
Thanks for the feedback, glad it was clear :)
Can you post a video onu continual pretraining of llms like Llama
Thanks for the great suggestion. I’ll be doing more content on fine-tuning so that will be a good topic to cover there.
clicked with low expectation, but wow what a gem. Great clarity with just the right amount of depth for beginners and intermediate learners.
Glad it was helpful :)
A bit disappointed with this video, there is no coding !
Thanks for the feedback. Yes this one stays relatively high-level since a coding video would be (much) longer.
For a hands-on guide I'd recommend Andrej Karpathy's video: th-cam.com/video/kCc8FmEb1nY/w-d-xo.html
Can a fine-tuned LLM be repurposed and re-fine-tuned for more than one task?
Yes it can! In fact, that is what OpenAI did with their RLHF technique to create their InstructGPT models
This video’s title is misleading
I'm sorry. The moral of the story was building an LLM from scratch is not feasible or desirable for most people.
However, that's not to say one can't build a smaller language model from scratch (
Thoroughly researched and referenced, clear explanations inclusive of examples. I will watch it again to take notes. Thanks so much!
Great to hear! Feel free to reach out with any questions or suggestions for future content :)
Thanks! I would have plenty of questions actually, but they are probably a bit too specific to make for a generally relevant video. I am exploring options for a few non-profit projects related to musical education and research. They need to integrate large bodies of text and produce precise referencing to what comes from where, so I was naively toying with the idea to perhaps produce a base model partially trained on the actual text in question. Which, I understood from the video, is a non-starter. So I will look into fine-tuning, RAG and prompt engineering. I suspect I will spend quite a lot of time watching your convent, given you covered quite a lot. I also learned quite a bit more from this specific video. Right now I am studying the basics, including a bit of the math involved, and it is a bit slow going, so I am quite grateful :)
@@aldotanca9430 That sounds like a really cool use case (I've been a musician for over 14 years)!
If you want to chat about more specific questions feel free to set up some office hours: calendly.com/shawhintalebi/office-hours
@@ShawhinTalebi That's very generous of you! I will book a slot, would love to chat, I think it would help me immensely to rule out blind alleys and at least get a well informed idea of what is feasible to attempt. I did notice the congas, piano and Hanon lurking in the background, so I suspected the topic will be interesting to you. It is about historical research, but it is also very applicable and creative for improvvisation. Perhaps I can compile a very short list of interesting resources, in case you want to check it out at some point for musical reasons :)
Hello , Thanks . Do you know is it possible to create an own LLM for own startup?
Of course this is possible. However, it is rarely necessary. I'd suggest seeking simpler (and cheaper) solutions before jumping to training an LLM from scratch.
@@ShawhinTalebi thanks
Great video!
Brilliant!
excellent job Shawhin. Merci.
Thanks Reza, glad you liked it!
Wow, what a great content, thanks for that!! In LLM Fine-tuning, is there also a suggestion table between number of trainable parameters and tokens used (dataset size)?
That’s a great question. While I haven’t come across such a table, good rule of thumb is 1k-10k examples depending on the use case.
@@ShawhinTalebi thanks for the quick reply! What about the number of trainable parameter, should we worry about that? What if my number of examples is smaller than that let's say a 100 to 200?
@@echofloripa IMO you've got to work with what you've got. I've heard some people get sufficient performance from just 100-200 examples, but it ultimately comes down to what is acceptable for that particular use case. It might be worth a try.
Hope that helps!
Nice.
Thanks for putting together this short video. I enjoy learning this subject from you.
Thanks Ethan, glad you enjoyed it!
watching this right before my interview.
Good luck!
@@ShawhinTalebi cleared 1st round, now its on Thursday, i hope your luck brings me my dream job ❤️
Thanks for the great and pack expose. 😀
Glad it was helpful 😁
Love these videos! Keep it up Shaw!
Thank you 🙏
thankyou bro for your help
Happy to help!
very informative thx
Happy to help!
Hey Shaw - Thank you for coming up with this extensive video on building LLM from Scratch, it certainly gives a fair idea on, how some of the existing LLMs were created !