The EASIEST way to finetune LLAMA-v2 on local machine!
ฝัง
- เผยแพร่เมื่อ 19 ก.ค. 2023
- In this video, I'll show you the easiest, simplest and fastest way to fine tune llama-v2 on your local machine for a custom dataset! You can also use the tutorial to train/finetune any other Large Language Model (LLM). In this tutorial, we will be using autotrain-advanced.
AutoTrain Advanced github repo: github.com/huggingface/autotr...
Steps:
Install autotrain-advanced using pip:
- pip install autotrain-advanced
Setup (optional, required on google colab):
- autotrain setup --update-torch
Train:
autotrain llm --train --project_name my-llm --model meta-llama/Llama-2-7b-hf --data_path . --use_peft --use_int4 --learning_rate 2e-4 --train_batch_size 12 --num_train_epochs 3 --trainer sft
If you are on free version of colab, use this model instead: huggingface.co/abhishek/llama.... This is a smaller sharded version of llama-2-7b-hf by meta.
Please subscribe and like the video to help me keep motivated to make awesome videos like this one. :)
My book, Approaching (Almost) Any Machine Learning problem, is available for free here: bit.ly/approachingml
Follow me on:
Twitter: / abhi1thakur
LinkedIn: / abhi1thakur
Kaggle: kaggle.com/abhishek
Anyone comes across this in 2024 (jan ), the command switches with new autotrain version is autotrain llm --train --project-name josh-ops --model mistralai/Mistral-7B-Instruct-v0.2 --data-path . --use-peft --quantization int4 --lr 2e-4 --train-batch-size 12 --epochs 3 --trainer sft . Great, Video, thanks Abhishek
After finetuning it, how to run it?
That's Awesome, nothing better than this way of training large language model. Super easy ❤
Keeping it this simple is something very few people are able to do. Very well explained.
This can be understood by even a beginner. Atleast the execution if not the intuition behind it. Kudos
Thank you for the on-point tutorial.
Appreciate it, and request to continue making such videos🎉
Incredibly helpful video, I appreciate that you took the time to create this! Great stuff
Thank you Abhishek! This is phenomenal.
Amazing, tutorials at light speed! Llama 2 was just released! 😮
🙏🏽
😂😂Yeah exactly
This man is a genius lol
Astonishing content Man 🔥🔥🔥 🚀
Excellent, thank you so much. I will try.
Great video! Would be great if in some future vid you could go into depth on the training hyperparameters and perhaps also talk about what size your custom datasets should be.
sometimes I do that. however, this model would have taken wayy too long to train. im training a model as i type here and if i get good results ill share both model and params 🙂
@@abhishekkrthakur guess no good luck with the training :(
Thanks for sharing!
Great Job :)
Great tutorial! Can you also put up one video teaching on how to merge the fine tuned weights to the base model and do inference? Would like to see an end-to-end course. Thank you!
+1 on this question !!!!
any updates guys?? I really want to know how to merge the fine tuned model with the base model and do the inference. Do let me you have any resources or insights about the same
@@devyanshrastogi also need this answered. have you guys had any success?
There is only one thing I want to see. I want to see you using the final result and prove it actually works. Thank you.
Thanks for the tutorial! A couple questions for you. Is there an approach you're using to test quality and verity that the training data has influenced the weights in the model sufficiently to learn the new task? And second, can you use the same approach for unstructured training data such as using a large corpus of private data to do domain adaptation?
Informative video, hopefully one day I will get a task that requires me to finetune a LLM
or you can just do it for fun 🤗
thanks for sharing!
Nice job!
Please subscribe and like the video to help me keep motivated to make awesome videos like this one. :)
Please make a video on Llama2 + RAG (instead of finetuning)
Hi Abh
Your method is inspiring and commendable. How do we read the csv or json training dataset we prepared instead of the hugging face dataset you used?
Nice video Abhishek!
I am curious to know about custom data for LLMs. What is the ideal (good quality) data size (e.g., #rows), to fine-tune these models for good performance, not necessarily it should be big data of course.
Thanks!
Thank you brother
Thanks!
thank you very much
thanks bro ❤
As always very useful and short without wasting anyone's time. Thank you. Just I'm a bit confused about the prompt formatting you have used here - "### Instruction:
### Input:... etc" while Llama official is "[INST] {{ system_prompt }}{{ user_message }} [/INST]" and on TheBloke's page it says "SYSTEM: {system_prompt}
USER: {prompt}
ASSISTANT:"
I think this isn't mandatory, it is a suggestion.
Hi Abhishek, is the auto train using LORA or prompt tuning as the PEFT technique?
Thanks for this great explanation. Can you provide the link to you training data?
what kind of GPUs do you have? how big was your dataset and how long did it take to train? what is the smallest fine-tuning data set size that would be reasonable?
Thanks Kaggle grand master :) you got an channel.
Nice vid
Should we label input tokens to -100 to focus the training on the prediction?
I see no one doing it
Great Video. i love it and can't wait to try it. Now that Llama2 is out... is it better to FineTune a model or try to create your own Model?
Superb video! Does autotrain allow to export finetuned model as GGML file? Or can it be used with GGML file?
For fine-tuning of the large language models (llama-2-13b-chat), what should be the format(.text/.json/.csv) and structure (like should be an excel or docs file or prompt and response or instruction and output) of the training dataset? And also how to prepare or organise the tabular dataset for training purpose?
First thank you for the great tutorial , I have one question : I am trying to finetune the model on Japanese , do you have any advice for that ? I have tried the same script as you did but it didn't work; it produced some gibberish after the training finished , I am guessing it is a tokenizer problem, what do you think ?
The way Abhishek side eyes before stopping the video and resuming is is soo crazy 🤣🤣😅
lol. big screen. button too far 🤣
Thanks for the comprehensive tutorial. Can this be done using chat logs to build a clone of your friend? I have done this with GPT3.5 finetuning using prompt->response. The prompts are questions generated by ChatGPT based on the chat log message. Can the same thing be done with Instruction->Input->Response? Thank you very much man.
Great work Abhishek Thakur, it will be great if you made a video on how to run the fine-tuned model
already done. check out other videos on my channel
@@abhishekkrthakur can you please post the video link? there are many videos in your channel, it's hard to find which one you are talking about.
@@abhishekkrthakur I did fine tuning on the model, but I don't think I can run it on google colab with T4 since its show out of memory error!! Any suggestion?
@@AnandMoorthyJ did you find the video? im looking for the link aswell :)
Beautiful content, I have a side question, what tool you are using to have "copilot"-like suggestion in your terminal? Thx again for the video
I use Hugginfaces co pilot. - it works pretty well and super easy to set up and free..
@@jessem2176 Thanks for the recommendation, but did you mean HuggingFace Personal-copilot Blog?
Thank you for this interesting video. How could we change the encoding to ''latin-1' in order to train on french language ? thank you.
How do you set the training data? I see different people using different formats? Does it matter or is the only requirement that it has to be structured meaniningfully?
Great one! Two things - you didn't show how to configure the cli to enable access to the model. Secondly, it would be useful to know how to use aws for training. Thanks!
Hey, could you guide us how to train custom dataset on llama2? How to prepare our dataset for training?
Great tutorial! Thank you! Maybe I missed it but, at which point do you enter your HF token?
You dont. You login using "huggingface-cli login" command. There's also a similar command for notebooks and colab. :)
How is your ipython giving suggestions? I want the same set up. Please make a video on these types of set up that assists in coding and other processes.
Hey Abhishek, Thanks for sharing this amazing tutorial. Can I do this on my RTX 3070Ti 8GB GPU? If yes, what batch size would be preferable?
8GB sounds a bit low for this. maybe try bs=1 or 2? but tbh, im not sure if it will work. Might work fine for a smaller model!
I know it could be too much but could you also make a video of how to prepare the data? I have like 'question' and 'answer' but I am strugging to make it to a trainable data set into that kind of csv so I could use it!
Thank you ,what is diff between instruction and input
Hi Abhishek,
Much appreciated. How would things change if we were to use simple fine tuning? That is, just a large single code file to learn from, to tune code-llama, phi2, etc..
I trained model using autotrain in same way as you suggested and model file is stored.
Now I need to use this model for prediction. Can you shed some light on this as well?
Thanks for this great video, but how to fine tune or train for question answer data set
I just bought a RTX 4090 Founders Edition. Could you tell on a particular example were I could run into limits with card when training LLMs locally? I personally think that I'm safe for the next few years and I will not run in any problems.
nice video. it great for a beginner to learn how to fine tune LLAMA2 locally. It would be better if you can share the code and dataset.
Can you make a video for fine tuning in silicon macs ?
Thanks for the nice tutorial. Could you show how to perform inference the finetuned model?
yes. its in one of my previous videos :) thanks!
@@abhishekkrthakur Which one?
Is there a way to run it on CPU? Thanks sir, I love your work
Do you have to use [INST]...[/INST] for indicating the instructions? I think the original Llama 2 model was trained with these tags, so I am a bit puzzled if you have to use the tags in the csv or they are added internally ?!
in this video, im finetuning the base model. you can finetune it anyway you want. you can even take the chat model and finetune it this way. if you are using a different format for finetuning, you must use the same format while inference in order to get the best results.
Thank you for the video. I am new to this, so I am trying to set it up on my windows PC. When I am trying to install the latest version of autotrain-advanced==0.6.2, I get an error saying: trition==2.0.0.post1 cannot be found. Which I believe is only available on Linux. So is it possible to use autotrain-advanced on windows?
Thank for the tutorial!
How can I fine-tune the language model with a domain-specific unlabeled dataset to improve performance on that specific domain? Is it effective to leave the instruction and input empty and only use domain-specific text for the output?
Hey, Can we connect. Even I need help on similar topic. We can discuss more ...
Hey! Fantastic video, but i do not understand at all the division into smaller chunks of the model in order to work in free version of collab, can you explain it? Thank you!
chunks are loaded into ram first. since larger chunks didnt fit in ram with all the other stuff, i created a version with smaller shards :)
How to shard as you mentioned towards the end?
Thanks for the tutorial! How would you modify the code to train on a dataset containing a single column of text? i.e. trying to perform domain-specific additional pretraining?
I would remove the peft portion to do full finetuning, anything else?
Hey, Can we connect. Even I need help on similar topic. We can discuss more ...
Hi, I'm looking into something similar. Did you find a good way to do this?
hello, thank you. i really need to know: does this pad appropriately? also, how does it internally split it into prompt completion? Can i make up roles like ### System? does it complete only the last message?
Please show how you used the trained mode for inference
Yes. Please!
"These days", LOL. We should all be training our own LLMs.
Thank you. Can you please also show us how to train on GPU #3 or GPU#1 or both GPU#1&3 but not in GPU #0 in a multi GPU machine?
CUDA_VISIBLE_DEVICES=0 autotrain llm --train ..... will run it on gpu 0
CUDA_VISIBLE_DEVICES=1,3 autotrain llm --train ..... will run it on gpu 1 and 3
It's very useful for me , can you share you instruction dataset or just tell me where I can look for some small and good dataset , thank's again !
I already found some tiny dataset from huggingface , I will follow your step , that's interesting thing !
Thanks for the video. i have a doubt that how can i make a package of it so others can also use it offline by just installing the application
Great tutorial, how can i run the fine tuned model on inference data?
there are couple of videos on my channel for that
When will I be able to see the model that I have trained? Once all the training is complete?
Amazing content. One Q left: how can I run the model locally in inference mode after training? Anyone have a command for that?
th-cam.com/video/o1BCq1KJULM/w-d-xo.html
Hello @abishekkrthakur can I train summarization models with autotrain advanced?
Thanks for this awesome video! Also, if we were fine-tuning llamaV2 for instruction/chat use-case, we would just replace Input, Response with User, Assistant respectively in the dataset, that's it right?
You should check for Alpaca Format.
Can we do this on a CPU? I am using a GGmL model.. please share your thoughts
How do you deal with response in the dataset that has newline characters?
I am running out of memory when testing it on the free-version of google colab, did you use the exact same tuning parameters as described in the video?
yes. you can reduce batch size. note, you need to use different model path if you are on colab or it will run out of memory. see description for more details
Very good tutorial. On what machine are you running this? I am trying to run it on a Macbook pro M1 but I keep getting "ValueError: No GPU found. Please install CUDA and try again." I have tried to do this directly on Huggingface and got "error 400: bad request"...so I cloned autotrain and ran it locally...still getting error 400. Do you have any pointers?
Same error
Hi I have a question , it´s posible to use "autotrain" without cuda, with rocm support of AMD GPU ?
How do you convert it to work with Ollama? I setup the model file and it doesnt seem to know anything from my training.
Nice video. What if we just want to fine tune on plain text, like a bunch of lines in a csv file?
yeap
Would it just be one column then?@@abhishekkrthakur
Amazing!
How can we use the model for inference?
th-cam.com/video/o1BCq1KJULM/w-d-xo.html
@@abhishekkrthakursorry, but output of this video and the video you linked doesn't match really
@@abhishekkrthakur I dont think its relevant since you use peft and 4bit. The model loading in inference should be different when you have an adapter model. Would be really really great if you show us the inference of the model you trained on the video
Where can i find to code you worte and your dataset? I would like to use it as an exemple for testing
hey Abhishek, I am a full stack developer and interested in AI. I love to code. I tried learning DS but found no interest in juggling with data. How should i learn?
What GPUs were you running?
Is it possible to train a model, transformer using this library from scratch? If not maybe you could make a video on how to train a LLM from scratch using a custom dataset?
Yesss pls @abhishek
which machine is recommended for fine-tuning LLAMA? windows?
Thanks for the tutorial.
On a Linux 5.15.0-71-generic #78-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux machine, I get following error when training llm with the small dataset. File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 2819, in from_pretrained
raise ValueError(
ValueError:
Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit
the quantized model. If you want to dispatch the model on the CPU or the disk while keeping
these modules in 32-bit, you need to set `load_in_8bit_fp32_cpu_offload=True` and pass a custom
`device_map` to `from_pretrained`.
What could be the problem? Is it possible to share the data.csv that you have with single row that I can take as reference to test my own data?
Thanks! Is there a way to push the trained model to hub after its trained and not using the --push_to_hub while training? Also, when I try to use push to hub, I get a "you don't have rights to create a model under this namespace". I am using a read token to access the llama model. Do I need to change it to a write token? Is it possible to use two separate tokens? (sorry, I'm super new to Huggingface) Any help is much appreciated. Thanks!
yes. you need to use a write token. you can remove push to hub and then push the model manually using git commands if you wish
Why do we need --mixed-precsion and --quantization both? Aren't they both doing the same thing?
can i train llama3 also with these steps?
Can this be done on Apple Silicon, I have M2 with 24G memory?
How can i load the model for inference if i stop training after 1 epoch out of 3 epochs.
Is Gpu Machine necessary for llama 7b to be trained?
Hi when we tried fine tuning MPT LLM using autotrain, getting the error ValueError: MPTForCausalLM does not support gradient checkpointing. Any help you can offer on this pleas?
I have llm downloaded can I fine tune it with downloading from huggingface.
If i understand this model is training in the local machine? and is saved locally too ? Or model continues on hugging face servers?
its training locally. its also saved locally. if you use --push-to-hub arg, it will push model to huggingface servers
Any chance you have the python scripts available somewhere? Thanks in advance.
Any idea to fine-tune it on Apple chip M1/M2? Just have an installation issues with auto train-advanced 😢
Would love to know how to do this as well!
Bumping because running into so many issues with M1. Cannot believe how little resources are available for M1 right now given that macOS is so widely used in data science
How to change this for tasks like classification?